You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "zyratlo (via GitHub)" <gi...@apache.org> on 2026/08/27 03:26:33 UTC
[PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
zyratlo opened a new pull request, #8032:
URL: https://github.com/apache/texera/pull/8032
### What changes were proposed in this PR?
`notebook-migration-service` read one Jupyter URL and token from configuration as process-wide values. That is safe only under the per-user-pod model, where each user runs their own pod. A single global instance would hand every user the same Jupyter and the same token. This resolves both per user instead.
This is stage 2 of the staged roadmap in the architectural note on #5258. Stage 1 (#7390) and stage 4 (#7671) are merged. Stage 3 (#8006) depends on this one.
**Registry table**
A new `user_jupyter` table holds one row per user whose Jupyter has been provisioned, keyed on `uid`. Both the internal and public URLs are stored rather than derived at read time, matching how `workflow_computing_unit.uri` is the recorded source of truth for a computing unit's address. Storing them lets the deployment change its addressing scheme without a code change, which is what #8006 needs in order to choose between per-user hostnames and a path prefix.
**Derived tokens**
Each user's token is `HMAC-SHA256(secret, uid)`, computed from `storage.jupyter.token-secret`. No credential is stored at rest, any replica of the service derives the same value, and rotation is a secret change. The service refuses to start when per-user Jupyter is on with an empty secret: an empty HMAC key is publicly known, so tokens would still look distinct per user while anyone could derive anyone else's.
**Fallback gated on a flag, not on row absence**
With `kubernetes.jupyter-enabled` off, every user resolves to the statically configured Jupyter, which is how the single-node (#7932) and local-dev (#7934) deployments run one shared JupyterLab. With it on, a user with no row has nothing provisioned yet and gets an unavailable response. Falling back to the shared server there would hand an unprovisioned user somebody else's notebooks, and would let `delete-notebook` remove another user's file. The flag is deliberately separate from `kubernetes.enabled`, so a deployment can run computing units on Kubernetes without per-user Jupyter.
**Provisioning**
`JupyterKubernetesClient` mirrors the computing unit's `KubernetesClient`, including the hostname and subdomain pair that makes the pod FQDN resolve through a headless service. `get-jupyter-iframe-url` and `set-notebook` provision on demand; `get-jupyter-url` and `delete-notebook` resolve only, so reading a URL or deleting a file never starts a pod. A registered pod that stops answering is discarded and rebuilt, since the row would otherwise outlive the pod and point every later request at nothing. Two concurrent first requests are resolved by the primary key: the loser keeps the winner's row, which holds the same uid-derived addresses.
The uid always comes from the authenticated session and never from a request body, so one user cannot address another's Jupyter.
Two notes on the implementation. The Kubernetes client is built on demand rather than at object initialisation, unlike the computing unit's, because the single-node and local-dev deployments have no cluster to build one against and never provision. Adding `io.fabric8:kubernetes-client` also raises three transitive versions over Dropwizard's pins (`slf4j-api` to 2.0.13, `jackson-dataformat-yaml` and `jackson-datatype-jsr310` to 2.17.0); `LICENSE-binary` claims are updated to match.
Out of scope, and left to #8006: the Helm templates, the NetworkPolicy, and the global Deployment. Also out of scope is reclaiming idle pods. A dead pod self-heals on next use, but nothing reaps one that is merely unused, and the row's `ON DELETE CASCADE` drops the registration without deleting the pod. Until #8006 supplies an ingress, `kubernetes.jupyter-public-url-template` defaults to empty and the browser is handed the in-cluster address.
### Any related issues, documentation, discussions?
Closes #7665
Parent issue #4301
### How was this PR tested?
`NotebookMigrationService/test` passes (72) and `Config/test` passes (71). `Test/compile` across the whole repo is clean, as is `scalafmtCheckAll`.
New coverage: token derivation is deterministic per uid, distinct across 50 uids, and changes with the secret; the start-up guard rejects an empty secret only when the feature is on. Resolution falls back with the feature off, returns the row with it on, returns nothing for an unregistered user, and never returns one user's Jupyter to another. Provisioning is exercised against a stubbed Kubernetes client across every branch: fresh provision, reuse of a live pod, rebuild of a pod that stopped answering, cleanup when readiness never arrives, public URL templating, and the concurrent-registration race.
The existing `withFakeJupyter` suite is untouched and still passes on the stub bound to `localhost:9100`, which is the point of the flag-gated fallback: test users have no registry row and take the configured path exactly as before.
Applied `sql/texera_ddl.sql` to a scratch database the way CI does, confirmed the table shape, and confirmed that deleting a user removes their registration row. `check_binary_deps.py` passes under both the PR-mode and the strict nightly invocations, and `NOTICE-binary` matches generator output byte for byte.
Not tested on a live cluster. The chart has no notebook-migration or Jupyter templates yet, and the in-cluster pod address does not resolve from outside the cluster, so end to end Kubernetes validation belongs to #8006.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8032:
URL: https://github.com/apache/texera/pull/8032#issuecomment-5433998282
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/8032?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `71.03448%` with `42 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 93.23%. Comparing base ([`9fe0657`](https://app.codecov.io/gh/apache/texera/commit/9fe06578f54c8d79488bb16337b2fe24d8d68dae?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dbbba4f`](https://app.codecov.io/gh/apache/texera/commit/dbbba4ffac418d0b27532932c4f2a3ca7f69c59a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/texera/pull/8032?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../texera/service/util/JupyterKubernetesClient.scala](https://app.codecov.io/gh/apache/texera/pull/8032?src=pr&el=tree&filepath=notebook-migration-service%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Fservice%2Futil%2FJupyterKubernetesClient.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bm90ZWJvb2stbWlncmF0aW9uLXNlcnZpY2Uvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS90ZXhlcmEvc2VydmljZS91dGlsL0p1cHl0ZXJLdWJlcm5ldGVzQ2xpZW50LnNjYWxh) | 17.24% | [24 Missing :warning: ](https://app.codecov.io/gh/apache/texera/pull/8032?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...pache/texera/service/util/JupyterProvisioner.scala](https://app.codecov.io/gh/apache/texera/pull/8032?src=pr&el=tree&filepath=notebook-migration-service%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Fservice%2Futil%2FJupyterProvisioner.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bm90ZWJvb2stbWlncmF0aW9uLXNlcnZpY2Uvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS90ZXhlcmEvc2VydmljZS91dGlsL0p1cHl0ZXJQcm92aXNpb25lci5zY2FsYQ==) | 80.00% | [4 Missing and 8 partials :warning: ](https://app.codecov.io/gh/apache/texera/pull/8032?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...a/service/resource/NotebookMigrationResource.scala](https://app.codecov.io/gh/apache/texera/pull/8032?src=pr&el=tree&filepath=notebook-migration-service%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Fservice%2Fresource%2FNotebookMigrationResource.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bm90ZWJvb2stbWlncmF0aW9uLXNlcnZpY2Uvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS90ZXhlcmEvc2VydmljZS9yZXNvdXJjZS9Ob3RlYm9va01pZ3JhdGlvblJlc291cmNlLnNjYWxh) | 82.35% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/texera/pull/8032?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../apache/texera/service/util/JupyterEndpoints.scala](https://app.codecov.io/gh/apache/texera/pull/8032?src=pr&el=tree&filepath=notebook-migration-service%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Fservice%2Futil%2FJupyterEndpoints.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bm90ZWJvb2stbWlncmF0aW9uLXNlcnZpY2Uvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS90ZXhlcmEvc2VydmljZS91dGlsL0p1cHl0ZXJFbmRwb2ludHMuc2NhbGE=) | 83.33% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/texera/pull/8032?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../org/apache/texera/service/util/JupyterProbe.scala](https://app.codecov.io/gh/apache/texera/pull/8032?src=pr&el=tree&filepath=notebook-migration-service%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Fservice%2Futil%2FJupyterProbe.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bm90ZWJvb2stbWlncmF0aW9uLXNlcnZpY2Uvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS90ZXhlcmEvc2VydmljZS91dGlsL0p1cHl0ZXJQcm9iZS5zY2FsYQ==) | 90.90% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/texera/pull/8032?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/texera/service/util/JupyterTokenDeriver.scala](https://app.codecov.io/gh/apache/texera/pull/8032?src=pr&el=tree&filepath=notebook-migration-service%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Fservice%2Futil%2FJupyterTokenDeriver.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bm90ZWJvb2stbWlncmF0aW9uLXNlcnZpY2Uvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS90ZXhlcmEvc2VydmljZS91dGlsL0p1cHl0ZXJUb2tlbkRlcml2ZXIuc2NhbGE=) | 92.30% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/texera/pull/8032?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #8032 +/- ##
============================================
- Coverage 93.29% 93.23% -0.07%
- Complexity 4678 4708 +30
============================================
Files 1179 1185 +6
Lines 47713 47832 +119
Branches 5314 5330 +16
============================================
+ Hits 44515 44594 +79
- Misses 1732 1763 +31
- Partials 1466 1475 +9
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/8032/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/8032/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | | |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/8032/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `99.32% <ø> (ø)` | | Carriedforward from [9fe0657](https://app.codecov.io/gh/apache/texera/commit/9fe06578f54c8d79488bb16337b2fe24d8d68dae?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/8032/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `89.42% <ø> (ø)` | | Carriedforward from [9fe0657](https://app.codecov.io/gh/apache/texera/commit/9fe06578f54c8d79488bb16337b2fe24d8d68dae?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/8032/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.67% <ø> (ø)` | | Carriedforward from [9fe0657](https://app.codecov.io/gh/apache/texera/commit/9fe06578f54c8d79488bb16337b2fe24d8d68dae?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/8032/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `86.73% <ø> (ø)` | | |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/8032/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `86.70% <ø> (ø)` | | Carriedforward from [9fe0657](https://app.codecov.io/gh/apache/texera/commit/9fe06578f54c8d79488bb16337b2fe24d8d68dae?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/8032/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `95.77% <ø> (ø)` | | Carriedforward from [9fe0657](https://app.codecov.io/gh/apache/texera/commit/9fe06578f54c8d79488bb16337b2fe24d8d68dae?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/8032/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.26% <71.03%> (-4.05%)` | :arrow_down: | |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/8032/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `97.78% <ø> (ø)` | | Carriedforward from [9fe0657](https://app.codecov.io/gh/apache/texera/commit/9fe06578f54c8d79488bb16337b2fe24d8d68dae?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/8032/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.19% <ø> (ø)` | | Carriedforward from [9fe0657](https://app.codecov.io/gh/apache/texera/commit/9fe06578f54c8d79488bb16337b2fe24d8d68dae?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/8032?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3908684707
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.common.config.{KubernetesConfig, StorageConfig}
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.daos.UserJupyterDao
+import org.apache.texera.dao.jooq.generated.tables.pojos.UserJupyter
+import org.jooq.exception.DataAccessException
+
+import scala.util.control.NonFatal
+
+/**
+ * Brings a user's JupyterLab into existence and registers where it lives.
+ *
+ * Dependencies are constructor parameters so the provisioning logic can be tested without a
+ * cluster; the companion object binds the production ones.
+ */
+class JupyterProvisioner(
+ kubernetesClient: => JupyterKubernetesClient,
+ isReachable: String => Boolean,
+ publicUrlTemplate: String,
+ readinessTimeoutMillis: Long,
+ readinessPollMillis: Long
+) extends LazyLogging {
+
+ // By-name above, forced once here, so no client is built unless a provision happens.
+ private lazy val kubernetes = kubernetesClient
+
+ /**
+ * The user's Jupyter, starting one if they have none. None means it could not be made
+ * ready, which callers report the same as an unreachable server.
+ *
+ * A registered pod that no longer answers is discarded and rebuilt: the row would otherwise
+ * outlive the pod and point every later request at nothing.
+ */
+ def ensure(
+ uid: Int,
+ jupyterEnabled: Boolean = KubernetesConfig.jupyterEnabled,
+ fallback: JupyterEndpoints = JupyterEndpoints.configured,
+ tokenSecret: String = StorageConfig.jupyterTokenSecret
+ ): Option[JupyterEndpoints] = {
+ if (!jupyterEnabled) return Some(fallback)
+
+ val token = JupyterTokenDeriver.derive(uid, tokenSecret)
+ JupyterEndpointResolver.resolve(uid, jupyterEnabled = true, tokenSecret = tokenSecret) match {
+ case Some(endpoints) if isReachable(endpoints.internalUrl) => Some(endpoints)
Review Comment:
Fixed in 8678d18b0. The reuse check now calls JupyterProbe.isAuthorized with the derived
token rather than the unauthenticated liveness probe, so a pod started under an old secret
fails the check and is rebuilt. Readiness polling uses that same authorized check, so a new
pod is only accepted once it answers to the token it was actually given.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3918478703
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala:
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.fabric8.kubernetes.client.KubernetesClientException
+import org.apache.texera.common.config.{KubernetesConfig, StorageConfig}
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.daos.UserJupyterDao
+import org.apache.texera.dao.jooq.generated.tables.pojos.UserJupyter
+import org.jooq.exception.DataAccessException
+
+import scala.util.control.NonFatal
+
+/**
+ * Brings a user's JupyterLab into existence and registers where it lives.
+ *
+ * Dependencies are constructor parameters so the provisioning logic can be tested without a
+ * cluster; the companion object binds the production ones.
+ */
+class JupyterProvisioner(
+ kubernetesClient: => JupyterKubernetesClient,
+ accepts: (String, String) => Boolean,
+ publicUrlTemplate: String,
+ readinessTimeoutMillis: Long,
+ readinessPollMillis: Long
+) extends LazyLogging {
+
+ // By-name above, forced once here, so no client is built unless a provision happens.
+ private lazy val kubernetes = kubernetesClient
+
+ /**
+ * The user's Jupyter, starting one if they have none. None means it could not be made
+ * ready, which callers report the same as an unreachable server.
+ *
+ * A registered pod that no longer answers is discarded and rebuilt: the row would otherwise
+ * outlive the pod and point every later request at nothing.
+ */
+ def ensure(
+ uid: Int,
+ jupyterEnabled: Boolean = KubernetesConfig.jupyterEnabled,
+ fallback: JupyterEndpoints = JupyterEndpoints.configured,
+ tokenSecret: String = StorageConfig.jupyterTokenSecret
+ ): Option[JupyterEndpoints] = {
+ if (!jupyterEnabled) return Some(fallback)
+
+ val token = JupyterTokenDeriver.derive(uid, tokenSecret)
+ JupyterEndpointResolver.resolve(uid, jupyterEnabled = true, tokenSecret = tokenSecret) match {
+ case Some(endpoints) if accepts(endpoints.internalUrl, token) => Some(endpoints)
+ case Some(endpoints) =>
+ // Either the pod is gone, or it predates a token-secret rotation and no longer
+ // accepts the derived token. Both are unusable and both are fixed by rebuilding.
+ logger.warn(
+ s"Jupyter for user $uid is registered at ${endpoints.internalUrl} but does not "
+ + "accept its current token; rebuilding it"
+ )
+ discard(uid)
+ provision(uid, token)
+ case None => provision(uid, token)
+ }
+ }
+
+ private def provision(uid: Int, token: String): Option[JupyterEndpoints] = {
+ val internalUrl = s"http://${kubernetes.generatePodURI(uid)}"
+ val endpoints = JupyterEndpoints(internalUrl, publicUrlFor(uid, internalUrl), token)
+ try {
+ createIfAbsent(uid, token)
+ if (!waitUntilAccepting(internalUrl, token)) {
+ logger.error(s"Jupyter for user $uid did not become ready; removing the pod")
+ kubernetes.deletePod(uid)
+ None
+ } else {
+ register(endpoints, uid)
+ Some(endpoints)
+ }
+ } catch {
+ case NonFatal(e) =>
+ logger.error(s"Failed to provision Jupyter for user $uid", e)
+ None
+ }
+ }
+
+ // Two concurrent first requests can both find the pod absent and both create it. The
+ // loser's create returns 409, and the winner's pod is the one it wanted anyway, since both
+ // derive the same token from the same uid, so the conflict is a success. Mirrors
+ // register()'s handling of a duplicate row.
+ private def createIfAbsent(uid: Int, token: String): Unit = {
+ if (kubernetes.podExists(uid)) return
+ try kubernetes.createPod(uid, token)
+ catch {
+ case e: KubernetesClientException if e.getCode == 409 =>
+ logger.info(s"Jupyter pod for user $uid was created concurrently; using that one")
+ }
+ }
+
+ /** Browser-facing address; the in-cluster name does not resolve from the browser. */
+ private def publicUrlFor(uid: Int, internalUrl: String): String =
+ if (publicUrlTemplate.isEmpty) internalUrl
+ else publicUrlTemplate.replace("{uid}", uid.toString)
+
+ // Waits for the pod to accept its own token rather than merely to answer, so a pod that
+ // starts with the wrong token never gets registered.
+ private def waitUntilAccepting(internalUrl: String, token: String): Boolean = {
+ val deadline = System.currentTimeMillis() + readinessTimeoutMillis
+ var ready = accepts(internalUrl, token)
+ while (!ready && System.currentTimeMillis() < deadline) {
+ Thread.sleep(readinessPollMillis)
+ ready = accepts(internalUrl, token)
+ }
+ ready
+ }
+
+ private def register(endpoints: JupyterEndpoints, uid: Int): Unit = {
+ val row = new UserJupyter
+ row.setUid(uid)
+ row.setInternalUrl(endpoints.internalUrl)
+ row.setPublicUrl(endpoints.publicUrl)
+ try dao().insert(row)
+ catch {
+ // Two concurrent first requests can both provision. uid is the primary key, so the
+ // loser trips 23505; the winner's row holds the same uid-derived addresses, so leaving
+ // it in place is correct.
+ case e: DataAccessException if e.sqlState == "23505" =>
+ logger.info(s"Jupyter for user $uid was registered concurrently; keeping that row")
+ }
+ }
+
+ private def discard(uid: Int): Unit = {
+ try kubernetes.deletePod(uid)
Review Comment:
Fixed in d688d2ec9. deletePod now deletes with withGracePeriod(0), and discard polls podExists until the pod is gone before returning, so the rebuild no longer finds the name taken. discard reports whether the pod actually went and ensure declines to rebuild if it did not, since otherwise a stuck pod lands back in the readiness poll you described. The stub now models a Terminating window, and verified on a real cluster: deleting a pod mid-Terminating and requesting the iframe URL rebuilds in 3.2 seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8032:
URL: https://github.com/apache/texera/pull/8032#issuecomment-5433980422
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@tanishqgandhi1908`, `@aicam`, `@Ma77Ball`
You can notify them by mentioning `@tanishqgandhi1908`, `@aicam`, `@Ma77Ball` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3918485401
##########
sql/updates/45.sql:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+\c texera_db
+
+SET search_path TO texera_db;
+
Review Comment:
No change needed. sql/docker-compose.yml strips these before Liquibase runs, via sed 's/^\\c.*//g;/^SET search_path/d', so the directives never reach JDBC. They stay in the file because it is also meant to be runnable by psql directly, which is how bin/k8s applies texera_ddl.sql. Every migration on main carries the same two lines, 44 included, and CI does not run Liquibase at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on PR #8032:
URL: https://github.com/apache/texera/pull/8032#issuecomment-5457062585
Worth recording that this has been exercised end to end, not only unit tested.
Stage 3 (#8006) is stacked on this branch, and deploying it to a local Minikube cluster ran every decision in this PR against real infrastructure. Calico was used deliberately, because Minikube's default CNI creates NetworkPolicy objects without enforcing them, which would make an isolation test look correct while enforcing nothing.
What that confirmed:
* Provisioning creates the pod on demand and the resource quota moves to match. The pod's labels, hostname and subdomain line up with what the headless service selects, so the FQDN this PR records actually resolves.
* The token a user receives matches an independent HMAC derivation of the same uid, computed outside the JVM from the configured secret. Two users receive different tokens.
* The stale row path works. Deleting a pod without touching its registry row makes the next request log the recorded address as unreachable, discard the row, and provision a replacement.
* Per-user isolation holds. One user's pod cannot reach another's on 8888 by pod IP or by DNS, in either direction, while the same pod reaches another namespace in the same probe. Removing the NetworkPolicy makes the connection succeed and restoring it blocks again, so the denial is attributable to the policy rather than to incidental CNI behaviour.
Three things only a real deployment surfaced, all fixed in the stage 3 branch rather than here, since none of them are defects in this code:
* Deriving the browser-facing origin from a gateway hostname alone is not enough. A deployment reached by port-forward or NodePort has no hostname, so the origin has to be settable directly.
* A proxy that rewrites the Host header breaks JupyterLab's own cross-origin check on cookie-authenticated requests, which silently prevents the kernel from starting.
* The gateway route serving the LLM endpoints had no explicit timeout, so Envoy's 15 second default cut off completions that take longer and were succeeding upstream.
On the Codecov result: the twelve lines it reports are partials, not uncovered lines. The per-file breakdown reads 0 Missing throughout. Ten are scala-logging macro branches, where `logger.warn(...)` expands to an `isWarnEnabled` check that would need the suite run at two log levels to cover both sides. The other two are a pattern-match dispatch branch and a case class synthetic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
mengw15 commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3911696348
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala:
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.fabric8.kubernetes.client.KubernetesClientException
+import org.apache.texera.common.config.{KubernetesConfig, StorageConfig}
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.daos.UserJupyterDao
+import org.apache.texera.dao.jooq.generated.tables.pojos.UserJupyter
+import org.jooq.exception.DataAccessException
+
+import scala.util.control.NonFatal
+
+/**
+ * Brings a user's JupyterLab into existence and registers where it lives.
+ *
+ * Dependencies are constructor parameters so the provisioning logic can be tested without a
+ * cluster; the companion object binds the production ones.
+ */
+class JupyterProvisioner(
+ kubernetesClient: => JupyterKubernetesClient,
+ accepts: (String, String) => Boolean,
+ publicUrlTemplate: String,
+ readinessTimeoutMillis: Long,
+ readinessPollMillis: Long
+) extends LazyLogging {
+
+ // By-name above, forced once here, so no client is built unless a provision happens.
+ private lazy val kubernetes = kubernetesClient
+
+ /**
+ * The user's Jupyter, starting one if they have none. None means it could not be made
+ * ready, which callers report the same as an unreachable server.
+ *
+ * A registered pod that no longer answers is discarded and rebuilt: the row would otherwise
+ * outlive the pod and point every later request at nothing.
+ */
+ def ensure(
+ uid: Int,
+ jupyterEnabled: Boolean = KubernetesConfig.jupyterEnabled,
+ fallback: JupyterEndpoints = JupyterEndpoints.configured,
+ tokenSecret: String = StorageConfig.jupyterTokenSecret
+ ): Option[JupyterEndpoints] = {
+ if (!jupyterEnabled) return Some(fallback)
+
+ val token = JupyterTokenDeriver.derive(uid, tokenSecret)
+ JupyterEndpointResolver.resolve(uid, jupyterEnabled = true, tokenSecret = tokenSecret) match {
+ case Some(endpoints) if accepts(endpoints.internalUrl, token) => Some(endpoints)
+ case Some(endpoints) =>
+ // Either the pod is gone, or it predates a token-secret rotation and no longer
+ // accepts the derived token. Both are unusable and both are fixed by rebuilding.
+ logger.warn(
+ s"Jupyter for user $uid is registered at ${endpoints.internalUrl} but does not "
+ + "accept its current token; rebuilding it"
+ )
+ discard(uid)
+ provision(uid, token)
+ case None => provision(uid, token)
+ }
+ }
+
+ private def provision(uid: Int, token: String): Option[JupyterEndpoints] = {
+ val internalUrl = s"http://${kubernetes.generatePodURI(uid)}"
+ val endpoints = JupyterEndpoints(internalUrl, publicUrlFor(uid, internalUrl), token)
+ try {
+ createIfAbsent(uid, token)
+ if (!waitUntilAccepting(internalUrl, token)) {
+ logger.error(s"Jupyter for user $uid did not become ready; removing the pod")
+ kubernetes.deletePod(uid)
+ None
+ } else {
+ register(endpoints, uid)
+ Some(endpoints)
+ }
+ } catch {
+ case NonFatal(e) =>
+ logger.error(s"Failed to provision Jupyter for user $uid", e)
+ None
+ }
+ }
+
+ // Two concurrent first requests can both find the pod absent and both create it. The
+ // loser's create returns 409, and the winner's pod is the one it wanted anyway, since both
+ // derive the same token from the same uid, so the conflict is a success. Mirrors
+ // register()'s handling of a duplicate row.
+ private def createIfAbsent(uid: Int, token: String): Unit = {
+ if (kubernetes.podExists(uid)) return
+ try kubernetes.createPod(uid, token)
+ catch {
+ case e: KubernetesClientException if e.getCode == 409 =>
+ logger.info(s"Jupyter pod for user $uid was created concurrently; using that one")
+ }
+ }
+
+ /** Browser-facing address; the in-cluster name does not resolve from the browser. */
+ private def publicUrlFor(uid: Int, internalUrl: String): String =
+ if (publicUrlTemplate.isEmpty) internalUrl
+ else publicUrlTemplate.replace("{uid}", uid.toString)
+
+ // Waits for the pod to accept its own token rather than merely to answer, so a pod that
+ // starts with the wrong token never gets registered.
+ private def waitUntilAccepting(internalUrl: String, token: String): Boolean = {
+ val deadline = System.currentTimeMillis() + readinessTimeoutMillis
+ var ready = accepts(internalUrl, token)
+ while (!ready && System.currentTimeMillis() < deadline) {
+ Thread.sleep(readinessPollMillis)
+ ready = accepts(internalUrl, token)
+ }
+ ready
+ }
+
+ private def register(endpoints: JupyterEndpoints, uid: Int): Unit = {
+ val row = new UserJupyter
+ row.setUid(uid)
+ row.setInternalUrl(endpoints.internalUrl)
+ row.setPublicUrl(endpoints.publicUrl)
+ try dao().insert(row)
+ catch {
+ // Two concurrent first requests can both provision. uid is the primary key, so the
+ // loser trips 23505; the winner's row holds the same uid-derived addresses, so leaving
+ // it in place is correct.
+ case e: DataAccessException if e.sqlState == "23505" =>
+ logger.info(s"Jupyter for user $uid was registered concurrently; keeping that row")
+ }
+ }
+
+ private def discard(uid: Int): Unit = {
+ try kubernetes.deletePod(uid)
Review Comment:
The rotation fix makes rebuilds a normal path rather than crash recovery — after a secret rotation, every user's next request goes through `discard()` → `provision()`. On a real cluster that first rebuild fails: `deletePod` returns while the pod is still Terminating (default grace period), `podExists` counts a Terminating pod as present, so `createIfAbsent` skips the create and `waitUntilAccepting` polls the dying pod until the 60s timeout — the user waits the full minute, gets "unavailable", and only a retry succeeds. The stubbed client deletes synchronously, so the rebuild spec can't see this.
Making `discard` wait the pod out would close it: delete with `withGracePeriod(0)` (a discarded pod has nothing worth a graceful shutdown) and poll `podExists` until false before returning. Treating Terminating as absent wouldn't be enough on its own — the create would then hit 409 AlreadyExists, which `createIfAbsent` reads as benign concurrency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8032:
URL: https://github.com/apache/texera/pull/8032#issuecomment-5434022366
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 0 better · 🔴 7 worse · ⚪ 8 noise (<±5%) · 0 without baseline
> Compared against main `57a4230` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/33036377169)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 399 | 0.244 | 24,626/32,815/32,815 us | 🔴 -9.9% / 🔴 +137.0% |
| 🔴 | bs=100 sw=10 sl=64 | 802 | 0.49 | 121,133/179,574/179,574 us | 🔴 +24.9% / 🔴 +91.7% |
| ⚪ | bs=1000 sw=10 sl=64 | 937 | 0.572 | 1,063,545/1,185,558/1,185,558 us | ⚪ within ±5% / 🔴 +31.3% |
<details><summary>Baseline details</summary>
Latest main `57a4230` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 399 tuples/sec | 443 tuples/sec | 894.86 tuples/sec | -9.9% | -55.4% |
| bs=10 sw=10 sl=64 | MB/s | 0.244 MB/s | 0.27 MB/s | 0.546 MB/s | -9.6% | -55.3% |
| bs=10 sw=10 sl=64 | p50 | 24,626 us | 23,000 us | 11,139 us | +7.1% | +121.1% |
| bs=10 sw=10 sl=64 | p95 | 32,815 us | 30,612 us | 13,843 us | +7.2% | +137.0% |
| bs=10 sw=10 sl=64 | p99 | 32,815 us | 30,612 us | 16,839 us | +7.2% | +94.9% |
| bs=100 sw=10 sl=64 | throughput | 802 tuples/sec | 822 tuples/sec | 1,166 tuples/sec | -2.4% | -31.2% |
| bs=100 sw=10 sl=64 | MB/s | 0.49 MB/s | 0.501 MB/s | 0.711 MB/s | -2.2% | -31.1% |
| bs=100 sw=10 sl=64 | p50 | 121,133 us | 119,783 us | 87,509 us | +1.1% | +38.4% |
| bs=100 sw=10 sl=64 | p95 | 179,574 us | 143,740 us | 93,675 us | +24.9% | +91.7% |
| bs=100 sw=10 sl=64 | p99 | 179,574 us | 143,740 us | 102,153 us | +24.9% | +75.8% |
| bs=1000 sw=10 sl=64 | throughput | 937 tuples/sec | 915 tuples/sec | 1,198 tuples/sec | +2.4% | -21.8% |
| bs=1000 sw=10 sl=64 | MB/s | 0.572 MB/s | 0.559 MB/s | 0.731 MB/s | +2.3% | -21.8% |
| bs=1000 sw=10 sl=64 | p50 | 1,063,545 us | 1,089,176 us | 859,766 us | -2.4% | +23.7% |
| bs=1000 sw=10 sl=64 | p95 | 1,185,558 us | 1,172,513 us | 902,897 us | +1.1% | +31.3% |
| bs=1000 sw=10 sl=64 | p99 | 1,185,558 us | 1,172,513 us | 937,957 us | +1.1% | +26.4% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,500.97,200,128000,399,0.244,24625.81,32814.90,32814.90
1,100,10,64,20,2493.35,2000,1280000,802,0.490,121133.35,179574.29,179574.29
2,1000,10,64,20,21346.70,20000,12800000,937,0.572,1063545.08,1185557.97,1185557.97
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3918491805
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala:
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.fabric8.kubernetes.client.KubernetesClientException
+import org.apache.texera.common.config.{KubernetesConfig, StorageConfig}
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.daos.UserJupyterDao
+import org.apache.texera.dao.jooq.generated.tables.pojos.UserJupyter
+import org.jooq.exception.DataAccessException
+
+import scala.util.control.NonFatal
+
+/**
+ * Brings a user's JupyterLab into existence and registers where it lives.
+ *
+ * Dependencies are constructor parameters so the provisioning logic can be tested without a
+ * cluster; the companion object binds the production ones.
+ */
+class JupyterProvisioner(
+ kubernetesClient: => JupyterKubernetesClient,
+ accepts: (String, String) => Boolean,
+ publicUrlTemplate: String,
+ readinessTimeoutMillis: Long,
+ readinessPollMillis: Long
+) extends LazyLogging {
+
+ // By-name above, forced once here, so no client is built unless a provision happens.
+ private lazy val kubernetes = kubernetesClient
+
+ /**
+ * The user's Jupyter, starting one if they have none. None means it could not be made
+ * ready, which callers report the same as an unreachable server.
+ *
+ * A registered pod that no longer answers is discarded and rebuilt: the row would otherwise
+ * outlive the pod and point every later request at nothing.
+ */
+ def ensure(
+ uid: Int,
+ jupyterEnabled: Boolean = KubernetesConfig.jupyterEnabled,
+ fallback: JupyterEndpoints = JupyterEndpoints.configured,
+ tokenSecret: String = StorageConfig.jupyterTokenSecret
+ ): Option[JupyterEndpoints] = {
+ if (!jupyterEnabled) return Some(fallback)
+
+ val token = JupyterTokenDeriver.derive(uid, tokenSecret)
+ JupyterEndpointResolver.resolve(uid, jupyterEnabled = true, tokenSecret = tokenSecret) match {
+ case Some(endpoints) if accepts(endpoints.internalUrl, token) => Some(endpoints)
+ case Some(endpoints) =>
+ // Either the pod is gone, or it predates a token-secret rotation and no longer
+ // accepts the derived token. Both are unusable and both are fixed by rebuilding.
+ logger.warn(
+ s"Jupyter for user $uid is registered at ${endpoints.internalUrl} but does not "
+ + "accept its current token; rebuilding it"
+ )
+ discard(uid)
+ provision(uid, token)
+ case None => provision(uid, token)
+ }
+ }
+
+ private def provision(uid: Int, token: String): Option[JupyterEndpoints] = {
+ val internalUrl = s"http://${kubernetes.generatePodURI(uid)}"
+ val endpoints = JupyterEndpoints(internalUrl, publicUrlFor(uid, internalUrl), token)
+ try {
+ createIfAbsent(uid, token)
+ if (!waitUntilAccepting(internalUrl, token)) {
+ logger.error(s"Jupyter for user $uid did not become ready; removing the pod")
+ kubernetes.deletePod(uid)
+ None
+ } else {
+ register(endpoints, uid)
+ Some(endpoints)
+ }
+ } catch {
+ case NonFatal(e) =>
+ logger.error(s"Failed to provision Jupyter for user $uid", e)
+ None
+ }
+ }
+
+ // Two concurrent first requests can both find the pod absent and both create it. The
+ // loser's create returns 409, and the winner's pod is the one it wanted anyway, since both
+ // derive the same token from the same uid, so the conflict is a success. Mirrors
+ // register()'s handling of a duplicate row.
+ private def createIfAbsent(uid: Int, token: String): Unit = {
+ if (kubernetes.podExists(uid)) return
+ try kubernetes.createPod(uid, token)
+ catch {
+ case e: KubernetesClientException if e.getCode == 409 =>
+ logger.info(s"Jupyter pod for user $uid was created concurrently; using that one")
+ }
+ }
+
+ /** Browser-facing address; the in-cluster name does not resolve from the browser. */
+ private def publicUrlFor(uid: Int, internalUrl: String): String =
+ if (publicUrlTemplate.isEmpty) internalUrl
+ else publicUrlTemplate.replace("{uid}", uid.toString)
+
+ // Waits for the pod to accept its own token rather than merely to answer, so a pod that
+ // starts with the wrong token never gets registered.
+ private def waitUntilAccepting(internalUrl: String, token: String): Boolean = {
+ val deadline = System.currentTimeMillis() + readinessTimeoutMillis
+ var ready = accepts(internalUrl, token)
+ while (!ready && System.currentTimeMillis() < deadline) {
+ Thread.sleep(readinessPollMillis)
+ ready = accepts(internalUrl, token)
+ }
+ ready
+ }
Review Comment:
Bounded in 5077f9333 rather than made asynchronous. A semaphore caps concurrent provisions at 16, well under Dropwizard's 1024 worker threads, and requests past the cap are refused immediately instead of queueing behind a readiness wait, so the pool can no longer be drained. The fast path stays outside the permit, so a user with a working pod is unaffected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3908689113
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterKubernetesClient.scala:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import io.fabric8.kubernetes.api.model.{
+ EnvVarBuilder,
+ Pod,
+ PodBuilder,
+ Quantity,
+ ResourceRequirementsBuilder
+}
+import io.fabric8.kubernetes.client.KubernetesClientBuilder
+import org.apache.texera.common.config.KubernetesConfig
+
+/**
+ * Thin wrapper over the fabric8 client for per-user JupyterLab pods, mirroring the computing
+ * unit's KubernetesClient. The fabric8 client is a constructor parameter rather than a global
+ * so tests can exercise the naming and addressing without a live cluster.
+ */
+class JupyterKubernetesClient(client: io.fabric8.kubernetes.client.KubernetesClient) {
+
+ private val namespace: String = KubernetesConfig.jupyterNamespace
+ private val podNamePrefix = "jupyter"
+
+ def generatePodName(uid: Int): String = s"$podNamePrefix-$uid"
+
+ /** The in-cluster address of a user's pod, resolvable via the headless service. */
+ def generatePodURI(uid: Int): String =
+ s"${generatePodName(uid)}.${KubernetesConfig.jupyterServiceName}.$namespace.svc.cluster.local:${KubernetesConfig.jupyterPortNumber}"
+
+ def podExists(uid: Int): Boolean = getPodByName(generatePodName(uid)).isDefined
+
+ def getPodByName(podName: String): Option[Pod] =
+ Option(client.pods().inNamespace(namespace).withName(podName).get())
+
+ /**
+ * Starts a user's JupyterLab. The token is passed as JUPYTER_TOKEN, which is what the image's
+ * start-texera-jupyter.sh reads, so each pod ends up with its owner's token and no other.
+ * Hostname and subdomain are what make generatePodURI resolve.
+ */
+ def createPod(uid: Int, token: String): Pod = {
+ val podName = generatePodName(uid)
+
+ val resources = new ResourceRequirementsBuilder()
+ .addToLimits("cpu", new Quantity(KubernetesConfig.jupyterCpuLimit))
+ .addToLimits("memory", new Quantity(KubernetesConfig.jupyterMemoryLimit))
+ .build()
+
+ val pod = new PodBuilder()
+ .withNewMetadata()
+ .withName(podName)
+ .withNamespace(namespace)
+ .addToLabels("type", "jupyter")
+ .addToLabels("uid", uid.toString)
+ .addToLabels("name", podName)
+ .endMetadata()
+ .withNewSpec()
+ .addNewContainer()
+ .withName("jupyter")
+ .withImage(KubernetesConfig.jupyterImageName)
+ .withImagePullPolicy(KubernetesConfig.computingUnitImagePullPolicy)
+ .addNewPort()
+ .withContainerPort(KubernetesConfig.jupyterPortNumber)
+ .endPort()
+ .withEnv(
+ new EnvVarBuilder().withName("JUPYTER_TOKEN").withValue(token).build()
Review Comment:
This layer owns it. b62c1d8ae adds a kubernetes.jupyter-texera-origin setting, empty by
default, passed into the pod as TEXERA_ORIGIN. Empty leaves the image's local development
default in place, and #8006 supplies the real origin from the Helm chart. Verified on the
cluster in #8006, where the iframe loads and cell click sync works.
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.common.config.{KubernetesConfig, StorageConfig}
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.daos.UserJupyterDao
+import org.apache.texera.dao.jooq.generated.tables.pojos.UserJupyter
+import org.jooq.exception.DataAccessException
+
+import scala.util.control.NonFatal
+
+/**
+ * Brings a user's JupyterLab into existence and registers where it lives.
+ *
+ * Dependencies are constructor parameters so the provisioning logic can be tested without a
+ * cluster; the companion object binds the production ones.
+ */
+class JupyterProvisioner(
+ kubernetesClient: => JupyterKubernetesClient,
+ isReachable: String => Boolean,
+ publicUrlTemplate: String,
+ readinessTimeoutMillis: Long,
+ readinessPollMillis: Long
+) extends LazyLogging {
+
+ // By-name above, forced once here, so no client is built unless a provision happens.
+ private lazy val kubernetes = kubernetesClient
+
+ /**
+ * The user's Jupyter, starting one if they have none. None means it could not be made
+ * ready, which callers report the same as an unreachable server.
+ *
+ * A registered pod that no longer answers is discarded and rebuilt: the row would otherwise
+ * outlive the pod and point every later request at nothing.
+ */
+ def ensure(
+ uid: Int,
+ jupyterEnabled: Boolean = KubernetesConfig.jupyterEnabled,
+ fallback: JupyterEndpoints = JupyterEndpoints.configured,
+ tokenSecret: String = StorageConfig.jupyterTokenSecret
+ ): Option[JupyterEndpoints] = {
+ if (!jupyterEnabled) return Some(fallback)
+
+ val token = JupyterTokenDeriver.derive(uid, tokenSecret)
+ JupyterEndpointResolver.resolve(uid, jupyterEnabled = true, tokenSecret = tokenSecret) match {
+ case Some(endpoints) if isReachable(endpoints.internalUrl) => Some(endpoints)
+ case Some(endpoints) =>
+ logger.warn(
+ s"Jupyter for user $uid is registered at ${endpoints.internalUrl} but "
+ + "unreachable; rebuilding it"
+ )
+ discard(uid)
+ provision(uid, token)
+ case None => provision(uid, token)
+ }
+ }
+
+ private def provision(uid: Int, token: String): Option[JupyterEndpoints] = {
+ val internalUrl = s"http://${kubernetes.generatePodURI(uid)}"
+ val endpoints = JupyterEndpoints(internalUrl, publicUrlFor(uid, internalUrl), token)
+ try {
+ if (!kubernetes.podExists(uid)) kubernetes.createPod(uid, token)
Review Comment:
Fixed in bb67c3511. createIfAbsent now catches KubernetesClientException with code 409 and
falls through to the readiness wait, so the loser of two concurrent first requests reuses the
winner's pod instead of reporting unavailable. That mirrors register()'s 23505 handling, and
the reasoning carries over cleanly since both writers produce an identical pod spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3884441110
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.common.config.{KubernetesConfig, StorageConfig}
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.daos.UserJupyterDao
+import org.apache.texera.dao.jooq.generated.tables.pojos.UserJupyter
+import org.jooq.exception.DataAccessException
+
+import scala.util.control.NonFatal
+
+/**
+ * Brings a user's JupyterLab into existence and registers where it lives.
+ *
+ * Dependencies are constructor parameters so the provisioning logic can be tested without a
+ * cluster; the companion object binds the production ones.
+ */
+class JupyterProvisioner(
+ kubernetesClient: => JupyterKubernetesClient,
+ isReachable: String => Boolean,
+ publicUrlTemplate: String,
+ readinessTimeoutMillis: Long,
+ readinessPollMillis: Long
+) extends LazyLogging {
+
+ // By-name above, forced once here, so no client is built unless a provision happens.
+ private lazy val kubernetes = kubernetesClient
+
+ /**
+ * The user's Jupyter, starting one if they have none. None means it could not be made
+ * ready, which callers report the same as an unreachable server.
+ *
+ * A registered pod that no longer answers is discarded and rebuilt: the row would otherwise
+ * outlive the pod and point every later request at nothing.
+ */
+ def ensure(
+ uid: Int,
+ jupyterEnabled: Boolean = KubernetesConfig.jupyterEnabled,
+ fallback: JupyterEndpoints = JupyterEndpoints.configured,
+ tokenSecret: String = StorageConfig.jupyterTokenSecret
+ ): Option[JupyterEndpoints] = {
+ if (!jupyterEnabled) return Some(fallback)
+
+ val token = JupyterTokenDeriver.derive(uid, tokenSecret)
+ JupyterEndpointResolver.resolve(uid, jupyterEnabled = true, tokenSecret = tokenSecret) match {
+ case Some(endpoints) if isReachable(endpoints.internalUrl) => Some(endpoints)
Review Comment:
After `token-secret` rotation, the resolver supplies the newly derived token, but this unauthenticated liveness check still accepts the existing pod. That pod was started with the old `JUPYTER_TOKEN`, so iframe, upload, and delete requests use a token it rejects and the pod is never rebuilt. Verify the derived token when deciding whether to reuse a pod, or persist a token-version fingerprint and recreate on mismatch.
This issue also appears in the following locations of the same file:
- line 69
- line 79
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3911768155
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterTokenDeriver.scala:
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import org.apache.texera.common.config.{KubernetesConfig, StorageConfig}
+
+import javax.crypto.Mac
+import javax.crypto.spec.SecretKeySpec
+import java.nio.charset.StandardCharsets.UTF_8
+
+/**
+ * Derives each user's JupyterLab token from a server-held secret instead of storing one.
+ * The value is stable for a uid until the secret changes, so any replica of this service
+ * derives the same token and no credential is kept at rest.
+ */
+object JupyterTokenDeriver {
+
+ private val algorithm = "HmacSHA256"
+
+ // 128 bits of the digest, which is ample for a token and keeps the URL short.
+ private val tokenLength = 32
+
+ def derive(uid: Int, secret: String = StorageConfig.jupyterTokenSecret): String = {
+ require(secret.nonEmpty, "cannot derive a Jupyter token from an empty secret")
+ val mac = Mac.getInstance(algorithm)
+ mac.init(new SecretKeySpec(secret.getBytes(UTF_8), algorithm))
+ mac.doFinal(uid.toString.getBytes(UTF_8)).map("%02x".format(_)).mkString.take(tokenLength)
+ }
Review Comment:
The hex encoding uses `"%02x".format(_)` directly on `Byte` values; because `Byte` is signed, formatting can produce sign-extended hex (e.g., `ffffffff...`) rather than a 2-digit byte representation, skewing/truncating the token unpredictably. Mask each byte with `0xff` (or use a known-correct hex encoder) before formatting to ensure the derived token is a correct truncation of the HMAC digest.
##########
sql/updates/45.sql:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+\c texera_db
+
+SET search_path TO texera_db;
+
Review Comment:
`\c texera_db` is a psql meta-command and will fail when executed by Liquibase/JDBC (`sqlFile` in `changelog.xml`). To keep this migration runnable in CI/prod, remove `\c` (and typically the explicit `SET search_path`) and rely on the configured database/schema, or use Liquibase-supported attributes (e.g., `defaultSchemaName`) rather than psql directives.
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProbe.scala:
##########
@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import java.net.{HttpURLConnection, URL}
+
+/** Liveness check for a Jupyter server. */
+object JupyterProbe {
+
+ private val timeoutMillis = 2000
+
+ /**
+ * Whether Jupyter on `internalUrl` accepts `token`. Distinct from isAvailable: a pod
+ * started with a superseded token is still perfectly alive, so liveness alone cannot
+ * tell a healthy pod from one whose token no longer matches the derived value. Probes
+ * an endpoint that requires authentication, so 403 means the token is wrong.
+ */
+ def isAuthorized(internalUrl: String, token: String): Boolean = {
+ var conn: HttpURLConnection = null
+ try {
+ conn = new URL(s"$internalUrl/api/contents")
+ .openConnection()
+ .asInstanceOf[HttpURLConnection]
+ conn.setRequestMethod("GET")
+ conn.setRequestProperty("Authorization", s"token $token")
+ conn.setConnectTimeout(timeoutMillis)
+ conn.setReadTimeout(timeoutMillis)
+ conn.getResponseCode == 200
+ } catch {
+ case _: Exception => false
+ } finally {
+ if (conn != null) conn.disconnect()
+ }
Review Comment:
Catching `Exception` will also swallow `InterruptedException`, which can break cooperative shutdown/cancellation and leave threads in an interrupted state without restoring it. Prefer catching `scala.util.control.NonFatal` and, if an interrupt is caught, re-set the interrupt flag before returning.
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala:
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.fabric8.kubernetes.client.KubernetesClientException
+import org.apache.texera.common.config.{KubernetesConfig, StorageConfig}
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.daos.UserJupyterDao
+import org.apache.texera.dao.jooq.generated.tables.pojos.UserJupyter
+import org.jooq.exception.DataAccessException
+
+import scala.util.control.NonFatal
+
+/**
+ * Brings a user's JupyterLab into existence and registers where it lives.
+ *
+ * Dependencies are constructor parameters so the provisioning logic can be tested without a
+ * cluster; the companion object binds the production ones.
+ */
+class JupyterProvisioner(
+ kubernetesClient: => JupyterKubernetesClient,
+ accepts: (String, String) => Boolean,
+ publicUrlTemplate: String,
+ readinessTimeoutMillis: Long,
+ readinessPollMillis: Long
+) extends LazyLogging {
+
+ // By-name above, forced once here, so no client is built unless a provision happens.
+ private lazy val kubernetes = kubernetesClient
+
+ /**
+ * The user's Jupyter, starting one if they have none. None means it could not be made
+ * ready, which callers report the same as an unreachable server.
+ *
+ * A registered pod that no longer answers is discarded and rebuilt: the row would otherwise
+ * outlive the pod and point every later request at nothing.
+ */
+ def ensure(
+ uid: Int,
+ jupyterEnabled: Boolean = KubernetesConfig.jupyterEnabled,
+ fallback: JupyterEndpoints = JupyterEndpoints.configured,
+ tokenSecret: String = StorageConfig.jupyterTokenSecret
+ ): Option[JupyterEndpoints] = {
+ if (!jupyterEnabled) return Some(fallback)
+
+ val token = JupyterTokenDeriver.derive(uid, tokenSecret)
+ JupyterEndpointResolver.resolve(uid, jupyterEnabled = true, tokenSecret = tokenSecret) match {
+ case Some(endpoints) if accepts(endpoints.internalUrl, token) => Some(endpoints)
+ case Some(endpoints) =>
+ // Either the pod is gone, or it predates a token-secret rotation and no longer
+ // accepts the derived token. Both are unusable and both are fixed by rebuilding.
+ logger.warn(
+ s"Jupyter for user $uid is registered at ${endpoints.internalUrl} but does not "
+ + "accept its current token; rebuilding it"
+ )
+ discard(uid)
+ provision(uid, token)
+ case None => provision(uid, token)
+ }
+ }
+
+ private def provision(uid: Int, token: String): Option[JupyterEndpoints] = {
+ val internalUrl = s"http://${kubernetes.generatePodURI(uid)}"
+ val endpoints = JupyterEndpoints(internalUrl, publicUrlFor(uid, internalUrl), token)
+ try {
+ createIfAbsent(uid, token)
+ if (!waitUntilAccepting(internalUrl, token)) {
+ logger.error(s"Jupyter for user $uid did not become ready; removing the pod")
+ kubernetes.deletePod(uid)
+ None
+ } else {
+ register(endpoints, uid)
+ Some(endpoints)
+ }
+ } catch {
+ case NonFatal(e) =>
+ logger.error(s"Failed to provision Jupyter for user $uid", e)
+ None
+ }
+ }
+
+ // Two concurrent first requests can both find the pod absent and both create it. The
+ // loser's create returns 409, and the winner's pod is the one it wanted anyway, since both
+ // derive the same token from the same uid, so the conflict is a success. Mirrors
+ // register()'s handling of a duplicate row.
+ private def createIfAbsent(uid: Int, token: String): Unit = {
+ if (kubernetes.podExists(uid)) return
+ try kubernetes.createPod(uid, token)
+ catch {
+ case e: KubernetesClientException if e.getCode == 409 =>
+ logger.info(s"Jupyter pod for user $uid was created concurrently; using that one")
+ }
+ }
+
+ /** Browser-facing address; the in-cluster name does not resolve from the browser. */
+ private def publicUrlFor(uid: Int, internalUrl: String): String =
+ if (publicUrlTemplate.isEmpty) internalUrl
+ else publicUrlTemplate.replace("{uid}", uid.toString)
+
+ // Waits for the pod to accept its own token rather than merely to answer, so a pod that
+ // starts with the wrong token never gets registered.
+ private def waitUntilAccepting(internalUrl: String, token: String): Boolean = {
+ val deadline = System.currentTimeMillis() + readinessTimeoutMillis
+ var ready = accepts(internalUrl, token)
+ while (!ready && System.currentTimeMillis() < deadline) {
+ Thread.sleep(readinessPollMillis)
+ ready = accepts(internalUrl, token)
+ }
+ ready
+ }
Review Comment:
This blocks the calling thread (potentially a Jersey request thread) with `Thread.sleep` loops for up to `readinessTimeoutMillis` (60s in production). Under load, concurrent first-time provisions can exhaust the HTTP worker pool and degrade overall service availability. Consider offloading provisioning/waiting to a dedicated executor/background job and returning a non-blocking response (e.g., `202`/`503` with retry guidance) until ready.
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala:
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.fabric8.kubernetes.client.KubernetesClientException
+import org.apache.texera.common.config.{KubernetesConfig, StorageConfig}
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.daos.UserJupyterDao
+import org.apache.texera.dao.jooq.generated.tables.pojos.UserJupyter
+import org.jooq.exception.DataAccessException
+
+import scala.util.control.NonFatal
+
+/**
+ * Brings a user's JupyterLab into existence and registers where it lives.
+ *
+ * Dependencies are constructor parameters so the provisioning logic can be tested without a
+ * cluster; the companion object binds the production ones.
+ */
+class JupyterProvisioner(
+ kubernetesClient: => JupyterKubernetesClient,
+ accepts: (String, String) => Boolean,
+ publicUrlTemplate: String,
+ readinessTimeoutMillis: Long,
+ readinessPollMillis: Long
+) extends LazyLogging {
+
+ // By-name above, forced once here, so no client is built unless a provision happens.
+ private lazy val kubernetes = kubernetesClient
+
+ /**
+ * The user's Jupyter, starting one if they have none. None means it could not be made
+ * ready, which callers report the same as an unreachable server.
+ *
+ * A registered pod that no longer answers is discarded and rebuilt: the row would otherwise
+ * outlive the pod and point every later request at nothing.
+ */
+ def ensure(
+ uid: Int,
+ jupyterEnabled: Boolean = KubernetesConfig.jupyterEnabled,
+ fallback: JupyterEndpoints = JupyterEndpoints.configured,
+ tokenSecret: String = StorageConfig.jupyterTokenSecret
+ ): Option[JupyterEndpoints] = {
+ if (!jupyterEnabled) return Some(fallback)
+
+ val token = JupyterTokenDeriver.derive(uid, tokenSecret)
+ JupyterEndpointResolver.resolve(uid, jupyterEnabled = true, tokenSecret = tokenSecret) match {
+ case Some(endpoints) if accepts(endpoints.internalUrl, token) => Some(endpoints)
+ case Some(endpoints) =>
+ // Either the pod is gone, or it predates a token-secret rotation and no longer
+ // accepts the derived token. Both are unusable and both are fixed by rebuilding.
+ logger.warn(
+ s"Jupyter for user $uid is registered at ${endpoints.internalUrl} but does not "
+ + "accept its current token; rebuilding it"
+ )
+ discard(uid)
+ provision(uid, token)
+ case None => provision(uid, token)
+ }
+ }
+
+ private def provision(uid: Int, token: String): Option[JupyterEndpoints] = {
+ val internalUrl = s"http://${kubernetes.generatePodURI(uid)}"
+ val endpoints = JupyterEndpoints(internalUrl, publicUrlFor(uid, internalUrl), token)
+ try {
+ createIfAbsent(uid, token)
+ if (!waitUntilAccepting(internalUrl, token)) {
+ logger.error(s"Jupyter for user $uid did not become ready; removing the pod")
+ kubernetes.deletePod(uid)
+ None
+ } else {
+ register(endpoints, uid)
+ Some(endpoints)
+ }
+ } catch {
+ case NonFatal(e) =>
+ logger.error(s"Failed to provision Jupyter for user $uid", e)
+ None
+ }
+ }
+
+ // Two concurrent first requests can both find the pod absent and both create it. The
+ // loser's create returns 409, and the winner's pod is the one it wanted anyway, since both
+ // derive the same token from the same uid, so the conflict is a success. Mirrors
+ // register()'s handling of a duplicate row.
+ private def createIfAbsent(uid: Int, token: String): Unit = {
+ if (kubernetes.podExists(uid)) return
+ try kubernetes.createPod(uid, token)
+ catch {
+ case e: KubernetesClientException if e.getCode == 409 =>
+ logger.info(s"Jupyter pod for user $uid was created concurrently; using that one")
+ }
+ }
+
+ /** Browser-facing address; the in-cluster name does not resolve from the browser. */
+ private def publicUrlFor(uid: Int, internalUrl: String): String =
+ if (publicUrlTemplate.isEmpty) internalUrl
+ else publicUrlTemplate.replace("{uid}", uid.toString)
+
+ // Waits for the pod to accept its own token rather than merely to answer, so a pod that
+ // starts with the wrong token never gets registered.
+ private def waitUntilAccepting(internalUrl: String, token: String): Boolean = {
+ val deadline = System.currentTimeMillis() + readinessTimeoutMillis
+ var ready = accepts(internalUrl, token)
+ while (!ready && System.currentTimeMillis() < deadline) {
+ Thread.sleep(readinessPollMillis)
+ ready = accepts(internalUrl, token)
+ }
+ ready
+ }
+
+ private def register(endpoints: JupyterEndpoints, uid: Int): Unit = {
+ val row = new UserJupyter
+ row.setUid(uid)
+ row.setInternalUrl(endpoints.internalUrl)
+ row.setPublicUrl(endpoints.publicUrl)
+ try dao().insert(row)
+ catch {
+ // Two concurrent first requests can both provision. uid is the primary key, so the
+ // loser trips 23505; the winner's row holds the same uid-derived addresses, so leaving
+ // it in place is correct.
+ case e: DataAccessException if e.sqlState == "23505" =>
+ logger.info(s"Jupyter for user $uid was registered concurrently; keeping that row")
+ }
+ }
+
+ private def discard(uid: Int): Unit = {
+ try kubernetes.deletePod(uid)
+ catch { case NonFatal(e) => logger.warn(s"Could not delete stale Jupyter pod for $uid", e) }
+ dao().deleteById(uid)
+ }
+
+ private def dao() = new UserJupyterDao(SqlServer.getInstance().createDSLContext().configuration())
Review Comment:
`dao()` constructs a new `DSLContext` (and DAO) on every call; `ensure()`/`provision()` can call `dao()` multiple times (e.g., `register`, `discard`). This creates avoidable overhead on hot paths. A concrete improvement is to create the DAO once per `ensure()`/`provision()` invocation (or reuse a single DAO/DSLContext if your `SqlServer` provider is thread-safe for that usage) and pass it to helper methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on PR #8032:
URL: https://github.com/apache/texera/pull/8032#issuecomment-5531265739
> LGTM, all threads addressed and verified. Before merging: main has taken 45–47, so the migration needs another renumber to 48 (file + changelog id), plus the merge conflicts.
I've numbered to 48 in [6babcf4](https://github.com/apache/texera/pull/8032/commits/6babcf425acee88c798faa2d5f83d942afcefa95) and merged main in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on PR #8032:
URL: https://github.com/apache/texera/pull/8032#issuecomment-5456781818
Flagging that tests are a large share of this PR: 728 of the 1380 added lines, about 53%, or a 1.40:1 ratio against production code.
Most of that came from a coverage pass after Codecov reported 42 missing lines on the first push. The biggest single file is `JupyterKubernetesClientSpec` at 188 lines, and much of it is Mockito scaffolding rather than assertions. fabric8's fluent API returns type variables, so `RETURNS_DEEP_STUBS` cannot be used and every step of each chain is stubbed by hand. That mirrors what `KubernetesClientSpec` in `computing-unit-managing-service` already does. The payoff is that the pod spec is now asserted field by field, including the hostname and subdomain pair that makes the pod FQDN resolve, which nothing covered before.
The six new files under `util/` are at 100% line coverage with no missed lines. The branch partials that remain are `scala-logging` macro expansions, which would need the suite run at two log levels to cover, so I left them rather than contort the tests.
Worth noting the production side is smaller than the 520 figure suggests: `NotebookMigrationResource.scala` shows 49 added against 49 deleted because `JupyterEndpoints` and the reachability probe moved out into `util/`, so the genuinely new logic is the 451 lines across the six new files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3918494364
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala:
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.fabric8.kubernetes.client.KubernetesClientException
+import org.apache.texera.common.config.{KubernetesConfig, StorageConfig}
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.daos.UserJupyterDao
+import org.apache.texera.dao.jooq.generated.tables.pojos.UserJupyter
+import org.jooq.exception.DataAccessException
+
+import scala.util.control.NonFatal
+
+/**
+ * Brings a user's JupyterLab into existence and registers where it lives.
+ *
+ * Dependencies are constructor parameters so the provisioning logic can be tested without a
+ * cluster; the companion object binds the production ones.
+ */
+class JupyterProvisioner(
+ kubernetesClient: => JupyterKubernetesClient,
+ accepts: (String, String) => Boolean,
+ publicUrlTemplate: String,
+ readinessTimeoutMillis: Long,
+ readinessPollMillis: Long
+) extends LazyLogging {
+
+ // By-name above, forced once here, so no client is built unless a provision happens.
+ private lazy val kubernetes = kubernetesClient
+
+ /**
+ * The user's Jupyter, starting one if they have none. None means it could not be made
+ * ready, which callers report the same as an unreachable server.
+ *
+ * A registered pod that no longer answers is discarded and rebuilt: the row would otherwise
+ * outlive the pod and point every later request at nothing.
+ */
+ def ensure(
+ uid: Int,
+ jupyterEnabled: Boolean = KubernetesConfig.jupyterEnabled,
+ fallback: JupyterEndpoints = JupyterEndpoints.configured,
+ tokenSecret: String = StorageConfig.jupyterTokenSecret
+ ): Option[JupyterEndpoints] = {
+ if (!jupyterEnabled) return Some(fallback)
+
+ val token = JupyterTokenDeriver.derive(uid, tokenSecret)
+ JupyterEndpointResolver.resolve(uid, jupyterEnabled = true, tokenSecret = tokenSecret) match {
+ case Some(endpoints) if accepts(endpoints.internalUrl, token) => Some(endpoints)
+ case Some(endpoints) =>
+ // Either the pod is gone, or it predates a token-secret rotation and no longer
+ // accepts the derived token. Both are unusable and both are fixed by rebuilding.
+ logger.warn(
+ s"Jupyter for user $uid is registered at ${endpoints.internalUrl} but does not "
+ + "accept its current token; rebuilding it"
+ )
+ discard(uid)
+ provision(uid, token)
+ case None => provision(uid, token)
+ }
+ }
+
+ private def provision(uid: Int, token: String): Option[JupyterEndpoints] = {
+ val internalUrl = s"http://${kubernetes.generatePodURI(uid)}"
+ val endpoints = JupyterEndpoints(internalUrl, publicUrlFor(uid, internalUrl), token)
+ try {
+ createIfAbsent(uid, token)
+ if (!waitUntilAccepting(internalUrl, token)) {
+ logger.error(s"Jupyter for user $uid did not become ready; removing the pod")
+ kubernetes.deletePod(uid)
+ None
+ } else {
+ register(endpoints, uid)
+ Some(endpoints)
+ }
+ } catch {
+ case NonFatal(e) =>
+ logger.error(s"Failed to provision Jupyter for user $uid", e)
+ None
+ }
+ }
+
+ // Two concurrent first requests can both find the pod absent and both create it. The
+ // loser's create returns 409, and the winner's pod is the one it wanted anyway, since both
+ // derive the same token from the same uid, so the conflict is a success. Mirrors
+ // register()'s handling of a duplicate row.
+ private def createIfAbsent(uid: Int, token: String): Unit = {
+ if (kubernetes.podExists(uid)) return
+ try kubernetes.createPod(uid, token)
+ catch {
+ case e: KubernetesClientException if e.getCode == 409 =>
+ logger.info(s"Jupyter pod for user $uid was created concurrently; using that one")
+ }
+ }
+
+ /** Browser-facing address; the in-cluster name does not resolve from the browser. */
+ private def publicUrlFor(uid: Int, internalUrl: String): String =
+ if (publicUrlTemplate.isEmpty) internalUrl
+ else publicUrlTemplate.replace("{uid}", uid.toString)
+
+ // Waits for the pod to accept its own token rather than merely to answer, so a pod that
+ // starts with the wrong token never gets registered.
+ private def waitUntilAccepting(internalUrl: String, token: String): Boolean = {
+ val deadline = System.currentTimeMillis() + readinessTimeoutMillis
+ var ready = accepts(internalUrl, token)
+ while (!ready && System.currentTimeMillis() < deadline) {
+ Thread.sleep(readinessPollMillis)
+ ready = accepts(internalUrl, token)
+ }
+ ready
+ }
+
+ private def register(endpoints: JupyterEndpoints, uid: Int): Unit = {
+ val row = new UserJupyter
+ row.setUid(uid)
+ row.setInternalUrl(endpoints.internalUrl)
+ row.setPublicUrl(endpoints.publicUrl)
+ try dao().insert(row)
+ catch {
+ // Two concurrent first requests can both provision. uid is the primary key, so the
+ // loser trips 23505; the winner's row holds the same uid-derived addresses, so leaving
+ // it in place is correct.
+ case e: DataAccessException if e.sqlState == "23505" =>
+ logger.info(s"Jupyter for user $uid was registered concurrently; keeping that row")
+ }
+ }
+
+ private def discard(uid: Int): Unit = {
+ try kubernetes.deletePod(uid)
+ catch { case NonFatal(e) => logger.warn(s"Could not delete stale Jupyter pod for $uid", e) }
+ dao().deleteById(uid)
+ }
+
+ private def dao() = new UserJupyterDao(SqlServer.getInstance().createDSLContext().configuration())
Review Comment:
No change needed. SqlServer.createDSLContext() is `def createDSLContext(): DSLContext = context`, a field read, so no DSLContext is built per call; what is allocated is one UserJupyterDao holding the shared configuration. It also is not a hot path: dao() runs at most twice per ensure and only when a provision or rebuild happened, never on the fast path. Per-call DAO construction is the existing idiom, as in WorkflowResource's `private def workflowDao = new WorkflowDao(context.configuration)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on PR #8032:
URL: https://github.com/apache/texera/pull/8032#issuecomment-5457065877
/request-review @mengw15
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
mengw15 commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3885841705
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala:
##########
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.common.config.{KubernetesConfig, StorageConfig}
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.daos.UserJupyterDao
+import org.apache.texera.dao.jooq.generated.tables.pojos.UserJupyter
+import org.jooq.exception.DataAccessException
+
+import scala.util.control.NonFatal
+
+/**
+ * Brings a user's JupyterLab into existence and registers where it lives.
+ *
+ * Dependencies are constructor parameters so the provisioning logic can be tested without a
+ * cluster; the companion object binds the production ones.
+ */
+class JupyterProvisioner(
+ kubernetesClient: => JupyterKubernetesClient,
+ isReachable: String => Boolean,
+ publicUrlTemplate: String,
+ readinessTimeoutMillis: Long,
+ readinessPollMillis: Long
+) extends LazyLogging {
+
+ // By-name above, forced once here, so no client is built unless a provision happens.
+ private lazy val kubernetes = kubernetesClient
+
+ /**
+ * The user's Jupyter, starting one if they have none. None means it could not be made
+ * ready, which callers report the same as an unreachable server.
+ *
+ * A registered pod that no longer answers is discarded and rebuilt: the row would otherwise
+ * outlive the pod and point every later request at nothing.
+ */
+ def ensure(
+ uid: Int,
+ jupyterEnabled: Boolean = KubernetesConfig.jupyterEnabled,
+ fallback: JupyterEndpoints = JupyterEndpoints.configured,
+ tokenSecret: String = StorageConfig.jupyterTokenSecret
+ ): Option[JupyterEndpoints] = {
+ if (!jupyterEnabled) return Some(fallback)
+
+ val token = JupyterTokenDeriver.derive(uid, tokenSecret)
+ JupyterEndpointResolver.resolve(uid, jupyterEnabled = true, tokenSecret = tokenSecret) match {
+ case Some(endpoints) if isReachable(endpoints.internalUrl) => Some(endpoints)
+ case Some(endpoints) =>
+ logger.warn(
+ s"Jupyter for user $uid is registered at ${endpoints.internalUrl} but "
+ + "unreachable; rebuilding it"
+ )
+ discard(uid)
+ provision(uid, token)
+ case None => provision(uid, token)
+ }
+ }
+
+ private def provision(uid: Int, token: String): Option[JupyterEndpoints] = {
+ val internalUrl = s"http://${kubernetes.generatePodURI(uid)}"
+ val endpoints = JupyterEndpoints(internalUrl, publicUrlFor(uid, internalUrl), token)
+ try {
+ if (!kubernetes.podExists(uid)) kubernetes.createPod(uid, token)
Review Comment:
This is check-then-act on a pod named by uid, so two concurrent first requests can both see it absent and both create; the loser's `.create()` throws 409, lands in the `NonFatal` catch, and that user gets "unavailable" on what should be a success. `register()` below handles exactly this shape for the row — catches 23505 and keeps the winner's, since both writers produce identical content — and the same reasoning applies here: catching the 409 and falling through to `waitUntilReachable` would make the two paths consistent. The computing-unit client never hits this because its pods are keyed on `cuid`, which is never contended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3918489196
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProbe.scala:
##########
@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import java.net.{HttpURLConnection, URL}
+
+/** Liveness check for a Jupyter server. */
+object JupyterProbe {
+
+ private val timeoutMillis = 2000
+
+ /**
+ * Whether Jupyter on `internalUrl` accepts `token`. Distinct from isAvailable: a pod
+ * started with a superseded token is still perfectly alive, so liveness alone cannot
+ * tell a healthy pod from one whose token no longer matches the derived value. Probes
+ * an endpoint that requires authentication, so 403 means the token is wrong.
+ */
+ def isAuthorized(internalUrl: String, token: String): Boolean = {
+ var conn: HttpURLConnection = null
+ try {
+ conn = new URL(s"$internalUrl/api/contents")
+ .openConnection()
+ .asInstanceOf[HttpURLConnection]
+ conn.setRequestMethod("GET")
+ conn.setRequestProperty("Authorization", s"token $token")
+ conn.setConnectTimeout(timeoutMillis)
+ conn.setReadTimeout(timeoutMillis)
+ conn.getResponseCode == 200
+ } catch {
+ case _: Exception => false
+ } finally {
+ if (conn != null) conn.disconnect()
+ }
Review Comment:
Changed in ce3194462, though not for the reason given: InterruptedException is checked and nothing in the HttpURLConnection path declares it, and java.net socket reads are not interruptible, so no interrupt can reach that catch today. NonFatal is still the right idiom here, it is what JupyterProvisioner and NotebookMigrationResource already use, and it makes a future port to java.net.http.HttpClient safe, since send does declare InterruptedException. NonFatal excludes it, so the interrupt propagates rather than being caught and a flag restored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
mengw15 merged PR #8032:
URL: https://github.com/apache/texera/pull/8032
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3918491805
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterProvisioner.scala:
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import com.typesafe.scalalogging.LazyLogging
+import io.fabric8.kubernetes.client.KubernetesClientException
+import org.apache.texera.common.config.{KubernetesConfig, StorageConfig}
+import org.apache.texera.dao.SqlServer
+import org.apache.texera.dao.jooq.generated.tables.daos.UserJupyterDao
+import org.apache.texera.dao.jooq.generated.tables.pojos.UserJupyter
+import org.jooq.exception.DataAccessException
+
+import scala.util.control.NonFatal
+
+/**
+ * Brings a user's JupyterLab into existence and registers where it lives.
+ *
+ * Dependencies are constructor parameters so the provisioning logic can be tested without a
+ * cluster; the companion object binds the production ones.
+ */
+class JupyterProvisioner(
+ kubernetesClient: => JupyterKubernetesClient,
+ accepts: (String, String) => Boolean,
+ publicUrlTemplate: String,
+ readinessTimeoutMillis: Long,
+ readinessPollMillis: Long
+) extends LazyLogging {
+
+ // By-name above, forced once here, so no client is built unless a provision happens.
+ private lazy val kubernetes = kubernetesClient
+
+ /**
+ * The user's Jupyter, starting one if they have none. None means it could not be made
+ * ready, which callers report the same as an unreachable server.
+ *
+ * A registered pod that no longer answers is discarded and rebuilt: the row would otherwise
+ * outlive the pod and point every later request at nothing.
+ */
+ def ensure(
+ uid: Int,
+ jupyterEnabled: Boolean = KubernetesConfig.jupyterEnabled,
+ fallback: JupyterEndpoints = JupyterEndpoints.configured,
+ tokenSecret: String = StorageConfig.jupyterTokenSecret
+ ): Option[JupyterEndpoints] = {
+ if (!jupyterEnabled) return Some(fallback)
+
+ val token = JupyterTokenDeriver.derive(uid, tokenSecret)
+ JupyterEndpointResolver.resolve(uid, jupyterEnabled = true, tokenSecret = tokenSecret) match {
+ case Some(endpoints) if accepts(endpoints.internalUrl, token) => Some(endpoints)
+ case Some(endpoints) =>
+ // Either the pod is gone, or it predates a token-secret rotation and no longer
+ // accepts the derived token. Both are unusable and both are fixed by rebuilding.
+ logger.warn(
+ s"Jupyter for user $uid is registered at ${endpoints.internalUrl} but does not "
+ + "accept its current token; rebuilding it"
+ )
+ discard(uid)
+ provision(uid, token)
+ case None => provision(uid, token)
+ }
+ }
+
+ private def provision(uid: Int, token: String): Option[JupyterEndpoints] = {
+ val internalUrl = s"http://${kubernetes.generatePodURI(uid)}"
+ val endpoints = JupyterEndpoints(internalUrl, publicUrlFor(uid, internalUrl), token)
+ try {
+ createIfAbsent(uid, token)
+ if (!waitUntilAccepting(internalUrl, token)) {
+ logger.error(s"Jupyter for user $uid did not become ready; removing the pod")
+ kubernetes.deletePod(uid)
+ None
+ } else {
+ register(endpoints, uid)
+ Some(endpoints)
+ }
+ } catch {
+ case NonFatal(e) =>
+ logger.error(s"Failed to provision Jupyter for user $uid", e)
+ None
+ }
+ }
+
+ // Two concurrent first requests can both find the pod absent and both create it. The
+ // loser's create returns 409, and the winner's pod is the one it wanted anyway, since both
+ // derive the same token from the same uid, so the conflict is a success. Mirrors
+ // register()'s handling of a duplicate row.
+ private def createIfAbsent(uid: Int, token: String): Unit = {
+ if (kubernetes.podExists(uid)) return
+ try kubernetes.createPod(uid, token)
+ catch {
+ case e: KubernetesClientException if e.getCode == 409 =>
+ logger.info(s"Jupyter pod for user $uid was created concurrently; using that one")
+ }
+ }
+
+ /** Browser-facing address; the in-cluster name does not resolve from the browser. */
+ private def publicUrlFor(uid: Int, internalUrl: String): String =
+ if (publicUrlTemplate.isEmpty) internalUrl
+ else publicUrlTemplate.replace("{uid}", uid.toString)
+
+ // Waits for the pod to accept its own token rather than merely to answer, so a pod that
+ // starts with the wrong token never gets registered.
+ private def waitUntilAccepting(internalUrl: String, token: String): Boolean = {
+ val deadline = System.currentTimeMillis() + readinessTimeoutMillis
+ var ready = accepts(internalUrl, token)
+ while (!ready && System.currentTimeMillis() < deadline) {
+ Thread.sleep(readinessPollMillis)
+ ready = accepts(internalUrl, token)
+ }
+ ready
+ }
Review Comment:
Bounded in 5077f9333 rather than made asynchronous. A semaphore caps concurrent provisions at 16, well under Dropwizard's 1024 worker threads, and requests past the cap are refused immediately instead of queueing behind a readiness wait, so the pool can no longer be drained. The fast path stays outside the permit, so a user with a working pod is unaffected. Returning 202 and polling is the better end state and matches how the computing unit service provisions, but it needs a frontend change, so it belongs in its own issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "zyratlo (via GitHub)" <gi...@apache.org>.
zyratlo commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3918481715
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterTokenDeriver.scala:
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import org.apache.texera.common.config.{KubernetesConfig, StorageConfig}
+
+import javax.crypto.Mac
+import javax.crypto.spec.SecretKeySpec
+import java.nio.charset.StandardCharsets.UTF_8
+
+/**
+ * Derives each user's JupyterLab token from a server-held secret instead of storing one.
+ * The value is stable for a uid until the secret changes, so any replica of this service
+ * derives the same token and no credential is kept at rest.
+ */
+object JupyterTokenDeriver {
+
+ private val algorithm = "HmacSHA256"
+
+ // 128 bits of the digest, which is ample for a token and keeps the URL short.
+ private val tokenLength = 32
+
+ def derive(uid: Int, secret: String = StorageConfig.jupyterTokenSecret): String = {
+ require(secret.nonEmpty, "cannot derive a Jupyter token from an empty secret")
+ val mac = Mac.getInstance(algorithm)
+ mac.init(new SecretKeySpec(secret.getBytes(UTF_8), algorithm))
+ mac.doFinal(uid.toString.getBytes(UTF_8)).map("%02x".format(_)).mkString.take(tokenLength)
+ }
Review Comment:
No change needed. Java's Formatter adds 2^8 to a negative value for %x and %o with a Byte argument, so "%02x".format on java.lang.Byte -1 yields "ff", not "ffffffff". Scala's format takes Any*, so the Byte boxes to java.lang.Byte and never widens to Int; masking would be a no-op. The spec could not have caught it either way, so d688d2ec9 adds a case pinning derive against an independently computed HMAC whose digest starts 0xa5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(notebook-migration): resolve the Jupyter URL and token per user [texera]
Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
mengw15 commented on code in PR #8032:
URL: https://github.com/apache/texera/pull/8032#discussion_r3885854629
##########
notebook-migration-service/src/main/scala/org/apache/texera/service/util/JupyterKubernetesClient.scala:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.texera.service.util
+
+import io.fabric8.kubernetes.api.model.{
+ EnvVarBuilder,
+ Pod,
+ PodBuilder,
+ Quantity,
+ ResourceRequirementsBuilder
+}
+import io.fabric8.kubernetes.client.KubernetesClientBuilder
+import org.apache.texera.common.config.KubernetesConfig
+
+/**
+ * Thin wrapper over the fabric8 client for per-user JupyterLab pods, mirroring the computing
+ * unit's KubernetesClient. The fabric8 client is a constructor parameter rather than a global
+ * so tests can exercise the naming and addressing without a live cluster.
+ */
+class JupyterKubernetesClient(client: io.fabric8.kubernetes.client.KubernetesClient) {
+
+ private val namespace: String = KubernetesConfig.jupyterNamespace
+ private val podNamePrefix = "jupyter"
+
+ def generatePodName(uid: Int): String = s"$podNamePrefix-$uid"
+
+ /** The in-cluster address of a user's pod, resolvable via the headless service. */
+ def generatePodURI(uid: Int): String =
+ s"${generatePodName(uid)}.${KubernetesConfig.jupyterServiceName}.$namespace.svc.cluster.local:${KubernetesConfig.jupyterPortNumber}"
+
+ def podExists(uid: Int): Boolean = getPodByName(generatePodName(uid)).isDefined
+
+ def getPodByName(podName: String): Option[Pod] =
+ Option(client.pods().inNamespace(namespace).withName(podName).get())
+
+ /**
+ * Starts a user's JupyterLab. The token is passed as JUPYTER_TOKEN, which is what the image's
+ * start-texera-jupyter.sh reads, so each pod ends up with its owner's token and no other.
+ * Hostname and subdomain are what make generatePodURI resolve.
+ */
+ def createPod(uid: Int, token: String): Pod = {
+ val podName = generatePodName(uid)
+
+ val resources = new ResourceRequirementsBuilder()
+ .addToLimits("cpu", new Quantity(KubernetesConfig.jupyterCpuLimit))
+ .addToLimits("memory", new Quantity(KubernetesConfig.jupyterMemoryLimit))
+ .build()
+
+ val pod = new PodBuilder()
+ .withNewMetadata()
+ .withName(podName)
+ .withNamespace(namespace)
+ .addToLabels("type", "jupyter")
+ .addToLabels("uid", uid.toString)
+ .addToLabels("name", podName)
+ .endMetadata()
+ .withNewSpec()
+ .addNewContainer()
+ .withName("jupyter")
+ .withImage(KubernetesConfig.jupyterImageName)
+ .withImagePullPolicy(KubernetesConfig.computingUnitImagePullPolicy)
+ .addNewPort()
+ .withContainerPort(KubernetesConfig.jupyterPortNumber)
+ .endPort()
+ .withEnv(
+ new EnvVarBuilder().withName("JUPYTER_TOKEN").withValue(token).build()
Review Comment:
The pod's env carries only `JUPYTER_TOKEN`, but `start-texera-jupyter.sh` also reads `TEXERA_ORIGIN` — it feeds both the CSP `frame-ancestors` header and the postMessage origin in `custom.js`, and defaults to `http://localhost:4200` when unset. If that default applies inside a pod, any real deployment would refuse the iframe and cell-click sync would silently break — the same pair #7934 just fixed for local-dev.
Since #8006 runs on this branch against a real cluster, could you check whether the pods there actually get an origin, and if so where it comes from? If it's handled in #8006's templates that's fine — just seems worth settling which layer owns it while the pod spec is being defined here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org