You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "mengw15 (via GitHub)" <gi...@apache.org> on 2026/04/23 01:38:40 UTC
[PR] fix: gate iceberg catalog credentials by catalog type in CU and Python worker [texera]
mengw15 opened a new pull request, #4478:
URL: https://github.com/apache/texera/pull/4478
What changes were proposed in this PR?
When the active Iceberg catalog is rest, only REST credentials are forwarded across process
boundaries; when it is postgres, only Postgres credentials are forwarded. Previously both backends'
credentials were forwarded unconditionally, exposing dead-weight credentials to consumers that never
use them.
Two transmission paths are gated:
1. K8s CU pod env vars — ComputingUnitManagingResource.scala: extract icebergEnvironmentVariables def
that branches on StorageConfig.icebergCatalogType and only emits the active backend's env keys
(STORAGE_ICEBERG_CATALOG_REST_* for rest; STORAGE_ICEBERG_CATALOG_POSTGRES_* for postgres).
2. Python UDF subprocess argv — PythonWorkflowWorker.scala: the 6 iceberg credential slots passed to
texera_run_python_worker.py are gated by catalog type. The inactive backend's slots are passed as
empty strings to keep the positional-arg layout stable, so the Python entrypoint needs no change (it
already only consumes the active backend's fields via iceberg_catalog_instance.py).
Any related issues, documentation, discussions?
Closes #4477
How was this PR tested?
- sbt ComputingUnitManagingService/compile WorkflowExecutionService/compile passes.
- Manual verification
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.7)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: pass different environment variables to computing unit and Python worker based on the catalog type [texera]
Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
mengw15 merged PR #4478:
URL: https://github.com/apache/texera/pull/4478
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: gate iceberg catalog credentials by catalog type in CU and Python worker [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
bobbai00 commented on code in PR #4478:
URL: https://github.com/apache/texera/pull/4478#discussion_r3134224954
##########
amber/src/main/scala/org/apache/texera/amber/engine/architecture/pythonworker/PythonWorkflowWorker.scala:
##########
@@ -168,6 +168,11 @@ class PythonWorkflowWorker(
private def startPythonProcess(): Unit = {
val udfEntryScriptPath: String =
pythonSrcDirectory.resolve("texera_run_python_worker.py").toString
+ // Only forward credentials for the active iceberg catalog backend; the
+ // inactive backend's slots are passed as empty strings to keep the
+ // positional arg layout stable for texera_run_python_worker.py.
Review Comment:
remove the sentences like "to keep the positional arg layout...", The comment should be something like "Set the Iceberg related env vars based on the catalog type".
##########
computing-unit-managing-service/src/main/scala/org/apache/texera/service/resource/ComputingUnitManagingResource.scala:
##########
@@ -65,49 +65,65 @@ object ComputingUnitManagingResource {
.getInstance()
.createDSLContext()
+ // Iceberg env vars are gated by catalog type so each CU only receives the
Review Comment:
Such comment is unnecessary. Remove them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: gate iceberg catalog credentials by catalog type in CU and Python worker [texera]
Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
mengw15 commented on code in PR #4478:
URL: https://github.com/apache/texera/pull/4478#discussion_r3134394662
##########
amber/src/main/scala/org/apache/texera/amber/engine/architecture/pythonworker/PythonWorkflowWorker.scala:
##########
@@ -168,6 +168,11 @@ class PythonWorkflowWorker(
private def startPythonProcess(): Unit = {
val udfEntryScriptPath: String =
pythonSrcDirectory.resolve("texera_run_python_worker.py").toString
+ // Only forward credentials for the active iceberg catalog backend; the
+ // inactive backend's slots are passed as empty strings to keep the
+ // positional arg layout stable for texera_run_python_worker.py.
Review Comment:
Adopted, slight tweak — used "arguments" instead of "env vars" since these are actually positional CLI args passed to the Python subprocess
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org