You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "shubhluck (via GitHub)" <gi...@apache.org> on 2026/08/18 11:45:19 UTC
[PR] SPARK-25078 Standalone does not work with spark.authenticate.secret a… [spark]
shubhluck opened a new pull request, #58078:
URL: https://github.com/apache/spark/pull/58078
…nd deploy-mode=cluster
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'common/utils/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
Fix `SecurityManager.initializeAuth()` so that in standalone cluster mode it accepts the auth secret from the `_SPARK_AUTH_SECRET` environment variable in addition to the `spark.authenticate.secret `config key.
Today, the non-YARN / non-local / non-K8s branch of `initializeAuth()` only checks `sparkConf.contains(SPARK_AUTH_SECRET_CONF)`:
```
case _ =>
require(sparkConf.contains(SPARK_AUTH_SECRET_CONF),
s"A secret key must be specified via the $SPARK_AUTH_SECRET_CONF config.")
return
```
This is inconsistent with `getSecretKey()` a few lines above, which already reads from both sources — conf and `_SPARK_AUTH_SECRET` env var — before failing. The check is broadened to match:
```
require(sparkConf.contains(SPARK_AUTH_SECRET_CONF) ||
sparkConf.getenv(ENV_AUTH_SECRET) != null,
s"A secret key must be specified via the $SPARK_AUTH_SECRET_CONF config.")
```
No other code path, master type, wire format, or secret-propagation mechanism is touched. The actual secret used for the SASL handshake still comes from the same source it always has (`getSecretKey()` — unchanged).
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(31, 31, 31); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">This fixes SPARK-25078 (<a href="https://issues.apache.org/jira/browse/SPARK-25078" target="_blank" rel="noopener noreferrer" style="color: rgb(77, 170, 252); text-decoration: rgb(77, 170, 252);">https://issues.apache.org/jira/browse/SPARK-25078</a>). Submitting a driver in standalone cluster mode with authentication enabled fails immediately even when everything is
configured correctly:</p><div class="codeBlockWrapper_-a7MRw" style="position: relative; margin: 8px 0px; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(31, 31, 31); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><button class="copyButton_CEmTFw copyButton_-a7MRw" title="Copy code" aria-label="Copy code to clipboard" style="color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; background: none 0% 0% / auto repeat scroll padding-box border-box rgb(31, 31, 31); border-co
lor: rgb(69, 69, 69); border-style: solid; border-width: 1px; border-image: none 100% / 1 / 0 stretch; cursor: pointer; opacity: 1; display: flex; border-radius: 4px; justify-content: center; align-items: center; padding: 4px; transition: opacity 0.15s, background 0.15s; position: absolute; top: 4px; right: 4px;"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 20 20" fill="currentColor" aria-hidden="true" data-slot="icon" class="copyIcon_CEmTFw"><path fill-rule="evenodd" d="M15.988 3.012A2.25 2.25 0 0 1 18 5.25v6.5A2.25 2.25 0 0 1 15.75 14H13.5v-3.379a3 3 0 0 0-.879-2.121l-3.12-3.121a3 3 0 0 0-1.402-.791 2.252 2.252 0 0 1 1.913-1.576A2.25 2.25 0 0 1 12.25 1h1.5a2.25 2.25 0 0 1 2.238 2.012ZM11.5 3.25a.75.75 0 0 1 .75-.75h1.5a.75.75 0 0 1 .75.75v.25h-3v-.25Z" clip-rule="evenodd"></path><path d="M3.5 6A1.5 1.5 0 0 0 2 7.5v9A1.5 1.5 0 0 0 3.5 18h7a1.5 1.5 0 0 0 1.5-1.5v-5.879a1.5 1.5 0 0 0-.44-1.06L8.44 6.439A1.5 1.5 0 0 0 7.378 6H3.5Z"></path></svg></button><pre style="overflow-x:
auto; white-space: pre; box-sizing: border-box; border-radius: 4px; max-width: 100%; margin: 0px; padding: 8px;"><code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 0px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">$ spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://host:7077 \
--deploy-mode cluster \
--conf spark.authenticate=true \
--conf spark.authenticate.secret=secret \
/path/to/spark-examples.jar 100
Driver stderr:
java.lang.IllegalArgumentException:
A secret key must be specified via the spark.authenticate.secret config.
</code></pre></div><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(31, 31, 31); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">The reason:</p><ol style="padding-inline-start: 2em; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-ind
ent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(31, 31, 31); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li style="unicode-bidi: plaintext;"><code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">spark-submit</code><span> </span>deliberately strips<span> </span><code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">spark.authenticate.secret</code><span> </span>from the SparkConf/javaOpts before the driver process is launched — see<span> </span><code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-brea
k: break-word; font-size: 0.9em;">CommandUtils.buildCommandSeq</code><span> </span>in<span> </span><code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala</code>, which both:<ul style="padding-inline-start: 2em;"><li style="unicode-bidi: plaintext;">filters<span> </span><code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">-Dspark.authenticate.secret=…</code><span> </span>out of<span> </span><code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">javaOpts</code>, and</li><li style="unicode-bidi: plaintext;">injects the master/worker-side se
cret into the driver's environment as<span> </span><code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">_SPARK_AUTH_SECRET</code>.</li></ul></li><li style="unicode-bidi: plaintext;">So by design, in standalone cluster mode the driver receives the secret via the env var, not via SparkConf.</li><li style="unicode-bidi: plaintext;"><code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">initializeAuth()</code><span> </span>only inspects the conf and throws before ever consulting the env var — even though<span> </span><code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">getSecretKey()</code><span> </span>later
would have found it there.</li></ol><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(31, 31, 31); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Standalone cluster mode + authentication has been unusable out-of-the-box since the introduction of this validation. The only known workaround is programmatically injecting a dummy config value inside the driver code (<code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px;
border-radius: 3px; word-break: break-word; font-size: 0.9em;">conf.setIfMissing("spark.authenticate.secret", "placeholder")</code>) purely to satisfy the misplaced check; the real secret from the env var is used for the actual handshake anyway. That workaround requires modifying every application, which is neither discoverable nor viable for prebuilt jars.</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(31, 31, 31); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">There i
s no security implication: the fix does not introduce a new secret source, does not change what travels over the network, and does not weaken the require — it only aligns the pre-check with the source <code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">getSecretKey()</code> was already trusting.</p><h3 style="unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(31, 31, 31); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Does this PR intro
duce<span> </span><em>any</em><span> </span>user-facing change?</h3><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(31, 31, 31); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Yes — but only in the fix direction. Behavior matrix for the standalone (<code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">spark://…</code>) branch of <code style="fon
t-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">initializeAuth()</code> with <code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">spark.authenticate=true</code>:</p>
Secret source at driver startup | Before | After
-- | -- | --
spark.authenticate.secret in SparkConf | pass | pass
_SPARK_AUTH_SECRET env var only (standalone cluster mode) | throws IllegalArgumentException | passes
Both present | pass | pass
Neither present | throws | throws
<p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(31, 31, 31); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Net effect: <code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">spark-submit --master spark://… --deploy-mode cluster</code> with <code style="font-family: monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2
px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">spark.authenticate=true</code> now works without the in-code dummy-secret workaround. No configuration change is required from users.</p>
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### How was this patch tested?
Two new unit tests in core/src/test/scala/org/apache/spark/SecurityManagerSuite.scala:
`SPARK-25078: standalone cluster mode accepts secret from env var `— sets master to spark://…, provides the secret only via _SPARK_AUTH_SECRET (using the existing SparkConfWithEnv helper), and asserts that initializeAuth() succeeds and getSecretKey() returns the env-var value.
`SPARK-25078: standalone cluster mode still fails when no secret is provided `— regression guard confirming that with neither the conf key nor the env var set, initializeAuth() still throws IllegalArgumentException.
Run locally:
`./build/sbt "core/testOnly org.apache.spark.SecurityManagerSuite"`
Also manually validated on a standalone cluster: `spark-submit --deploy-mode cluster --conf spark.authenticate=true --conf spark.authenticate.secret=`… succeeds after the patch and fails before it, matching the reproducer above.
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
No
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] SPARK-25078 Standalone does not work with spark.authenticate.secret a… [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58078:
URL: https://github.com/apache/spark/pull/58078#issuecomment-5336282704
e.g.
> [SPARK-25078][<COMPONENT>] Standalone does not work with spark.authenticate.secret and deploy-mode=cluster
instead of
> SPARK-25078 Standalone does not work with spark.authenticate.secret a…
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-25078][DEPLOY] Standalone does not work with spark.authenticate.secret and deploy-mode=cluster [spark]
Posted by "shubhluck (via GitHub)" <gi...@apache.org>.
shubhluck commented on PR #58078:
URL: https://github.com/apache/spark/pull/58078#issuecomment-5336345174
Thanks @uros-b I have updated the PR title
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org