You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/09/30 15:35:07 UTC
[PR] [SPARK-59895] Add `migration_guide.md` [spark-kubernetes-operator]
dongjoon-hyun opened a new pull request, #914:
URL: https://github.com/apache/spark-kubernetes-operator/pull/914
### What changes were proposed in this pull request?
This PR aims to add `docs/migration_guide.md` to track the behavior changes since [1.0.0](https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0) (2026-07-23) and how to restore the previous behavior where possible.
- Like the [Apache Spark migration guides](https://spark.apache.org/docs/latest/core-migration-guide.html), it has one `Upgrading from X.Y to X.(Y+1)` section per minor release, with `Since 1.1.0, ... To restore the behavior before 1.1.0, ...` items and their JIRA links.
- New features which are disabled by default, e.g. K8s events, the Kueue integration and `spec.suspend`, are not listed. Neither are changes which only affect building the project, CI, tests or examples.
- Add a guideline to `AGENTS.md` that a PR which changes a user-visible behavior of the latest release updates the guide in the same PR.
The `Upgrading from 1.0 to 1.1` section covers the following.
| Area | Items |
| --- | --- |
| Kubernetes and CRDs | K8s 1.35+ recommendation, replacing the CRDs for `spec.suspend` and the `Suspended` state |
| Helm Chart | Deprecated `enable` keys, `operatorRbac.configManagement.create`, `events` RBAC verbs, `jvmArgs` |
| SparkApplication | Retries on `Retry-After` and unanswered requests, `restartCounterResetMillis`, `ResourceReleased` after the last attempt, early driver pod failures |
| SparkCluster | External shuffle service, worker NetworkPolicy, `terminationGracePeriodSeconds`, `dnsConfig`, retryable failures, `Failed` message |
| Docker Image | `alpine:3.24` without `libstdc++` |
| Java API | `Constants`, `ApplicationStatus.terminateOrRestart`, `ClusterStateSummary.Suspended` |
| Downgrading to 1.0 | `Suspended` clusters, `spec.suspend`, Kueue `Workload`s |
### Why are the changes needed?
Since 1.0.0, several changes in `main` alter the behavior of an existing deployment. For example, SPARK-58835 enables the external shuffle service of `SparkCluster` workers by default. SPARK-58426 makes the default `jvmArgs` take effect, so the maximum heap of the operator grows from 25% to 80% of its memory limit. These changes are described only in their own PRs, so users upgrading from 1.0 have no single place to find them or to learn how to restore the previous behavior. In addition, `helm upgrade` does not update CRDs, so the new CRD fields of 1.1 need a manual step.
### Does this PR introduce _any_ user-facing change?
No behavior change. This is a documentation-only change.
### How was this patch tested?
Manual review against the code of 1.0.0 and `main`. Also passed `markdownlint`.
In addition, I verified the following manually.
- `helm template` with the dynamic config enabled and `operatorRbac.configManagement.create=false` renders the config monitor `Role` and `RoleBinding` with the chart of 1.0.0, but not with the chart of `main`.
- `-XX:+PrintFlagsFinal` on JDK 26 with the default `jvmArgs`:
| Flag | 1.0.0 (one argument) | `main` (split arguments) |
| --- | --- | --- |
| `MaxRAMPercentage` | 25 (default) | 80 |
| `UseParallelGC` | false (default) | true |
| `AlwaysPreTouch` | false (default) | true |
| `UseCompactObjectHeaders` | false (default) | true |
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59895] Add `migration_guide.md` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #914:
URL: https://github.com/apache/spark-kubernetes-operator/pull/914#issuecomment-5914751140
This looks like this.
- https://github.com/dongjoon-hyun/spark-kubernetes-operator/blob/SPARK-59895/docs/migration_guide.md
<img width="740" height="526" alt="Screenshot 2026-09-30 at 08 48 48" src="https://github.com/user-attachments/assets/6a2135b7-58f4-444c-8ef6-96cb713efba9" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59895] Add `migration_guide.md` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #914:
URL: https://github.com/apache/spark-kubernetes-operator/pull/914#issuecomment-5917700083
Thank you, @peter-toth !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59895] Add `migration_guide.md` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #914:
URL: https://github.com/apache/spark-kubernetes-operator/pull/914#discussion_r4148077231
##########
docs/migration_guide.md:
##########
@@ -0,0 +1,152 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Migration Guide
+
+This document lists the behavior changes of Apache Spark K8s Operator releases which may affect an
+existing deployment, and how to restore the previous behavior where possible. When upgrading across
+several releases, read every section in between.
+
+New features which are disabled by default are not listed, and neither are changes which only
+affect building the project, CI, tests or examples.
+
+## Upgrading from 1.0 to 1.1
+
+### Kubernetes and CRDs
+
+- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34 ([SPARK-59692](https://issues.apache.org/jira/browse/SPARK-59692)).
+- Since 1.1.0, the CRDs have `spec.suspend` for `SparkApplication` and `SparkCluster`, and the
+ `Suspended` state for `SparkCluster`. Since `helm upgrade` does not update CRDs, replace them
+ before upgrading the operator, e.g. with the CRDs of the Helm chart `1.9.0`, which deploys
+ 1.1.0. Otherwise, the API server rejects or drops `spec.suspend`, and rejects the `Suspended`
+ state ([SPARK-59475](https://issues.apache.org/jira/browse/SPARK-59475), [SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+
+ ```bash
+ helm repo update
+ helm pull spark/spark-kubernetes-operator --version 1.9.0 --untar
+ kubectl replace -f spark-kubernetes-operator/crds/
+ ```
+
+### Helm Chart
+
+- Since 1.1.0, `operatorDeployment.networkPolicy.enable` and
+ `operatorConfiguration.dynamicConfig.enable` are deprecated in favor of `enabled`, and
+ `helm install` and `helm upgrade` print a warning for them. They are still honored until chart
+ `2.0.0`: a feature is enabled when either key is `true`, so a legacy `enable: true` must be
+ removed to disable it ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59540](https://issues.apache.org/jira/browse/SPARK-59540)).
+- Since 1.1.0, `operatorRbac.configManagement.create: false` takes effect. The chart then no longer
+ creates the Role and RoleBinding for the `configMap` source of the dynamic config, which 1.0
+ created regardless. To restore the behavior before 1.1.0, set it to `true`, the default
+ ([SPARK-59537](https://issues.apache.org/jira/browse/SPARK-59537)).
+- Since 1.1.0, the operator ClusterRole and Roles grant only `get`, `create` and `patch` on
+ `events`, which are the verbs the operator uses, instead of all verbs. If other workloads use
+ the operator ServiceAccount and need `list`, `watch`, `update` or `delete` on `events`, grant
+ them with another Role or ClusterRole ([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)).
+- Since 1.1.0, each option of `operatorDeployment.operatorPod.operatorContainer.jvmArgs` is passed
+ to the operator JVM as its own argument. 1.0 passed the whole value as one argument, which the
+ JVM took as a single `-Dfile.encoding` system property, so the other options had no effect. The
+ default options now take effect, e.g. the Parallel GC and a heap of 80% of the container memory
+ limit (`2Gi` by default), fully touched at startup, instead of at most 25%. Keep a memory limit
+ on the operator container, since the percentages apply to the node memory otherwise. The options
+ also take precedence over `JAVA_TOOL_OPTIONS` and `JDK_JAVA_OPTIONS` in `operatorContainer.env`,
+ and cannot contain a space, since the value is split on spaces. To restore the behavior before
+ 1.1.0, set `jvmArgs` to `-Dfile.encoding=UTF8` ([SPARK-58426](https://issues.apache.org/jira/browse/SPARK-58426)).
+
+### SparkApplication
+
+- Since 1.1.0, when the operator retries creating the driver resources of an application, it waits
+ as long as the `Retry-After` hint of the API server asks, capped at
+ `spark.kubernetes.operator.api.secondaryResourceCreateMaxBackoffMillis`. 1.0 ignored the hint,
+ and retried errors other than `409` and `429`, e.g. `503`, right away. A request which the API
+ server never answered, e.g. on a connection reset or a timeout, is retried with exponential
+ backoff too, while 1.0 failed the attempt with `SchedulingFailure` at once. The number of
+ attempts is still bounded by `spark.kubernetes.operator.api.secondaryResourceCreateMaxAttempts`
+ ([SPARK-59092](https://issues.apache.org/jira/browse/SPARK-59092), [SPARK-59694](https://issues.apache.org/jira/browse/SPARK-59694)).
+- Since 1.1.0, the duration of an attempt which is compared with `restartCounterResetMillis` is
+ measured from the first state after `Submitted` or `ScheduledToRestart`, normally
+ `DriverRequested`, instead of from `Submitted` or `ScheduledToRestart`. The restart backoff no
+ longer counts, so an attempt which reset the restart counters in 1.0 may not reset them, and the
+ application may reach `maxRestartAttempts` earlier. To keep a similar threshold, decrease
+ `restartCounterResetMillis` by the restart backoff, e.g. `restartBackoffMillis`
+ ([SPARK-59542](https://issues.apache.org/jira/browse/SPARK-59542)).
+- Since 1.1.0, an application which is configured to restart ends in `ResourceReleased` instead of
+ `TerminatedWithoutReleaseResources` after its last attempt, even if its `resourceRetainPolicy`
+ retains the resources, since they are released at the end of every attempt. So
+ `ttlAfterStopMillis` takes effect after the last attempt, while 1.0 applied it only once
+ `resourceRetainDurationMillis` elapsed, or never if the retain duration was disabled. To keep
+ such an application longer, increase `ttlAfterStopMillis`, or set it to a negative value
+ ([SPARK-59732](https://issues.apache.org/jira/browse/SPARK-59732)).
Review Comment:
Thank you, you are right. I missed that `isRetainDurationEnabled()` is true when only `ttlAfterStopMillis` is set, so the TTL deletes such an app at the same time in both versions. I applied your suggestion in [`3fb90cf`](https://github.com/apache/spark-kubernetes-operator/commit/3fb90cfdbf5340e423ee4e504db1d38b3319fdc5).
##########
docs/migration_guide.md:
##########
@@ -0,0 +1,152 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Migration Guide
+
+This document lists the behavior changes of Apache Spark K8s Operator releases which may affect an
+existing deployment, and how to restore the previous behavior where possible. When upgrading across
+several releases, read every section in between.
+
+New features which are disabled by default are not listed, and neither are changes which only
+affect building the project, CI, tests or examples.
+
+## Upgrading from 1.0 to 1.1
+
+### Kubernetes and CRDs
+
+- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34 ([SPARK-59692](https://issues.apache.org/jira/browse/SPARK-59692)).
+- Since 1.1.0, the CRDs have `spec.suspend` for `SparkApplication` and `SparkCluster`, and the
+ `Suspended` state for `SparkCluster`. Since `helm upgrade` does not update CRDs, replace them
+ before upgrading the operator, e.g. with the CRDs of the Helm chart `1.9.0`, which deploys
+ 1.1.0. Otherwise, the API server rejects or drops `spec.suspend`, and rejects the `Suspended`
+ state ([SPARK-59475](https://issues.apache.org/jira/browse/SPARK-59475), [SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+
+ ```bash
+ helm repo update
+ helm pull spark/spark-kubernetes-operator --version 1.9.0 --untar
+ kubectl replace -f spark-kubernetes-operator/crds/
+ ```
+
+### Helm Chart
+
+- Since 1.1.0, `operatorDeployment.networkPolicy.enable` and
+ `operatorConfiguration.dynamicConfig.enable` are deprecated in favor of `enabled`, and
+ `helm install` and `helm upgrade` print a warning for them. They are still honored until chart
+ `2.0.0`: a feature is enabled when either key is `true`, so a legacy `enable: true` must be
+ removed to disable it ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59540](https://issues.apache.org/jira/browse/SPARK-59540)).
+- Since 1.1.0, `operatorRbac.configManagement.create: false` takes effect. The chart then no longer
+ creates the Role and RoleBinding for the `configMap` source of the dynamic config, which 1.0
+ created regardless. To restore the behavior before 1.1.0, set it to `true`, the default
+ ([SPARK-59537](https://issues.apache.org/jira/browse/SPARK-59537)).
+- Since 1.1.0, the operator ClusterRole and Roles grant only `get`, `create` and `patch` on
+ `events`, which are the verbs the operator uses, instead of all verbs. If other workloads use
+ the operator ServiceAccount and need `list`, `watch`, `update` or `delete` on `events`, grant
+ them with another Role or ClusterRole ([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)).
+- Since 1.1.0, each option of `operatorDeployment.operatorPod.operatorContainer.jvmArgs` is passed
+ to the operator JVM as its own argument. 1.0 passed the whole value as one argument, which the
+ JVM took as a single `-Dfile.encoding` system property, so the other options had no effect. The
+ default options now take effect, e.g. the Parallel GC and a heap of 80% of the container memory
+ limit (`2Gi` by default), fully touched at startup, instead of at most 25%. Keep a memory limit
+ on the operator container, since the percentages apply to the node memory otherwise. The options
+ also take precedence over `JAVA_TOOL_OPTIONS` and `JDK_JAVA_OPTIONS` in `operatorContainer.env`,
+ and cannot contain a space, since the value is split on spaces. To restore the behavior before
+ 1.1.0, set `jvmArgs` to `-Dfile.encoding=UTF8` ([SPARK-58426](https://issues.apache.org/jira/browse/SPARK-58426)).
+
+### SparkApplication
Review Comment:
Thank you for confirming.
##########
docs/migration_guide.md:
##########
@@ -0,0 +1,152 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Migration Guide
+
+This document lists the behavior changes of Apache Spark K8s Operator releases which may affect an
+existing deployment, and how to restore the previous behavior where possible. When upgrading across
+several releases, read every section in between.
+
+New features which are disabled by default are not listed, and neither are changes which only
+affect building the project, CI, tests or examples.
+
+## Upgrading from 1.0 to 1.1
+
+### Kubernetes and CRDs
+
+- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34 ([SPARK-59692](https://issues.apache.org/jira/browse/SPARK-59692)).
+- Since 1.1.0, the CRDs have `spec.suspend` for `SparkApplication` and `SparkCluster`, and the
+ `Suspended` state for `SparkCluster`. Since `helm upgrade` does not update CRDs, replace them
+ before upgrading the operator, e.g. with the CRDs of the Helm chart `1.9.0`, which deploys
+ 1.1.0. Otherwise, the API server rejects or drops `spec.suspend`, and rejects the `Suspended`
+ state ([SPARK-59475](https://issues.apache.org/jira/browse/SPARK-59475), [SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+
+ ```bash
+ helm repo update
+ helm pull spark/spark-kubernetes-operator --version 1.9.0 --untar
+ kubectl replace -f spark-kubernetes-operator/crds/
+ ```
+
+### Helm Chart
+
+- Since 1.1.0, `operatorDeployment.networkPolicy.enable` and
+ `operatorConfiguration.dynamicConfig.enable` are deprecated in favor of `enabled`, and
+ `helm install` and `helm upgrade` print a warning for them. They are still honored until chart
+ `2.0.0`: a feature is enabled when either key is `true`, so a legacy `enable: true` must be
+ removed to disable it ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59540](https://issues.apache.org/jira/browse/SPARK-59540)).
+- Since 1.1.0, `operatorRbac.configManagement.create: false` takes effect. The chart then no longer
+ creates the Role and RoleBinding for the `configMap` source of the dynamic config, which 1.0
+ created regardless. To restore the behavior before 1.1.0, set it to `true`, the default
+ ([SPARK-59537](https://issues.apache.org/jira/browse/SPARK-59537)).
+- Since 1.1.0, the operator ClusterRole and Roles grant only `get`, `create` and `patch` on
+ `events`, which are the verbs the operator uses, instead of all verbs. If other workloads use
+ the operator ServiceAccount and need `list`, `watch`, `update` or `delete` on `events`, grant
+ them with another Role or ClusterRole ([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)).
+- Since 1.1.0, each option of `operatorDeployment.operatorPod.operatorContainer.jvmArgs` is passed
+ to the operator JVM as its own argument. 1.0 passed the whole value as one argument, which the
+ JVM took as a single `-Dfile.encoding` system property, so the other options had no effect. The
+ default options now take effect, e.g. the Parallel GC and a heap of 80% of the container memory
+ limit (`2Gi` by default), fully touched at startup, instead of at most 25%. Keep a memory limit
+ on the operator container, since the percentages apply to the node memory otherwise. The options
+ also take precedence over `JAVA_TOOL_OPTIONS` and `JDK_JAVA_OPTIONS` in `operatorContainer.env`,
+ and cannot contain a space, since the value is split on spaces. To restore the behavior before
+ 1.1.0, set `jvmArgs` to `-Dfile.encoding=UTF8` ([SPARK-58426](https://issues.apache.org/jira/browse/SPARK-58426)).
Review Comment:
Thank you for the catch. I reproduced it with `helm template` of this chart with the default values of chart `1.8.0`, and added your item as the first item of the `Helm Chart` section in [`3fb90cf`](https://github.com/apache/spark-kubernetes-operator/commit/3fb90cfdbf5340e423ee4e504db1d38b3319fdc5), since it blocks the upgrade itself.
##########
docs/migration_guide.md:
##########
@@ -0,0 +1,152 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Migration Guide
+
+This document lists the behavior changes of Apache Spark K8s Operator releases which may affect an
+existing deployment, and how to restore the previous behavior where possible. When upgrading across
+several releases, read every section in between.
+
+New features which are disabled by default are not listed, and neither are changes which only
+affect building the project, CI, tests or examples.
+
+## Upgrading from 1.0 to 1.1
+
+### Kubernetes and CRDs
+
+- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34 ([SPARK-59692](https://issues.apache.org/jira/browse/SPARK-59692)).
+- Since 1.1.0, the CRDs have `spec.suspend` for `SparkApplication` and `SparkCluster`, and the
+ `Suspended` state for `SparkCluster`. Since `helm upgrade` does not update CRDs, replace them
+ before upgrading the operator, e.g. with the CRDs of the Helm chart `1.9.0`, which deploys
+ 1.1.0. Otherwise, the API server rejects or drops `spec.suspend`, and rejects the `Suspended`
+ state ([SPARK-59475](https://issues.apache.org/jira/browse/SPARK-59475), [SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+
+ ```bash
+ helm repo update
+ helm pull spark/spark-kubernetes-operator --version 1.9.0 --untar
+ kubectl replace -f spark-kubernetes-operator/crds/
+ ```
+
+### Helm Chart
+
+- Since 1.1.0, `operatorDeployment.networkPolicy.enable` and
+ `operatorConfiguration.dynamicConfig.enable` are deprecated in favor of `enabled`, and
+ `helm install` and `helm upgrade` print a warning for them. They are still honored until chart
+ `2.0.0`: a feature is enabled when either key is `true`, so a legacy `enable: true` must be
+ removed to disable it ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59540](https://issues.apache.org/jira/browse/SPARK-59540)).
+- Since 1.1.0, `operatorRbac.configManagement.create: false` takes effect. The chart then no longer
+ creates the Role and RoleBinding for the `configMap` source of the dynamic config, which 1.0
+ created regardless. To restore the behavior before 1.1.0, set it to `true`, the default
+ ([SPARK-59537](https://issues.apache.org/jira/browse/SPARK-59537)).
+- Since 1.1.0, the operator ClusterRole and Roles grant only `get`, `create` and `patch` on
+ `events`, which are the verbs the operator uses, instead of all verbs. If other workloads use
+ the operator ServiceAccount and need `list`, `watch`, `update` or `delete` on `events`, grant
+ them with another Role or ClusterRole ([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)).
Review Comment:
Right, it was not `*`. Instead of referring to the verbs on pods, I listed the verbs which only 1.0 granted, `list`, `watch`, `update` and `delete`, in [`3fb90cf`](https://github.com/apache/spark-kubernetes-operator/commit/3fb90cfdbf5340e423ee4e504db1d38b3319fdc5).
##########
docs/migration_guide.md:
##########
@@ -0,0 +1,152 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Migration Guide
+
+This document lists the behavior changes of Apache Spark K8s Operator releases which may affect an
+existing deployment, and how to restore the previous behavior where possible. When upgrading across
+several releases, read every section in between.
+
+New features which are disabled by default are not listed, and neither are changes which only
+affect building the project, CI, tests or examples.
+
+## Upgrading from 1.0 to 1.1
+
+### Kubernetes and CRDs
+
+- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34 ([SPARK-59692](https://issues.apache.org/jira/browse/SPARK-59692)).
+- Since 1.1.0, the CRDs have `spec.suspend` for `SparkApplication` and `SparkCluster`, and the
+ `Suspended` state for `SparkCluster`. Since `helm upgrade` does not update CRDs, replace them
+ before upgrading the operator, e.g. with the CRDs of the Helm chart `1.9.0`, which deploys
+ 1.1.0. Otherwise, the API server rejects or drops `spec.suspend`, and rejects the `Suspended`
+ state ([SPARK-59475](https://issues.apache.org/jira/browse/SPARK-59475), [SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+
+ ```bash
+ helm repo update
+ helm pull spark/spark-kubernetes-operator --version 1.9.0 --untar
+ kubectl replace -f spark-kubernetes-operator/crds/
+ ```
+
+### Helm Chart
+
+- Since 1.1.0, `operatorDeployment.networkPolicy.enable` and
+ `operatorConfiguration.dynamicConfig.enable` are deprecated in favor of `enabled`, and
+ `helm install` and `helm upgrade` print a warning for them. They are still honored until chart
+ `2.0.0`: a feature is enabled when either key is `true`, so a legacy `enable: true` must be
+ removed to disable it ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59540](https://issues.apache.org/jira/browse/SPARK-59540)).
+- Since 1.1.0, `operatorRbac.configManagement.create: false` takes effect. The chart then no longer
+ creates the Role and RoleBinding for the `configMap` source of the dynamic config, which 1.0
+ created regardless. To restore the behavior before 1.1.0, set it to `true`, the default
+ ([SPARK-59537](https://issues.apache.org/jira/browse/SPARK-59537)).
+- Since 1.1.0, the operator ClusterRole and Roles grant only `get`, `create` and `patch` on
+ `events`, which are the verbs the operator uses, instead of all verbs. If other workloads use
+ the operator ServiceAccount and need `list`, `watch`, `update` or `delete` on `events`, grant
+ them with another Role or ClusterRole ([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)).
+- Since 1.1.0, each option of `operatorDeployment.operatorPod.operatorContainer.jvmArgs` is passed
+ to the operator JVM as its own argument. 1.0 passed the whole value as one argument, which the
+ JVM took as a single `-Dfile.encoding` system property, so the other options had no effect. The
+ default options now take effect, e.g. the Parallel GC and a heap of 80% of the container memory
+ limit (`2Gi` by default), fully touched at startup, instead of at most 25%. Keep a memory limit
Review Comment:
Thanks. I added your sentence in [`3fb90cf`](https://github.com/apache/spark-kubernetes-operator/commit/3fb90cfdbf5340e423ee4e504db1d38b3319fdc5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59895] Add `migration_guide.md` [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #914:
URL: https://github.com/apache/spark-kubernetes-operator/pull/914#discussion_r4147478499
##########
docs/migration_guide.md:
##########
@@ -0,0 +1,152 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Migration Guide
+
+This document lists the behavior changes of Apache Spark K8s Operator releases which may affect an
+existing deployment, and how to restore the previous behavior where possible. When upgrading across
+several releases, read every section in between.
+
+New features which are disabled by default are not listed, and neither are changes which only
+affect building the project, CI, tests or examples.
+
+## Upgrading from 1.0 to 1.1
+
+### Kubernetes and CRDs
+
+- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34 ([SPARK-59692](https://issues.apache.org/jira/browse/SPARK-59692)).
+- Since 1.1.0, the CRDs have `spec.suspend` for `SparkApplication` and `SparkCluster`, and the
+ `Suspended` state for `SparkCluster`. Since `helm upgrade` does not update CRDs, replace them
+ before upgrading the operator, e.g. with the CRDs of the Helm chart `1.9.0`, which deploys
+ 1.1.0. Otherwise, the API server rejects or drops `spec.suspend`, and rejects the `Suspended`
+ state ([SPARK-59475](https://issues.apache.org/jira/browse/SPARK-59475), [SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+
+ ```bash
+ helm repo update
+ helm pull spark/spark-kubernetes-operator --version 1.9.0 --untar
+ kubectl replace -f spark-kubernetes-operator/crds/
+ ```
+
+### Helm Chart
+
+- Since 1.1.0, `operatorDeployment.networkPolicy.enable` and
+ `operatorConfiguration.dynamicConfig.enable` are deprecated in favor of `enabled`, and
+ `helm install` and `helm upgrade` print a warning for them. They are still honored until chart
+ `2.0.0`: a feature is enabled when either key is `true`, so a legacy `enable: true` must be
+ removed to disable it ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59540](https://issues.apache.org/jira/browse/SPARK-59540)).
+- Since 1.1.0, `operatorRbac.configManagement.create: false` takes effect. The chart then no longer
+ creates the Role and RoleBinding for the `configMap` source of the dynamic config, which 1.0
+ created regardless. To restore the behavior before 1.1.0, set it to `true`, the default
+ ([SPARK-59537](https://issues.apache.org/jira/browse/SPARK-59537)).
+- Since 1.1.0, the operator ClusterRole and Roles grant only `get`, `create` and `patch` on
+ `events`, which are the verbs the operator uses, instead of all verbs. If other workloads use
+ the operator ServiceAccount and need `list`, `watch`, `update` or `delete` on `events`, grant
+ them with another Role or ClusterRole ([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)).
+- Since 1.1.0, each option of `operatorDeployment.operatorPod.operatorContainer.jvmArgs` is passed
+ to the operator JVM as its own argument. 1.0 passed the whole value as one argument, which the
+ JVM took as a single `-Dfile.encoding` system property, so the other options had no effect. The
+ default options now take effect, e.g. the Parallel GC and a heap of 80% of the container memory
+ limit (`2Gi` by default), fully touched at startup, instead of at most 25%. Keep a memory limit
+ on the operator container, since the percentages apply to the node memory otherwise. The options
+ also take precedence over `JAVA_TOOL_OPTIONS` and `JDK_JAVA_OPTIONS` in `operatorContainer.env`,
+ and cannot contain a space, since the value is split on spaces. To restore the behavior before
+ 1.1.0, set `jvmArgs` to `-Dfile.encoding=UTF8` ([SPARK-58426](https://issues.apache.org/jira/browse/SPARK-58426)).
+
+### SparkApplication
+
+- Since 1.1.0, when the operator retries creating the driver resources of an application, it waits
+ as long as the `Retry-After` hint of the API server asks, capped at
+ `spark.kubernetes.operator.api.secondaryResourceCreateMaxBackoffMillis`. 1.0 ignored the hint,
+ and retried errors other than `409` and `429`, e.g. `503`, right away. A request which the API
+ server never answered, e.g. on a connection reset or a timeout, is retried with exponential
+ backoff too, while 1.0 failed the attempt with `SchedulingFailure` at once. The number of
+ attempts is still bounded by `spark.kubernetes.operator.api.secondaryResourceCreateMaxAttempts`
+ ([SPARK-59092](https://issues.apache.org/jira/browse/SPARK-59092), [SPARK-59694](https://issues.apache.org/jira/browse/SPARK-59694)).
+- Since 1.1.0, the duration of an attempt which is compared with `restartCounterResetMillis` is
+ measured from the first state after `Submitted` or `ScheduledToRestart`, normally
+ `DriverRequested`, instead of from `Submitted` or `ScheduledToRestart`. The restart backoff no
+ longer counts, so an attempt which reset the restart counters in 1.0 may not reset them, and the
+ application may reach `maxRestartAttempts` earlier. To keep a similar threshold, decrease
+ `restartCounterResetMillis` by the restart backoff, e.g. `restartBackoffMillis`
+ ([SPARK-59542](https://issues.apache.org/jira/browse/SPARK-59542)).
+- Since 1.1.0, an application which is configured to restart ends in `ResourceReleased` instead of
+ `TerminatedWithoutReleaseResources` after its last attempt, even if its `resourceRetainPolicy`
+ retains the resources, since they are released at the end of every attempt. So
+ `ttlAfterStopMillis` takes effect after the last attempt, while 1.0 applied it only once
+ `resourceRetainDurationMillis` elapsed, or never if the retain duration was disabled. To keep
+ such an application longer, increase `ttlAfterStopMillis`, or set it to a negative value
+ ([SPARK-59732](https://issues.apache.org/jira/browse/SPARK-59732)).
Review Comment:
**Finding 1.** The TTL half of this item doesn't match 1.0.0. There, `ttlAfterStopMillis` deleted such an app at the same time as `main` does. With `t0` as the end of the last attempt:
- 1.0 wrote `TerminatedWithoutReleaseResources` at `t0`.
- `checkEarlyExitForTerminatedApp` released it on demand once `computeEffectiveRetainDurationMillis()` passed. That is `min(resourceRetainDurationMillis, ttlAfterStopMillis)`, or the TTL alone when the retain duration is `-1`.
- `isRetainDurationEnabled()` is true whenever the TTL is set, so the "never" case can't happen.
- The next reconcile took the `ResourceReleased` branch. It measured the TTL from `getLastObservedStateBeforeTermination`, which is the `TerminatedWithoutReleaseResources` entry at `t0`.
- On `main`, the app goes to `ResourceReleased` at `t0`. The same helper reads the `Succeeded` / `Failed` entry, also at `t0`.
1.0.0's own `AppCleanUpStepTest` pins both 1.0 hops with the TTL-only spec `exceedRetainDurationFromTtl`. `checkEarlyExitForAppTerminatedWithoutReleaseResourcesExceededRetainDuration` releases the app, and `checkEarlyExitForResourceReleasedAppWithExceededTTL` deletes the CR. So increasing `ttlAfterStopMillis` would delete the app later than 1.0 did, not keep the 1.0 timing.
What changes is the final state itself. `resourceRetainDurationMillis` also no longer keeps such an app in `TerminatedWithoutReleaseResources` for a while:
```suggestion
`resourceRetainDurationMillis` no longer applies to it. `ttlAfterStopMillis` still deletes it at
the same time as in 1.0. A tool which waits for `TerminatedWithoutReleaseResources` has to wait
for `ResourceReleased` instead ([SPARK-59732](https://issues.apache.org/jira/browse/SPARK-59732)).
```
##########
docs/migration_guide.md:
##########
@@ -0,0 +1,152 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Migration Guide
+
+This document lists the behavior changes of Apache Spark K8s Operator releases which may affect an
+existing deployment, and how to restore the previous behavior where possible. When upgrading across
+several releases, read every section in between.
+
+New features which are disabled by default are not listed, and neither are changes which only
+affect building the project, CI, tests or examples.
+
+## Upgrading from 1.0 to 1.1
+
+### Kubernetes and CRDs
+
+- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34 ([SPARK-59692](https://issues.apache.org/jira/browse/SPARK-59692)).
+- Since 1.1.0, the CRDs have `spec.suspend` for `SparkApplication` and `SparkCluster`, and the
+ `Suspended` state for `SparkCluster`. Since `helm upgrade` does not update CRDs, replace them
+ before upgrading the operator, e.g. with the CRDs of the Helm chart `1.9.0`, which deploys
+ 1.1.0. Otherwise, the API server rejects or drops `spec.suspend`, and rejects the `Suspended`
+ state ([SPARK-59475](https://issues.apache.org/jira/browse/SPARK-59475), [SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+
+ ```bash
+ helm repo update
+ helm pull spark/spark-kubernetes-operator --version 1.9.0 --untar
+ kubectl replace -f spark-kubernetes-operator/crds/
+ ```
+
+### Helm Chart
+
+- Since 1.1.0, `operatorDeployment.networkPolicy.enable` and
+ `operatorConfiguration.dynamicConfig.enable` are deprecated in favor of `enabled`, and
+ `helm install` and `helm upgrade` print a warning for them. They are still honored until chart
+ `2.0.0`: a feature is enabled when either key is `true`, so a legacy `enable: true` must be
+ removed to disable it ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59540](https://issues.apache.org/jira/browse/SPARK-59540)).
+- Since 1.1.0, `operatorRbac.configManagement.create: false` takes effect. The chart then no longer
+ creates the Role and RoleBinding for the `configMap` source of the dynamic config, which 1.0
+ created regardless. To restore the behavior before 1.1.0, set it to `true`, the default
+ ([SPARK-59537](https://issues.apache.org/jira/browse/SPARK-59537)).
+- Since 1.1.0, the operator ClusterRole and Roles grant only `get`, `create` and `patch` on
+ `events`, which are the verbs the operator uses, instead of all verbs. If other workloads use
+ the operator ServiceAccount and need `list`, `watch`, `update` or `delete` on `events`, grant
+ them with another Role or ClusterRole ([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)).
+- Since 1.1.0, each option of `operatorDeployment.operatorPod.operatorContainer.jvmArgs` is passed
+ to the operator JVM as its own argument. 1.0 passed the whole value as one argument, which the
+ JVM took as a single `-Dfile.encoding` system property, so the other options had no effect. The
+ default options now take effect, e.g. the Parallel GC and a heap of 80% of the container memory
+ limit (`2Gi` by default), fully touched at startup, instead of at most 25%. Keep a memory limit
+ on the operator container, since the percentages apply to the node memory otherwise. The options
+ also take precedence over `JAVA_TOOL_OPTIONS` and `JDK_JAVA_OPTIONS` in `operatorContainer.env`,
+ and cannot contain a space, since the value is split on spaces. To restore the behavior before
+ 1.1.0, set `jvmArgs` to `-Dfile.encoding=UTF8` ([SPARK-58426](https://issues.apache.org/jira/browse/SPARK-58426)).
Review Comment:
**Finding 3.** `helm upgrade --reuse-values` from chart `1.8.0` to this chart fails. With that flag, Helm renders the new templates with the old chart's defaults (`chart.Values = oldVals` in `Upgrade.reuseValues`, `pkg/action/upgrade.go`, unchanged in v3.22.0 and v4.3.0). Those defaults lack three keys that `values.schema.json` now requires:
- `operatorDeployment.networkPolicy.enabled` and `operatorConfiguration.dynamicConfig.enabled`, from SPARK-59504.
- `operatorRbac.kueue`, from SPARK-59519.
I reproduced it with a copy of this chart whose `values.yaml` is replaced by the one of `1.0.0`. That is what `--reuse-values` renders for a release without overrides:
```
$ helm template spark ./chart-with-1.0.0-values
Error: values don't meet the specifications of the schema(s) in the following chart(s):
spark-kubernetes-operator:
- at '/operatorRbac': missing property 'kueue'
- at '/operatorConfiguration/dynamicConfig': missing property 'enabled'
- at '/operatorDeployment/networkPolicy': missing property 'enabled'
```
With the schema removed, the templates fail next, on `.Values.operatorRbac.kueue.enabled` in `templates/tests/test-rbac.yaml` (nil pointer). So a chart-side fix would need both the schema and the templates to tolerate the missing keys, and a doc item is the cheaper fix. The unmodified chart renders fine, also with `--set operatorDeployment.networkPolicy.enable=true`. That is what `--reset-then-reuse-values` (Helm 3.14+) does: the new defaults plus the release's own values.
```suggestion
1.1.0, set `jvmArgs` to `-Dfile.encoding=UTF8` ([SPARK-58426](https://issues.apache.org/jira/browse/SPARK-58426)).
- Since 1.1.0, `helm upgrade --reuse-values` from chart `1.8.0` fails. It renders the new chart
with the default values of chart `1.8.0`, which lack `operatorDeployment.networkPolicy.enabled`,
`operatorConfiguration.dynamicConfig.enabled` and `operatorRbac.kueue`, and the chart schema
requires them now. Use `--reset-then-reuse-values` of Helm 3.14 or newer instead, or pass the
values file again with `-f` ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59519](https://issues.apache.org/jira/browse/SPARK-59519)).
```
##########
docs/migration_guide.md:
##########
@@ -0,0 +1,152 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Migration Guide
+
+This document lists the behavior changes of Apache Spark K8s Operator releases which may affect an
+existing deployment, and how to restore the previous behavior where possible. When upgrading across
+several releases, read every section in between.
+
+New features which are disabled by default are not listed, and neither are changes which only
+affect building the project, CI, tests or examples.
+
+## Upgrading from 1.0 to 1.1
+
+### Kubernetes and CRDs
+
+- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34 ([SPARK-59692](https://issues.apache.org/jira/browse/SPARK-59692)).
+- Since 1.1.0, the CRDs have `spec.suspend` for `SparkApplication` and `SparkCluster`, and the
+ `Suspended` state for `SparkCluster`. Since `helm upgrade` does not update CRDs, replace them
+ before upgrading the operator, e.g. with the CRDs of the Helm chart `1.9.0`, which deploys
+ 1.1.0. Otherwise, the API server rejects or drops `spec.suspend`, and rejects the `Suspended`
+ state ([SPARK-59475](https://issues.apache.org/jira/browse/SPARK-59475), [SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+
+ ```bash
+ helm repo update
+ helm pull spark/spark-kubernetes-operator --version 1.9.0 --untar
+ kubectl replace -f spark-kubernetes-operator/crds/
+ ```
+
+### Helm Chart
+
+- Since 1.1.0, `operatorDeployment.networkPolicy.enable` and
+ `operatorConfiguration.dynamicConfig.enable` are deprecated in favor of `enabled`, and
+ `helm install` and `helm upgrade` print a warning for them. They are still honored until chart
+ `2.0.0`: a feature is enabled when either key is `true`, so a legacy `enable: true` must be
+ removed to disable it ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59540](https://issues.apache.org/jira/browse/SPARK-59540)).
+- Since 1.1.0, `operatorRbac.configManagement.create: false` takes effect. The chart then no longer
+ creates the Role and RoleBinding for the `configMap` source of the dynamic config, which 1.0
+ created regardless. To restore the behavior before 1.1.0, set it to `true`, the default
+ ([SPARK-59537](https://issues.apache.org/jira/browse/SPARK-59537)).
+- Since 1.1.0, the operator ClusterRole and Roles grant only `get`, `create` and `patch` on
+ `events`, which are the verbs the operator uses, instead of all verbs. If other workloads use
+ the operator ServiceAccount and need `list`, `watch`, `update` or `delete` on `events`, grant
+ them with another Role or ClusterRole ([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)).
+- Since 1.1.0, each option of `operatorDeployment.operatorPod.operatorContainer.jvmArgs` is passed
+ to the operator JVM as its own argument. 1.0 passed the whole value as one argument, which the
+ JVM took as a single `-Dfile.encoding` system property, so the other options had no effect. The
+ default options now take effect, e.g. the Parallel GC and a heap of 80% of the container memory
+ limit (`2Gi` by default), fully touched at startup, instead of at most 25%. Keep a memory limit
+ on the operator container, since the percentages apply to the node memory otherwise. The options
+ also take precedence over `JAVA_TOOL_OPTIONS` and `JDK_JAVA_OPTIONS` in `operatorContainer.env`,
+ and cannot contain a space, since the value is split on spaces. To restore the behavior before
+ 1.1.0, set `jvmArgs` to `-Dfile.encoding=UTF8` ([SPARK-58426](https://issues.apache.org/jira/browse/SPARK-58426)).
+
+### SparkApplication
Review Comment:
**Finding 2.** SPARK-59892 ([`3b32bb3`](https://github.com/apache/spark-kubernetes-operator/commit/3b32bb387d2b912106d6ed654b5086183999d1d9)) merged at 15:42 UTC today, seven minutes after this PR was opened, so the base `bd15465` doesn't have it. It changes a 1.0 default:
- `spark.kubernetes.operator.reconciler.retry.maxIntervalSeconds` defaults to `-1`, documented as unlimited.
- 1.0.0 did nothing for a non-positive value, so JOSDK's own cap of 15187 ms applied.
- `main` now calls `withoutMaxInterval()`, so with the other defaults the retries grow to about 16 minutes apart.
Its PR already names the way back, setting it to `15`. It applies to both reconcilers, so it may fit a section of its own before this one:
```suggestion
### Operator
- Since 1.1.0, `spark.kubernetes.operator.reconciler.retry.maxIntervalSeconds`, `-1` by default,
means no limit on the interval between the retries of a failed reconciliation. 1.0 capped it at
about 15 seconds. With the other defaults, the retries now grow to about 16 minutes apart. To
restore the behavior before 1.1.0, set it to `15` ([SPARK-59892](https://issues.apache.org/jira/browse/SPARK-59892)).
### SparkApplication
```
##########
docs/migration_guide.md:
##########
@@ -0,0 +1,152 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Migration Guide
+
+This document lists the behavior changes of Apache Spark K8s Operator releases which may affect an
+existing deployment, and how to restore the previous behavior where possible. When upgrading across
+several releases, read every section in between.
+
+New features which are disabled by default are not listed, and neither are changes which only
+affect building the project, CI, tests or examples.
+
+## Upgrading from 1.0 to 1.1
+
+### Kubernetes and CRDs
+
+- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34 ([SPARK-59692](https://issues.apache.org/jira/browse/SPARK-59692)).
+- Since 1.1.0, the CRDs have `spec.suspend` for `SparkApplication` and `SparkCluster`, and the
+ `Suspended` state for `SparkCluster`. Since `helm upgrade` does not update CRDs, replace them
+ before upgrading the operator, e.g. with the CRDs of the Helm chart `1.9.0`, which deploys
+ 1.1.0. Otherwise, the API server rejects or drops `spec.suspend`, and rejects the `Suspended`
+ state ([SPARK-59475](https://issues.apache.org/jira/browse/SPARK-59475), [SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+
+ ```bash
+ helm repo update
+ helm pull spark/spark-kubernetes-operator --version 1.9.0 --untar
+ kubectl replace -f spark-kubernetes-operator/crds/
+ ```
+
+### Helm Chart
+
+- Since 1.1.0, `operatorDeployment.networkPolicy.enable` and
+ `operatorConfiguration.dynamicConfig.enable` are deprecated in favor of `enabled`, and
+ `helm install` and `helm upgrade` print a warning for them. They are still honored until chart
+ `2.0.0`: a feature is enabled when either key is `true`, so a legacy `enable: true` must be
+ removed to disable it ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59540](https://issues.apache.org/jira/browse/SPARK-59540)).
+- Since 1.1.0, `operatorRbac.configManagement.create: false` takes effect. The chart then no longer
+ creates the Role and RoleBinding for the `configMap` source of the dynamic config, which 1.0
+ created regardless. To restore the behavior before 1.1.0, set it to `true`, the default
+ ([SPARK-59537](https://issues.apache.org/jira/browse/SPARK-59537)).
+- Since 1.1.0, the operator ClusterRole and Roles grant only `get`, `create` and `patch` on
+ `events`, which are the verbs the operator uses, instead of all verbs. If other workloads use
+ the operator ServiceAccount and need `list`, `watch`, `update` or `delete` on `events`, grant
+ them with another Role or ClusterRole ([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)).
Review Comment:
**Finding 4.** 1.0.0 listed `events` in the same rule as `pods`, `services`, `configmaps` and `persistentvolumeclaims`, with `get`, `list`, `watch`, `create`, `update`, `patch` and `delete`. That is not all verbs (`deletecollection` was never granted). Someone auditing RBAC may read "all verbs" as `*`.
```suggestion
`events`, which are the verbs the operator uses, instead of the verbs they grant on pods. If
other workloads use the operator ServiceAccount and need `list`, `watch`, `update` or `delete`
on `events`, grant them with another Role or ClusterRole ([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)).
```
##########
docs/migration_guide.md:
##########
@@ -0,0 +1,152 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Migration Guide
+
+This document lists the behavior changes of Apache Spark K8s Operator releases which may affect an
+existing deployment, and how to restore the previous behavior where possible. When upgrading across
+several releases, read every section in between.
+
+New features which are disabled by default are not listed, and neither are changes which only
+affect building the project, CI, tests or examples.
+
+## Upgrading from 1.0 to 1.1
+
+### Kubernetes and CRDs
+
+- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34 ([SPARK-59692](https://issues.apache.org/jira/browse/SPARK-59692)).
+- Since 1.1.0, the CRDs have `spec.suspend` for `SparkApplication` and `SparkCluster`, and the
+ `Suspended` state for `SparkCluster`. Since `helm upgrade` does not update CRDs, replace them
+ before upgrading the operator, e.g. with the CRDs of the Helm chart `1.9.0`, which deploys
+ 1.1.0. Otherwise, the API server rejects or drops `spec.suspend`, and rejects the `Suspended`
+ state ([SPARK-59475](https://issues.apache.org/jira/browse/SPARK-59475), [SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+
+ ```bash
+ helm repo update
+ helm pull spark/spark-kubernetes-operator --version 1.9.0 --untar
+ kubectl replace -f spark-kubernetes-operator/crds/
+ ```
+
+### Helm Chart
+
+- Since 1.1.0, `operatorDeployment.networkPolicy.enable` and
+ `operatorConfiguration.dynamicConfig.enable` are deprecated in favor of `enabled`, and
+ `helm install` and `helm upgrade` print a warning for them. They are still honored until chart
+ `2.0.0`: a feature is enabled when either key is `true`, so a legacy `enable: true` must be
+ removed to disable it ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59540](https://issues.apache.org/jira/browse/SPARK-59540)).
+- Since 1.1.0, `operatorRbac.configManagement.create: false` takes effect. The chart then no longer
+ creates the Role and RoleBinding for the `configMap` source of the dynamic config, which 1.0
+ created regardless. To restore the behavior before 1.1.0, set it to `true`, the default
+ ([SPARK-59537](https://issues.apache.org/jira/browse/SPARK-59537)).
+- Since 1.1.0, the operator ClusterRole and Roles grant only `get`, `create` and `patch` on
+ `events`, which are the verbs the operator uses, instead of all verbs. If other workloads use
+ the operator ServiceAccount and need `list`, `watch`, `update` or `delete` on `events`, grant
+ them with another Role or ClusterRole ([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)).
+- Since 1.1.0, each option of `operatorDeployment.operatorPod.operatorContainer.jvmArgs` is passed
+ to the operator JVM as its own argument. 1.0 passed the whole value as one argument, which the
+ JVM took as a single `-Dfile.encoding` system property, so the other options had no effect. The
+ default options now take effect, e.g. the Parallel GC and a heap of 80% of the container memory
+ limit (`2Gi` by default), fully touched at startup, instead of at most 25%. Keep a memory limit
Review Comment:
**Finding 5.** The default `jvmArgs` also carry `-XX:+CrashOnOutOfMemoryError -XX:ErrorFile=/dev/stderr` (`values.yaml:44`). In 1.0 an `OutOfMemoryError` left the JVM running. Now it aborts the JVM, so the operator pod restarts. That is the one new default an operator team would notice in an incident.
```suggestion
limit (`2Gi` by default), fully touched at startup, instead of at most 25%. An
`OutOfMemoryError` now crashes the JVM, so the container restarts. Keep a memory limit
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59895] Add `migration_guide.md` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #914:
URL: https://github.com/apache/spark-kubernetes-operator/pull/914#issuecomment-5917489181
Thank you for the review, @peter-toth. Here is the summary of the changes for your findings.
| Finding | Resolution |
| --- | --- |
| 1. TTL part of the SPARK-59732 item | Fixed. `resourceRetainDurationMillis` no longer applies, `ttlAfterStopMillis` deletes such an app at the same time as in 1.0, and only the final state changes. |
| 2. No item for SPARK-59892 | Already covered by the `Operator` section of [`f1e7d03`](https://github.com/apache/spark-kubernetes-operator/commit/f1e7d032c5614da3cecdd153668bbd0caa960d15). |
| 3. `helm upgrade --reuse-values` from chart `1.8.0` fails | Added as the first item of the `Helm Chart` section, with `--reset-then-reuse-values` and `-f` as the ways to upgrade. |
| 4. 1.0 did not grant all verbs on `events` | Fixed. The item lists `list`, `watch`, `update` and `delete`, which only 1.0 granted. |
| 5. The `jvmArgs` item leaves out `-XX:+CrashOnOutOfMemoryError` | Added that an `OutOfMemoryError` now crashes the JVM, so the container restarts. |
Findings 1, 3, 4 and 5 are addressed in [`3fb90cf`](https://github.com/apache/spark-kubernetes-operator/commit/3fb90cfdbf5340e423ee4e504db1d38b3319fdc5), and the PR description is updated accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59895] Add `migration_guide.md` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #914: [SPARK-59895] Add `migration_guide.md`
URL: https://github.com/apache/spark-kubernetes-operator/pull/914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59895] Add `migration_guide.md` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #914:
URL: https://github.com/apache/spark-kubernetes-operator/pull/914#issuecomment-5914739051
cc @peter-toth , @jiangzho , @TQJADE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59895] Add `migration_guide.md` [spark-kubernetes-operator]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #914:
URL: https://github.com/apache/spark-kubernetes-operator/pull/914#discussion_r4148018292
##########
docs/migration_guide.md:
##########
@@ -0,0 +1,152 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Migration Guide
+
+This document lists the behavior changes of Apache Spark K8s Operator releases which may affect an
+existing deployment, and how to restore the previous behavior where possible. When upgrading across
+several releases, read every section in between.
+
+New features which are disabled by default are not listed, and neither are changes which only
+affect building the project, CI, tests or examples.
+
+## Upgrading from 1.0 to 1.1
+
+### Kubernetes and CRDs
+
+- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34 ([SPARK-59692](https://issues.apache.org/jira/browse/SPARK-59692)).
+- Since 1.1.0, the CRDs have `spec.suspend` for `SparkApplication` and `SparkCluster`, and the
+ `Suspended` state for `SparkCluster`. Since `helm upgrade` does not update CRDs, replace them
+ before upgrading the operator, e.g. with the CRDs of the Helm chart `1.9.0`, which deploys
+ 1.1.0. Otherwise, the API server rejects or drops `spec.suspend`, and rejects the `Suspended`
+ state ([SPARK-59475](https://issues.apache.org/jira/browse/SPARK-59475), [SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)).
+
+ ```bash
+ helm repo update
+ helm pull spark/spark-kubernetes-operator --version 1.9.0 --untar
+ kubectl replace -f spark-kubernetes-operator/crds/
+ ```
+
+### Helm Chart
+
+- Since 1.1.0, `operatorDeployment.networkPolicy.enable` and
+ `operatorConfiguration.dynamicConfig.enable` are deprecated in favor of `enabled`, and
+ `helm install` and `helm upgrade` print a warning for them. They are still honored until chart
+ `2.0.0`: a feature is enabled when either key is `true`, so a legacy `enable: true` must be
+ removed to disable it ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59540](https://issues.apache.org/jira/browse/SPARK-59540)).
+- Since 1.1.0, `operatorRbac.configManagement.create: false` takes effect. The chart then no longer
+ creates the Role and RoleBinding for the `configMap` source of the dynamic config, which 1.0
+ created regardless. To restore the behavior before 1.1.0, set it to `true`, the default
+ ([SPARK-59537](https://issues.apache.org/jira/browse/SPARK-59537)).
+- Since 1.1.0, the operator ClusterRole and Roles grant only `get`, `create` and `patch` on
+ `events`, which are the verbs the operator uses, instead of all verbs. If other workloads use
+ the operator ServiceAccount and need `list`, `watch`, `update` or `delete` on `events`, grant
+ them with another Role or ClusterRole ([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)).
+- Since 1.1.0, each option of `operatorDeployment.operatorPod.operatorContainer.jvmArgs` is passed
+ to the operator JVM as its own argument. 1.0 passed the whole value as one argument, which the
+ JVM took as a single `-Dfile.encoding` system property, so the other options had no effect. The
+ default options now take effect, e.g. the Parallel GC and a heap of 80% of the container memory
+ limit (`2Gi` by default), fully touched at startup, instead of at most 25%. Keep a memory limit
+ on the operator container, since the percentages apply to the node memory otherwise. The options
+ also take precedence over `JAVA_TOOL_OPTIONS` and `JDK_JAVA_OPTIONS` in `operatorContainer.env`,
+ and cannot contain a space, since the value is split on spaces. To restore the behavior before
+ 1.1.0, set `jvmArgs` to `-Dfile.encoding=UTF8` ([SPARK-58426](https://issues.apache.org/jira/browse/SPARK-58426)).
+
+### SparkApplication
Review Comment:
This is already covered by [`f1e7d03`](https://github.com/apache/spark-kubernetes-operator/commit/f1e7d032c5614da3cecdd153668bbd0caa960d15), which was pushed while I was reviewing `cd807ff`. Please treat this finding as moot. Findings 1, 3, 4 and 5 still apply to `f1e7d03`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59895] Add `migration_guide.md` [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #914:
URL: https://github.com/apache/spark-kubernetes-operator/pull/914#issuecomment-5917721386
Merged to main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org