You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiangzho (via GitHub)" <gi...@apache.org> on 2024/08/29 23:02:54 UTC
[PR] [SPARK-49464] Add docs for operator [spark-kubernetes-operator]
jiangzho opened a new pull request, #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, use `Draft` feature of GitHub Action.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If there is design documentation, please add the link.
3. If there is a discussion in the mailing list, please add the link.
-->
This PR includes
1. Operator docs under `docs/` for configuration, architecture, operations, and metrics.
2. A module to automatically generate config properties table based on `org.apache.spark.k8s.operator.config.SparkOperatorConf`
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
3. If you fix a bug, you can clarify why it is a bug.
-->
Operator docs are necessary for users to understand the design and getting started with the operator installation
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No - new release
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
CIs
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1744840447
##########
docs/operations.md:
##########
@@ -0,0 +1,132 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+### Compatibility
+
+- JDK17
+- Operator used fabric8 which assumes to be compatible with available k8s versions. However for using status subresource, please use k8s version 1.14 or above.
+- Spark versions 3.4 or above
+
+## Manage Your Spark Operator
Review Comment:
Remove this section because it's duplicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762473124
##########
docs/configuration.md:
##########
@@ -0,0 +1,59 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Configuration
+
+## Configure Operator
+
+Spark Operator supports different ways to configure the behavior:
+
+* **spark-operator.properties** provided when deploying the operator. In addition to the
+ [property file](../build-tools/helm/spark-kubernetes-operator/conf/spark-operator.
+ properties), it is also possible to override or append config properties in helm [Values
+ files](../build-tools/helm/spark-kubernetes-operator/values.yaml).
+* **System Properties** : when provided as system properties (e.g. via -D options to the
+ operator JVM), it overrides the values provided in property file.
+* **Hot property loading** : when enabled, a
+ [configmap](https://kubernetes.io/docs/concepts/configuration/configmap/) would be created with
+ the operator in the same namespace. Operator can monitor updates performed on the configmap. Hot
+ properties reloading takes higher precedence comparing with default properties override.
+ - An example use case: operator use hot properties to figure the list of namespace(s) to
+ operate Spark applications. The hot properties config map can be updated and
+ maintained by user or additional microservice to tune the operator behavior without
+ rebooting it.
+ - Please be advised that not all properties can be hot-loaded and honored at runtime.
+ Refer the list of [supported properties](./config_properties.md) for more details.
+
+To enable hot properties loading, update the **helm chart values file** with
+
+```
+
+operatorConfiguration:
+ spark-operator.properties: |+
+ spark.operator.dynamic.config.enabled=true
+ # ... all other config overides...
+ dynamicConfig:
+ create: true
+
+```
+
+## Config Metrics Publishing Behavior
+
+Spark Operator uses the same source & sink interface as Apache Spark. You may
Review Comment:
If Spark has corresponding doc, it is better to add a hyper link here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762475001
##########
docs/metrics_logging.md:
##########
@@ -0,0 +1,109 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Metrics
+
+Spark operator,
+following [Apache Spark](https://spark.apache.org/docs/latest/monitoring.html#metrics),
+has a configurable metrics system based on
+the [Dropwizard Metrics Library](https://metrics.dropwizard.io/4.2.25/). Note that Spark Operator
+does not have Spark UI, MetricsServlet
+and PrometheusServlet from org.apache.spark.metrics.sink package are not supported. If you are
+interested in Prometheus metrics exporting, please take a look at below section `Forward Metrics to Prometheus`
Review Comment:
Add hyper link?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1744843093
##########
docs/operations.md:
##########
@@ -0,0 +1,132 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+### Compatibility
+
+- JDK17
+- Operator used fabric8 which assumes to be compatible with available k8s versions. However for using status subresource, please use k8s version 1.14 or above.
+- Spark versions 3.4 or above
+
+## Manage Your Spark Operator
+
+The operator installation is managed by a helm chart. To install run:
+
+```
+helm install spark-kubernetes-operator \
+ -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ build-tools/helm/spark-kubernetes-operator/
+```
+
+Alternatively to install the operator (and also the helm chart) to a specific namespace:
+
+```
+helm install spark-kubernetes-operator \
+ -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ build-tools/helm/spark-kubernetes-operator/ \
+ --namespace spark-system --create-namespace
+```
+
+Note that in this case you will need to update the namespace in the examples accordingly.
+
+### Spark Application Namespaces
+
+By default, Spark applications are created in the same namespace as the operator deployment.
+You many also configure the chart deployment to add necessary RBAC resources for
+applications to enable them running in additional namespaces.
+
+## Overriding configuration parameters during Helm install
+
+Helm provides different ways to override the default installation parameters (contained
+in `values.yaml`) for the Helm chart.
+
+To override single parameters you can use `--set`, for example:
+
+```
+helm install --set image.repository=<my_registory>/spark-kubernetes-operator \
+ -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ build-tools/helm/spark-kubernetes-operator/
+```
+
+You can also provide multiple custom values file by using the `-f` flag, the latest takes
+higher precedence:
+
+```
+helm install spark-kubernetes-operator \
+ -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ -f my_values.yaml \
+ build-tools/helm/spark-kubernetes-operator/
+```
+
+The configurable parameters of the Helm chart and which default values as detailed in the
+following table:
+
+| Parameters | Description | Default value |
+|----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
+| image.repository | The image repository of spark-kubernetes-operator. | spark-kubernetes-operator |
+| image.pullPolicy | The image pull policy of spark-kubernetes-operator. | IfNotPresent |
+| image.tag | The image tag of spark-kubernetes-operator. | |
+| image.digest | The image digest of spark-kubernetes-operator. If set then it takes precedence and the image tag will be ignored. | |
+| imagePullSecrets | The image pull secrets of spark-kubernetes-operator. | |
+| operatorDeployment.replica | Operator replica count. Must be 1 unless leader election is configured. | 1 |
+| operatorDeployment.strategy.type | Operator pod upgrade strategy. Must be Recreate unless leader election is configured. | Recreate |
+| operatorDeployment.operatorPod.annotations | Custom annotations to be added to the operator pod | |
+| operatorDeployment.operatorPod.labels | Custom labels to be added to the operator pod | |
+| operatorDeployment.operatorPod.nodeSelector | Custom nodeSelector to be added to the operator pod. | |
+| operatorDeployment.operatorPod.topologySpreadConstraints | Custom topologySpreadConstraints to be added to the operator pod. | |
+| operatorDeployment.operatorPod.dnsConfig | DNS configuration to be used by the operator pod. | |
+| operatorDeployment.operatorPod.volumes | Additional volumes to be added to the operator pod. | |
+| operatorDeployment.operatorPod.priorityClassName | Priority class name to be used for the operator pod | |
+| operatorDeployment.operatorPod.securityContext | Security context overrides for the operator pod | |
+| operatorDeployment.operatorContainer.jvmArgs | JVM arg override for the operator container. | `-XX:+UseG1GC -Xms3G -Xmx3G -Dfile.encoding=UTF8` |
+| operatorDeployment.operatorContainer.env | Custom env to be added to the operator container. | |
+| operatorDeployment.operatorContainer.envFrom | Custom envFrom to be added to the operator container, e.g. for downward API. | |
+| operatorDeployment.operatorContainer.probes | Probe config for the operator container. | |
+| operatorDeployment.operatorContainer.securityContext | Security context overrides for the operator container. | run as non root for baseline secuirty standard compliance |
+| operatorDeployment.operatorContainer.resources | Resources for the operator container. | memory 4Gi, ephemeral storage 2Gi and 1 cpu |
+| operatorDeployment.additionalContainers | Additional containers to be added to the operator pod, e.g. sidecar. | |
+| operatorRbac.serviceAccount.create | Whether to create service account for operator to use. | |
+| operatorRbac.clusterRole.create | Whether to create ClusterRole for operator to use. | true |
+| operatorRbac.clusterRoleBinding.create | Whether to create ClusterRoleBinding for operator to use. | true |
+| operatorRbac.role.create | Whether to create Role for operator to use. At least one of `clusterRole.create` or `role.create` should be enabled | true |
+| operatorRbac.roleBinding.create | Whether to create RoleBinding for operator to use. At least one of `clusterRoleBinding.create` or `roleBinding.create` should be enabled | true |
+| operatorRbac.clusterRole.configManagement.roleName | Role name for operator configuration management (hot property loading and leader election) | `spark-operator-config-role` |
+| appResources.namespaces.create | Whether to create dedicated namespaces for Spark apps. | `spark-operator-config-role-binding` |
Review Comment:
Shall we add `clusterResources` first before adding this document? It looks a little weird because the document is missing one of the part while Apache Spark Operator supports both `SparkApp` CRD and `SparkCluster` CRD.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762265641
##########
docs/operator_probes.md:
##########
@@ -0,0 +1,82 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Operator Probes
+
+In Kubernetes world, the kubelet uses readiness probes to know when a container is ready to
+start accepting traffic, and it uses liveness probes to know when to restart a container. Here
+for Spark Operators, we provided those as below by default. You can override the values in
+values.yaml if you use Helm Chart to deploy the Spark Operator.
+
+```
+ports:
+- containerPort: 18080
+ name: probe-port
+livenessProbe:
+ httpGet:
+ port: probe-port
+ path: /healthz
+ initialDelaySeconds: 30
+ periodSeconds: 10
+readinessProbe:
+ httpGet:
+ port: probe-port
+ path: /readyz
+ failureThreshold: 30
+ periodSeconds: 10
+```
+
+## Operator Readiness Probe
+
+A readiness probe helps to determine whether current instances can serve the traffic.
+Therefore, Spark Operator's readiness probe has to make sure both operator has started and also
+need to verify the existence of required rbac access.
+
+## Operator Health(Liveness) Probe
+
+A built-in health endpoint that serves as the information source for Operator liveness. Since
+Java Operator SDK provides [runtimeInfo](https://javaoperatorsdk.io/docs/features#runtime-info)
+to check the actual health of event sources. Spark Operator' s healthProbe will check:
+
+* operator runtimeInfo health state
+* Sentinel resources health state
+
+### Operator Sentinel Resource
Review Comment:
Please move this section only into `operations.md` and remove `operator_probles.md` completely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762477153
##########
docs/metrics_logging.md:
##########
@@ -0,0 +1,109 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Metrics
+
+Spark operator,
+following [Apache Spark](https://spark.apache.org/docs/latest/monitoring.html#metrics),
+has a configurable metrics system based on
+the [Dropwizard Metrics Library](https://metrics.dropwizard.io/4.2.25/). Note that Spark Operator
+does not have Spark UI, MetricsServlet
+and PrometheusServlet from org.apache.spark.metrics.sink package are not supported. If you are
+interested in Prometheus metrics exporting, please take a look at below section `Forward Metrics to Prometheus`
+
+## JVM Metrics
+
+Spark Operator collects JVM metrics
+via [Codahale JVM Metrics](https://javadoc.io/doc/com.codahale.metrics/metrics-jvm/latest/index.html)
+
+- BufferPoolMetricSet
+- FileDescriptorRatioGauge
+- GarbageCollectorMetricSet
+- MemoryUsageGaugeSet
+- ThreadStatesGaugeSet
+
+## Kubernetes Client Metrics
+
+| Metrics Name | Type | Description |
+|-----------------------------------------------------------|------------|--------------------------------------------------------------------------------------------------------------------------|
+| kubernetes.client.http.request | Meter | Tracking the rates of HTTP request sent to the Kubernetes API Server |
+| kubernetes.client.http.response | Meter | Tracking the rates of HTTP response from the Kubernetes API Server |
+| kubernetes.client.http.response.failed | Meter | Tracking the rates of HTTP requests which have no response from the Kubernetes API Server |
+| kubernetes.client.http.response.latency.nanos | Histograms | Measures the statistical distribution of HTTP response latency from the Kubernetes API Server |
+| kubernetes.client.http.response.<ResponseCode> | Meter | Tracking the rates of HTTP response based on response code from the Kubernetes API Server |
+| kubernetes.client.http.request.<RequestMethod> | Meter | Tracking the rates of HTTP request based type of method to the Kubernetes API Server |
+| kubernetes.client.http.response.1xx | Meter | Tracking the rates of HTTP Code 1xx responses (informational) received from the Kubernetes API Server per response code. |
+| kubernetes.client.http.response.2xx | Meter | Tracking the rates of HTTP Code 2xx responses (success) received from the Kubernetes API Server per response code. |
+| kubernetes.client.http.response.3xx | Meter | Tracking the rates of HTTP Code 3xx responses (redirection) received from the Kubernetes API Server per response code. |
+| kubernetes.client.http.response.4xx | Meter | Tracking the rates of HTTP Code 4xx responses (client error) received from the Kubernetes API Server per response code. |
+| kubernetes.client.http.response.5xx | Meter | Tracking the rates of HTTP Code 5xx responses (server error) received from the Kubernetes API Server per response code. |
+| kubernetes.client.<ResourceName>.<Method> | Meter | Tracking the rates of HTTP request for a combination of one Kubernetes resource and one http method |
+| kubernetes.client.<NamespaceName>.<ResourceName>.<Method> | Meter | Tracking the rates of HTTP request for a combination of one namespace-scoped Kubernetes resource and one http method |
+
+## Forward Metrics to Prometheus
+
+In this section, we will show you how to forward spark operator metrics
Review Comment:
```suggestion
In this section, we will show you how to forward Spark Operator metrics
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1779321617
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
Review Comment:
Thanks a lot for the review and sorry for the late response!
I updated this to try a best-effort to cover SparkCluster as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762259132
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
+* Operator launches driver and observes its status
+* Operator observes driver-spawn resources (e.g. executors) till app terminates
+* Operator releases all Spark-app owned resources to cluster
+* The SparkApplication CR can be (re)applied on the cluster any time - e.g. to issue proactive
+ termination of an application. The Operator makes continuous adjustments to imitate the
+ desired state until the
+ current state becomes the desired state. All lifecycle management operations are realized
+ using this very simple
+ principle in the Operator.
+
+The Operator is built with the Java Operator SDK and uses the Native Kubernetes Integration for
+launching Spark deployments and submitting jobs under the hood. The Java Operator SDK is a
+higher level
+framework and related tooling to support writing Kubernetes Operators in Java. Both the Java
+Operator SDK and Spark’s native
+kubernetes integration itself is using the Fabric8 Kubernetes Client to interact with the
+Kubernetes API Server.
+
+## State Transition
+
+[<img src="resources/state.png">](resources/state.png)
+
+* Spark application are expected to run from submitted to succeeded before releasing resources
+* User may configure the app CR to time-out after given threshold of time
+* In addition, user may configure the app CR to skip releasing resources after terminated. This is
+ typically used at dev phase: pods / configmaps. etc would be kept for debugging. They have
Review Comment:
` pods / configmaps. etc `?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762479437
##########
docs/metrics_logging.md:
##########
@@ -0,0 +1,109 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Metrics
+
+Spark operator,
+following [Apache Spark](https://spark.apache.org/docs/latest/monitoring.html#metrics),
+has a configurable metrics system based on
+the [Dropwizard Metrics Library](https://metrics.dropwizard.io/4.2.25/). Note that Spark Operator
+does not have Spark UI, MetricsServlet
+and PrometheusServlet from org.apache.spark.metrics.sink package are not supported. If you are
+interested in Prometheus metrics exporting, please take a look at below section `Forward Metrics to Prometheus`
+
+## JVM Metrics
+
+Spark Operator collects JVM metrics
+via [Codahale JVM Metrics](https://javadoc.io/doc/com.codahale.metrics/metrics-jvm/latest/index.html)
+
+- BufferPoolMetricSet
+- FileDescriptorRatioGauge
+- GarbageCollectorMetricSet
+- MemoryUsageGaugeSet
+- ThreadStatesGaugeSet
+
+## Kubernetes Client Metrics
+
+| Metrics Name | Type | Description |
+|-----------------------------------------------------------|------------|--------------------------------------------------------------------------------------------------------------------------|
+| kubernetes.client.http.request | Meter | Tracking the rates of HTTP request sent to the Kubernetes API Server |
+| kubernetes.client.http.response | Meter | Tracking the rates of HTTP response from the Kubernetes API Server |
+| kubernetes.client.http.response.failed | Meter | Tracking the rates of HTTP requests which have no response from the Kubernetes API Server |
+| kubernetes.client.http.response.latency.nanos | Histograms | Measures the statistical distribution of HTTP response latency from the Kubernetes API Server |
+| kubernetes.client.http.response.<ResponseCode> | Meter | Tracking the rates of HTTP response based on response code from the Kubernetes API Server |
+| kubernetes.client.http.request.<RequestMethod> | Meter | Tracking the rates of HTTP request based type of method to the Kubernetes API Server |
+| kubernetes.client.http.response.1xx | Meter | Tracking the rates of HTTP Code 1xx responses (informational) received from the Kubernetes API Server per response code. |
+| kubernetes.client.http.response.2xx | Meter | Tracking the rates of HTTP Code 2xx responses (success) received from the Kubernetes API Server per response code. |
+| kubernetes.client.http.response.3xx | Meter | Tracking the rates of HTTP Code 3xx responses (redirection) received from the Kubernetes API Server per response code. |
+| kubernetes.client.http.response.4xx | Meter | Tracking the rates of HTTP Code 4xx responses (client error) received from the Kubernetes API Server per response code. |
+| kubernetes.client.http.response.5xx | Meter | Tracking the rates of HTTP Code 5xx responses (server error) received from the Kubernetes API Server per response code. |
+| kubernetes.client.<ResourceName>.<Method> | Meter | Tracking the rates of HTTP request for a combination of one Kubernetes resource and one http method |
+| kubernetes.client.<NamespaceName>.<ResourceName>.<Method> | Meter | Tracking the rates of HTTP request for a combination of one namespace-scoped Kubernetes resource and one http method |
+
+## Forward Metrics to Prometheus
+
+In this section, we will show you how to forward spark operator metrics
+to [Prometheus](https://prometheus.io).
+
+* Modify the
+ build-tools/helm/spark-kubernetes-operator/values.yaml file' s metrics properties section:
Review Comment:
```suggestion
* Modify the metrics properties section in the file
`build-tools/helm/spark-kubernetes-operator/values.yaml`:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762257265
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
+* Operator launches driver and observes its status
+* Operator observes driver-spawn resources (e.g. executors) till app terminates
+* Operator releases all Spark-app owned resources to cluster
+* The SparkApplication CR can be (re)applied on the cluster any time - e.g. to issue proactive
+ termination of an application. The Operator makes continuous adjustments to imitate the
+ desired state until the
+ current state becomes the desired state. All lifecycle management operations are realized
+ using this very simple
+ principle in the Operator.
+
+The Operator is built with the Java Operator SDK and uses the Native Kubernetes Integration for
Review Comment:
- Please add a hyperlink for `Java Operator SDK` and remove `the`.
- What is `the Native Kubernetes Integration`? When you refer with `the`, you had better add a hyperlink for clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762253536
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
Review Comment:
Let's not assume that the users are unaware of `kubectl` and `helm`. It's too common in these days, even in the ASF projects.
> Users can interact with the operator using the kubectl or k8s API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1779322246
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
+* Operator launches driver and observes its status
+* Operator observes driver-spawn resources (e.g. executors) till app terminates
+* Operator releases all Spark-app owned resources to cluster
+* The SparkApplication CR can be (re)applied on the cluster any time - e.g. to issue proactive
+ termination of an application. The Operator makes continuous adjustments to imitate the
+ desired state until the
+ current state becomes the desired state. All lifecycle management operations are realized
+ using this very simple
+ principle in the Operator.
+
+The Operator is built with the Java Operator SDK and uses the Native Kubernetes Integration for
+launching Spark deployments and submitting jobs under the hood. The Java Operator SDK is a
+higher level
+framework and related tooling to support writing Kubernetes Operators in Java. Both the Java
+Operator SDK and Spark’s native
+kubernetes integration itself is using the Fabric8 Kubernetes Client to interact with the
+Kubernetes API Server.
+
+## State Transition
+
+[<img src="resources/state.png">](resources/state.png)
+
+* Spark application are expected to run from submitted to succeeded before releasing resources
+* User may configure the app CR to time-out after given threshold of time
Review Comment:
The application diagram tried to cover timeout blocks as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762258048
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
+* Operator launches driver and observes its status
+* Operator observes driver-spawn resources (e.g. executors) till app terminates
+* Operator releases all Spark-app owned resources to cluster
+* The SparkApplication CR can be (re)applied on the cluster any time - e.g. to issue proactive
+ termination of an application. The Operator makes continuous adjustments to imitate the
+ desired state until the
+ current state becomes the desired state. All lifecycle management operations are realized
+ using this very simple
+ principle in the Operator.
+
+The Operator is built with the Java Operator SDK and uses the Native Kubernetes Integration for
+launching Spark deployments and submitting jobs under the hood. The Java Operator SDK is a
+higher level
+framework and related tooling to support writing Kubernetes Operators in Java. Both the Java
+Operator SDK and Spark’s native
+kubernetes integration itself is using the Fabric8 Kubernetes Client to interact with the
+Kubernetes API Server.
+
+## State Transition
+
+[<img src="resources/state.png">](resources/state.png)
Review Comment:
`png` is not editable. When we need to update this in the future, how can we do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762483310
##########
docs/spark_custom_resources.md:
##########
@@ -0,0 +1,203 @@
+## Spark Operator API
Review Comment:
License header?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1744839631
##########
docs/operations.md:
##########
@@ -0,0 +1,132 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+### Compatibility
+
+- JDK17
+- Operator used fabric8 which assumes to be compatible with available k8s versions. However for using status subresource, please use k8s version 1.14 or above.
Review Comment:
As I pinged you already, the K8s eco-system is moving faster in the public environment.
- https://github.com/apache/spark/pull/47990
- https://github.com/apache/spark-kubernetes-operator/pull/117
Just FYI, in the community, please don't claim which you didn't test explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1744837381
##########
docs/operations.md:
##########
@@ -0,0 +1,132 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+### Compatibility
+
+- JDK17
+- Operator used fabric8 which assumes to be compatible with available k8s versions. However for using status subresource, please use k8s version 1.14 or above.
+- Spark versions 3.4 or above
Review Comment:
Apache Spark 3.4.x reaches the end-of-life very soon (2024-10-13).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762259132
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
+* Operator launches driver and observes its status
+* Operator observes driver-spawn resources (e.g. executors) till app terminates
+* Operator releases all Spark-app owned resources to cluster
+* The SparkApplication CR can be (re)applied on the cluster any time - e.g. to issue proactive
+ termination of an application. The Operator makes continuous adjustments to imitate the
+ desired state until the
+ current state becomes the desired state. All lifecycle management operations are realized
+ using this very simple
+ principle in the Operator.
+
+The Operator is built with the Java Operator SDK and uses the Native Kubernetes Integration for
+launching Spark deployments and submitting jobs under the hood. The Java Operator SDK is a
+higher level
+framework and related tooling to support writing Kubernetes Operators in Java. Both the Java
+Operator SDK and Spark’s native
+kubernetes integration itself is using the Fabric8 Kubernetes Client to interact with the
+Kubernetes API Server.
+
+## State Transition
+
+[<img src="resources/state.png">](resources/state.png)
+
+* Spark application are expected to run from submitted to succeeded before releasing resources
+* User may configure the app CR to time-out after given threshold of time
+* In addition, user may configure the app CR to skip releasing resources after terminated. This is
+ typically used at dev phase: pods / configmaps. etc would be kept for debugging. They have
Review Comment:
` pods / configmaps. etc `? Although the meaning is clear, could you revise this grammatically?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762479753
##########
docs/metrics_logging.md:
##########
@@ -0,0 +1,109 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Metrics
+
+Spark operator,
+following [Apache Spark](https://spark.apache.org/docs/latest/monitoring.html#metrics),
+has a configurable metrics system based on
+the [Dropwizard Metrics Library](https://metrics.dropwizard.io/4.2.25/). Note that Spark Operator
+does not have Spark UI, MetricsServlet
+and PrometheusServlet from org.apache.spark.metrics.sink package are not supported. If you are
+interested in Prometheus metrics exporting, please take a look at below section `Forward Metrics to Prometheus`
+
+## JVM Metrics
+
+Spark Operator collects JVM metrics
+via [Codahale JVM Metrics](https://javadoc.io/doc/com.codahale.metrics/metrics-jvm/latest/index.html)
+
+- BufferPoolMetricSet
+- FileDescriptorRatioGauge
+- GarbageCollectorMetricSet
+- MemoryUsageGaugeSet
+- ThreadStatesGaugeSet
+
+## Kubernetes Client Metrics
+
+| Metrics Name | Type | Description |
+|-----------------------------------------------------------|------------|--------------------------------------------------------------------------------------------------------------------------|
+| kubernetes.client.http.request | Meter | Tracking the rates of HTTP request sent to the Kubernetes API Server |
+| kubernetes.client.http.response | Meter | Tracking the rates of HTTP response from the Kubernetes API Server |
+| kubernetes.client.http.response.failed | Meter | Tracking the rates of HTTP requests which have no response from the Kubernetes API Server |
+| kubernetes.client.http.response.latency.nanos | Histograms | Measures the statistical distribution of HTTP response latency from the Kubernetes API Server |
+| kubernetes.client.http.response.<ResponseCode> | Meter | Tracking the rates of HTTP response based on response code from the Kubernetes API Server |
+| kubernetes.client.http.request.<RequestMethod> | Meter | Tracking the rates of HTTP request based type of method to the Kubernetes API Server |
+| kubernetes.client.http.response.1xx | Meter | Tracking the rates of HTTP Code 1xx responses (informational) received from the Kubernetes API Server per response code. |
+| kubernetes.client.http.response.2xx | Meter | Tracking the rates of HTTP Code 2xx responses (success) received from the Kubernetes API Server per response code. |
+| kubernetes.client.http.response.3xx | Meter | Tracking the rates of HTTP Code 3xx responses (redirection) received from the Kubernetes API Server per response code. |
+| kubernetes.client.http.response.4xx | Meter | Tracking the rates of HTTP Code 4xx responses (client error) received from the Kubernetes API Server per response code. |
+| kubernetes.client.http.response.5xx | Meter | Tracking the rates of HTTP Code 5xx responses (server error) received from the Kubernetes API Server per response code. |
+| kubernetes.client.<ResourceName>.<Method> | Meter | Tracking the rates of HTTP request for a combination of one Kubernetes resource and one http method |
+| kubernetes.client.<NamespaceName>.<ResourceName>.<Method> | Meter | Tracking the rates of HTTP request for a combination of one namespace-scoped Kubernetes resource and one http method |
+
+## Forward Metrics to Prometheus
+
+In this section, we will show you how to forward spark operator metrics
+to [Prometheus](https://prometheus.io).
+
+* Modify the
+ build-tools/helm/spark-kubernetes-operator/values.yaml file' s metrics properties section:
+
+```properties
+metrics.properties:|+
+ spark.metrics.conf.operator.sink.prometheus.class=org.apache.spark.kubernetes.operator.metrics.
+sink.PrometheusPullModelSink
+```
+
+* Install the Spark Operator
Review Comment:
```suggestion
* Install Spark Operator
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1761650851
##########
build-tools/docs-utils/build.gradle:
##########
@@ -38,3 +38,5 @@ tasks.register('generateConfPropsDoc', Exec) {
description = "Generate config properties doc for operator"
commandLine "java", "-classpath", sourceSets.main.runtimeClasspath.getAsPath(), javaMainClass, docsPath
}
+
+build.finalizedBy(generateConfPropsDoc)
Review Comment:
This ensures the generated doc is updated per `gradle build` when new conf is introduced, if any
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762255803
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
+* Operator launches driver and observes its status
+* Operator observes driver-spawn resources (e.g. executors) till app terminates
+* Operator releases all Spark-app owned resources to cluster
+* The SparkApplication CR can be (re)applied on the cluster any time - e.g. to issue proactive
+ termination of an application. The Operator makes continuous adjustments to imitate the
+ desired state until the
+ current state becomes the desired state. All lifecycle management operations are realized
+ using this very simple
+ principle in the Operator.
+
+The Operator is built with the Java Operator SDK and uses the Native Kubernetes Integration for
+launching Spark deployments and submitting jobs under the hood. The Java Operator SDK is a
+higher level
Review Comment:
New line after two words (`higher level`) looks a little weird to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762260209
##########
docs/configuration.md:
##########
@@ -0,0 +1,59 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Configuration
+
+## Configure Operator
+
+Spark Operator supports different ways to configure the behavior:
+
+* **spark-operator.properties** provided when deploying the operator. In addition to the
+ [property file](../build-tools/helm/spark-kubernetes-operator/conf/spark-operator.
+ properties), it is also possible to override or append config properties in helm [Values
+ files](../build-tools/helm/spark-kubernetes-operator/values.yaml).
+* **System Properties** : when provided as system properties (e.g. via -D options to the
+ operator JVM), it overrides the values provided in property file.
+* **Hot property loading** : when enabled, a
+ [configmap](https://kubernetes.io/docs/concepts/configuration/configmap/) would be created with
+ the operator in the same namespace. Operator can monitor updates performed on the configmap. Hot
+ properties reloading takes higher precedence comparing with default properties override.
+ - An example use case: operator use hot properties to figure the list of namespace(s) to
+ operate Spark applications. The hot properties config map can be updated and
+ maintained by user or additional microservice to tune the operator behavior without
+ rebooting it.
+ - Please be advised that not all properties can be hot-loaded and honored at runtime.
+ Refer the list of [supported properties](./config_properties.md) for more details.
+
+To enable hot properties loading, update the **helm chart values file** with
+
+```
+
Review Comment:
Redundant empty line.
##########
docs/configuration.md:
##########
@@ -0,0 +1,59 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Configuration
+
+## Configure Operator
+
+Spark Operator supports different ways to configure the behavior:
+
+* **spark-operator.properties** provided when deploying the operator. In addition to the
+ [property file](../build-tools/helm/spark-kubernetes-operator/conf/spark-operator.
+ properties), it is also possible to override or append config properties in helm [Values
+ files](../build-tools/helm/spark-kubernetes-operator/values.yaml).
+* **System Properties** : when provided as system properties (e.g. via -D options to the
+ operator JVM), it overrides the values provided in property file.
+* **Hot property loading** : when enabled, a
+ [configmap](https://kubernetes.io/docs/concepts/configuration/configmap/) would be created with
+ the operator in the same namespace. Operator can monitor updates performed on the configmap. Hot
+ properties reloading takes higher precedence comparing with default properties override.
+ - An example use case: operator use hot properties to figure the list of namespace(s) to
+ operate Spark applications. The hot properties config map can be updated and
+ maintained by user or additional microservice to tune the operator behavior without
+ rebooting it.
+ - Please be advised that not all properties can be hot-loaded and honored at runtime.
+ Refer the list of [supported properties](./config_properties.md) for more details.
+
+To enable hot properties loading, update the **helm chart values file** with
+
+```
+
+operatorConfiguration:
+ spark-operator.properties: |+
+ spark.operator.dynamic.config.enabled=true
+ # ... all other config overides...
+ dynamicConfig:
+ create: true
+
Review Comment:
Redundant empty line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762254410
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
+* Operator launches driver and observes its status
+* Operator observes driver-spawn resources (e.g. executors) till app terminates
+* Operator releases all Spark-app owned resources to cluster
+* The SparkApplication CR can be (re)applied on the cluster any time - e.g. to issue proactive
Review Comment:
`CR` -> `CRD`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762254307
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
Review Comment:
Let's be clear that this is one of two cases: `SparkApplication` and `SparkCluster` .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add docs for operator [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#issuecomment-2329648122
Gentle ping, @jiangzho .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add docs for operator [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1739905044
##########
spark-operator-docs/build.gradle:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+ext {
+ javaMainClass = "org.apache.spark.k8s.operator.utils.ConfOptionDocGenerator"
+ docsPath = System.getProperty("user.dir") + "/docs"
+}
+
+dependencies {
+ implementation project(":spark-operator")
+ implementation("org.projectlombok:lombok:$lombokVersion")
Review Comment:
To @jiangzho , it seems that your repository is outdated a little.
Apache Spark Kubernetes Operator follows the Gradle Version Catalog. Please rebase your repository and refer the following commit.
- https://github.com/apache/spark-kubernetes-operator/pull/109
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1744837509
##########
docs/operations.md:
##########
@@ -0,0 +1,132 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+### Compatibility
+
+- JDK17
Review Comment:
Java 17 and 21.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#issuecomment-2347427819
Thank you for updating this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762258810
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
+* Operator launches driver and observes its status
+* Operator observes driver-spawn resources (e.g. executors) till app terminates
+* Operator releases all Spark-app owned resources to cluster
+* The SparkApplication CR can be (re)applied on the cluster any time - e.g. to issue proactive
+ termination of an application. The Operator makes continuous adjustments to imitate the
+ desired state until the
+ current state becomes the desired state. All lifecycle management operations are realized
+ using this very simple
+ principle in the Operator.
+
+The Operator is built with the Java Operator SDK and uses the Native Kubernetes Integration for
+launching Spark deployments and submitting jobs under the hood. The Java Operator SDK is a
+higher level
+framework and related tooling to support writing Kubernetes Operators in Java. Both the Java
+Operator SDK and Spark’s native
+kubernetes integration itself is using the Fabric8 Kubernetes Client to interact with the
+Kubernetes API Server.
+
+## State Transition
+
+[<img src="resources/state.png">](resources/state.png)
+
+* Spark application are expected to run from submitted to succeeded before releasing resources
+* User may configure the app CR to time-out after given threshold of time
Review Comment:
Is this in the diagram?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762265952
##########
docs/spark_custom_resources.md:
##########
@@ -0,0 +1,203 @@
+## Spark Operator API
+
+The core user facing API of the Spark Kubernetes Operator is the `SparkApplication` and
+`SparkCluster` Custom Resources Definition (CRD). Spark custom resource extends
+standard k8s API, defines Spark Application spec and tracks status.
+
+Once the Spark Operator is installed and running in your Kubernetes environment, it will
+continuously watch SparkApplication(s) and SparkCluster(s) submitted, via k8s API client or
+kubectl by the user, orchestrate secondary resources (pods, configmaps .etc).
+
+Please check out the [quickstart](../README.md) as well for installing operator.
+
+## SparkApplication
+
+SparkApplication can be defined in YAML format and with bare minimal required fields in
+order to start:
+
+```
+apiVersion: spark.apache.org/v1alpha1
+kind: SparkApplication
+metadata:
+ name: pi
+spec:
+ mainClass: "org.apache.spark.examples.SparkPi"
+ jars: "local:///opt/spark/examples/jars/spark-examples_2.13-4.0.0-preview1.jar"
+ sparkConf:
+ spark.executor.instances: "5"
+ applicationTolerations:
+ resourceRetainPolicy: OnFailure
+ runtimeVersions:
+ scalaVersion: "2.13"
+ sparkVersion: "4.0.0-preview1"
+
Review Comment:
Redundant empty link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762262077
##########
docs/metrics_logging.md:
##########
@@ -0,0 +1,109 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Metrics
Review Comment:
Why do we have a new file for this? I'd recommend to include this content into `configuration.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762264951
##########
docs/operator_probes.md:
##########
@@ -0,0 +1,82 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Operator Probes
Review Comment:
Please remove this section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762260832
##########
docs/configuration.md:
##########
@@ -0,0 +1,59 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Configuration
+
+## Configure Operator
+
+Spark Operator supports different ways to configure the behavior:
+
+* **spark-operator.properties** provided when deploying the operator. In addition to the
+ [property file](../build-tools/helm/spark-kubernetes-operator/conf/spark-operator.
+ properties), it is also possible to override or append config properties in helm [Values
+ files](../build-tools/helm/spark-kubernetes-operator/values.yaml).
+* **System Properties** : when provided as system properties (e.g. via -D options to the
+ operator JVM), it overrides the values provided in property file.
+* **Hot property loading** : when enabled, a
+ [configmap](https://kubernetes.io/docs/concepts/configuration/configmap/) would be created with
+ the operator in the same namespace. Operator can monitor updates performed on the configmap. Hot
+ properties reloading takes higher precedence comparing with default properties override.
+ - An example use case: operator use hot properties to figure the list of namespace(s) to
+ operate Spark applications. The hot properties config map can be updated and
+ maintained by user or additional microservice to tune the operator behavior without
+ rebooting it.
+ - Please be advised that not all properties can be hot-loaded and honored at runtime.
+ Refer the list of [supported properties](./config_properties.md) for more details.
+
+To enable hot properties loading, update the **helm chart values file** with
+
+```
+
+operatorConfiguration:
+ spark-operator.properties: |+
+ spark.operator.dynamic.config.enabled=true
+ # ... all other config overides...
+ dynamicConfig:
+ create: true
+
+```
+
+## Config Metrics Publishing Behavior
+
+Spark Operator uses the same source & sink interface as Apache Spark. You may
Review Comment:
`source & sink` -> `source and sink`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762257492
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
+* Operator launches driver and observes its status
+* Operator observes driver-spawn resources (e.g. executors) till app terminates
+* Operator releases all Spark-app owned resources to cluster
+* The SparkApplication CR can be (re)applied on the cluster any time - e.g. to issue proactive
+ termination of an application. The Operator makes continuous adjustments to imitate the
+ desired state until the
+ current state becomes the desired state. All lifecycle management operations are realized
+ using this very simple
+ principle in the Operator.
+
+The Operator is built with the Java Operator SDK and uses the Native Kubernetes Integration for
+launching Spark deployments and submitting jobs under the hood. The Java Operator SDK is a
+higher level
+framework and related tooling to support writing Kubernetes Operators in Java. Both the Java
+Operator SDK and Spark’s native
+kubernetes integration itself is using the Fabric8 Kubernetes Client to interact with the
+Kubernetes API Server.
+
+## State Transition
Review Comment:
Please add `SparkCluster` transition state diagram.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762254921
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
+* Operator launches driver and observes its status
+* Operator observes driver-spawn resources (e.g. executors) till app terminates
+* Operator releases all Spark-app owned resources to cluster
+* The SparkApplication CR can be (re)applied on the cluster any time - e.g. to issue proactive
+ termination of an application. The Operator makes continuous adjustments to imitate the
+ desired state until the
+ current state becomes the desired state. All lifecycle management operations are realized
+ using this very simple
+ principle in the Operator.
Review Comment:
Please remove `All lifecycle management operations are realized using this very simple principle in the Operator.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762254410
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
+* Operator launches driver and observes its status
+* Operator observes driver-spawn resources (e.g. executors) till app terminates
+* Operator releases all Spark-app owned resources to cluster
+* The SparkApplication CR can be (re)applied on the cluster any time - e.g. to issue proactive
Review Comment:
`CR` -> `CRD`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762251259
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
Review Comment:
This was correct, but not as of now because we add `SparkCluster` CRD.
I guess we need to revise `README.md`, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1746362479
##########
settings.gradle:
##########
@@ -20,3 +20,4 @@ rootProject.name = 'apache-spark-kubernetes-operator'
include 'spark-operator-api'
include 'spark-submission-worker'
include 'spark-operator'
+include 'spark-operator-docs'
Review Comment:
Refactored to SPARK-49527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1757677158
##########
docs/operations.md:
##########
@@ -0,0 +1,132 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+### Compatibility
+
+- JDK17
+- Operator used fabric8 which assumes to be compatible with available k8s versions. However for using status subresource, please use k8s version 1.14 or above.
+- Spark versions 3.4 or above
+
+## Manage Your Spark Operator
+
+The operator installation is managed by a helm chart. To install run:
+
+```
+helm install spark-kubernetes-operator \
+ -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ build-tools/helm/spark-kubernetes-operator/
+```
+
+Alternatively to install the operator (and also the helm chart) to a specific namespace:
+
+```
+helm install spark-kubernetes-operator \
+ -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ build-tools/helm/spark-kubernetes-operator/ \
+ --namespace spark-system --create-namespace
+```
+
+Note that in this case you will need to update the namespace in the examples accordingly.
+
+### Spark Application Namespaces
+
+By default, Spark applications are created in the same namespace as the operator deployment.
+You many also configure the chart deployment to add necessary RBAC resources for
+applications to enable them running in additional namespaces.
+
+## Overriding configuration parameters during Helm install
+
+Helm provides different ways to override the default installation parameters (contained
+in `values.yaml`) for the Helm chart.
+
+To override single parameters you can use `--set`, for example:
+
+```
+helm install --set image.repository=<my_registory>/spark-kubernetes-operator \
+ -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ build-tools/helm/spark-kubernetes-operator/
+```
+
+You can also provide multiple custom values file by using the `-f` flag, the latest takes
+higher precedence:
+
+```
+helm install spark-kubernetes-operator \
+ -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ -f my_values.yaml \
+ build-tools/helm/spark-kubernetes-operator/
+```
+
+The configurable parameters of the Helm chart and which default values as detailed in the
+following table:
+
+| Parameters | Description | Default value |
+|----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
+| image.repository | The image repository of spark-kubernetes-operator. | spark-kubernetes-operator |
+| image.pullPolicy | The image pull policy of spark-kubernetes-operator. | IfNotPresent |
+| image.tag | The image tag of spark-kubernetes-operator. | |
+| image.digest | The image digest of spark-kubernetes-operator. If set then it takes precedence and the image tag will be ignored. | |
+| imagePullSecrets | The image pull secrets of spark-kubernetes-operator. | |
+| operatorDeployment.replica | Operator replica count. Must be 1 unless leader election is configured. | 1 |
+| operatorDeployment.strategy.type | Operator pod upgrade strategy. Must be Recreate unless leader election is configured. | Recreate |
+| operatorDeployment.operatorPod.annotations | Custom annotations to be added to the operator pod | |
+| operatorDeployment.operatorPod.labels | Custom labels to be added to the operator pod | |
+| operatorDeployment.operatorPod.nodeSelector | Custom nodeSelector to be added to the operator pod. | |
+| operatorDeployment.operatorPod.topologySpreadConstraints | Custom topologySpreadConstraints to be added to the operator pod. | |
+| operatorDeployment.operatorPod.dnsConfig | DNS configuration to be used by the operator pod. | |
+| operatorDeployment.operatorPod.volumes | Additional volumes to be added to the operator pod. | |
+| operatorDeployment.operatorPod.priorityClassName | Priority class name to be used for the operator pod | |
+| operatorDeployment.operatorPod.securityContext | Security context overrides for the operator pod | |
+| operatorDeployment.operatorContainer.jvmArgs | JVM arg override for the operator container. | `-XX:+UseG1GC -Xms3G -Xmx3G -Dfile.encoding=UTF8` |
+| operatorDeployment.operatorContainer.env | Custom env to be added to the operator container. | |
+| operatorDeployment.operatorContainer.envFrom | Custom envFrom to be added to the operator container, e.g. for downward API. | |
+| operatorDeployment.operatorContainer.probes | Probe config for the operator container. | |
+| operatorDeployment.operatorContainer.securityContext | Security context overrides for the operator container. | run as non root for baseline secuirty standard compliance |
+| operatorDeployment.operatorContainer.resources | Resources for the operator container. | memory 4Gi, ephemeral storage 2Gi and 1 cpu |
+| operatorDeployment.additionalContainers | Additional containers to be added to the operator pod, e.g. sidecar. | |
+| operatorRbac.serviceAccount.create | Whether to create service account for operator to use. | |
+| operatorRbac.clusterRole.create | Whether to create ClusterRole for operator to use. | true |
+| operatorRbac.clusterRoleBinding.create | Whether to create ClusterRoleBinding for operator to use. | true |
+| operatorRbac.role.create | Whether to create Role for operator to use. At least one of `clusterRole.create` or `role.create` should be enabled | true |
+| operatorRbac.roleBinding.create | Whether to create RoleBinding for operator to use. At least one of `clusterRoleBinding.create` or `roleBinding.create` should be enabled | true |
+| operatorRbac.clusterRole.configManagement.roleName | Role name for operator configuration management (hot property loading and leader election) | `spark-operator-config-role` |
+| appResources.namespaces.create | Whether to create dedicated namespaces for Spark apps. | `spark-operator-config-role-binding` |
Review Comment:
Actually +1 for the point - `appResources` can be a bit misleading, since it may serve both `SparkApp` and `SparkCluster`. It was introduced to indicate this is for running Spark workload (comparing other resources created for operator deployment itself).
I shall fix this by SPARK-49623
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1744839631
##########
docs/operations.md:
##########
@@ -0,0 +1,132 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+### Compatibility
+
+- JDK17
+- Operator used fabric8 which assumes to be compatible with available k8s versions. However for using status subresource, please use k8s version 1.14 or above.
Review Comment:
As I pinged you already, the K8s eco-system is moving faster in the public environment.
- https://github.com/apache/spark/pull/47990
- https://github.com/apache/spark-kubernetes-operator/pull/117
Just FYI, in the community, please don't claim which you didn't test at all. AFAIK, nobody tests K8s 1.14 in these days.
##########
docs/operations.md:
##########
@@ -0,0 +1,132 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+### Compatibility
+
+- JDK17
+- Operator used fabric8 which assumes to be compatible with available k8s versions. However for using status subresource, please use k8s version 1.14 or above.
Review Comment:
As I pinged you already, the K8s eco-system is moving faster in the public environment.
- https://github.com/apache/spark/pull/47990
- https://github.com/apache/spark-kubernetes-operator/pull/117
Just FYI, in the community, please don't claim which you didn't test at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1744847803
##########
settings.gradle:
##########
@@ -20,3 +20,4 @@ rootProject.name = 'apache-spark-kubernetes-operator'
include 'spark-operator-api'
include 'spark-submission-worker'
include 'spark-operator'
+include 'spark-operator-docs'
Review Comment:
Shall we move this into `build-tools` directory? In addition, `spark-operator-docs` sounds like a little overclaim because this has only `ConfOptionDocGenerator` while the documentations has more contents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1746362098
##########
docs/operations.md:
##########
@@ -0,0 +1,132 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+### Compatibility
+
+- JDK17
+- Operator used fabric8 which assumes to be compatible with available k8s versions. However for using status subresource, please use k8s version 1.14 or above.
+- Spark versions 3.4 or above
+
+## Manage Your Spark Operator
+
+The operator installation is managed by a helm chart. To install run:
+
+```
+helm install spark-kubernetes-operator \
+ -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ build-tools/helm/spark-kubernetes-operator/
+```
+
+Alternatively to install the operator (and also the helm chart) to a specific namespace:
+
+```
+helm install spark-kubernetes-operator \
+ -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ build-tools/helm/spark-kubernetes-operator/ \
+ --namespace spark-system --create-namespace
+```
+
+Note that in this case you will need to update the namespace in the examples accordingly.
+
+### Spark Application Namespaces
+
+By default, Spark applications are created in the same namespace as the operator deployment.
+You many also configure the chart deployment to add necessary RBAC resources for
+applications to enable them running in additional namespaces.
+
+## Overriding configuration parameters during Helm install
+
+Helm provides different ways to override the default installation parameters (contained
+in `values.yaml`) for the Helm chart.
+
+To override single parameters you can use `--set`, for example:
+
+```
+helm install --set image.repository=<my_registory>/spark-kubernetes-operator \
+ -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ build-tools/helm/spark-kubernetes-operator/
+```
+
+You can also provide multiple custom values file by using the `-f` flag, the latest takes
+higher precedence:
+
+```
+helm install spark-kubernetes-operator \
+ -f build-tools/helm/spark-kubernetes-operator/values.yaml \
+ -f my_values.yaml \
+ build-tools/helm/spark-kubernetes-operator/
+```
+
+The configurable parameters of the Helm chart and which default values as detailed in the
+following table:
+
+| Parameters | Description | Default value |
+|----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
+| image.repository | The image repository of spark-kubernetes-operator. | spark-kubernetes-operator |
+| image.pullPolicy | The image pull policy of spark-kubernetes-operator. | IfNotPresent |
+| image.tag | The image tag of spark-kubernetes-operator. | |
+| image.digest | The image digest of spark-kubernetes-operator. If set then it takes precedence and the image tag will be ignored. | |
+| imagePullSecrets | The image pull secrets of spark-kubernetes-operator. | |
+| operatorDeployment.replica | Operator replica count. Must be 1 unless leader election is configured. | 1 |
+| operatorDeployment.strategy.type | Operator pod upgrade strategy. Must be Recreate unless leader election is configured. | Recreate |
+| operatorDeployment.operatorPod.annotations | Custom annotations to be added to the operator pod | |
+| operatorDeployment.operatorPod.labels | Custom labels to be added to the operator pod | |
+| operatorDeployment.operatorPod.nodeSelector | Custom nodeSelector to be added to the operator pod. | |
+| operatorDeployment.operatorPod.topologySpreadConstraints | Custom topologySpreadConstraints to be added to the operator pod. | |
+| operatorDeployment.operatorPod.dnsConfig | DNS configuration to be used by the operator pod. | |
+| operatorDeployment.operatorPod.volumes | Additional volumes to be added to the operator pod. | |
+| operatorDeployment.operatorPod.priorityClassName | Priority class name to be used for the operator pod | |
+| operatorDeployment.operatorPod.securityContext | Security context overrides for the operator pod | |
+| operatorDeployment.operatorContainer.jvmArgs | JVM arg override for the operator container. | `-XX:+UseG1GC -Xms3G -Xmx3G -Dfile.encoding=UTF8` |
+| operatorDeployment.operatorContainer.env | Custom env to be added to the operator container. | |
+| operatorDeployment.operatorContainer.envFrom | Custom envFrom to be added to the operator container, e.g. for downward API. | |
+| operatorDeployment.operatorContainer.probes | Probe config for the operator container. | |
+| operatorDeployment.operatorContainer.securityContext | Security context overrides for the operator container. | run as non root for baseline secuirty standard compliance |
+| operatorDeployment.operatorContainer.resources | Resources for the operator container. | memory 4Gi, ephemeral storage 2Gi and 1 cpu |
+| operatorDeployment.additionalContainers | Additional containers to be added to the operator pod, e.g. sidecar. | |
+| operatorRbac.serviceAccount.create | Whether to create service account for operator to use. | |
+| operatorRbac.clusterRole.create | Whether to create ClusterRole for operator to use. | true |
+| operatorRbac.clusterRoleBinding.create | Whether to create ClusterRoleBinding for operator to use. | true |
+| operatorRbac.role.create | Whether to create Role for operator to use. At least one of `clusterRole.create` or `role.create` should be enabled | true |
+| operatorRbac.roleBinding.create | Whether to create RoleBinding for operator to use. At least one of `clusterRoleBinding.create` or `roleBinding.create` should be enabled | true |
+| operatorRbac.clusterRole.configManagement.roleName | Role name for operator configuration management (hot property loading and leader election) | `spark-operator-config-role` |
+| appResources.namespaces.create | Whether to create dedicated namespaces for Spark apps. | `spark-operator-config-role-binding` |
Review Comment:
Yep! Add a short field in Spark Custom Resources page to start with. Also created SPARK-49528 to better doc the template support for clusters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762253913
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
Review Comment:
Please try to rewrite the above paragraph into a single sentence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1762252867
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
Review Comment:
Let's remove CRD details. It distracts Apache Spark K8s Operator explanation. We had better link to K8s CRD document.
> For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
used, and the registration process sometimes takes a few seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1779322115
##########
docs/architecture.md:
##########
@@ -0,0 +1,64 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Design & Architecture
+
+**Spark-Kubernetes-Operator** (Operator) acts as a control plane to manage the complete
+deployment lifecycle of Spark applications. The Operator can be installed on a Kubernetes
+cluster using Helm. In most production environments it is typically deployed in a designated
+namespace and controls Spark deployments in one or more managed namespaces. The custom resource
+definition (CRD) that describes the schema of a SparkApplication is a cluster wide resource.
+For a CRD, the declaration must be registered before any resources of that CRDs kind(s) can be
+used, and the registration process sometimes takes a few seconds.
+
+Users can interact with the operator using the kubectl or k8s API. The Operator continuously
+tracks cluster events relating to the SparkApplication custom resources. When the operator
+receives a new resource update, it will take action to adjust the Kubernetes cluster to the
+desired state as part of its reconciliation loop. The initial loop consists of the following
+high-level steps:
+
+* User submits a SparkApplication custom resource(CR) using kubectl / API
+* Operator launches driver and observes its status
+* Operator observes driver-spawn resources (e.g. executors) till app terminates
+* Operator releases all Spark-app owned resources to cluster
+* The SparkApplication CR can be (re)applied on the cluster any time - e.g. to issue proactive
+ termination of an application. The Operator makes continuous adjustments to imitate the
+ desired state until the
+ current state becomes the desired state. All lifecycle management operations are realized
+ using this very simple
+ principle in the Operator.
+
+The Operator is built with the Java Operator SDK and uses the Native Kubernetes Integration for
+launching Spark deployments and submitting jobs under the hood. The Java Operator SDK is a
+higher level
+framework and related tooling to support writing Kubernetes Operators in Java. Both the Java
+Operator SDK and Spark’s native
+kubernetes integration itself is using the Fabric8 Kubernetes Client to interact with the
+Kubernetes API Server.
+
+## State Transition
+
+[<img src="resources/state.png">](resources/state.png)
Review Comment:
Diagrams are created with [draw.io](https://github.com/jgraph/drawio) which allows import and update png for simple diagrams. Would you suggest we add this in docs as well ? It's not user facing but may help future works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49464] Add documentations [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #113:
URL: https://github.com/apache/spark-kubernetes-operator/pull/113#discussion_r1779322642
##########
docs/spark_custom_resources.md:
##########
@@ -0,0 +1,203 @@
+## Spark Operator API
Review Comment:
Thanks for the catch!
We have disabled license header check for markdowns, but I added this back for files under `/docs` for consistency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org