You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiangzho (via GitHub)" <gi...@apache.org> on 2024/07/24 00:22:28 UTC
[PR] [SPARK-48984]Add Controller Metrics System and Utils [spark-kubernetes-operator]
jiangzho opened a new pull request, #23:
URL: https://github.com/apache/spark-kubernetes-operator/pull/23
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, use `Draft` feature of GitHub Action.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If there is design documentation, please add the link.
3. If there is a discussion in the mailing list, please add the link.
-->
This is a breakdown PR of #12 - defines metrics system and utils classes to be used by the reconcilers.
It also refactors a few methods in previous utils class `org.apache.spark.k8s.operator.reconciler.SparkReconcilerUtils` into common utils package.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Breakdown PRs help us to move with more flexibility.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
CIs
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48984]Add Controller Metrics System and Utils [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #23: [SPARK-48984]Add Controller Metrics System and Utils
URL: https://github.com/apache/spark-kubernetes-operator/pull/23
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48984]Add Controller Metrics System and Utils [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #23:
URL: https://github.com/apache/spark-kubernetes-operator/pull/23#discussion_r1690622477
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/source/OperatorJosdkMetrics.java:
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.metrics.source;
+
+import static io.javaoperatorsdk.operator.api.reconciler.Constants.CONTROLLER_NAME;
+
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+
+import com.codahale.metrics.Counter;
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.Histogram;
+import com.codahale.metrics.MetricRegistry;
+import io.fabric8.kubernetes.api.model.HasMetadata;
+import io.javaoperatorsdk.operator.api.monitoring.Metrics;
+import io.javaoperatorsdk.operator.api.reconciler.Constants;
+import io.javaoperatorsdk.operator.api.reconciler.RetryInfo;
+import io.javaoperatorsdk.operator.processing.Controller;
+import io.javaoperatorsdk.operator.processing.GroupVersionKind;
+import io.javaoperatorsdk.operator.processing.event.Event;
+import io.javaoperatorsdk.operator.processing.event.ResourceID;
+import io.javaoperatorsdk.operator.processing.event.source.controller.ResourceAction;
+import io.javaoperatorsdk.operator.processing.event.source.controller.ResourceEvent;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.BaseResource;
+import org.apache.spark.k8s.operator.SparkApplication;
+import org.apache.spark.metrics.source.Source;
+import org.apache.spark.util.Clock;
+import org.apache.spark.util.SystemClock;
+
+@Slf4j
+public class OperatorJosdkMetrics implements Source, Metrics {
+ public static final String FINISHED = "finished";
+ public static final String CLEANUP = "cleanup";
+ public static final String FAILED = "failed";
+ public static final String RETRIES = "retries";
+ private final Map<String, Histogram> histograms = new ConcurrentHashMap<>();
+ private final Map<String, Counter> counters = new ConcurrentHashMap<>();
+ private final Map<String, Gauge<?>> gauges = new ConcurrentHashMap<>();
+ private static final String RECONCILIATION = "reconciliation";
+ private static final String RESOURCE = "resource";
+ private static final String EVENT = "event";
+ private static final String SUCCESS = "success";
+ private static final String FAILURE = "failure";
+ private static final String EXCEPTION = "exception";
+ private static final String PREFIX = "operator.sdk";
+ private static final String RECONCILIATIONS = "reconciliations";
+ private static final String RECONCILIATIONS_EXECUTIONS = RECONCILIATIONS + ".executions";
+ private static final String RECONCILIATIONS_QUEUE_SIZE = RECONCILIATIONS + ".queue.size";
+ private static final String SIZE = "size";
+
+ private final Clock clock;
+ private final MetricRegistry metricRegistry;
+
+ public OperatorJosdkMetrics() {
+ this.clock = new SystemClock();
+ this.metricRegistry = new MetricRegistry();
+ }
+
+ @Override
+ public String sourceName() {
+ return PREFIX;
+ }
+
+ @Override
+ public MetricRegistry metricRegistry() {
+ return metricRegistry;
+ }
+
+ @Override
+ public void controllerRegistered(Controller<? extends HasMetadata> controller) {
+ // no-op
+ log.debug("Controller has been registered");
+ }
+
+ @Override
+ public void receivedEvent(Event event, Map<String, Object> metadata) {
+ log.debug("received event {}, metadata {}", event, metadata);
+ if (event instanceof ResourceEvent) {
+ final ResourceAction action = ((ResourceEvent) event).getAction();
+ final Optional<Class<? extends BaseResource<?, ?, ?, ?, ?>>> resource =
+ getResourceClass(metadata);
+ final Optional<String> namespaceOptional = event.getRelatedCustomResourceID().getNamespace();
+ resource.ifPresent(
+ aClass -> getCounter(aClass, action.name().toLowerCase(), RESOURCE, EVENT).inc());
+ if (resource.isPresent() && namespaceOptional.isPresent()) {
+ getCounter(
+ resource.get(),
+ namespaceOptional.get(),
+ action.name().toLowerCase(),
+ RESOURCE,
+ EVENT)
+ .inc();
+ }
+ }
+ }
+
+ @Override
+ public <T> T timeControllerExecution(ControllerExecution<T> execution) throws Exception {
+ log.debug("Time controller execution");
+ final String name = execution.controllerName();
+ final ResourceID resourceID = execution.resourceID();
+ final Optional<String> namespaceOptional = resourceID.getNamespace();
+ final Map<String, Object> metadata = execution.metadata();
+ final Optional<Class<? extends BaseResource<?, ?, ?, ?, ?>>> resourceClass =
+ getResourceClass(metadata);
+ final String execName = execution.name();
+
+ long startTime = clock.getTimeMillis();
+ try {
+ T result = execution.execute();
+ final String successType = execution.successTypeName(result);
+ if (resourceClass.isPresent()) {
+ getHistogram(resourceClass.get(), name, execName, successType).update(toSeconds(startTime));
+ getCounter(resourceClass.get(), name, execName, SUCCESS, successType).inc();
+ if (namespaceOptional.isPresent()) {
+ getHistogram(resourceClass.get(), namespaceOptional.get(), name, execName, successType)
+ .update(toSeconds(startTime));
+ getCounter(
+ resourceClass.get(),
+ namespaceOptional.get(),
+ name,
+ execName,
+ SUCCESS,
+ successType)
+ .inc();
+ }
+ }
+ return result;
+ } catch (Exception e) {
Review Comment:
This one was not narrowed down for metrics purpose only (so that it would be captured by the metrics). The exception is thrown at the end to ensure it's handled. There's a unit test added for this purpose as well.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/MetricsSystem.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.metrics;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.codahale.metrics.MetricFilter;
+import com.codahale.metrics.MetricRegistry;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.metrics.source.OperatorJvmSource;
+import org.apache.spark.metrics.sink.Sink;
+import org.apache.spark.metrics.source.Source;
+
+@Slf4j
+public class MetricsSystem {
+ private final AtomicBoolean running = new AtomicBoolean(false);
+ @Getter private final Set<Sink> sinks;
+ @Getter private final Set<Source> sources;
+ @Getter private final MetricRegistry registry;
+ @Getter private final Properties properties;
+ // PrometheusPullModelHandler is registered by default, metrics exposed via http port
+ @Getter private final PrometheusPullModelHandler prometheusPullModelHandler;
+ private final Map<String, SinkProps> sinkPropertiesMap;
+
+ public MetricsSystem() {
+ this(new Properties());
+ }
+
+ public MetricsSystem(Properties properties) {
+ this.sources = new HashSet<>();
+ this.sinks = new HashSet<>();
+ this.registry = new MetricRegistry();
+ this.properties = properties;
+ this.sinkPropertiesMap = MetricsSystemFactory.parseSinkProperties(this.properties);
+ // Add default sinks
+ this.prometheusPullModelHandler = new PrometheusPullModelHandler(new Properties(), registry);
+ this.sinks.add(prometheusPullModelHandler);
+ }
+
+ public void start() {
+ if (running.get()) {
+ throw new IllegalStateException(
+ "Attempting to start a MetricsSystem that is already running");
+ }
+ running.set(true);
+ registerDefaultSources();
+ registerSinks();
+ sinks.forEach(Sink::start);
+ }
+
+ public void stop() {
+ if (running.get()) {
+ sinks.forEach(Sink::stop);
+ registry.removeMatching(MetricFilter.ALL);
+ } else {
+ log.error("Stopping a MetricsSystem that is not running");
Review Comment:
As recommended by PMD, we enabled according gradle task to [enforce guard log statement](https://pmd.github.io/pmd/pmd_rules_java_bestpractices.html#guardlogstatement) when associate String creation and manipulation. I think this is not reported as no string mani
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48984]Add Controller Metrics System and Utils [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #23:
URL: https://github.com/apache/spark-kubernetes-operator/pull/23#discussion_r1690622804
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
+ .enableDynamicOverride(false)
+ .description(
+ "Max number of reconcile loops triggered within the rate limiter refresh "
+ + "period for each resource. Setting the limit <= 0 disables the "
+ + "limiter.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description("Initial interval(in seconds) of retries on unhandled controller errors.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
Review Comment:
A best effort estimate for putting some initial delay (to avoid noisy neighbor problem)
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
+ .enableDynamicOverride(false)
+ .description(
+ "Max number of reconcile loops triggered within the rate limiter refresh "
+ + "period for each resource. Setting the limit <= 0 disables the "
+ + "limiter.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description("Initial interval(in seconds) of retries on unhandled controller errors.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Double> RETRY_INTERVAL_MULTIPLIER =
+ ConfigOption.<Double>builder()
+ .key("spark.kubernetes.operator.retry.intervalMultiplier")
+ .enableDynamicOverride(false)
+ .description(
+ "Interval multiplier of retries on unhandled controller errors. Setting "
+ + "this to 1 for linear retry.")
Review Comment:
I don't think the underlying library has a sanity check for non positive values (as other rate limiting param). For this param in particular, naughty user would get max internal taking effect when setting this. https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/retry/GenericRetryExecution.java#L23 . which is reasonable
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
+ .enableDynamicOverride(false)
+ .description(
+ "Max number of reconcile loops triggered within the rate limiter refresh "
+ + "period for each resource. Setting the limit <= 0 disables the "
+ + "limiter.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description("Initial interval(in seconds) of retries on unhandled controller errors.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Double> RETRY_INTERVAL_MULTIPLIER =
+ ConfigOption.<Double>builder()
+ .key("spark.kubernetes.operator.retry.intervalMultiplier")
+ .enableDynamicOverride(false)
+ .description(
+ "Interval multiplier of retries on unhandled controller errors. Setting "
+ + "this to 1 for linear retry.")
+ .typeParameterClass(Double.class)
+ .defaultValue(1.5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Max interval(in seconds) of retries on unhandled controller errors. "
+ + "Set to non-positive for unlimited.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(-1)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_ATTEMPTS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Max attempts of retries on unhandled controller errors. Setting this to "
+ + "non-positive value means no retry.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Long> RETRY_ATTEMPT_AFTER_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.attemptAfterSeconds")
Review Comment:
`When initialIntervalSeconds and intervalMultiplier is used` - they are for reconciler behavior
`When attemptAfterSeconds is used` - this is for requests made to API server.
To avoid further confusion, lemme use another namespace for the latter.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
+ .enableDynamicOverride(false)
+ .description(
+ "Max number of reconcile loops triggered within the rate limiter refresh "
+ + "period for each resource. Setting the limit <= 0 disables the "
+ + "limiter.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description("Initial interval(in seconds) of retries on unhandled controller errors.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Double> RETRY_INTERVAL_MULTIPLIER =
+ ConfigOption.<Double>builder()
+ .key("spark.kubernetes.operator.retry.intervalMultiplier")
+ .enableDynamicOverride(false)
+ .description(
+ "Interval multiplier of retries on unhandled controller errors. Setting "
+ + "this to 1 for linear retry.")
+ .typeParameterClass(Double.class)
+ .defaultValue(1.5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Max interval(in seconds) of retries on unhandled controller errors. "
+ + "Set to non-positive for unlimited.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(-1)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_ATTEMPTS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Max attempts of retries on unhandled controller errors. Setting this to "
+ + "non-positive value means no retry.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Long> RETRY_ATTEMPT_AFTER_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.attemptAfterSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Default time (in seconds) to wait till next request. This would be used if "
+ + "server does not set Retry-After in response. Setting this to non-positive "
+ + "number means immediate retry.")
+ .typeParameterClass(Long.class)
+ .defaultValue(1L)
+ .build();
+
+ public static final ConfigOption<Long> STATUS_PATCH_MAX_RETRY =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.maxStatusPatchAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Maximal number of retry attempts of requests to k8s server for resource "
+ + "status update. This would be performed on top of k8s client "
+ + "spark.kubernetes.operator.retry.maxAttempts to overcome potential "
+ + "conflicting update on the same SparkApplication. This should be positive "
+ + "number.")
+ .typeParameterClass(Long.class)
+ .defaultValue(3L)
+ .build();
+
+ public static final ConfigOption<Long> SECONDARY_RESOURCE_CREATE_MAX_ATTEMPTS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.secondaryResourceCreateMaxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Maximal number of retry attempts of requesting secondary resource for Spark "
+ + "application. This would be performed on top of k8s client "
+ + "spark.kubernetes.operator.retry.maxAttempts to overcome potential "
+ + "conflicting reconcile on the same SparkApplication. This should be "
+ + "positive number")
+ .typeParameterClass(Long.class)
+ .defaultValue(3L)
+ .build();
+
+ public static final ConfigOption<Boolean> JOSDK_METRICS_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.josdkMetricsEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, the josdk metrics will be added in metrics source and "
+ + "configured for operator.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Boolean> KUBERNETES_CLIENT_METRICS_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.clientMetricsEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable KubernetesClient metrics for measuring the HTTP traffic to "
+ + "the Kubernetes API Server. Since the metrics is collected "
+ + "via Okhttp interceptors, can be disabled when opt in "
+ + "customized interceptors.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Boolean>
+ KUBERNETES_CLIENT_METRICS_GROUP_BY_RESPONSE_CODE_GROUP_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.clientMetricsGroupByResponseCodeEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, additional metrics group by http response code group(1xx, "
+ + "2xx, 3xx, 4xx, 5xx) received from API server will be added. Users "
+ + "can disable it when their monitoring system can combine lower level "
+ + "kubernetes.client.http.response.<3-digit-response-code> metrics.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Integer> OPERATOR_METRICS_PORT =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.metrics.port")
+ .enableDynamicOverride(false)
+ .description("The port used for checking metrics")
+ .typeParameterClass(Integer.class)
+ .defaultValue(19090)
+ .build();
+
+ public static final ConfigOption<Integer> OPERATOR_PROBE_PORT =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.probePort")
+ .enableDynamicOverride(false)
+ .description("The port used for health/readiness check probe status.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(19091)
+ .build();
+
+ public static final ConfigOption<Integer> SENTINEL_EXECUTOR_SERVICE_POOL_SIZE =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.sentinelExecutorPoolSize")
+ .typeParameterClass(Integer.class)
+ .description(
+ "Size of executor service in Sentinel Managers to check the health "
+ + "of sentinel resources.")
+ .defaultValue(3)
+ .enableDynamicOverride(false)
+ .build();
+
+ public static final ConfigOption<Integer> SENTINEL_RESOURCE_RECONCILIATION_DELAY =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.sentinelResourceReconciliationDelaySeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Allowed max time(seconds) between spec update and reconciliation "
+ + "for sentinel resources.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(60)
+ .build();
+
+ public static final ConfigOption<Boolean> LEADER_ELECTION_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.leaderElection.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable leader election for the operator to allow running standby instances. When "
+ + "this is disabled, only one operator instance is expected to be up and "
+ + "running at any time (replica = 1) to avoid race condition.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> LEADER_ELECTION_LEASE_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.leaderElection.leaseName")
+ .enableDynamicOverride(false)
+ .description(
+ "Leader election lease name, must be unique for leases in the same namespace.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-operator-lease")
+ .build();
+
+ public static final ConfigOption<Integer> LEADER_ELECTION_LEASE_DURATION_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.leaderElection.leaseDurationSeconds")
+ .enableDynamicOverride(false)
+ .description("Leader election lease duration in seconds, non-negative.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1200)
Review Comment:
Thanks for the cache. This was introduced in prod version of operator which initial sync can take a while considering the load. But for general 20min is quite big to identify something is not right - updated this to 5 minutes and adjusted the rest of leader election params accordingly.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
+ .enableDynamicOverride(false)
+ .description(
+ "Max number of reconcile loops triggered within the rate limiter refresh "
+ + "period for each resource. Setting the limit <= 0 disables the "
+ + "limiter.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description("Initial interval(in seconds) of retries on unhandled controller errors.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Double> RETRY_INTERVAL_MULTIPLIER =
+ ConfigOption.<Double>builder()
+ .key("spark.kubernetes.operator.retry.intervalMultiplier")
+ .enableDynamicOverride(false)
+ .description(
+ "Interval multiplier of retries on unhandled controller errors. Setting "
+ + "this to 1 for linear retry.")
+ .typeParameterClass(Double.class)
+ .defaultValue(1.5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Max interval(in seconds) of retries on unhandled controller errors. "
+ + "Set to non-positive for unlimited.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(-1)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_ATTEMPTS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Max attempts of retries on unhandled controller errors. Setting this to "
+ + "non-positive value means no retry.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Long> RETRY_ATTEMPT_AFTER_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.attemptAfterSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Default time (in seconds) to wait till next request. This would be used if "
+ + "server does not set Retry-After in response. Setting this to non-positive "
+ + "number means immediate retry.")
+ .typeParameterClass(Long.class)
+ .defaultValue(1L)
+ .build();
+
+ public static final ConfigOption<Long> STATUS_PATCH_MAX_RETRY =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.maxStatusPatchAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Maximal number of retry attempts of requests to k8s server for resource "
+ + "status update. This would be performed on top of k8s client "
+ + "spark.kubernetes.operator.retry.maxAttempts to overcome potential "
+ + "conflicting update on the same SparkApplication. This should be positive "
+ + "number.")
+ .typeParameterClass(Long.class)
+ .defaultValue(3L)
+ .build();
+
+ public static final ConfigOption<Long> SECONDARY_RESOURCE_CREATE_MAX_ATTEMPTS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.secondaryResourceCreateMaxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Maximal number of retry attempts of requesting secondary resource for Spark "
+ + "application. This would be performed on top of k8s client "
+ + "spark.kubernetes.operator.retry.maxAttempts to overcome potential "
+ + "conflicting reconcile on the same SparkApplication. This should be "
+ + "positive number")
+ .typeParameterClass(Long.class)
+ .defaultValue(3L)
+ .build();
+
+ public static final ConfigOption<Boolean> JOSDK_METRICS_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.josdkMetricsEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, the josdk metrics will be added in metrics source and "
+ + "configured for operator.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Boolean> KUBERNETES_CLIENT_METRICS_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.clientMetricsEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable KubernetesClient metrics for measuring the HTTP traffic to "
+ + "the Kubernetes API Server. Since the metrics is collected "
+ + "via Okhttp interceptors, can be disabled when opt in "
+ + "customized interceptors.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Boolean>
+ KUBERNETES_CLIENT_METRICS_GROUP_BY_RESPONSE_CODE_GROUP_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.clientMetricsGroupByResponseCodeEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, additional metrics group by http response code group(1xx, "
+ + "2xx, 3xx, 4xx, 5xx) received from API server will be added. Users "
+ + "can disable it when their monitoring system can combine lower level "
+ + "kubernetes.client.http.response.<3-digit-response-code> metrics.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Integer> OPERATOR_METRICS_PORT =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.metrics.port")
+ .enableDynamicOverride(false)
+ .description("The port used for checking metrics")
+ .typeParameterClass(Integer.class)
+ .defaultValue(19090)
+ .build();
+
+ public static final ConfigOption<Integer> OPERATOR_PROBE_PORT =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.probePort")
+ .enableDynamicOverride(false)
+ .description("The port used for health/readiness check probe status.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(19091)
+ .build();
+
+ public static final ConfigOption<Integer> SENTINEL_EXECUTOR_SERVICE_POOL_SIZE =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.sentinelExecutorPoolSize")
+ .typeParameterClass(Integer.class)
+ .description(
+ "Size of executor service in Sentinel Managers to check the health "
+ + "of sentinel resources.")
+ .defaultValue(3)
+ .enableDynamicOverride(false)
+ .build();
+
+ public static final ConfigOption<Integer> SENTINEL_RESOURCE_RECONCILIATION_DELAY =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.sentinelResourceReconciliationDelaySeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Allowed max time(seconds) between spec update and reconciliation "
+ + "for sentinel resources.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(60)
+ .build();
+
+ public static final ConfigOption<Boolean> LEADER_ELECTION_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.leaderElection.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable leader election for the operator to allow running standby instances. When "
+ + "this is disabled, only one operator instance is expected to be up and "
+ + "running at any time (replica = 1) to avoid race condition.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> LEADER_ELECTION_LEASE_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.leaderElection.leaseName")
+ .enableDynamicOverride(false)
+ .description(
+ "Leader election lease name, must be unique for leases in the same namespace.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-operator-lease")
+ .build();
+
+ public static final ConfigOption<Integer> LEADER_ELECTION_LEASE_DURATION_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.leaderElection.leaseDurationSeconds")
+ .enableDynamicOverride(false)
+ .description("Leader election lease duration in seconds, non-negative.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1200)
+ .build();
+
+ public static final ConfigOption<Integer> LEADER_ELECTION_RENEW_DEADLINE_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.leaderElection.renewDeadlineSeconds")
+ .enableDynamicOverride(false)
+ .description("Leader election renew deadline in seconds, non-negative.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(600)
Review Comment:
We do not guarantee actually - in this case RuntimeException would be thrown for invalid config. Added a few more description for renewal deadline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48984]Add Controller Metrics System and Utils [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #23:
URL: https://github.com/apache/spark-kubernetes-operator/pull/23#discussion_r1690846851
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/MetricsSystem.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.metrics;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.codahale.metrics.MetricFilter;
+import com.codahale.metrics.MetricRegistry;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.metrics.source.OperatorJvmSource;
+import org.apache.spark.metrics.sink.Sink;
+import org.apache.spark.metrics.source.Source;
+
+@Slf4j
+public class MetricsSystem {
+ private final AtomicBoolean running = new AtomicBoolean(false);
+ @Getter private final Set<Sink> sinks;
+ @Getter private final Set<Source> sources;
+ @Getter private final MetricRegistry registry;
+ @Getter private final Properties properties;
+ // PrometheusPullModelHandler is registered by default, metrics exposed via http port
+ @Getter private final PrometheusPullModelHandler prometheusPullModelHandler;
+ private final Map<String, SinkProps> sinkPropertiesMap;
+
+ public MetricsSystem() {
+ this(new Properties());
+ }
+
+ public MetricsSystem(Properties properties) {
+ this.sources = new HashSet<>();
+ this.sinks = new HashSet<>();
+ this.registry = new MetricRegistry();
+ this.properties = properties;
+ this.sinkPropertiesMap = MetricsSystemFactory.parseSinkProperties(this.properties);
+ // Add default sinks
+ this.prometheusPullModelHandler = new PrometheusPullModelHandler(new Properties(), registry);
+ this.sinks.add(prometheusPullModelHandler);
+ }
+
+ public void start() {
+ if (running.get()) {
+ throw new IllegalStateException(
+ "Attempting to start a MetricsSystem that is already running");
+ }
+ running.set(true);
+ registerDefaultSources();
+ registerSinks();
+ sinks.forEach(Sink::start);
+ }
+
+ public void stop() {
+ if (running.get()) {
+ sinks.forEach(Sink::stop);
+ registry.removeMatching(MetricFilter.ALL);
+ } else {
+ log.error("Stopping a MetricsSystem that is not running");
Review Comment:
Ack. It makes sense to me too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48984]Add Controller Metrics System and Utils [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #23:
URL: https://github.com/apache/spark-kubernetes-operator/pull/23#discussion_r1690843741
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -195,18 +195,18 @@ public class SparkOperatorConf {
.defaultValue(5)
.build();
- public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ public static final ConfigOption<Integer> RECONCILER_RETRY_INITIAL_INTERVAL_SECONDS =
ConfigOption.<Integer>builder()
- .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .key("spark.kubernetes.operator.reconciler.retry.initialIntervalSeconds")
.enableDynamicOverride(false)
.description("Initial interval(in seconds) of retries on unhandled controller errors.")
.typeParameterClass(Integer.class)
.defaultValue(5)
.build();
- public static final ConfigOption<Double> RETRY_INTERVAL_MULTIPLIER =
+ public static final ConfigOption<Double> RECONCILER_RETRY_INTERVAL_MULTIPLIER =
ConfigOption.<Double>builder()
- .key("spark.kubernetes.operator.retry.intervalMultiplier")
+ .key("spark.kubernetes.operator.reconciler.retry.intervalMultiplier")
Review Comment:
Thank you for changing the namespace of `retry.*`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48984]Add Controller Metrics System and Utils [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #23:
URL: https://github.com/apache/spark-kubernetes-operator/pull/23#issuecomment-2249509883
Merged to main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48984]Add Controller Metrics System and Utils [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #23:
URL: https://github.com/apache/spark-kubernetes-operator/pull/23#discussion_r1689009925
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
Review Comment:
Thank you for supporting this syntax.
I saw the implementation.
```
if ("*".equals(str)) {
return Collections.emptySet();
}
```
Although I understand that you meant the following, the description about `unset` could be a little misleading because this configuration has a default value, `default`.
```
spark.kubernetes.operator.watchedNamespaces=*
spark.kubernetes.operator.watchedNamespaces= # blank
```
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
Review Comment:
`this` -> `This`
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
Review Comment:
Can we keep the pattern consistently? For example, since the name is `spark.kubernetes.operator.reconciler`,
`SPARK_APP_RECONCILE_INTERVAL_SECONDS` -> `RECONCILER_SPARK_APP_INTERVAL_SECONDS`?
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
+ .enableDynamicOverride(false)
+ .description(
+ "Max number of reconcile loops triggered within the rate limiter refresh "
+ + "period for each resource. Setting the limit <= 0 disables the "
+ + "limiter.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description("Initial interval(in seconds) of retries on unhandled controller errors.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
Review Comment:
QQ. Why 5 seconds?
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
Review Comment:
Since `rateLimiter` already includes `limit`, can we be more specific?
For example, `limitForPeriod` -> `maxLoopForPeriod`? WDYT?
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
+ .enableDynamicOverride(false)
+ .description(
+ "Max number of reconcile loops triggered within the rate limiter refresh "
+ + "period for each resource. Setting the limit <= 0 disables the "
+ + "limiter.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description("Initial interval(in seconds) of retries on unhandled controller errors.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Double> RETRY_INTERVAL_MULTIPLIER =
+ ConfigOption.<Double>builder()
+ .key("spark.kubernetes.operator.retry.intervalMultiplier")
+ .enableDynamicOverride(false)
+ .description(
+ "Interval multiplier of retries on unhandled controller errors. Setting "
+ + "this to 1 for linear retry.")
Review Comment:
Since it's `Double`, what happens when we have `0`?
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
Review Comment:
To use `Enabled` here and `spark.kubernetes.operator.metrics.clientMetricsEnabled`, we need to change the following for consistency.
```
- spark.kubernetes.operator.terminateOnInformerFailure
+ spark.kubernetes.operator.terminateOnInformerFailureEnabled
```
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
+ .enableDynamicOverride(false)
+ .description(
+ "Max number of reconcile loops triggered within the rate limiter refresh "
+ + "period for each resource. Setting the limit <= 0 disables the "
+ + "limiter.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description("Initial interval(in seconds) of retries on unhandled controller errors.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Double> RETRY_INTERVAL_MULTIPLIER =
+ ConfigOption.<Double>builder()
+ .key("spark.kubernetes.operator.retry.intervalMultiplier")
+ .enableDynamicOverride(false)
+ .description(
+ "Interval multiplier of retries on unhandled controller errors. Setting "
+ + "this to 1 for linear retry.")
+ .typeParameterClass(Double.class)
+ .defaultValue(1.5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Max interval(in seconds) of retries on unhandled controller errors. "
+ + "Set to non-positive for unlimited.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(-1)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_ATTEMPTS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Max attempts of retries on unhandled controller errors. Setting this to "
+ + "non-positive value means no retry.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Long> RETRY_ATTEMPT_AFTER_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.attemptAfterSeconds")
Review Comment:
Could you clarify
- When `initialIntervalSeconds` and `intervalMultiplier` is used
- When `attemptAfterSeconds` is used
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/source/OperatorJosdkMetrics.java:
##########
@@ -0,0 +1,312 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.metrics.source;
+
+import static io.javaoperatorsdk.operator.api.reconciler.Constants.CONTROLLER_NAME;
+
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+
+import com.codahale.metrics.Counter;
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.Histogram;
+import com.codahale.metrics.MetricRegistry;
+import io.fabric8.kubernetes.api.model.HasMetadata;
+import io.javaoperatorsdk.operator.api.monitoring.Metrics;
+import io.javaoperatorsdk.operator.api.reconciler.Constants;
+import io.javaoperatorsdk.operator.api.reconciler.RetryInfo;
+import io.javaoperatorsdk.operator.processing.Controller;
+import io.javaoperatorsdk.operator.processing.GroupVersionKind;
+import io.javaoperatorsdk.operator.processing.event.Event;
+import io.javaoperatorsdk.operator.processing.event.ResourceID;
+import io.javaoperatorsdk.operator.processing.event.source.controller.ResourceAction;
+import io.javaoperatorsdk.operator.processing.event.source.controller.ResourceEvent;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.BaseResource;
+import org.apache.spark.k8s.operator.SparkApplication;
+import org.apache.spark.metrics.source.Source;
+import org.apache.spark.util.Clock;
+import org.apache.spark.util.SystemClock;
+
+@Slf4j
+public class OperatorJosdkMetrics implements Source, Metrics {
+ public static final String FINISHED = "finished";
+ public static final String CLEANUP = "cleanup";
+ public static final String FAILED = "failed";
+ public static final String RETRIES = "retries";
+ private final Map<String, Histogram> histograms = new ConcurrentHashMap<>();
+ private final Map<String, Counter> counters = new ConcurrentHashMap<>();
+ private final Map<String, Gauge<?>> gauges = new ConcurrentHashMap<>();
+ private static final String RECONCILIATION = "reconciliation";
+ private static final String RESOURCE = "resource";
+ private static final String EVENT = "event";
+ private static final String SUCCESS = "success";
+ private static final String FAILURE = "failure";
+ private static final String EXCEPTION = "exception";
+ private static final String PREFIX = "operator.sdk";
+ private static final String RECONCILIATIONS = "reconciliations";
+ private static final String RECONCILIATIONS_EXECUTIONS = RECONCILIATIONS + ".executions";
+ private static final String RECONCILIATIONS_QUEUE_SIZE = RECONCILIATIONS + ".queue.size";
+ private static final String SIZE = "size";
+
+ private final Clock clock;
+ private final MetricRegistry metricRegistry;
+
+ public OperatorJosdkMetrics() {
+ this.clock = new SystemClock();
+ this.metricRegistry = new MetricRegistry();
+ }
+
+ @Override
+ public String sourceName() {
+ return PREFIX;
+ }
+
+ @Override
+ public MetricRegistry metricRegistry() {
+ return metricRegistry;
+ }
+
+ @Override
+ public void controllerRegistered(Controller<? extends HasMetadata> controller) {
+ // no-op
+ log.debug("Controller has been registered");
+ }
+
+ @Override
+ public void receivedEvent(Event event, Map<String, Object> metadata) {
+ log.debug("received event {}, metadata {}", event, metadata);
+ if (event instanceof ResourceEvent) {
+ final ResourceAction action = ((ResourceEvent) event).getAction();
+ final Optional<Class<? extends BaseResource<?, ?, ?, ?, ?>>> resource =
+ getResourceClass(metadata);
+ final Optional<String> namespaceOptional = event.getRelatedCustomResourceID().getNamespace();
+ resource.ifPresent(
+ aClass -> getCounter(aClass, action.name().toLowerCase(), RESOURCE, EVENT).inc());
+ if (resource.isPresent() && namespaceOptional.isPresent()) {
+ getCounter(
+ resource.get(),
+ namespaceOptional.get(),
+ action.name().toLowerCase(),
+ RESOURCE,
+ EVENT)
+ .inc();
+ }
+ }
+ }
+
+ @Override
+ public <T> T timeControllerExecution(ControllerExecution<T> execution) throws Exception {
+ log.debug("Time controller execution");
+ final String name = execution.controllerName();
+ final ResourceID resourceID = execution.resourceID();
+ final Optional<String> namespaceOptional = resourceID.getNamespace();
+ final Map<String, Object> metadata = execution.metadata();
+ final Optional<Class<? extends BaseResource<?, ?, ?, ?, ?>>> resourceClass =
+ getResourceClass(metadata);
+ final String execName = execution.name();
+
+ long startTime = clock.getTimeMillis();
+ try {
+ T result = execution.execute();
+ final String successType = execution.successTypeName(result);
+ if (resourceClass.isPresent()) {
+ getHistogram(resourceClass.get(), name, execName, successType).update(toSeconds(startTime));
+ getCounter(resourceClass.get(), name, execName, SUCCESS, successType).inc();
+ if (namespaceOptional.isPresent()) {
+ getHistogram(resourceClass.get(), namespaceOptional.get(), name, execName, successType)
+ .update(toSeconds(startTime));
+ getCounter(
+ resourceClass.get(),
+ namespaceOptional.get(),
+ name,
+ execName,
+ SUCCESS,
+ successType)
+ .inc();
+ }
+ }
+ return result;
+ } catch (Exception e) {
Review Comment:
Is this a narrowest exception?
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/ClassLoadingUtils.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.utils;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+
+import org.apache.spark.k8s.operator.listeners.BaseStatusListener;
+
+@Slf4j
+public class ClassLoadingUtils {
+ /**
+ * load status listener classes for given type of BaseStatusListener
+ *
+ * @param clazz Class name of status listener. e.g. SparkAppStatusListener
+ * @param implementationClassNames Comma-separated implementation class names of given type of
+ * StatusListener
+ * @return list of listeners loaded
+ * @param <T> Class of status listener. e.g. SparkAppStatusListener
+ */
+ public static <T extends BaseStatusListener<?, ?>> List<T> getStatusListener(
+ Class<T> clazz, String implementationClassNames) {
+ List<T> listeners = new ArrayList<>();
+ if (StringUtils.isNotBlank(implementationClassNames)) {
+ try {
+ Set<String> listenerNames = Utils.sanitizeCommaSeparatedStrAsSet(implementationClassNames);
+ for (String name : listenerNames) {
+ Class listenerClass = Class.forName(name);
+ if (clazz.isAssignableFrom(listenerClass)) {
+ listeners.add((T) listenerClass.getConstructor().newInstance());
+ }
+ }
+ } catch (Exception e) {
Review Comment:
Is this the narrowest one?
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
+ .enableDynamicOverride(false)
+ .description(
+ "Max number of reconcile loops triggered within the rate limiter refresh "
+ + "period for each resource. Setting the limit <= 0 disables the "
+ + "limiter.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description("Initial interval(in seconds) of retries on unhandled controller errors.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Double> RETRY_INTERVAL_MULTIPLIER =
+ ConfigOption.<Double>builder()
+ .key("spark.kubernetes.operator.retry.intervalMultiplier")
+ .enableDynamicOverride(false)
+ .description(
+ "Interval multiplier of retries on unhandled controller errors. Setting "
+ + "this to 1 for linear retry.")
+ .typeParameterClass(Double.class)
+ .defaultValue(1.5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Max interval(in seconds) of retries on unhandled controller errors. "
+ + "Set to non-positive for unlimited.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(-1)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_ATTEMPTS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Max attempts of retries on unhandled controller errors. Setting this to "
+ + "non-positive value means no retry.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Long> RETRY_ATTEMPT_AFTER_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.attemptAfterSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Default time (in seconds) to wait till next request. This would be used if "
+ + "server does not set Retry-After in response. Setting this to non-positive "
+ + "number means immediate retry.")
+ .typeParameterClass(Long.class)
+ .defaultValue(1L)
+ .build();
+
+ public static final ConfigOption<Long> STATUS_PATCH_MAX_RETRY =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.maxStatusPatchAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Maximal number of retry attempts of requests to k8s server for resource "
+ + "status update. This would be performed on top of k8s client "
+ + "spark.kubernetes.operator.retry.maxAttempts to overcome potential "
+ + "conflicting update on the same SparkApplication. This should be positive "
+ + "number.")
+ .typeParameterClass(Long.class)
+ .defaultValue(3L)
+ .build();
+
+ public static final ConfigOption<Long> SECONDARY_RESOURCE_CREATE_MAX_ATTEMPTS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.secondaryResourceCreateMaxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Maximal number of retry attempts of requesting secondary resource for Spark "
+ + "application. This would be performed on top of k8s client "
+ + "spark.kubernetes.operator.retry.maxAttempts to overcome potential "
+ + "conflicting reconcile on the same SparkApplication. This should be "
+ + "positive number")
+ .typeParameterClass(Long.class)
+ .defaultValue(3L)
+ .build();
+
+ public static final ConfigOption<Boolean> JOSDK_METRICS_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.josdkMetricsEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, the josdk metrics will be added in metrics source and "
+ + "configured for operator.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Boolean> KUBERNETES_CLIENT_METRICS_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.clientMetricsEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable KubernetesClient metrics for measuring the HTTP traffic to "
+ + "the Kubernetes API Server. Since the metrics is collected "
+ + "via Okhttp interceptors, can be disabled when opt in "
+ + "customized interceptors.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Boolean>
+ KUBERNETES_CLIENT_METRICS_GROUP_BY_RESPONSE_CODE_GROUP_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.clientMetricsGroupByResponseCodeEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, additional metrics group by http response code group(1xx, "
+ + "2xx, 3xx, 4xx, 5xx) received from API server will be added. Users "
+ + "can disable it when their monitoring system can combine lower level "
+ + "kubernetes.client.http.response.<3-digit-response-code> metrics.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Integer> OPERATOR_METRICS_PORT =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.metrics.port")
+ .enableDynamicOverride(false)
+ .description("The port used for checking metrics")
+ .typeParameterClass(Integer.class)
+ .defaultValue(19090)
+ .build();
+
+ public static final ConfigOption<Integer> OPERATOR_PROBE_PORT =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.probePort")
+ .enableDynamicOverride(false)
+ .description("The port used for health/readiness check probe status.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(19091)
+ .build();
+
+ public static final ConfigOption<Integer> SENTINEL_EXECUTOR_SERVICE_POOL_SIZE =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.sentinelExecutorPoolSize")
+ .typeParameterClass(Integer.class)
+ .description(
+ "Size of executor service in Sentinel Managers to check the health "
+ + "of sentinel resources.")
+ .defaultValue(3)
+ .enableDynamicOverride(false)
+ .build();
+
+ public static final ConfigOption<Integer> SENTINEL_RESOURCE_RECONCILIATION_DELAY =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.sentinelResourceReconciliationDelaySeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Allowed max time(seconds) between spec update and reconciliation "
+ + "for sentinel resources.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(60)
+ .build();
+
+ public static final ConfigOption<Boolean> LEADER_ELECTION_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.leaderElection.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable leader election for the operator to allow running standby instances. When "
+ + "this is disabled, only one operator instance is expected to be up and "
+ + "running at any time (replica = 1) to avoid race condition.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> LEADER_ELECTION_LEASE_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.leaderElection.leaseName")
+ .enableDynamicOverride(false)
+ .description(
+ "Leader election lease name, must be unique for leases in the same namespace.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-operator-lease")
+ .build();
+
+ public static final ConfigOption<Integer> LEADER_ELECTION_LEASE_DURATION_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.leaderElection.leaseDurationSeconds")
+ .enableDynamicOverride(false)
+ .description("Leader election lease duration in seconds, non-negative.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1200)
Review Comment:
Why 1200?
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
+ .enableDynamicOverride(false)
+ .description(
+ "Max number of reconcile loops triggered within the rate limiter refresh "
+ + "period for each resource. Setting the limit <= 0 disables the "
+ + "limiter.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description("Initial interval(in seconds) of retries on unhandled controller errors.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Double> RETRY_INTERVAL_MULTIPLIER =
+ ConfigOption.<Double>builder()
+ .key("spark.kubernetes.operator.retry.intervalMultiplier")
+ .enableDynamicOverride(false)
+ .description(
+ "Interval multiplier of retries on unhandled controller errors. Setting "
+ + "this to 1 for linear retry.")
+ .typeParameterClass(Double.class)
+ .defaultValue(1.5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Max interval(in seconds) of retries on unhandled controller errors. "
+ + "Set to non-positive for unlimited.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(-1)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_ATTEMPTS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Max attempts of retries on unhandled controller errors. Setting this to "
+ + "non-positive value means no retry.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Long> RETRY_ATTEMPT_AFTER_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.attemptAfterSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Default time (in seconds) to wait till next request. This would be used if "
+ + "server does not set Retry-After in response. Setting this to non-positive "
+ + "number means immediate retry.")
+ .typeParameterClass(Long.class)
+ .defaultValue(1L)
+ .build();
+
+ public static final ConfigOption<Long> STATUS_PATCH_MAX_RETRY =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.maxStatusPatchAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Maximal number of retry attempts of requests to k8s server for resource "
+ + "status update. This would be performed on top of k8s client "
+ + "spark.kubernetes.operator.retry.maxAttempts to overcome potential "
+ + "conflicting update on the same SparkApplication. This should be positive "
+ + "number.")
+ .typeParameterClass(Long.class)
+ .defaultValue(3L)
+ .build();
+
+ public static final ConfigOption<Long> SECONDARY_RESOURCE_CREATE_MAX_ATTEMPTS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.secondaryResourceCreateMaxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Maximal number of retry attempts of requesting secondary resource for Spark "
+ + "application. This would be performed on top of k8s client "
+ + "spark.kubernetes.operator.retry.maxAttempts to overcome potential "
+ + "conflicting reconcile on the same SparkApplication. This should be "
+ + "positive number")
+ .typeParameterClass(Long.class)
+ .defaultValue(3L)
+ .build();
+
+ public static final ConfigOption<Boolean> JOSDK_METRICS_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.josdkMetricsEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, the josdk metrics will be added in metrics source and "
+ + "configured for operator.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Boolean> KUBERNETES_CLIENT_METRICS_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.clientMetricsEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable KubernetesClient metrics for measuring the HTTP traffic to "
+ + "the Kubernetes API Server. Since the metrics is collected "
+ + "via Okhttp interceptors, can be disabled when opt in "
+ + "customized interceptors.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Boolean>
+ KUBERNETES_CLIENT_METRICS_GROUP_BY_RESPONSE_CODE_GROUP_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.clientMetricsGroupByResponseCodeEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, additional metrics group by http response code group(1xx, "
+ + "2xx, 3xx, 4xx, 5xx) received from API server will be added. Users "
+ + "can disable it when their monitoring system can combine lower level "
+ + "kubernetes.client.http.response.<3-digit-response-code> metrics.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Integer> OPERATOR_METRICS_PORT =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.metrics.port")
+ .enableDynamicOverride(false)
+ .description("The port used for checking metrics")
+ .typeParameterClass(Integer.class)
+ .defaultValue(19090)
+ .build();
+
+ public static final ConfigOption<Integer> OPERATOR_PROBE_PORT =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.probePort")
+ .enableDynamicOverride(false)
+ .description("The port used for health/readiness check probe status.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(19091)
+ .build();
+
+ public static final ConfigOption<Integer> SENTINEL_EXECUTOR_SERVICE_POOL_SIZE =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.sentinelExecutorPoolSize")
+ .typeParameterClass(Integer.class)
+ .description(
+ "Size of executor service in Sentinel Managers to check the health "
+ + "of sentinel resources.")
+ .defaultValue(3)
+ .enableDynamicOverride(false)
+ .build();
+
+ public static final ConfigOption<Integer> SENTINEL_RESOURCE_RECONCILIATION_DELAY =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.sentinelResourceReconciliationDelaySeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Allowed max time(seconds) between spec update and reconciliation "
+ + "for sentinel resources.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(60)
+ .build();
+
+ public static final ConfigOption<Boolean> LEADER_ELECTION_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.leaderElection.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable leader election for the operator to allow running standby instances. When "
+ + "this is disabled, only one operator instance is expected to be up and "
+ + "running at any time (replica = 1) to avoid race condition.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> LEADER_ELECTION_LEASE_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.leaderElection.leaseName")
+ .enableDynamicOverride(false)
+ .description(
+ "Leader election lease name, must be unique for leases in the same namespace.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-operator-lease")
+ .build();
+
+ public static final ConfigOption<Integer> LEADER_ELECTION_LEASE_DURATION_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.leaderElection.leaseDurationSeconds")
+ .enableDynamicOverride(false)
+ .description("Leader election lease duration in seconds, non-negative.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1200)
+ .build();
+
+ public static final ConfigOption<Integer> LEADER_ELECTION_RENEW_DEADLINE_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.leaderElection.renewDeadlineSeconds")
+ .enableDynamicOverride(false)
+ .description("Leader election renew deadline in seconds, non-negative.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(600)
+ .build();
+
+ public static final ConfigOption<Integer> LEADER_ELECTION_RETRY_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.leaderElection.retryPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Leader election retry period in seconds, non-negative.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(180)
+ .build();
+
+ public static LeaderElectionConfiguration getLeaderElectionConfig() {
Review Comment:
I believe we need more validation logic in this method in order to prevent a user mistake.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
+ .enableDynamicOverride(false)
+ .description(
+ "Max number of reconcile loops triggered within the rate limiter refresh "
+ + "period for each resource. Setting the limit <= 0 disables the "
+ + "limiter.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description("Initial interval(in seconds) of retries on unhandled controller errors.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Double> RETRY_INTERVAL_MULTIPLIER =
+ ConfigOption.<Double>builder()
+ .key("spark.kubernetes.operator.retry.intervalMultiplier")
+ .enableDynamicOverride(false)
+ .description(
+ "Interval multiplier of retries on unhandled controller errors. Setting "
+ + "this to 1 for linear retry.")
+ .typeParameterClass(Double.class)
+ .defaultValue(1.5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Max interval(in seconds) of retries on unhandled controller errors. "
+ + "Set to non-positive for unlimited.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(-1)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_ATTEMPTS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Max attempts of retries on unhandled controller errors. Setting this to "
+ + "non-positive value means no retry.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Long> RETRY_ATTEMPT_AFTER_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.attemptAfterSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Default time (in seconds) to wait till next request. This would be used if "
+ + "server does not set Retry-After in response. Setting this to non-positive "
+ + "number means immediate retry.")
+ .typeParameterClass(Long.class)
+ .defaultValue(1L)
+ .build();
+
+ public static final ConfigOption<Long> STATUS_PATCH_MAX_RETRY =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.maxStatusPatchAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Maximal number of retry attempts of requests to k8s server for resource "
+ + "status update. This would be performed on top of k8s client "
+ + "spark.kubernetes.operator.retry.maxAttempts to overcome potential "
+ + "conflicting update on the same SparkApplication. This should be positive "
+ + "number.")
+ .typeParameterClass(Long.class)
+ .defaultValue(3L)
+ .build();
+
+ public static final ConfigOption<Long> SECONDARY_RESOURCE_CREATE_MAX_ATTEMPTS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.secondaryResourceCreateMaxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Maximal number of retry attempts of requesting secondary resource for Spark "
+ + "application. This would be performed on top of k8s client "
+ + "spark.kubernetes.operator.retry.maxAttempts to overcome potential "
+ + "conflicting reconcile on the same SparkApplication. This should be "
+ + "positive number")
+ .typeParameterClass(Long.class)
+ .defaultValue(3L)
+ .build();
+
+ public static final ConfigOption<Boolean> JOSDK_METRICS_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.josdkMetricsEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, the josdk metrics will be added in metrics source and "
+ + "configured for operator.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Boolean> KUBERNETES_CLIENT_METRICS_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.clientMetricsEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable KubernetesClient metrics for measuring the HTTP traffic to "
+ + "the Kubernetes API Server. Since the metrics is collected "
+ + "via Okhttp interceptors, can be disabled when opt in "
+ + "customized interceptors.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Boolean>
+ KUBERNETES_CLIENT_METRICS_GROUP_BY_RESPONSE_CODE_GROUP_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.clientMetricsGroupByResponseCodeEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, additional metrics group by http response code group(1xx, "
+ + "2xx, 3xx, 4xx, 5xx) received from API server will be added. Users "
+ + "can disable it when their monitoring system can combine lower level "
+ + "kubernetes.client.http.response.<3-digit-response-code> metrics.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Integer> OPERATOR_METRICS_PORT =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.metrics.port")
+ .enableDynamicOverride(false)
+ .description("The port used for checking metrics")
+ .typeParameterClass(Integer.class)
+ .defaultValue(19090)
+ .build();
+
+ public static final ConfigOption<Integer> OPERATOR_PROBE_PORT =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.probePort")
+ .enableDynamicOverride(false)
+ .description("The port used for health/readiness check probe status.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(19091)
+ .build();
+
+ public static final ConfigOption<Integer> SENTINEL_EXECUTOR_SERVICE_POOL_SIZE =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.sentinelExecutorPoolSize")
+ .typeParameterClass(Integer.class)
+ .description(
+ "Size of executor service in Sentinel Managers to check the health "
+ + "of sentinel resources.")
+ .defaultValue(3)
+ .enableDynamicOverride(false)
+ .build();
+
+ public static final ConfigOption<Integer> SENTINEL_RESOURCE_RECONCILIATION_DELAY =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.sentinelResourceReconciliationDelaySeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Allowed max time(seconds) between spec update and reconciliation "
+ + "for sentinel resources.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(60)
+ .build();
+
+ public static final ConfigOption<Boolean> LEADER_ELECTION_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.leaderElection.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable leader election for the operator to allow running standby instances. When "
+ + "this is disabled, only one operator instance is expected to be up and "
+ + "running at any time (replica = 1) to avoid race condition.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> LEADER_ELECTION_LEASE_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.leaderElection.leaseName")
+ .enableDynamicOverride(false)
+ .description(
+ "Leader election lease name, must be unique for leases in the same namespace.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-operator-lease")
+ .build();
+
+ public static final ConfigOption<Integer> LEADER_ELECTION_LEASE_DURATION_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.leaderElection.leaseDurationSeconds")
+ .enableDynamicOverride(false)
+ .description("Leader election lease duration in seconds, non-negative.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1200)
+ .build();
+
+ public static final ConfigOption<Integer> LEADER_ELECTION_RENEW_DEADLINE_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.leaderElection.renewDeadlineSeconds")
+ .enableDynamicOverride(false)
+ .description("Leader election renew deadline in seconds, non-negative.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(600)
+ .build();
+
+ public static final ConfigOption<Integer> LEADER_ELECTION_RETRY_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.leaderElection.retryPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Leader election retry period in seconds, non-negative.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(180)
+ .build();
+
+ public static LeaderElectionConfiguration getLeaderElectionConfig() {
+ return new LeaderElectionConfiguration(
+ LEADER_ELECTION_LEASE_NAME.getValue(),
+ OPERATOR_NAMESPACE.getValue(),
+ Duration.ofSeconds(ensurePositiveIntFor(LEADER_ELECTION_LEASE_DURATION_SECONDS)),
+ Duration.ofSeconds(ensurePositiveIntFor(LEADER_ELECTION_RENEW_DEADLINE_SECONDS)),
+ Duration.ofSeconds(ensurePositiveIntFor(LEADER_ELECTION_RETRY_PERIOD_SECONDS)));
+ }
+
+ public static GenericRetry getOperatorRetry() {
+ GenericRetry genericRetry =
+ new GenericRetry()
+ .setMaxAttempts(ensureNonNegativeIntFor(RETRY_MAX_ATTEMPTS))
+ .setInitialInterval(
+ Duration.ofSeconds(ensureNonNegativeIntFor(RETRY_INITIAL_INTERVAL_SECONDS))
+ .toMillis())
+ .setIntervalMultiplier(RETRY_INTERVAL_MULTIPLIER.getValue());
+ if (RETRY_MAX_INTERVAL_SECONDS.getValue() > 0) {
+ genericRetry.setMaxInterval(
+ Duration.ofSeconds(RETRY_MAX_INTERVAL_SECONDS.getValue()).toMillis());
+ } else {
+ log.info("Reconciler retry policy is configured with unlimited max attempts");
+ }
+ return genericRetry;
+ }
+
+ public static RateLimiter<?> getOperatorRateLimiter() {
+ return new LinearRateLimiter(
+ Duration.ofSeconds(ensureNonNegativeIntFor(RATE_LIMITER_REFRESH_PERIOD_SECONDS)),
+ ensureNonNegativeIntFor(RATE_LIMITER_LIMIT_FOR_PERIOD));
+ }
+
+ private static int ensureNonNegativeIntFor(ConfigOption<Integer> configOption) {
+ return ensureValid(configOption.getValue(), configOption.getDescription(), 0, 0);
+ }
+
+ private static int ensurePositiveIntFor(ConfigOption<Integer> configOption) {
+ return ensureValid(configOption.getValue(), configOption.getDescription(), 1, 1);
+ }
+
+ private static int ensureValid(int value, String description, int minValue, int defaultValue) {
+ if (value < minValue) {
+ if (defaultValue < minValue) {
+ throw new IllegalArgumentException(
+ "Default value for " + description + " must be greater than " + minValue);
+ }
+ log.warn(
+ "Requested {} should be greater than {}. Requested: {}, using {} (default) " + "instead",
Review Comment:
Shall we remove `" + "` like the following?
```java
- " + "instead"
+ instead"
```
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
Review Comment:
`min` -> `minutes`
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/MetricsSystem.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.metrics;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.codahale.metrics.MetricFilter;
+import com.codahale.metrics.MetricRegistry;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.metrics.source.OperatorJvmSource;
+import org.apache.spark.metrics.sink.Sink;
+import org.apache.spark.metrics.source.Source;
+
+@Slf4j
+public class MetricsSystem {
+ private final AtomicBoolean running = new AtomicBoolean(false);
+ @Getter private final Set<Sink> sinks;
+ @Getter private final Set<Source> sources;
+ @Getter private final MetricRegistry registry;
+ @Getter private final Properties properties;
+ // PrometheusPullModelHandler is registered by default, metrics exposed via http port
+ @Getter private final PrometheusPullModelHandler prometheusPullModelHandler;
+ private final Map<String, SinkProps> sinkPropertiesMap;
+
+ public MetricsSystem() {
+ this(new Properties());
+ }
+
+ public MetricsSystem(Properties properties) {
+ this.sources = new HashSet<>();
+ this.sinks = new HashSet<>();
+ this.registry = new MetricRegistry();
+ this.properties = properties;
+ this.sinkPropertiesMap = MetricsSystemFactory.parseSinkProperties(this.properties);
+ // Add default sinks
+ this.prometheusPullModelHandler = new PrometheusPullModelHandler(new Properties(), registry);
+ this.sinks.add(prometheusPullModelHandler);
+ }
+
+ public void start() {
+ if (running.get()) {
+ throw new IllegalStateException(
+ "Attempting to start a MetricsSystem that is already running");
+ }
+ running.set(true);
+ registerDefaultSources();
+ registerSinks();
+ sinks.forEach(Sink::start);
+ }
+
+ public void stop() {
+ if (running.get()) {
+ sinks.forEach(Sink::stop);
+ registry.removeMatching(MetricFilter.ALL);
+ } else {
+ log.error("Stopping a MetricsSystem that is not running");
Review Comment:
I'm wondering when you use `log.isErrorEnabled`? If we want to check `log.isErrorEnabled`, we had better use it consistently.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
+ .enableDynamicOverride(false)
+ .description(
+ "Max number of reconcile loops triggered within the rate limiter refresh "
+ + "period for each resource. Setting the limit <= 0 disables the "
+ + "limiter.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description("Initial interval(in seconds) of retries on unhandled controller errors.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Double> RETRY_INTERVAL_MULTIPLIER =
+ ConfigOption.<Double>builder()
+ .key("spark.kubernetes.operator.retry.intervalMultiplier")
+ .enableDynamicOverride(false)
+ .description(
+ "Interval multiplier of retries on unhandled controller errors. Setting "
+ + "this to 1 for linear retry.")
+ .typeParameterClass(Double.class)
+ .defaultValue(1.5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Max interval(in seconds) of retries on unhandled controller errors. "
+ + "Set to non-positive for unlimited.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(-1)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_ATTEMPTS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Max attempts of retries on unhandled controller errors. Setting this to "
+ + "non-positive value means no retry.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Long> RETRY_ATTEMPT_AFTER_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.attemptAfterSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Default time (in seconds) to wait till next request. This would be used if "
+ + "server does not set Retry-After in response. Setting this to non-positive "
+ + "number means immediate retry.")
+ .typeParameterClass(Long.class)
+ .defaultValue(1L)
+ .build();
+
+ public static final ConfigOption<Long> STATUS_PATCH_MAX_RETRY =
Review Comment:
Please try to start with `RETRY_` prefix.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/SparkExceptionUtils.java:
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.utils;
+
+import io.fabric8.kubernetes.client.KubernetesClientException;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.exception.ExceptionUtils;
+
+public class SparkExceptionUtils {
+ public static boolean isConflictForExistingResource(KubernetesClientException e) {
+ return e != null
+ && e.getCode() == 409
+ && e.getStatus() != null
+ && StringUtils.isNotEmpty(e.getStatus().toString())
+ && e.getStatus().toString().toLowerCase().contains("alreadyexists");
Review Comment:
Just a question. When we have other string here for 409 code?
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<Boolean> TERMINATE_ON_INFORMER_FAILURE =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.terminateOnInformerFailure")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable to indicate informer errors should stop operator startup. If "
+ + "disabled, operator startup will ignore recoverable errors, "
+ + "caused for example by RBAC issues and will retry "
+ + "periodically.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_TERMINATION_TIMEOUT_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.terminationTimeoutSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Grace period for operator shutdown before reconciliation threads are killed.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(30)
+ .build();
+
+ public static final ConfigOption<Integer> RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.reconciler.parallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Thread pool size for Spark Operator reconcilers. Unbounded pool would be used if "
+ + "set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(50)
+ .build();
+
+ public static final ConfigOption<Long> RECONCILER_FOREGROUND_REQUEST_TIMEOUT_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.foregroundRequestTimeoutSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Timeout (in seconds) to for requests made to API server. this "
+ + "applies only to foreground requests.")
+ .typeParameterClass(Long.class)
+ .defaultValue(30L)
+ .build();
+
+ public static final ConfigOption<Long> SPARK_APP_RECONCILE_INTERVAL_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.reconciler.intervalSeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Interval (in seconds, non-negative) to reconcile Spark applications. Note that "
+ + "reconciliation is always expected to be triggered when app spec / status is "
+ + "updated. This interval controls the reconcile behavior of operator "
+ + "reconciliation even when there's no update on SparkApplication, e.g. to "
+ + "determine whether a hanging app needs to be proactively terminated. Thus "
+ + "this is recommended to set to above 2 min to avoid unnecessary no-op "
+ + "reconciliation.")
+ .typeParameterClass(Long.class)
+ .defaultValue(120L)
+ .build();
+
+ public static final ConfigOption<Boolean> TRIM_ATTEMPT_STATE_TRANSITION_HISTORY =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled")
+ .enableDynamicOverride(true)
+ .description(
+ "When enabled, operator would trim state transition history when a "
+ + "new attempt starts, keeping previous attempt summary only.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<String> SPARK_APP_STATUS_LISTENER_CLASS_NAMES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.reconciler.appStatusListenerClassNames")
+ .enableDynamicOverride(false)
+ .description("Comma-separated names of SparkAppStatusListener class implementations")
+ .typeParameterClass(String.class)
+ .defaultValue("")
+ .build();
+
+ public static final ConfigOption<Boolean> DYNAMIC_CONFIG_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, operator would use config map as source of truth for config "
+ + "property override. The config map need to be created in "
+ + "spark.kubernetes.operator.namespace, and labeled with operator name.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> DYNAMIC_CONFIG_SELECTOR =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.selector")
+ .enableDynamicOverride(false)
+ .description("The selector str applied to dynamic config map.")
+ .typeParameterClass(String.class)
+ .defaultValue(Utils.labelsAsStr(Utils.defaultOperatorConfigLabels()))
+ .build();
+
+ public static final ConfigOption<Integer> DYNAMIC_CONFIG_RECONCILER_PARALLELISM =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.dynamicConfig.reconcilerParallelism")
+ .enableDynamicOverride(false)
+ .description(
+ "Parallelism for dynamic config reconciler. Unbounded pool would be used "
+ + "if set to non-positive number.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_REFRESH_PERIOD_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.refreshPeriodSeconds")
+ .enableDynamicOverride(false)
+ .description("Operator rate limiter refresh period(in seconds) for each resource.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Integer> RATE_LIMITER_LIMIT_FOR_PERIOD =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.rateLimiter.limitForPeriod")
+ .enableDynamicOverride(false)
+ .description(
+ "Max number of reconcile loops triggered within the rate limiter refresh "
+ + "period for each resource. Setting the limit <= 0 disables the "
+ + "limiter.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_INITIAL_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.initialIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description("Initial interval(in seconds) of retries on unhandled controller errors.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(5)
+ .build();
+
+ public static final ConfigOption<Double> RETRY_INTERVAL_MULTIPLIER =
+ ConfigOption.<Double>builder()
+ .key("spark.kubernetes.operator.retry.intervalMultiplier")
+ .enableDynamicOverride(false)
+ .description(
+ "Interval multiplier of retries on unhandled controller errors. Setting "
+ + "this to 1 for linear retry.")
+ .typeParameterClass(Double.class)
+ .defaultValue(1.5)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_INTERVAL_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxIntervalSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Max interval(in seconds) of retries on unhandled controller errors. "
+ + "Set to non-positive for unlimited.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(-1)
+ .build();
+
+ public static final ConfigOption<Integer> RETRY_MAX_ATTEMPTS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.retry.maxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Max attempts of retries on unhandled controller errors. Setting this to "
+ + "non-positive value means no retry.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(15)
+ .build();
+
+ public static final ConfigOption<Long> RETRY_ATTEMPT_AFTER_SECONDS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.attemptAfterSeconds")
+ .enableDynamicOverride(false)
+ .description(
+ "Default time (in seconds) to wait till next request. This would be used if "
+ + "server does not set Retry-After in response. Setting this to non-positive "
+ + "number means immediate retry.")
+ .typeParameterClass(Long.class)
+ .defaultValue(1L)
+ .build();
+
+ public static final ConfigOption<Long> STATUS_PATCH_MAX_RETRY =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.maxStatusPatchAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Maximal number of retry attempts of requests to k8s server for resource "
+ + "status update. This would be performed on top of k8s client "
+ + "spark.kubernetes.operator.retry.maxAttempts to overcome potential "
+ + "conflicting update on the same SparkApplication. This should be positive "
+ + "number.")
+ .typeParameterClass(Long.class)
+ .defaultValue(3L)
+ .build();
+
+ public static final ConfigOption<Long> SECONDARY_RESOURCE_CREATE_MAX_ATTEMPTS =
+ ConfigOption.<Long>builder()
+ .key("spark.kubernetes.operator.retry.secondaryResourceCreateMaxAttempts")
+ .enableDynamicOverride(false)
+ .description(
+ "Maximal number of retry attempts of requesting secondary resource for Spark "
+ + "application. This would be performed on top of k8s client "
+ + "spark.kubernetes.operator.retry.maxAttempts to overcome potential "
+ + "conflicting reconcile on the same SparkApplication. This should be "
+ + "positive number")
+ .typeParameterClass(Long.class)
+ .defaultValue(3L)
+ .build();
+
+ public static final ConfigOption<Boolean> JOSDK_METRICS_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.josdkMetricsEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, the josdk metrics will be added in metrics source and "
+ + "configured for operator.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Boolean> KUBERNETES_CLIENT_METRICS_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.clientMetricsEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable KubernetesClient metrics for measuring the HTTP traffic to "
+ + "the Kubernetes API Server. Since the metrics is collected "
+ + "via Okhttp interceptors, can be disabled when opt in "
+ + "customized interceptors.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Boolean>
+ KUBERNETES_CLIENT_METRICS_GROUP_BY_RESPONSE_CODE_GROUP_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.metrics.clientMetricsGroupByResponseCodeEnabled")
+ .enableDynamicOverride(false)
+ .description(
+ "When enabled, additional metrics group by http response code group(1xx, "
+ + "2xx, 3xx, 4xx, 5xx) received from API server will be added. Users "
+ + "can disable it when their monitoring system can combine lower level "
+ + "kubernetes.client.http.response.<3-digit-response-code> metrics.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(true)
+ .build();
+
+ public static final ConfigOption<Integer> OPERATOR_METRICS_PORT =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.metrics.port")
+ .enableDynamicOverride(false)
+ .description("The port used for checking metrics")
+ .typeParameterClass(Integer.class)
+ .defaultValue(19090)
+ .build();
+
+ public static final ConfigOption<Integer> OPERATOR_PROBE_PORT =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.probePort")
+ .enableDynamicOverride(false)
+ .description("The port used for health/readiness check probe status.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(19091)
+ .build();
+
+ public static final ConfigOption<Integer> SENTINEL_EXECUTOR_SERVICE_POOL_SIZE =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.sentinelExecutorPoolSize")
+ .typeParameterClass(Integer.class)
+ .description(
+ "Size of executor service in Sentinel Managers to check the health "
+ + "of sentinel resources.")
+ .defaultValue(3)
+ .enableDynamicOverride(false)
+ .build();
+
+ public static final ConfigOption<Integer> SENTINEL_RESOURCE_RECONCILIATION_DELAY =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.health.sentinelResourceReconciliationDelaySeconds")
+ .enableDynamicOverride(true)
+ .description(
+ "Allowed max time(seconds) between spec update and reconciliation "
+ + "for sentinel resources.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(60)
+ .build();
+
+ public static final ConfigOption<Boolean> LEADER_ELECTION_ENABLED =
+ ConfigOption.<Boolean>builder()
+ .key("spark.kubernetes.operator.leaderElection.enabled")
+ .enableDynamicOverride(false)
+ .description(
+ "Enable leader election for the operator to allow running standby instances. When "
+ + "this is disabled, only one operator instance is expected to be up and "
+ + "running at any time (replica = 1) to avoid race condition.")
+ .typeParameterClass(Boolean.class)
+ .defaultValue(false)
+ .build();
+
+ public static final ConfigOption<String> LEADER_ELECTION_LEASE_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.leaderElection.leaseName")
+ .enableDynamicOverride(false)
+ .description(
+ "Leader election lease name, must be unique for leases in the same namespace.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-operator-lease")
+ .build();
+
+ public static final ConfigOption<Integer> LEADER_ELECTION_LEASE_DURATION_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.leaderElection.leaseDurationSeconds")
+ .enableDynamicOverride(false)
+ .description("Leader election lease duration in seconds, non-negative.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(1200)
+ .build();
+
+ public static final ConfigOption<Integer> LEADER_ELECTION_RENEW_DEADLINE_SECONDS =
+ ConfigOption.<Integer>builder()
+ .key("spark.kubernetes.operator.leaderElection.renewDeadlineSeconds")
+ .enableDynamicOverride(false)
+ .description("Leader election renew deadline in seconds, non-negative.")
+ .typeParameterClass(Integer.class)
+ .defaultValue(600)
Review Comment:
How do we guarantee the relation between `leaseDurationSeconds` and `renewDeadlineSeconds`?
And, could you describe that we need `leaseDurationSeconds` > `renewDeadlineSeconds` to prevent user mistakes?
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/MetricsSystem.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.metrics;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.codahale.metrics.MetricFilter;
+import com.codahale.metrics.MetricRegistry;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.metrics.source.OperatorJvmSource;
+import org.apache.spark.metrics.sink.Sink;
+import org.apache.spark.metrics.source.Source;
+
+@Slf4j
+public class MetricsSystem {
+ private final AtomicBoolean running = new AtomicBoolean(false);
+ @Getter private final Set<Sink> sinks;
+ @Getter private final Set<Source> sources;
+ @Getter private final MetricRegistry registry;
+ @Getter private final Properties properties;
+ // PrometheusPullModelHandler is registered by default, metrics exposed via http port
+ @Getter private final PrometheusPullModelHandler prometheusPullModelHandler;
+ private final Map<String, SinkProps> sinkPropertiesMap;
+
+ public MetricsSystem() {
+ this(new Properties());
+ }
+
+ public MetricsSystem(Properties properties) {
+ this.sources = new HashSet<>();
+ this.sinks = new HashSet<>();
+ this.registry = new MetricRegistry();
+ this.properties = properties;
+ this.sinkPropertiesMap = MetricsSystemFactory.parseSinkProperties(this.properties);
+ // Add default sinks
+ this.prometheusPullModelHandler = new PrometheusPullModelHandler(new Properties(), registry);
+ this.sinks.add(prometheusPullModelHandler);
+ }
+
+ public void start() {
+ if (running.get()) {
+ throw new IllegalStateException(
+ "Attempting to start a MetricsSystem that is already running");
+ }
+ running.set(true);
+ registerDefaultSources();
+ registerSinks();
+ sinks.forEach(Sink::start);
+ }
+
+ public void stop() {
+ if (running.get()) {
+ sinks.forEach(Sink::stop);
+ registry.removeMatching(MetricFilter.ALL);
+ } else {
+ log.error("Stopping a MetricsSystem that is not running");
+ }
+ running.set(false);
+ }
+
+ public void report() {
+ sinks.forEach(Sink::report);
+ }
+
+ protected void registerDefaultSources() {
+ registerSource(new OperatorJvmSource());
+ }
+
+ protected void registerSinks() {
+ log.info("sinkPropertiesMap: {}", sinkPropertiesMap);
+ sinkPropertiesMap
+ .values()
+ .forEach(
+ sinkProp -> {
+ try {
+ Class<Sink> sinkClass = (Class<Sink>) Class.forName(sinkProp.getClassName());
+ Sink sinkInstance;
+ sinkInstance =
+ sinkClass
+ .getConstructor(Properties.class, MetricRegistry.class)
+ .newInstance(sinkProp.getProperties(), registry);
+ sinks.add(sinkInstance);
+ } catch (InstantiationException
+ | IllegalAccessException
+ | IllegalArgumentException
+ | InvocationTargetException
+ | NoSuchMethodException
+ | SecurityException
+ | ClassNotFoundException e) {
+ if (log.isErrorEnabled()) {
+ log.error(
+ "Fail to create metrics sink for sink name {}, sink properties {}",
+ sinkProp.getClassName(),
+ sinkProp.getProperties());
+ }
+ throw new RuntimeException("Fail to create metrics sink", e);
+ }
+ });
+ }
+
+ public void registerSource(Source source) {
+ sources.add(source);
+ try {
+ String regName = MetricRegistry.name(source.sourceName());
+ registry.register(regName, source.metricRegistry());
+ } catch (IllegalArgumentException e) {
+ log.error("Metrics already registered", e);
+ }
+ }
+
+ @Data
+ public static class SinkProps {
Review Comment:
Can we rename `SinkProps` -> `SinkProperties`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48984]Add Controller Metrics System and Utils [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #23:
URL: https://github.com/apache/spark-kubernetes-operator/pull/23#discussion_r1690845284
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -226,9 +226,9 @@ public class SparkOperatorConf {
.defaultValue(-1)
.build();
- public static final ConfigOption<Integer> RETRY_MAX_ATTEMPTS =
+ public static final ConfigOption<Integer> API_RETRY_MAX_ATTEMPTS =
ConfigOption.<Integer>builder()
- .key("spark.kubernetes.operator.retry.maxAttempts")
+ .key("spark.kubernetes.operator.api.retryMaxAttempts")
Review Comment:
New namespace `spark.kubernetes.operator.api` could be misleading.
This is the backend K8s API-related configurations, right? If then, can we rename this more specifically in order to leave a room for `Apache Spark Kubernetes Operator`'s API-related configurations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48984]Add Controller Metrics System and Utils [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #23:
URL: https://github.com/apache/spark-kubernetes-operator/pull/23#issuecomment-2249502073
Thank you for updating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48984]Add Controller Metrics System and Utils [spark-kubernetes-operator]
Posted by "jiangzho (via GitHub)" <gi...@apache.org>.
jiangzho commented on code in PR #23:
URL: https://github.com/apache/spark-kubernetes-operator/pull/23#discussion_r1690622504
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/metrics/MetricsSystem.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.metrics;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.codahale.metrics.MetricFilter;
+import com.codahale.metrics.MetricRegistry;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.metrics.source.OperatorJvmSource;
+import org.apache.spark.metrics.sink.Sink;
+import org.apache.spark.metrics.source.Source;
+
+@Slf4j
+public class MetricsSystem {
+ private final AtomicBoolean running = new AtomicBoolean(false);
+ @Getter private final Set<Sink> sinks;
+ @Getter private final Set<Source> sources;
+ @Getter private final MetricRegistry registry;
+ @Getter private final Properties properties;
+ // PrometheusPullModelHandler is registered by default, metrics exposed via http port
+ @Getter private final PrometheusPullModelHandler prometheusPullModelHandler;
+ private final Map<String, SinkProps> sinkPropertiesMap;
+
+ public MetricsSystem() {
+ this(new Properties());
+ }
+
+ public MetricsSystem(Properties properties) {
+ this.sources = new HashSet<>();
+ this.sinks = new HashSet<>();
+ this.registry = new MetricRegistry();
+ this.properties = properties;
+ this.sinkPropertiesMap = MetricsSystemFactory.parseSinkProperties(this.properties);
+ // Add default sinks
+ this.prometheusPullModelHandler = new PrometheusPullModelHandler(new Properties(), registry);
+ this.sinks.add(prometheusPullModelHandler);
+ }
+
+ public void start() {
+ if (running.get()) {
+ throw new IllegalStateException(
+ "Attempting to start a MetricsSystem that is already running");
+ }
+ running.set(true);
+ registerDefaultSources();
+ registerSinks();
+ sinks.forEach(Sink::start);
+ }
+
+ public void stop() {
+ if (running.get()) {
+ sinks.forEach(Sink::stop);
+ registry.removeMatching(MetricFilter.ALL);
+ } else {
+ log.error("Stopping a MetricsSystem that is not running");
Review Comment:
As recommended by PMD, we enabled according gradle task to [enforce guard log statement](https://pmd.github.io/pmd/pmd_rules_java_bestpractices.html#guardlogstatement) when associate String creation and manipulation. I think this is not reported for no string manipulation
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/SparkExceptionUtils.java:
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.utils;
+
+import io.fabric8.kubernetes.client.KubernetesClientException;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.exception.ExceptionUtils;
+
+public class SparkExceptionUtils {
+ public static boolean isConflictForExistingResource(KubernetesClientException e) {
+ return e != null
+ && e.getCode() == 409
+ && e.getStatus() != null
+ && StringUtils.isNotEmpty(e.getStatus().toString())
+ && e.getStatus().toString().toLowerCase().contains("alreadyexists");
Review Comment:
This is used to handle only specific 409 cases (e.g. request timed out but the resource was actually created, then this help the reconciler to proceed).
For other 409 conflicts - quota or status patch .etc, they are handled separately in according reconcile logic.
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java:
##########
@@ -0,0 +1,463 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.k8s.operator.config;
+
+import java.time.Duration;
+
+import io.javaoperatorsdk.operator.api.config.LeaderElectionConfiguration;
+import io.javaoperatorsdk.operator.processing.event.rate.LinearRateLimiter;
+import io.javaoperatorsdk.operator.processing.event.rate.RateLimiter;
+import io.javaoperatorsdk.operator.processing.retry.GenericRetry;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.spark.k8s.operator.utils.Utils;
+
+/** Spark Operator Configuration options. */
+@Slf4j
+public class SparkOperatorConf {
+ public static final ConfigOption<String> OPERATOR_APP_NAME =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.name")
+ .enableDynamicOverride(false)
+ .description("Name of the operator.")
+ .typeParameterClass(String.class)
+ .defaultValue("spark-kubernetes-operator")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_NAMESPACE =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.namespace")
+ .enableDynamicOverride(false)
+ .description("Namespace that operator is deployed within.")
+ .typeParameterClass(String.class)
+ .defaultValue("default")
+ .build();
+
+ public static final ConfigOption<String> OPERATOR_WATCHED_NAMESPACES =
+ ConfigOption.<String>builder()
+ .key("spark.kubernetes.operator.watchedNamespaces")
+ .enableDynamicOverride(true)
+ .description(
+ "Comma-separated list of namespaces that the operator would be watching for "
+ + "Spark resources. If set to '*' or unset, operator would watch all namespaces.")
Review Comment:
Yep! I was hesitating about the 'unset' (which is under the hood can be considered as not setting namespaces), but indeed it can be quite confusing at properties level. I'd update this to advice user use `*` only instead of unset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48984]Add Controller Metrics System and Utils [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #23:
URL: https://github.com/apache/spark-kubernetes-operator/pull/23#discussion_r1690845960
##########
spark-operator/src/main/java/org/apache/spark/k8s/operator/utils/ClassLoadingUtils.java:
##########
@@ -51,7 +52,11 @@ public class ClassLoadingUtils {
listeners.add((T) listenerClass.getConstructor().newInstance());
}
}
- } catch (Exception e) {
Review Comment:
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org