You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2025/06/14 20:36:13 UTC
[PR] [SPARK-52481] Add `Spark History Server` example [spark-kubernetes-operator]
dongjoon-hyun opened a new pull request, #249:
URL: https://github.com/apache/spark-kubernetes-operator/pull/249
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, use `Draft` feature of GitHub Action.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If there is design documentation, please add the link.
3. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52481] Add `Spark History Server` example [spark-kubernetes-operator]
Posted by "melin (via GitHub)" <gi...@apache.org>.
melin commented on PR #249:
URL: https://github.com/apache/spark-kubernetes-operator/pull/249#issuecomment-2973419689
The Spark event log cannot be directly written to s3. There are the following errors:
WARN S3ABlockOutputStream: Application invoked the Syncable API against stream writing to logs/spark/events/eventlog_v2_spark-61bc831c6f774629a1cd1bc95d6a7288/events_1_spark-61bc831c6f774629a1cd1bc95d6a7288. This is unsupported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52481] Add `Spark History Server` example [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #249:
URL: https://github.com/apache/spark-kubernetes-operator/pull/249#issuecomment-2977384662
@melin , it's a warning, not an error.
> The Spark event log cannot be directly written to s3. There are the following errors:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52481] Add `Spark History Server` example [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #249:
URL: https://github.com/apache/spark-kubernetes-operator/pull/249#issuecomment-3020509196
Here is the PR, @Ferdinanddb .
- #264
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52481] Add `Spark History Server` example [spark-kubernetes-operator]
Posted by "Ferdinanddb (via GitHub)" <gi...@apache.org>.
Ferdinanddb commented on PR #249:
URL: https://github.com/apache/spark-kubernetes-operator/pull/249#issuecomment-3020482176
Hi @dongjoon-hyun , thank you for your work here.
I am trying to deploy a spark-history-server pod using your example, and I see it as running but I cannot port-forward it since the port 18080 is not being used by the pod (and so not used by the service).
Could you please tell me what should I do to be able to fix my issue?
I have the following config:
```yaml
apiVersion: spark.apache.org/v1
kind: SparkApplication
metadata:
name: spark-history-server
namespace: spark-operator
spec:
mainClass: "org.apache.spark.deploy.history.HistoryServer"
sparkConf:
# spark.jars.packages: "org.apache.hadoop:hadoop-aws:3.4.1"
spark.jars: "https://repo1.maven.org/maven2/com/google/cloud/bigdataoss/gcs-connector/3.1.1/gcs-connector-3.1.1-shaded.jar"
spark.jars.ivy: "/tmp/.ivy2.5.2"
spark.driver.memory: "2g"
spark.kubernetes.namespace: spark-operator
spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
spark.kubernetes.container.image: "apache/spark:4.0.0-java21-scala"
spark.history.fs.cleaner.enabled: "true"
spark.history.fs.cleaner.maxAge: "30d"
spark.history.fs.cleaner.maxNum: "100"
spark.history.fs.eventLog.rolling.maxFilesToRetain: "10"
spark.hadoop.fs.gs.impl: "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"
spark.hadoop.fs.AbstractFileSystem.gs.impl: "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS"
spark.hadoop.fs.gs.auth.service.account.enable: "true"
# spark.hadoop.fs.gs.auth.type: "COMPUTE_ENGINE"
spark.hadoop.fs.gs.project.id: "rcim-prod-data-core-0"
spark.hadoop.fs.defaultFS: "gs://SOME-BUCKET-GCS-spark-logs"
spark.history.fs.logDirectory: "gs://SOME-BUCKET-GCS-spark-logs"
runtimeVersions:
sparkVersion: "4.0.0"
applicationTolerations:
restartConfig:
restartPolicy: Always
maxRestartAttempts: 9223372036854775807
```
I changed the value of my GCS bucket name, but again: I don't see any errors in the pod's logs.
When I execute the following port-forward command I get this:
```
kubectl port-forward svc/spark-history-server-0-driver-svc 8080:spark-ui --namespace spark-operator
Forwarding from 127.0.0.1:8080 -> 4040
Forwarding from [::1]:8080 -> 4040
Handling connection for 8080
E0630 21:06:51.716348 2639361 portforward.go:424] "Unhandled Error" err="an error occurred forwarding 8080 -> 4040: error forwarding port 4040 to pod 76c12381718e2542ae6e371d8c28992dc9e541439cd53b832df5aa5aaaeddbac, uid : failed to execute portforward in network namespace \"/var/run/netns/cni-5208164a-41a3-2769-bfaf-6b6f31eb1c80\": failed to connect to localhost:4040 inside namespace \"76c12381718e2542ae6e371d8c28992dc9e541439cd53b832df5aa5aaaeddbac\", IPv4: dial tcp4 127.0.0.1:4040: connect: connection refused IPv6 dial tcp6 [::1]:4040: connect: cannot assign requested address "
```
Thank you very much if you can help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52481] Add `Spark History Server` example [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #249:
URL: https://github.com/apache/spark-kubernetes-operator/pull/249#issuecomment-2980831031
> > @melin , it's a warning, not an error.
> > > The Spark event log cannot be directly written to s3. There are the following errors:
> >
> >
> > FYI, Apache Spark 3.2+ has the following by default.
> >
> > * [[SPARK-35868][CORE] Add fs.s3a.downgrade.syncable.exceptions if not set spark#33044](https://github.com/apache/spark/pull/33044)
>
> spark.eventLog.dir cannot be directly set to s3 path, which is not very convenient. There are two solutions:
>
> 1. Write directly to the nfs shared storage
> 2. First, write to the pod locally. Once the task is completed, upload it to s3.
It seems that the above are not questions.
So, to be clear, No, your personal assessments and proposed solutions sounds wrong to me, @melin . It doesn't make sense at all for long running jobs and streaming jobs. In other words, that's not a community recommendation.
Apache Spark community has been using S3 directly as an Apache Spark event log directory for a long time.
BTW, Apache Spark PR is not a good place for this kind of Q&A. I'd recommend you to send to [dev@spark.apache.org](https://lists.apache.org/list.html?dev@spark.apache.org) if you have any difficulties or you want to discuss this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52481] Add `Spark History Server` example [spark-kubernetes-operator]
Posted by "melin (via GitHub)" <gi...@apache.org>.
melin commented on PR #249:
URL: https://github.com/apache/spark-kubernetes-operator/pull/249#issuecomment-2978572765
> @melin , it's a warning, not an error.
>
> > The Spark event log cannot be directly written to s3. There are the following errors:
>
> FYI, Apache Spark 3.2+ has the following by default.
>
> * [[SPARK-35868][CORE] Add fs.s3a.downgrade.syncable.exceptions if not set spark#33044](https://github.com/apache/spark/pull/33044)
spark.eventLog.dir cannot be directly set to s3 path, which is not very convenient. There are two solutions:
1. Write directly to the nfs shared storage
2. First, write to the pod locally. Once the task is completed, upload it to s3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52481] Add `Spark History Server` example [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #249:
URL: https://github.com/apache/spark-kubernetes-operator/pull/249#issuecomment-3020501759
You need to use `spark.ui.port`. Let me revise the example for you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52481] Add `Spark History Server` example [spark-kubernetes-operator]
Posted by "Ferdinanddb (via GitHub)" <gi...@apache.org>.
Ferdinanddb commented on PR #249:
URL: https://github.com/apache/spark-kubernetes-operator/pull/249#issuecomment-3020522574
@dongjoon-hyun thanks for the taking the time for this! I confirm that it now works :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52481] Add `Spark History Server` example [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #249:
URL: https://github.com/apache/spark-kubernetes-operator/pull/249#discussion_r2147300075
##########
examples/spark-history-server.yaml:
##########
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: spark.apache.org/v1beta1
+kind: SparkApplication
+metadata:
+ name: spark-history-server
+spec:
+ mainClass: "org.apache.spark.deploy.history.HistoryServer"
+ sparkConf:
+ spark.jars.packages: "org.apache.hadoop:hadoop-aws:3.4.1"
+ spark.jars.ivy: "/tmp/.ivy2.5.2"
+ spark.driver.memory: "2g"
+ spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
+ spark.kubernetes.container.image: "apache/spark:4.0.0-java21-scala"
+ spark.history.fs.logDirectory: "s3a://spark-events"
+ spark.history.fs.cleaner.enabled: "true"
+ spark.history.fs.cleaner.maxAge: "30d"
+ spark.history.fs.cleaner.maxNum: "100"
+ spark.history.fs.eventLog.rolling.maxFilesToRetain: "10"
+ spark.hadoop.fs.defaultFS: "s3a://spark-events"
+ spark.hadoop.fs.s3a.endpoint: "http://localstack:4566"
+ spark.hadoop.fs.s3a.path.style.access: "true"
+ spark.hadoop.fs.s3a.access.key: "test"
+ spark.hadoop.fs.s3a.secret.key: "test"
+ runtimeVersions:
+ sparkVersion: "4.0.0"
+ applicationTolerations:
+ restartConfig:
+ restartPolicy: Always
+ maxRestartAttempts: 9223372036854775807
Review Comment:
Since the default value of `maxRestartAttempts` is 3 (for normal jobs), I set this as `Long.MAX_VALUE` because SHS is supposed to run forever.
```bash
$ jshell
| Welcome to JShell -- Version 21.0.7
| For an introduction type: /help intro
jshell> Long.MAX_VALUE
$1 ==> 9223372036854775807
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52481] Add `Spark History Server` example [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #249:
URL: https://github.com/apache/spark-kubernetes-operator/pull/249#issuecomment-2973333062
Thank you, @viirya . Merged to main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52481] Add `Spark History Server` example [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #249: [SPARK-52481] Add `Spark History Server` example
URL: https://github.com/apache/spark-kubernetes-operator/pull/249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52481] Add `Spark History Server` example [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #249:
URL: https://github.com/apache/spark-kubernetes-operator/pull/249#issuecomment-2981015087
FYI, @melin . Here is the example.
- https://github.com/apache/spark-kubernetes-operator/pull/250
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org