You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Hyukjin Kwon <gu...@apache.org> on 2026/04/10 01:06:05 UTC
[ANNOUNCE] Announcing Apache Spark 4.2.0-preview4
Hi all,
To enable wide-scale community testing of the upcoming Spark 4.2.0 release,
the Apache Spark community has posted a Spark 4.2.0-preview4 release
<https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4>.
This preview is not a stable release in terms of either API or
functionality,
but it is meant to give the community early access to try the code that will
become Spark 4.2.0. If you would like to test the release, please download
<https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview3> it,
and send feedback using either mailing lists
<https://spark.apache.org/community.html> or JIRA
<https://issues.apache.org/jira/browse/SPARK/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel>.
The documentation is available at the link
<https://spark.apache.org/docs/4.2.0-preview3/>.
We’d like to thank our contributors and users for their contributions and
early feedback to this release. This release would not have been possible
without you.
To download Spark 4.2.0-preview4, head over to the download page:
https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4/
To view the documentation:
https://spark.apache.org/docs/preview/
Re: Regression in Spark 4.2.0-preview4
Posted by Enrico Minack <en...@gmx.de>.
Dongjoon,
Thanks for fixing this!
https://github.com/apache/spark/pull/55355
Enrico
Am 15.04.26 um 18:22 schrieb Dongjoon Hyun:
> `spark-app-selector` works fine in our CIs and my environment like the following. Could you double-check your environment, Enrico?
>
>
> $ bin/docker-image-tool.sh -r docker.io/apache -t 4.2.0-preview4 build
>
>
> $ bin/spark-submit \
> --master k8s://$K8S_MASTER \
> --deploy-mode cluster \
> -c spark.kubernetes.container.image=apache/spark:4.2.0-preview4 \
> -c spark.kubernetes.authenticate.driver.serviceAccountName=spark \
> --class org.apache.spark.examples.SparkPi \
> local:///opt/spark/examples/jars/spark-examples.jar 300000
>
>
> $ kubectl get pods --show-labels
> NAME READY STATUS RESTARTS AGE LABELS
> org-apache-spark-examples-sparkpi-8fea449d91f10411-driver 1/1 Running 0 6s spark-app-name=org-apache-spark-examples-sparkpi,spark-app-selector=spark-e88adf6c93e64834a26bd9162034f34b,spark-role=driver,spark-version=4.2.0-preview4
> spark-pi-e33c809d91f10f25-exec-1 1/1 Running 0 3s spark-app-name=spark-pi,spark-app-selector=spark-e88adf6c93e64834a26bd9162034f34b,spark-exec-id=1,spark-exec-resourceprofile-id=0,spark-role=executor,spark-version=4.2.0-preview4
> spark-pi-e33c809d91f10f25-exec-2 1/1 Running 0 3s spark-app-name=spark-pi,spark-app-selector=spark-e88adf6c93e64834a26bd9162034f34b,spark-exec-id=2,spark-exec-resourceprofile-id=0,spark-role=executor,spark-version=4.2.0-preview4
>
>
> Dongjoon.
>
> On 2026/04/14 09:04:44 Enrico Minack wrote:
>> Hi Spark-devs,
>>
>> Looks like 4.2.0-preview4 is affected by regression SPARK-56238.
>>
>> Building a docker image from 4.2.0-preview4 and running a Spark job on
>> Kubernetes fails due to a Spark app id mismatch in the selector.
>>
>> Setting an app id explicitly fixes this mismatch: --conf spark.app.id=abc
>>
>> Build Spark binaries and Docker image, load image into minikube:
>>
>> cd spark-4.2.0-preview4-bin-hadoop3
>> ./bin/docker-image-tool.sh -t "latest" build
>> docker save spark:latest -o spark.tar
>> minikube image load spark.tar
>>
>> Run Spark Shell:
>>
>> echo"spark.range(10).mapPartitions { it => Thread.sleep(60000); it }.collect" | ./bin/spark-shell --master k8s://https://192.168.49.2:8443--conf
>> spark.kubernetes.container.image=spark:latest --conf
>> spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa --conf
>> spark.kubernetes.executor.missingPodDetectDelta=1000
>>
>> The driver says:
>>
>> Spark context available as'sc' (master = k8s://https://192.168.49.2:8443, app id =
>> spark-c0f802279d8146c29b1ef3467b467590).
>>
>> The executor pods say:
>>
>> kubectl describe"pod/$(kubectl get pods | grep exec | grep Running | head -n 1 | cut -d " " -f 1)" | grep selec
>> spark-app-selector=spark-6c8d69be4f00410b884fd6e6417b872a
>>
>> this mismatch causes:
>>
>> 26/03/26 11:40:29 ERROR dispatcher-CoarseGrainedScheduler org.apache.spark.scheduler.TaskSchedulerImpl: Lost executor 1 on 10.244.0.55: The executor with ID 1 (registered at 1774521602640 ms) was not found in the cluster at the polling time (1774521629051 ms) which is after the accepted detect delta time (1000 ms) configured by `spark.kubernetes.executor.missingPodDetectDelta`. The executor may have been deleted but the driver missed the deletion event. Markingthis executor as failed.
>>
>>
>> Either the app id mismatch should be fixed, or the app id should be
>> mandatory in this setup.
>>
>> Cheers,
>> Enrico
>>
>>
>> Am 10.04.26 um 03:06 schrieb Hyukjin Kwon:
>>> Hi all,
>>>
>>> To enable wide-scale community testing of the upcoming Spark 4.2.0
>>> release,
>>> the Apache Spark community has posted a Spark 4.2.0-preview4 release
>>> <https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4>.
>>> This preview is not a stable release in terms of either API or
>>> functionality,
>>> but it is meant to give the community early access to try the code
>>> that will
>>> become Spark 4.2.0. If you would like to test the release, please
>>> download
>>> <https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview3> it,
>>> and send feedback using either mailing lists
>>> <https://spark.apache.org/community.html> or JIRA
>>> <https://issues.apache.org/jira/browse/SPARK/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel>.
>>> The documentation is available at the link
>>> <https://spark.apache.org/docs/4.2.0-preview3/>.
>>>
>>> We’d like to thank our contributors and users for their contributions
>>> and early feedback to this release. This release would not have been
>>> possible without you.
>>>
>>> To download Spark 4.2.0-preview4, head over to the download page:
>>> https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4/
>>> <https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4/>
>>>
>>> To view the documentation:
>>> https://spark.apache.org/docs/preview/
>>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Regression in Spark 4.2.0-preview4
Posted by Dongjoon Hyun <do...@apache.org>.
`spark-app-selector` works fine in our CIs and my environment like the following. Could you double-check your environment, Enrico?
$ bin/docker-image-tool.sh -r docker.io/apache -t 4.2.0-preview4 build
$ bin/spark-submit \
--master k8s://$K8S_MASTER \
--deploy-mode cluster \
-c spark.kubernetes.container.image=apache/spark:4.2.0-preview4 \
-c spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--class org.apache.spark.examples.SparkPi \
local:///opt/spark/examples/jars/spark-examples.jar 300000
$ kubectl get pods --show-labels
NAME READY STATUS RESTARTS AGE LABELS
org-apache-spark-examples-sparkpi-8fea449d91f10411-driver 1/1 Running 0 6s spark-app-name=org-apache-spark-examples-sparkpi,spark-app-selector=spark-e88adf6c93e64834a26bd9162034f34b,spark-role=driver,spark-version=4.2.0-preview4
spark-pi-e33c809d91f10f25-exec-1 1/1 Running 0 3s spark-app-name=spark-pi,spark-app-selector=spark-e88adf6c93e64834a26bd9162034f34b,spark-exec-id=1,spark-exec-resourceprofile-id=0,spark-role=executor,spark-version=4.2.0-preview4
spark-pi-e33c809d91f10f25-exec-2 1/1 Running 0 3s spark-app-name=spark-pi,spark-app-selector=spark-e88adf6c93e64834a26bd9162034f34b,spark-exec-id=2,spark-exec-resourceprofile-id=0,spark-role=executor,spark-version=4.2.0-preview4
Dongjoon.
On 2026/04/14 09:04:44 Enrico Minack wrote:
> Hi Spark-devs,
>
> Looks like 4.2.0-preview4 is affected by regression SPARK-56238.
>
> Building a docker image from 4.2.0-preview4 and running a Spark job on
> Kubernetes fails due to a Spark app id mismatch in the selector.
>
> Setting an app id explicitly fixes this mismatch: --conf spark.app.id=abc
>
> Build Spark binaries and Docker image, load image into minikube:
>
> cd spark-4.2.0-preview4-bin-hadoop3
> ./bin/docker-image-tool.sh -t "latest" build
> docker save spark:latest -o spark.tar
> minikube image load spark.tar
>
> Run Spark Shell:
>
> echo"spark.range(10).mapPartitions { it => Thread.sleep(60000); it }.collect" | ./bin/spark-shell --master k8s://https://192.168.49.2:8443--conf
> spark.kubernetes.container.image=spark:latest --conf
> spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa --conf
> spark.kubernetes.executor.missingPodDetectDelta=1000
>
> The driver says:
>
> Spark context available as'sc' (master = k8s://https://192.168.49.2:8443, app id =
> spark-c0f802279d8146c29b1ef3467b467590).
>
> The executor pods say:
>
> kubectl describe"pod/$(kubectl get pods | grep exec | grep Running | head -n 1 | cut -d " " -f 1)" | grep selec
> spark-app-selector=spark-6c8d69be4f00410b884fd6e6417b872a
>
> this mismatch causes:
>
> 26/03/26 11:40:29 ERROR dispatcher-CoarseGrainedScheduler org.apache.spark.scheduler.TaskSchedulerImpl: Lost executor 1 on 10.244.0.55: The executor with ID 1 (registered at 1774521602640 ms) was not found in the cluster at the polling time (1774521629051 ms) which is after the accepted detect delta time (1000 ms) configured by `spark.kubernetes.executor.missingPodDetectDelta`. The executor may have been deleted but the driver missed the deletion event. Markingthis executor as failed.
>
>
> Either the app id mismatch should be fixed, or the app id should be
> mandatory in this setup.
>
> Cheers,
> Enrico
>
>
> Am 10.04.26 um 03:06 schrieb Hyukjin Kwon:
> > Hi all,
> >
> > To enable wide-scale community testing of the upcoming Spark 4.2.0
> > release,
> > the Apache Spark community has posted a Spark 4.2.0-preview4 release
> > <https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4>.
> > This preview is not a stable release in terms of either API or
> > functionality,
> > but it is meant to give the community early access to try the code
> > that will
> > become Spark 4.2.0. If you would like to test the release, please
> > download
> > <https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview3> it,
> > and send feedback using either mailing lists
> > <https://spark.apache.org/community.html> or JIRA
> > <https://issues.apache.org/jira/browse/SPARK/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel>.
> > The documentation is available at the link
> > <https://spark.apache.org/docs/4.2.0-preview3/>.
> >
> > We’d like to thank our contributors and users for their contributions
> > and early feedback to this release. This release would not have been
> > possible without you.
> >
> > To download Spark 4.2.0-preview4, head over to the download page:
> > https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4/
> > <https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4/>
> >
> > To view the documentation:
> > https://spark.apache.org/docs/preview/
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Regression in Spark 4.2.0-preview4
Posted by Enrico Minack <en...@gmx.de>.
Hi Spark-devs,
Looks like 4.2.0-preview4 is affected by regression SPARK-56238.
Building a docker image from 4.2.0-preview4 and running a Spark job on
Kubernetes fails due to a Spark app id mismatch in the selector.
Setting an app id explicitly fixes this mismatch: --conf spark.app.id=abc
Build Spark binaries and Docker image, load image into minikube:
cd spark-4.2.0-preview4-bin-hadoop3
./bin/docker-image-tool.sh -t "latest" build
docker save spark:latest -o spark.tar
minikube image load spark.tar
Run Spark Shell:
echo"spark.range(10).mapPartitions { it => Thread.sleep(60000); it }.collect" | ./bin/spark-shell --master k8s://https://192.168.49.2:8443--conf
spark.kubernetes.container.image=spark:latest --conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa --conf
spark.kubernetes.executor.missingPodDetectDelta=1000
The driver says:
Spark context available as'sc' (master = k8s://https://192.168.49.2:8443, app id =
spark-c0f802279d8146c29b1ef3467b467590).
The executor pods say:
kubectl describe"pod/$(kubectl get pods | grep exec | grep Running | head -n 1 | cut -d " " -f 1)" | grep selec
spark-app-selector=spark-6c8d69be4f00410b884fd6e6417b872a
this mismatch causes:
26/03/26 11:40:29 ERROR dispatcher-CoarseGrainedScheduler org.apache.spark.scheduler.TaskSchedulerImpl: Lost executor 1 on 10.244.0.55: The executor with ID 1 (registered at 1774521602640 ms) was not found in the cluster at the polling time (1774521629051 ms) which is after the accepted detect delta time (1000 ms) configured by `spark.kubernetes.executor.missingPodDetectDelta`. The executor may have been deleted but the driver missed the deletion event. Markingthis executor as failed.
Either the app id mismatch should be fixed, or the app id should be
mandatory in this setup.
Cheers,
Enrico
Am 10.04.26 um 03:06 schrieb Hyukjin Kwon:
> Hi all,
>
> To enable wide-scale community testing of the upcoming Spark 4.2.0
> release,
> the Apache Spark community has posted a Spark 4.2.0-preview4 release
> <https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4>.
> This preview is not a stable release in terms of either API or
> functionality,
> but it is meant to give the community early access to try the code
> that will
> become Spark 4.2.0. If you would like to test the release, please
> download
> <https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview3> it,
> and send feedback using either mailing lists
> <https://spark.apache.org/community.html> or JIRA
> <https://issues.apache.org/jira/browse/SPARK/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel>.
> The documentation is available at the link
> <https://spark.apache.org/docs/4.2.0-preview3/>.
>
> We’d like to thank our contributors and users for their contributions
> and early feedback to this release. This release would not have been
> possible without you.
>
> To download Spark 4.2.0-preview4, head over to the download page:
> https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4/
> <https://dist.apache.org/repos/dist/release/spark/spark-4.2.0-preview4/>
>
> To view the documentation:
> https://spark.apache.org/docs/preview/