[jira] [Resolved] (SPARK-58649) Allow SparkApplication drivers ingress to SparkCluster workers

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun resolved SPARK-58649.
-----------------------------------
    Fix Version/s: kubernetes-operator-1.1.0
       Resolution: Fixed

Issue resolved by pull request 787
[https://github.com/apache/spark-kubernetes-operator/pull/787]

> Allow SparkApplication drivers ingress to SparkCluster workers  
> ----------------------------------------------------------------
>
>                 Key: SPARK-58649
>                 URL: https://issues.apache.org/jira/browse/SPARK-58649
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes
>    Affects Versions: kubernetes-operator-1.0.0
>         Environment:   
>            Reporter: Yannick Goetschel
>            Assignee: Yannick Goetschel
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: kubernetes-operator-1.1.0
>
>
> We run per-user Spark Connect sessions as SparkApplications attached to a shared
>   standalone SparkCluster. Most jobs are fine, but every so often one dies like this:
> {code:java}
> Job aborted due to stage failure: Task 11 in stage 2.0 failed 4 times, most recent
> failure: Lost task 11.3 in stage 2.0 (TID 86) (<executor-ip> executor 3):
> TaskResultLost (result lost from block manager)
> {code}
> We run per-user Spark Connect sessions as SparkApplications attached to a shared
> standalone SparkCluster. Most jobs are fine, but every so often one dies like this:
> {code:java}
>   Job aborted due to stage failure: Task 11 in stage 2.0 failed 4 times, most recent
>   failure: Lost task 11.3 in stage 2.0 (TID 86) (<executor-ip> executor 3):
>   TaskResultLost (result lost from block manager)
> {code}
> The driver log explains why only some of them:
> {code:java}
>  java.io.IOException: Connecting to /<executor-ip>:<ephemeral-port> timed out (120000 ms)
>         at org.apache.spark.network.client.TransportClientFactory.createClient(...)
>         at org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(...)
>         at org.apache.spark.network.shuffle.RetryingBlockTransferor.transferAllOutstanding(...)
>         at org.apache.spark.storage.BlockManager.fetchRemoteManagedBuffer(...)
>         at org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(...)
>  {code}
> The driver cannot reach the executors' block manager. That only matters once a task
> result goes over {{spark.task.maxDirectResultSize}} (1 MB): below it the result rides
> along inline in the RPC, above it the executor parks the result in its block manager and
> hands the driver an {{IndirectTaskResult}} to go and fetch. The tasks that stayed under
> the limit never noticed anything. The three that went over timed out, retried three
> times and gave up, and the stage aborted on the fourth attempt.
> The culprit is the worker NetworkPolicy the operator generates for every SparkCluster in
> {{SparkClusterResourceSpec#buildWorkerNetworkPolicy}}:
> {code:yaml}
>   podSelector:
>     matchLabels: {spark-role: worker, spark.operator/spark-cluster-name: <cluster>}
>   ingress:
>   - from:
>     - podSelector:
>         matchLabels: {spark.operator/spark-cluster-name: <cluster>}
>   policyTypes: [Ingress]
>   {code}
> Only pods carrying the cluster label may talk to the workers. But a SparkApplication
> attached to the cluster is a separate resource: its driver is labelled
> {{spark-role: driver}} and {{spark.operator/spark-app-name: <app>}}, never the cluster
> label. So the driver is locked out of its own executors.
> The executor-to-driver direction works perfectly well —
> executors register, tasks get dispatched, tasks run. Only the return path for large
> results is broken, so the cluster looks healthy right up until a job happens to cross
> the 1 MB threshold.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org