You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Yeachan Park <ye...@gmail.com> on 2022/04/21 09:01:01 UTC

[Spark Core]: Unexpectedly exiting executor while gracefully decommissioning

Hello all, we are running into some issues while attempting graceful
decommissioning of executors. We are running spark-thriftserver (3.2.0) on
Kubernetes (GKE 1.20.15-gke.2500). We enabled:

   - spark.decommission.enabled
   - spark.storage.decommission.rddBlocks.enabled
   - spark.storage.decommission.shuffleBlocks.enabled
   - spark.storage.decommission.enabled

and set spark.storage.decommission.fallbackStorage.path to a path in our
bucket.

The logs from the driver seems to suggest the decommissioning process
started but then unexpectedly exited and failed while the executor logs
seem to suggest that decommissioning was successful.

Attached are the error logs:

https://gist.github.com/yeachan153/9bfb2f0ab9ac7f292fb626186b014bbf


Thanks in advance.