You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Holden Karau (Jira)" <ji...@apache.org> on 2022/05/02 19:31:00 UTC
[jira] [Commented] (SPARK-38158) Rdd and shuffle blocks not migrated to new executors when decommission feature is enabled
[ https://issues.apache.org/jira/browse/SPARK-38158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17530916#comment-17530916 ]
Holden Karau commented on SPARK-38158:
--------------------------------------
Do you have executor logs?
> Rdd and shuffle blocks not migrated to new executors when decommission feature is enabled
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-38158
> URL: https://issues.apache.org/jira/browse/SPARK-38158
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Kubernetes
> Affects Versions: 3.2.0
> Environment: Spark on k8s
> Reporter: Mohan Patidar
> Priority: Major
> Attachments: sparkConf_Decommission_to_alternate_storage.conf, sparkConf_Decommission_to_executors.conf
>
>
> We’re using Spark 3.2.0 and we have enabled the spark decommission feature. As part of validating this feature, we wanted to check if the rdd blocks and shuffle blocks from the decommissioned executors are migrated to other executors.
> However, we could not see this happening. Below is the configuration we used.
> # *Spark Configuration used:*
> ** spark.local.dir /mnt/spark-ldir
> spark.decommission.enabled true
> spark.storage.decommission.enabled true
> spark.storage.decommission.rddBlocks.enabled true
> spark.storage.decommission.shuffleBlocks.enabled true
> spark.dynamicAllocation.enabled true
> # *Brought up spark-driver and executors on the different nodes.*
> NAME READY STATUS NODE
> decommission-driver 1/1 Running *Node1*
> gzip-compression-test-ae0b0b7e4d7fbe40-exec-1 1/1 Running *Node1*
> gzip-compression-test-ae0b0b7e4d7fbe40-exec-2 1/1 Running *Node2*
> gzip-compression-test-ae0b0b7e4d7fbe40-exec-3 1/1 Running *Node1*
> gzip-compression-test-ae0b0b7e4d7fbe40-exec-4 1/1 Running *Node2*
> gzip-compression-test-ae0b0b7e4d7fbe40-exec-5 1/1 Running *Node1*
> # *Bringdown Node2 so status of pods as are following.*
> NAME READY STATUS NODE
> decommission-driver 1/1 Running *Node1*
> gzip-compression-test-ae0b0b7e4d7fbe40-exec-1 1/1 Running *Node1*
> gzip-compression-test-ae0b0b7e4d7fbe40-exec-2 1/1 Terminating *Node2*
> gzip-compression-test-ae0b0b7e4d7fbe40-exec-3 1/1 Running *Node1*
> gzip-compression-test-ae0b0b7e4d7fbe40-exec-4 1/1 Terminating *Node2*
> gzip-compression-test-ae0b0b7e4d7fbe40-exec-5 1/1 Running *Node1*
> # *Driver logs:*
> {"type":"log", "level":"INFO", "time":"2022-01-12T08:55:28.296Z", "timezone":"UTC", "log":"Adding decommission script to lifecycle"}
> {"type":"log", "level":"INFO", "time":"2022-01-12T08:55:28.459Z", "timezone":"UTC", "log":"Adding decommission script to lifecycle"}
> {"type":"log", "level":"INFO", "time":"2022-01-12T08:55:28.564Z", "timezone":"UTC", "log":"Adding decommission script to lifecycle"}
> {"type":"log", "level":"INFO", "time":"2022-01-12T08:55:28.601Z", "timezone":"UTC", "log":"Adding decommission script to lifecycle"}
> {"type":"log", "level":"INFO", "time":"2022-01-12T08:55:28.667Z", "timezone":"UTC", "log":"Adding decommission script to lifecycle"}
> {"type":"log", "level":"INFO", "time":"2022-01-12T08:58:21.885Z", "timezone":"UTC", "log":"Notify executor 5 to decommissioning."}
> {"type":"log", "level":"INFO", "time":"2022-01-12T08:58:21.887Z", "timezone":"UTC", "log":"Notify executor 1 to decommissioning."}
> {"type":"log", "level":"INFO", "time":"2022-01-12T08:58:21.887Z", "timezone":"UTC", "log":"Notify executor 3 to decommissioning."}
> {"type":"log", "level":"INFO", "time":"2022-01-12T08:58:21.887Z", "timezone":"UTC", "log":"Mark BlockManagers (BlockManagerId(5, X.X.X.X, 33359, None), BlockManagerId(1, X.X.X.X, 38655, None), BlockManagerId(3, X.X.X.X, 35797, None)) as being decommissioning."}
> {"type":"log", "level":"INFO", "time":"2022-01-12T08:59:24.426Z", "timezone":"UTC", "log":"Executor 2 is removed. Remove reason statistics: (gracefully decommissioned: 0, decommision unfinished: 0, driver killed: 0, unexpectedly exited: 1)."}
> {"type":"log", "level":"INFO", "time":"2022-01-12T08:59:24.426Z", "timezone":"UTC", "log":"Executor 4 is removed. Remove reason statistics: (gracefully decommissioned: 0, decommision unfinished: 0, driver killed: 0, unexpectedly exited: 2)."}
> # *Verified by Execute into all live executors(1,3,5) and checked at location (/mnt/spark-ldir/) so only one blockManger id present, not seeing any other blockManager id copied to this location.*
> Example:
> \{*}${*}kubectl exec -it gzip-compression-test-ae0b0b7e4d7fbe40-exec-1 -n test bash
> $cd /mnt/spark-ldir/
> $ blockmgr-60872c99-e7d6-43ba-a43e-a97fc9f619ca
>
> Since the migration was not happening, we tried to use fallback storage option by specifying the hdfs storage. But unfortunately we could not see the rdd and shuffle blocks in this fallback storage location as well. Below is the configuration we used.
> **
> # *Spark Configuration Used:*
> spark.decommission.enabled true
> spark.storage.decommission.enabled true
> spark.storage.decommission.rddBlocks.enabled true
> spark.storage.decommission.shuffleBlocks.enabled true
> spark.storage.decommission.fallbackStorage.path hdfs://namenodeHA/tmp/fallbackstorage
> spark.dynamicAllocation.enabled true
> # *Brought up one spark-driver and one executor on the different nodes.*
> NAME READY NODE
> decommission-driver 1/1 Node1
> gzip-compression-test-49acf67e679f9259-exec-1 1/1 Node2
>
> # ** *Bringdown Node2 so status of pods as are following.*
> Example:
> NAME READY STATUS
> decommission-driver 1/1 Running
> gzip-compression-test-49acf67e679f9259-exec-1 1/1 Running
> gzip-compression-test-49acf67e679f9259-exec-1 1/1 Running
> gzip-compression-test-49acf67e679f9259-exec-1 1/1 Terminating
> #
> *Verified data migration on that storage fallback location:*
> Example:
> $ hdfs dfs -ls /tmp/fallbackstorage
> *Note:* still empty this location.
> # *Driver logs is here.*
> *related to fallback*
> \{"type":"log", "level":"INFO", "time":"2022-01-17T10:40:21.682Z", "timezone":"UTC", "log":"Registering BlockManager BlockManagerId(fallback, remote, 7337, None)"}
> \{"type":"log", "level":"INFO", "time":"2022-01-17T10:40:21.682Z", "timezone":"UTC", "log":"Registering block manager remote:7337 with 0.0 B RAM, BlockManagerId(fallback, remote, 7337, None)"}
> \{"type":"log", "level":"INFO", "time":"2022-01-17T10:40:21.682Z", "timezone":"UTC", "log":"Registered BlockManager BlockManagerId(fallback, remote, 7337, None)"}
> *related to decommissioning*
> ** {"type":"log", "level":"INFO", "time":"2022-01-17T10:40:21.661Z", "timezone":"UTC", "log":"Adding decommission script to lifecycle"}
> \{"type":"log", "level":"INFO", "time":"2022-01-17T10:46:17.952Z", "timezone":"UTC", "log":"Executor 1 is removed. Remove reason statistics: (gracefully decommissioned: 0, decommision unfinished: 0, driver killed: 0, unexpectedly exited: 1)."}
> {*}Note:{*}- Configuration files for both the scenarios are attached.
> Please Let us know if we are missing anything which is stopping the migration of rdd and shuffle blocks.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org