You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/10/12 08:40:00 UTC

[jira] [Commented] (SPARK-40464) Support automatic data format conversion for shuffle state db

    [ https://issues.apache.org/jira/browse/SPARK-40464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17616278#comment-17616278 ] 

Apache Spark commented on SPARK-40464:
--------------------------------------

User 'panbingkun' has created a pull request for this issue:
https://github.com/apache/spark/pull/38220

> Support automatic data format conversion for shuffle state db
> -------------------------------------------------------------
>
>                 Key: SPARK-40464
>                 URL: https://issues.apache.org/jira/browse/SPARK-40464
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core, YARN
>    Affects Versions: 3.4.0
>            Reporter: Yang Jie
>            Priority: Minor
>
> After SPARK-38888,  Spark starts to support use RocksDB as the storage type of shuffle state db, but when users switch storage types, the original data store in `LevelDB/RocksDB` will not be automatically converted to another kind of storage now. The original data store will be retained and the new type data store will be  created when switching storage types.
> This is not very friendly for migrating stock users to use RocksDB. If necessary, consider adding the function of automatic data format conversion:
>  * LevelDB -> RocksDB
>  * RocksDB -> LevelDB
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org