You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Chandni Singh (Jira)" <ji...@apache.org> on 2023/04/19 05:24:00 UTC
[jira] [Updated] (SPARK-43179) Add option for applications to control saving of metadata in External Shuffle Service LevelDB
[ https://issues.apache.org/jira/browse/SPARK-43179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chandni Singh updated SPARK-43179:
----------------------------------
Summary: Add option for applications to control saving of metadata in External Shuffle Service LevelDB (was: Allow applications to control whether their metadata gets saved by the shuffle server in the db)
> Add option for applications to control saving of metadata in External Shuffle Service LevelDB
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-43179
> URL: https://issues.apache.org/jira/browse/SPARK-43179
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 3.4.0
> Reporter: Chandni Singh
> Priority: Major
>
> Currently, the External Shuffle Service stores application metadata in LevelDB. This is necessary to enable the shuffle server to resume serving shuffle data for an application whose executors registered before the NodeManager restarts. However, the metadata includes the application secret, which is stored in LevelDB without encryption. This is a potential security risk, particularly for applications with high security requirements. While filesystem access control lists (ACLs) can help protect keys and certificates, they may not be sufficient for some use cases. In response, we have decided not to store metadata for these high-security applications in LevelDB. As a result, these applications may experience more failures in the event of a node restart, but we believe this trade-off is acceptable given the increased security risk.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org