[jira] [Updated] (SPARK-53980) Add `SparkConf.getAllWithPrefix(String, String => K)` API

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-53980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun updated SPARK-53980:
----------------------------------
    Summary: Add `SparkConf.getAllWithPrefix(String, String => K)` API  (was: Add more flexible getAllWithPrefix for SparkConf)

> Add `SparkConf.getAllWithPrefix(String, String => K)` API
> ---------------------------------------------------------
>
>                 Key: SPARK-53980
>                 URL: https://issues.apache.org/jira/browse/SPARK-53980
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 4.1.0
>            Reporter: Jiaan Geng
>            Assignee: Jiaan Geng
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.1.0
>
>
> We need to set some config related to S3 for our inner Spark. The implementation of the function show below.
> {code:java}
> private def setS3Configs(conf: SparkConf): Unit = {
>     val S3A_PREFIX = "spark.fs.s3a"
>     val SPARK_HADOOP_S3A_PREFIX = "spark.hadoop.fs.s3a"
>     val s3aConf = conf.getAllWithPrefix(S3A_PREFIX)
>     s3aConf
>       .foreach(
>         confPair => {
>           val keyWithoutPrefix = confPair._1
>           val oldKey = S3A_PREFIX + keyWithoutPrefix
>           val newKey = SPARK_HADOOP_S3A_PREFIX + keyWithoutPrefix
>           val value = confPair._2
>           (newKey, value)
>         })
>   }
> {code}
> These code seems redundant and complicated. The reason is getAllWithPrefix only return the suffix part.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org