[jira] [Created] (SPARK-53980) Add more flexible getAllWithPrefix for SparkConf

["Jiaan Geng (Jira)" <ji...@apache.org>]

Jiaan Geng created SPARK-53980:
----------------------------------

             Summary: Add more flexible getAllWithPrefix for SparkConf
                 Key: SPARK-53980
                 URL: https://issues.apache.org/jira/browse/SPARK-53980
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 4.1.0
            Reporter: Jiaan Geng
            Assignee: Jiaan Geng


We need to set some config related to S3 for our inner Spark. The implementation of the function show below.

{code:java}
private def setS3Configs(conf: SparkConf): Unit = {
    val S3A_PREFIX = "spark.fs.s3a"
    val SPARK_HADOOP_S3A_PREFIX = "spark.hadoop.fs.s3a"
    val s3aConf = conf.getAllWithPrefix(S3A_PREFIX)
    s3aConf
      .foreach(
        confPair => {
          val keyWithoutPrefix = confPair._1
          val oldKey = S3A_PREFIX + keyWithoutPrefix
          val newKey = SPARK_HADOOP_S3A_PREFIX + keyWithoutPrefix
          val value = confPair._2
          conf.remove(oldKey)
          conf.set(newKey, value)
          logInfo(s"Migrate $oldKey to $newKey")
        })
  }

{code}

These code seems redundant and complicated. The reason is getAllWithPrefix only return the suffix part.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org