[GitHub] spark issue #22887: [SPARK-25880][CORE] user set's hadoop conf should not ov...

[cxzl25 <gi...@git.apache.org>]

Github user cxzl25 commented on the issue:

    https://github.com/apache/spark/pull/22887
  
    user set hadoop conf can't overwrite spark-defaults.conf
    
    **SparkHadoopUtil.get.appendS3AndSparkHadoopConfigurations** overwrite the user-set spark.hadoop with the default configuration (sparkSession.sparkContext.conf)
    
    @gengliangwang @cloud-fan @gatorsmile  
    Could you please give some comments when you have time?
    Thanks so much.
    
    https://github.com/apache/spark/blob/80813e198033cd63cc6100ee6ffe7d1eb1dff27b/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala#L85-L89
    
    ## test:
    
    ### spark-defaults.conf
    ```
    spark.hadoop.mapreduce.input.fileinputformat.split.maxsize  2
    ```
    
    ### spark-shell
    
    ```scala
    
    val hadoopConfKey="mapreduce.input.fileinputformat.split.maxsize"
    spark.conf.get("spark.hadoop."+hadoopConfKey) // 2
    var hadoopConf=spark.sessionState.newHadoopConf
    hadoopConf.get(hadoopConfKey) // 2
    
    spark.conf.set(hadoopConfKey,1) // set 1
    hadoopConf=spark.sessionState.newHadoopConf
    hadoopConf.get(hadoopConfKey) // 1
    
    //org.apache.spark.sql.hive.HadoopTableReader append Conf
    org.apache.spark.deploy.SparkHadoopUtil.get.appendS3AndSparkHadoopConfigurations(spark.sparkContext.getConf, hadoopConf)
    
    //org.apache.spark.sql.hive.HadoopTableReader _broadcastedHadoopConf
    hadoopConf.get("mapreduce.input.fileinputformat.split.maxsize") // 2
    
    ```
    
    
    
    
    
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org