You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "guojh (JIRA)" <ji...@apache.org> on 2018/10/30 07:06:00 UTC
[jira] [Commented] (SPARK-25880) user set some hadoop
configurations can not work
[ https://issues.apache.org/jira/browse/SPARK-25880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668263#comment-16668263 ]
guojh commented on SPARK-25880:
-------------------------------
The root cause of this issue is that use the sparkContext's conf to construct the hadoop conf which is use to broadcasted. So, We should filter the properties that startwith 'spark.hadoop.'
> user set some hadoop configurations can not work
> ------------------------------------------------
>
> Key: SPARK-25880
> URL: https://issues.apache.org/jira/browse/SPARK-25880
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: guojh
> Priority: Major
>
> When user set some hadoop configuration in spark-defaults.conf, for instance:
> spark.hadoop.mapreduce.input.fileinputformat.split.maxsize 100000
> and then user use the spark-sql and use set command to overwrite this configuration, but it can not cover the value which set in the file of spark-defaults.conf.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org