You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (JIRA)" <ji...@apache.org> on 2018/08/20 17:13:00 UTC
[jira] [Resolved] (SPARK-23573) Create linter rule to prevent
misuse of SparkContext.hadoopConfiguration in SQL modules
[ https://issues.apache.org/jira/browse/SPARK-23573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li resolved SPARK-23573.
-----------------------------
Resolution: Duplicate
Assignee: Gengliang Wang
Fix Version/s: 2.4.0
> Create linter rule to prevent misuse of SparkContext.hadoopConfiguration in SQL modules
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-23573
> URL: https://issues.apache.org/jira/browse/SPARK-23573
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Xiao Li
> Assignee: Gengliang Wang
> Priority: Major
> Fix For: 2.4.0
>
>
> SparkContext.hadoopConfiguration should not be used directly in SQL module.
> Instead, one should always use sessionState.newHadoopConfiguration(), which blends in configs set per session.
> The idea is to add the linter rule to ban it.
> - Restrict the linter rule to the components that use SQL. use the parameter `scalastyleSources`?
> - Exclude genuinely valid uses, like e.g. in SessionState (ok, that can be done per case with scalastyle:off in the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org