You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Gengliang Wang (Jira)" <ji...@apache.org> on 2020/06/08 23:17:00 UTC
[jira] [Created] (SPARK-31935) Data source options should be
propagated in method `checkAndGlobPathIfNecessary`
Gengliang Wang created SPARK-31935:
--------------------------------------
Summary: Data source options should be propagated in method `checkAndGlobPathIfNecessary`
Key: SPARK-31935
URL: https://issues.apache.org/jira/browse/SPARK-31935
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.1.0
Reporter: Gengliang Wang
Data source options should be propagated into the hadoop configuration of method `checkAndGlobPathIfNecessary`
From org.apache.hadoop.fs.FileSystem.java:
{code:java}
public static FileSystem get(URI uri, Configuration conf) throws IOException {
String scheme = uri.getScheme();
String authority = uri.getAuthority();
if (scheme == null && authority == null) { // use default FS
return get(conf);
}
if (scheme != null && authority == null) { // no authority
URI defaultUri = getDefaultUri(conf);
if (scheme.equals(defaultUri.getScheme()) // if scheme matches default
&& defaultUri.getAuthority() != null) { // & default has authority
return get(defaultUri, conf); // return default
}
}
String disableCacheName = String.format("fs.%s.impl.disable.cache", scheme);
if (conf.getBoolean(disableCacheName, false)) {
return createFileSystem(uri, conf);
}
return CACHE.get(uri, conf);
}
{code}
With this, we can specify URI schema and authority related configurations for scanning file systems.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org