You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2021/08/27 20:00:05 UTC
[jira] [Resolved] (SPARK-36327) Spark sql creates staging dir
inside database directory rather than creating inside table directory
[ https://issues.apache.org/jira/browse/SPARK-36327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-36327.
-----------------------------------
Fix Version/s: 3.3.0
Resolution: Fixed
Issue resolved by pull request 33577
[https://github.com/apache/spark/pull/33577]
> Spark sql creates staging dir inside database directory rather than creating inside table directory
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-36327
> URL: https://issues.apache.org/jira/browse/SPARK-36327
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 3.1.2
> Reporter: Senthil Kumar
> Assignee: Senthil Kumar
> Priority: Minor
> Fix For: 3.3.0
>
>
> Spark sql creates staging dir inside database directory rather than creating inside table directory.
>
> This arises only when viewfs:// is configured. When the location is hdfs://, it doesn't occur.
>
> Based on further investigation in file *SaveAsHiveFile.scala*, I could see that the directory hierarchy has been not properly handled for viewFS condition.
> Parent path(db path) is passed rather than passing the actual directory(table location).
> {{
> // Mostly copied from Context.java#getExternalTmpPath of Hive 1.2
> private def newVersionExternalTempPath(
> path: Path,
> hadoopConf: Configuration,
> stagingDir: String): Path = {
> val extURI: URI = path.toUri
> if (extURI.getScheme == "viewfs")
> { getExtTmpPathRelTo(path.getParent, hadoopConf, stagingDir) }
> else
> { new Path(getExternalScratchDir(extURI, hadoopConf, stagingDir), "-ext-10000") }
> }
> }}
> Please refer below lines
> ===============================
> if (extURI.getScheme == "viewfs") {
> getExtTmpPathRelTo(path.getParent, hadoopConf, stagingDir)
> ===============================
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org