You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by lxsmnv <gi...@git.apache.org> on 2017/02/20 02:26:05 UTC
[GitHub] spark pull request #16995: [SPARK-19340][SQL] CSV file will result in an exc...
GitHub user lxsmnv opened a pull request:
https://github.com/apache/spark/pull/16995
[SPARK-19340][SQL] CSV file will result in an exception if the filename contains special characters
## What changes were proposed in this pull request?
The root cause of the problem is that when spark is inferring schema from the csv file, it tries to resolve the file path pattern more then once by calling DataSouce.resolveRelation each time.
So, if we have file path like:
<...>/test*
and the actual file with name: test{00-1}.txt
Then from the initial call of DataSouce.resolveRelation the pattern will be resolved to /<...>/test{00-1}.txt. When it tries to infer schema for csv file, it calls DataSouce.resolveRelation the second time. The second attempt to resolve the path pattern fails because the file name /<...>/test{00-1}.txt is considered as a pattern and not as actual file and if there no file that match that pattern the whole DataSouce.resolveRelation fails.
The idea behind the fix is quite straightforward:
The part of DataSouce.resolveRelation that creates Hadoop Relation based on a resolved(actual) file names moved to separate function createHadoopRelation. CSVFileFormat.createBaseDataset calls this new function instead of DataSouce.resolveRelation, that caused unnecessary file path resolution.
## How was this patch tested?
manual tests
This contribution is my original work and I license the work to the project under the project\u2019s open source license.
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lxsmnv/spark SPARK-19340
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16995.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16995
----
commit 507a929694653d49d1eb42398131743e0d004f65
Author: lxsmnv <al...@gmail.com>
Date: 2017-02-20T01:52:40Z
SPARK-19340 file path resolution for csv files fixed
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16995: [SPARK-19340][SQL] CSV file will result in an exc...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16995#discussion_r101961405
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -404,6 +386,35 @@ case class DataSource(
}
/**
+ * Creates Hadoop relation based on format and globbed file paths
+ * @param format format of the data source file
+ * @param globPaths Path to the file resolved by Hadoop library
+ * @return Hadoop relation object
+ */
+ def createHadoopRelation(format: FileFormat,
+ globPaths: Array[Path]): BaseRelation = {
+ val (dataSchema, partitionSchema) = getOrInferFileFormatSchema(format)
--- End diff --
You do twice `getOrInferFileFormatSchema`. One is before calling `createHadoopRelation`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16995: [SPARK-19340][SQL] CSV file will result in an exception ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16995
@lxsmnv What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16995: [SPARK-19340][SQL] CSV file will result in an exception ...
Posted by lxsmnv <gi...@git.apache.org>.
Github user lxsmnv commented on the issue:
https://github.com/apache/spark/pull/16995
There is a possible problem about adding file existence check to globPathIfNecessary that I have just realized. If the user will provide pattern that is exactly the same as existing file name e.g. we have file with name test* and user provides pattern test* meaning pattern but not exact file name. globPathIfNecessary with proposed modification will resolve it to only one file - test*. It will change the existing behaviour but I am not sure if this is practically a big issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16995: [SPARK-19340][SQL] CSV file will result in an exception ...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16995
Actually I have a simpler fix like this:
--- a/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
@@ -213,7 +213,12 @@ class SparkHadoopUtil extends Logging {
}
def globPathIfNecessary(pattern: Path): Seq[Path] = {
- if (isGlobPath(pattern)) globPath(pattern) else Seq(pattern)
+ val fs = pattern.getFileSystem(conf)
+ if (fs.exists(pattern)) {
+ Seq(pattern)
+ } else {
+ if (isGlobPath(pattern)) globPath(pattern) else Seq(pattern)
+ }
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16995: [SPARK-19340][SQL] CSV file will result in an exc...
Posted by lxsmnv <gi...@git.apache.org>.
Github user lxsmnv commented on a diff in the pull request:
https://github.com/apache/spark/pull/16995#discussion_r102012068
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -404,6 +386,35 @@ case class DataSource(
}
/**
+ * Creates Hadoop relation based on format and globbed file paths
+ * @param format format of the data source file
+ * @param globPaths Path to the file resolved by Hadoop library
+ * @return Hadoop relation object
+ */
+ def createHadoopRelation(format: FileFormat,
+ globPaths: Array[Path]): BaseRelation = {
+ val (dataSchema, partitionSchema) = getOrInferFileFormatSchema(format)
--- End diff --
@viirya I will fix this. Looks like merge issue.
@maropu I will add tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16995: [SPARK-19340][SQL] CSV file will result in an exception ...
Posted by lxsmnv <gi...@git.apache.org>.
Github user lxsmnv commented on the issue:
https://github.com/apache/spark/pull/16995
@viirya I've removed a duplicate call for getOrInferFileFormatSchema. Thanks for pointing out.
@maropu I've added the test case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16995: [SPARK-19340][SQL] CSV file will result in an exc...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16995
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16995: [SPARK-19340][SQL] CSV file will result in an exception ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16995
@lxsmnv, could you check if this is a more general problem? I suspect it is not a CSV specific issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16995: [SPARK-19340][SQL] CSV file will result in an exception ...
Posted by lxsmnv <gi...@git.apache.org>.
Github user lxsmnv commented on the issue:
https://github.com/apache/spark/pull/16995
However, resolving path patterns and checking file existence multiple times is a bit awkward but it is a more general problem - all this data source stuff needs refactoring.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16995: [SPARK-19340][SQL] CSV file will result in an exception ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16995
We are closing it due to inactivity. please do reopen if you want to push it forward. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16995: [SPARK-19340][SQL] CSV file will result in an exception ...
Posted by lxsmnv <gi...@git.apache.org>.
Github user lxsmnv commented on the issue:
https://github.com/apache/spark/pull/16995
@HyukjinKwon the problem that I have found was in CSVFileFormat. So its more of csv specific. However, it can be a problem of some other data source types, but not all - it will depend on data source implementation. If the same problem with other data source types exists, there can be some degree of a common nature of that problem for those datasource types and the fix may require a some significant amount of work and changes.
My fix is quite simple and doesn't introduce much changes. For now, I would suggest to merge it. Otherwise aiming a more generic solution now may end up in neither options being implemented.
If you tried to reproduce this issue with our datasource types, can you create a new ticket and provide the details about the tests you have done and I will have a look at it and think about more generic approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16995: [SPARK-19340][SQL] CSV file will result in an exception ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/16995
Could you add tests for this pr?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16995: [SPARK-19340][SQL] CSV file will result in an exception ...
Posted by lxsmnv <gi...@git.apache.org>.
Github user lxsmnv commented on the issue:
https://github.com/apache/spark/pull/16995
@viirya Looks good to me :) and more generic. Do you want me to update my pull request or you have the other pull request with that fix?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16995: [SPARK-19340][SQL] CSV file will result in an exception ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16995
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16995: [SPARK-19340][SQL] CSV file will result in an exception ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16995
Since this could cause the behavior change, how about we first close this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16995: [SPARK-19340][SQL] CSV file will result in an exc...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16995#discussion_r102133102
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -374,34 +374,42 @@ case class DataSource(
globPath
}.toArray
- val (dataSchema, partitionSchema) = getOrInferFileFormatSchema(format)
-
- val fileCatalog = if (sparkSession.sqlContext.conf.manageFilesourcePartitions &&
- catalogTable.isDefined && catalogTable.get.tracksPartitionsInCatalog) {
- val defaultTableSize = sparkSession.sessionState.conf.defaultSizeInBytes
- new CatalogFileIndex(
- sparkSession,
- catalogTable.get,
- catalogTable.get.stats.map(_.sizeInBytes.toLong).getOrElse(defaultTableSize))
- } else {
- new InMemoryFileIndex(sparkSession, globbedPaths, options, Some(partitionSchema))
- }
-
- HadoopFsRelation(
- fileCatalog,
- partitionSchema = partitionSchema,
- dataSchema = dataSchema.asNullable,
- bucketSpec = bucketSpec,
- format,
- caseInsensitiveOptions)(sparkSession)
-
+ createHadoopRelation(format, globbedPaths)
case _ =>
throw new AnalysisException(
s"$className is not a valid Spark SQL Data Source.")
}
relation
}
+ /**
+ * Creates Hadoop relation based on format and globbed file paths
+ * @param format format of the data source file
+ * @param globPaths Path to the file resolved by Hadoop library
+ * @return Hadoop relation object
+ */
+ def createHadoopRelation(format: FileFormat,
+ globPaths: Array[Path]): BaseRelation = {
--- End diff --
Let's make this inlined.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org