You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ma...@apache.org on 2014/08/28 00:05:48 UTC
git commit: [SQL] [SPARK-3236] Reading Parquet tables from Metastore
mangles location
Repository: spark
Updated Branches:
refs/heads/master 28d41d627 -> cc275f4b7
[SQL] [SPARK-3236] Reading Parquet tables from Metastore mangles location
Currently we do `relation.hiveQlTable.getDataLocation.getPath`, which returns the path-part of the URI (e.g., "s3n://my-bucket/my-path" => "/my-path"). We should do `relation.hiveQlTable.getDataLocation.toString` instead, as a URI's toString returns a faithful representation of the full URI, which can later be passed into a Hadoop Path.
Author: Aaron Davidson <aa...@databricks.com>
Closes #2150 from aarondav/parquet-location and squashes the following commits:
459f72c [Aaron Davidson] [SQL] [SPARK-3236] Reading Parquet tables from Metastore mangles location
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cc275f4b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/cc275f4b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/cc275f4b
Branch: refs/heads/master
Commit: cc275f4b7910f6d0ad266a43bac2fdae58e9739e
Parents: 28d41d6
Author: Aaron Davidson <aa...@databricks.com>
Authored: Wed Aug 27 15:05:47 2014 -0700
Committer: Michael Armbrust <mi...@databricks.com>
Committed: Wed Aug 27 15:05:47 2014 -0700
----------------------------------------------------------------------
.../src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/cc275f4b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
----------------------------------------------------------------------
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
index 10fa831..47e24f0 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
@@ -135,7 +135,7 @@ private[hive] trait HiveStrategies {
.fakeOutput(projectList.map(_.toAttribute)):: Nil
} else {
hiveContext
- .parquetFile(relation.hiveQlTable.getDataLocation.getPath)
+ .parquetFile(relation.hiveQlTable.getDataLocation.toString)
.lowerCase
.where(unresolvedOtherPredicates)
.select(unresolvedProjection:_*)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org