You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by rx...@apache.org on 2016/10/07 07:27:58 UTC
spark git commit: [SPARK-17805][PYSPARK] Fix in sqlContext.read.text
when pass in list of paths
Repository: spark
Updated Branches:
refs/heads/master 3713bb199 -> bcaa799cb
[SPARK-17805][PYSPARK] Fix in sqlContext.read.text when pass in list of paths
## What changes were proposed in this pull request?
If given a list of paths, `pyspark.sql.readwriter.text` will attempt to use an undefined variable `paths`. This change checks if the param `paths` is a basestring and then converts it to a list, so that the same variable `paths` can be used for both cases
## How was this patch tested?
Added unit test for reading list of files
Author: Bryan Cutler <cu...@gmail.com>
Closes #15379 from BryanCutler/sql-readtext-paths-SPARK-17805.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bcaa799c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bcaa799c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bcaa799c
Branch: refs/heads/master
Commit: bcaa799cb01289f73e9f48526e94653a07628983
Parents: 3713bb1
Author: Bryan Cutler <cu...@gmail.com>
Authored: Fri Oct 7 00:27:55 2016 -0700
Committer: Reynold Xin <rx...@databricks.com>
Committed: Fri Oct 7 00:27:55 2016 -0700
----------------------------------------------------------------------
python/pyspark/sql/readwriter.py | 4 ++--
python/pyspark/sql/tests.py | 6 ++++++
2 files changed, 8 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/bcaa799c/python/pyspark/sql/readwriter.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py
index 3ad6f80..91c2b17 100644
--- a/python/pyspark/sql/readwriter.py
+++ b/python/pyspark/sql/readwriter.py
@@ -289,8 +289,8 @@ class DataFrameReader(OptionUtils):
[Row(value=u'hello'), Row(value=u'this')]
"""
if isinstance(paths, basestring):
- path = [paths]
- return self._df(self._jreader.text(self._spark._sc._jvm.PythonUtils.toSeq(path)))
+ paths = [paths]
+ return self._df(self._jreader.text(self._spark._sc._jvm.PythonUtils.toSeq(paths)))
@since(2.0)
def csv(self, path, schema=None, sep=None, encoding=None, quote=None, escape=None,
http://git-wip-us.apache.org/repos/asf/spark/blob/bcaa799c/python/pyspark/sql/tests.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/tests.py b/python/pyspark/sql/tests.py
index c2171c2..a9e4555 100644
--- a/python/pyspark/sql/tests.py
+++ b/python/pyspark/sql/tests.py
@@ -1702,6 +1702,12 @@ class SQLTests(ReusedPySparkTestCase):
"does_not_exist",
lambda: spark.catalog.uncacheTable("does_not_exist"))
+ def test_read_text_file_list(self):
+ df = self.spark.read.text(['python/test_support/sql/text-test.txt',
+ 'python/test_support/sql/text-test.txt'])
+ count = df.count()
+ self.assertEquals(count, 4)
+
class HiveSparkSubmitTests(SparkSubmitTests):
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org