You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2020/07/14 23:49:06 UTC
[spark] branch branch-3.0 updated: [SPARK-32301][PYTHON][TESTS] Add
a test case for toPandas to work with empty partitioned Spark DataFrame
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new b896b17 [SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame
b896b17 is described below
commit b896b173eae6a2f323498debb32b0979441d0126
Author: HyukjinKwon <gu...@apache.org>
AuthorDate: Wed Jul 15 08:44:48 2020 +0900
[SPARK-32301][PYTHON][TESTS] Add a test case for toPandas to work with empty partitioned Spark DataFrame
### What changes were proposed in this pull request?
This PR proposes to port the test case from https://github.com/apache/spark/pull/29098 to branch-3.0 and master. In the master and branch-3.0, this was fixed together at https://github.com/apache/spark/commit/ecaa495b1fe532c36e952ccac42f4715809476af but no partition case is not being tested.
### Why are the changes needed?
To improve test coverage.
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
Unit test was forward-ported.
Closes #29099 from HyukjinKwon/SPARK-32300-1.
Authored-by: HyukjinKwon <gu...@apache.org>
Signed-off-by: HyukjinKwon <gu...@apache.org>
(cherry picked from commit 676d92ecceb3d46baa524c725b9f9a14450f1e9d)
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
python/pyspark/sql/tests/test_arrow.py | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/python/pyspark/sql/tests/test_arrow.py b/python/pyspark/sql/tests/test_arrow.py
index 1386f8d..7a41d24 100644
--- a/python/pyspark/sql/tests/test_arrow.py
+++ b/python/pyspark/sql/tests/test_arrow.py
@@ -421,6 +421,13 @@ class ArrowTests(ReusedSQLTestCase):
self.spark.createDataFrame(
pd.DataFrame({'a': [1, 2, 3]}, index=[2., 3., 4.])).distinct().count(), 3)
+ def test_no_partition_toPandas(self):
+ # SPARK-32301: toPandas should work from a Spark DataFrame with no partitions
+ # Forward-ported from SPARK-32300.
+ pdf = self.spark.sparkContext.emptyRDD().toDF("col1 int").toPandas()
+ self.assertEqual(len(pdf), 0)
+ self.assertEqual(list(pdf.columns), ["col1"])
+
@unittest.skipIf(
not have_pandas or not have_pyarrow,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org