You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2017/02/06 18:07:50 UTC
[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/16824
[SPARK-18069][PYTHON] Make PySpark doctests for SQL self-contained
## What changes were proposed in this pull request?
This PR proposes to make the examples in Python API documentation as self-contained.
These seem pretty common in Python API documentation, for example, [pandas](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.pivot.html#pandas.DataFrame.pivot) and [numpy](https://docs.scipy.org/doc/numpy/reference/arrays.ndarray.html).
**Before**
<img width="430" alt="2017-02-07 2 51 52" src="https://cloud.githubusercontent.com/assets/6477701/22659679/ded31cc0-ece1-11e6-983a-f1f1632beeda.png">
<img width="621" alt="2017-02-07 2 56 52" src="https://cloud.githubusercontent.com/assets/6477701/22659667/d7eda240-ece1-11e6-8082-e79924f69869.png">
**After**
<img width="634" alt="2017-02-07 2 52 03" src="https://cloud.githubusercontent.com/assets/6477701/22659672/db80d56c-ece1-11e6-8679-50484d410768.png">
<img width="709" alt="2017-02-07 2 56 33" src="https://cloud.githubusercontent.com/assets/6477701/22659666/d6649d66-ece1-11e6-88bf-310d8d0da5c4.png">
## How was this patch tested?
Manually tested the doctests.
```
./run-tests.py --python-executables=python2.6 --modules=pyspark-sql
./run-tests.py --python-executables=python3.6 --modules=pyspark-sql
```
Closes #15053
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-18069
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16824.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16824
----
commit 00c8af35b704e989ad8536490f310a35c2e721fb
Author: hyukjinkwon <gu...@gmail.com>
Date: 2017-02-06T15:42:04Z
Make doctests self-contained
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon closed the pull request at:
https://github.com/apache/spark/pull/16824
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16824#discussion_r99728558
--- Diff: python/pyspark/sql/dataframe.py ---
@@ -296,8 +312,10 @@ def show(self, n=20, truncate=True):
If set to a number greater than one, truncates long strings to length ``truncate``
and align cells right.
+ >>> from pyspark.sql import Row
+ >>> df = spark.createDataFrame([Row(name='Alice', age=2), Row(name='Bob', age=5)])
>>> df
- DataFrame[age: int, name: string]
+ DataFrame[age: bigint, name: string]
--- End diff --
This is the same. It seems `bigint` by default if we don't explicitly provide the schema.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16824
**[Test build #72461 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72461/testReport)** for PR 16824 at commit [`00c8af3`](https://github.com/apache/spark/commit/00c8af35b704e989ad8536490f310a35c2e721fb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16824#discussion_r99727122
--- Diff: python/pyspark/sql/dataframe.py ---
@@ -233,11 +245,13 @@ def schema(self):
def printSchema(self):
"""Prints out the schema in the tree format.
+ >>> from pyspark.sql import Row
+ >>> df = spark.createDataFrame([Row(name='Alice', age=2), Row(name='Bob', age=5)])
>>> df.printSchema()
root
- |-- age: integer (nullable = true)
+ |-- age: long (nullable = true)
--- End diff --
This one too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16824#discussion_r99729087
--- Diff: python/pyspark/sql/session.py ---
@@ -51,8 +51,10 @@ def toDF(self, schema=None, sampleRatio=None):
:param samplingRatio: the sample ratio of rows used for inferring
:return: a DataFrame
+ >>> from pyspark.sql import Row
+ >>> rdd = sc.parallelize([Row(field1=1, field2="row1"), Row(field1=2, field2="row2")])
>>> rdd.toDF().collect()
- [Row(name=u'Alice', age=1)]
+ [Row(field1=1, field2=u'row1'), Row(field1=2, field2=u'row2')]
--- End diff --
It seems doctest does not run this as the results were already wrong.
```python
>>> rdd = sc.parallelize(
... [Row(field1=1, field2="row1"),
... Row(field1=2, field2="row2"),
... Row(field1=3, field2="row3")])
>>> rdd.collect()
[Row(field1=1, field2='row1'), Row(field1=2, field2='row2'), Row(field1=3, field2='row3')]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16824
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72496/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16824
I'm slightly against the work to make this change happen and would rather focus on some other Python PRs - I'm not sure it improves readability of the test cases as much as planned but I'll defer to @davies judgement on this one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16824
cc @holden, @srowen and @mortada who were in the PR. Could three of you take a look when you have some time?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16824
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72462/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16824
**[Test build #72496 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72496/testReport)** for PR 16824 at commit [`59da0f1`](https://github.com/apache/spark/commit/59da0f1b3c477a05047f43c1eeea7c20a65cbf4b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16824
**[Test build #72461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72461/testReport)** for PR 16824 at commit [`00c8af3`](https://github.com/apache/spark/commit/00c8af35b704e989ad8536490f310a35c2e721fb).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16824
**[Test build #72462 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72462/testReport)** for PR 16824 at commit [`b3acaad`](https://github.com/apache/spark/commit/b3acaadfed5833c108c03aae7865b6ed2782169a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16824#discussion_r99728503
--- Diff: python/pyspark/sql/dataframe.py ---
@@ -247,9 +261,11 @@ def explain(self, extended=False):
:param extended: boolean, default ``False``. If ``False``, prints only the physical plan.
+ >>> from pyspark.sql import Row
+ >>> df = spark.createDataFrame([Row(name='Alice', age=2), Row(name='Bob', age=5)])
>>> df.explain()
== Physical Plan ==
- Scan ExistingRDD[age#0,name#1]
+ Scan ExistingRDD[age#...,name#...]
--- End diff --
We now create multiple dataframes so the columns are not always `#0` and `#1`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16824
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16824
@davies and @holdenk, what do you think about this change? I am fine if we can't decide it is worth for now. I can close this if any of you is not confident of this change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16824
**[Test build #72496 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72496/testReport)** for PR 16824 at commit [`59da0f1`](https://github.com/apache/spark/commit/59da0f1b3c477a05047f43c1eeea7c20a65cbf4b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16824
Thanks @holdenk, let me close this for now.
@davies, please give me your opinion. If you think it is worth I will reopen. If not, I will resolve the JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16824
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16824
gentle ping @holdenk (somehow writing `@holdenk` does not show your name ... <img width="255" alt="2017-02-11 11 45 01" src="https://cloud.githubusercontent.com/assets/6477701/22854544/3d5ec930-f0b4-11e6-82c8-195a725caaf4.png">)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16824
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72461/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16824
**[Test build #72462 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72462/testReport)** for PR 16824 at commit [`b3acaad`](https://github.com/apache/spark/commit/b3acaadfed5833c108c03aae7865b6ed2782169a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16824#discussion_r99728911
--- Diff: python/pyspark/sql/dataframe.py ---
@@ -553,10 +594,10 @@ def repartition(self, numPartitions, *cols):
+---+-----+
|age| name|
+---+-----+
- | 2|Alice|
| 5| Bob|
- | 2|Alice|
| 5| Bob|
+ | 2|Alice|
+ | 2|Alice|
--- End diff --
It seems the order is varied dependent on how we create the dataframe (as I saw the related JIRA I can point out if anyone wants). I haven't looked into this deeper as this variant seems not related to the tests in `repartition` API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16824
Just back from Spark Summit East, I'll try and take a look soon :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16824#discussion_r99727033
--- Diff: python/pyspark/sql/dataframe.py ---
@@ -218,8 +228,10 @@ def writeStream(self):
def schema(self):
"""Returns the schema of this :class:`DataFrame` as a :class:`pyspark.sql.types.StructType`.
+ >>> from pyspark.sql import Row
+ >>> df = spark.createDataFrame([Row(name='Alice', age=2), Row(name='Bob', age=5)])
>>> df.schema
- StructType(List(StructField(age,IntegerType,true),StructField(name,StringType,true)))
+ StructType(List(StructField(age,LongType,true),StructField(name,StringType,true)))
--- End diff --
This happened to be `LongType` as it seems numbers become a long type by default via PySpark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16824#discussion_r99711908
--- Diff: python/pyspark/sql/dataframe.py ---
@@ -1191,6 +1287,12 @@ def dropna(self, how='any', thresh=None, subset=None):
This overwrites the `how` parameter.
:param subset: optional list of column names to consider.
+
--- End diff --
Ops, let me remove this extra newline when I happen to push more commits.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16824: [SPARK-18069][PYTHON] Make PySpark doctests for SQL self...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16824
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org