You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by davies <gi...@git.apache.org> on 2015/02/11 09:46:48 UTC
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/4528
[SPARK-5677] [SPARK-5734] [SQL] [PySpark] Python DataFrame API remaining tasks
1. DataFrame.renameColumn
2. DataFrame.show() and _repr_
3. Use simpleString() rather than jsonValue in DataFrame.dtypes
4. createDataFrame from local Python data, including pandas.DataFrame
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark df3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4528.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4528
----
commit 6f94f25a18585ab4c59594eff8010cae06fcca72
Author: Davies Liu <da...@databricks.com>
Date: 2015-02-11T08:42:49Z
create DataFrame from local Python data
commit 6cbc1540847bb04190713518dcf90cb7a4475ddc
Author: Davies Liu <da...@databricks.com>
Date: 2015-02-11T08:43:44Z
dataframe.show() and improve dtypes
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73935150
[Test build #27295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27295/consoleFull) for PR 4528 at commit [`014acea`](https://github.com/apache/spark/commit/014acea03b9945fcea72251e7749e9ba5949c4ba).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4528#discussion_r24517268
--- Diff: python/pyspark/sql/context.py ---
@@ -294,9 +303,9 @@ def applySchema(self, rdd, schema):
df = self._ssql_ctx.applySchemaToPythonRDD(jrdd.rdd(), schema.json())
return DataFrame(df, self)
- def createDataFrame(self, rdd, schema=None, samplingRatio=None):
+ def createDataFrame(self, data, schema=None, samplingRatio=None):
--- End diff --
it is not just an RDD anymore
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73943049
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27293/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73929153
For functions taking a table name as an input parameter, can we make the parameter name consistent in this PR? There are a few places in Python where we call it `name`. But, in Scala/Java, we call it `tableName`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73849118
cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73954133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27295/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4528#discussion_r24480364
--- Diff: python/pyspark/sql/types.py ---
@@ -188,6 +199,8 @@ class IntegerType(PrimitiveType):
The data type representing int values.
"""
+ def simpleString(self):
+ return 'int'
--- End diff --
is it possible to just call the JVM function? Otherwise we might be inconsistent between the two languages in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4528#discussion_r24480267
--- Diff: python/pyspark/sql/context.py ---
@@ -336,19 +353,28 @@ def createDataFrame(self, rdd, schema=None, samplingRatio=None):
>>> df3.collect()
[Row(name=u'Alice', age=1)]
"""
- if isinstance(rdd, DataFrame):
+ if isinstance(data, DataFrame):
raise TypeError("rdd is already a DataFrame")
- if isinstance(schema, StructType):
- return self.applySchema(rdd, schema)
- else:
- if isinstance(schema, (list, tuple)):
- first = rdd.first()
- if not isinstance(first, (list, tuple)):
- raise ValueError("each row in `rdd` should be list or tuple")
- row_cls = Row(*schema)
- rdd = rdd.map(lambda r: row_cls(*r))
- return self.inferSchema(rdd, samplingRatio)
+ if has_pandas and isinstance(data, pandas.DataFrame):
+ data = self._sc.parallelize(data.to_records(index=False))
+ if schema is None:
+ schema = list(data.columns)
+
+ if not isinstance(data, RDD):
+ data = self._sc.parallelize(data)
--- End diff --
can we provide more type checking? e.g. throw a TypeError if data is not the proper type. maybe do that check in the very beginning.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/4528#discussion_r24518769
--- Diff: python/pyspark/sql/types.py ---
@@ -188,6 +199,8 @@ class IntegerType(PrimitiveType):
The data type representing int values.
"""
+ def simpleString(self):
+ return 'int'
--- End diff --
There are dumped as JSON, passed into JVM, parsed again.
The good part is that we could infer the schema in pure Python (also in workers, it's necessary to infer the schema on many rows of the RDD), the bad part is that we have duplicated code in Python and Scala.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4528#discussion_r24480198
--- Diff: python/pyspark/sql/context.py ---
@@ -336,19 +353,28 @@ def createDataFrame(self, rdd, schema=None, samplingRatio=None):
>>> df3.collect()
[Row(name=u'Alice', age=1)]
"""
- if isinstance(rdd, DataFrame):
+ if isinstance(data, DataFrame):
raise TypeError("rdd is already a DataFrame")
- if isinstance(schema, StructType):
- return self.applySchema(rdd, schema)
- else:
- if isinstance(schema, (list, tuple)):
- first = rdd.first()
- if not isinstance(first, (list, tuple)):
- raise ValueError("each row in `rdd` should be list or tuple")
- row_cls = Row(*schema)
- rdd = rdd.map(lambda r: row_cls(*r))
- return self.inferSchema(rdd, samplingRatio)
+ if has_pandas and isinstance(data, pandas.DataFrame):
--- End diff --
would it ever be possible in Python for has_pandas to be false, but a pandas.DataFrame is passed in?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73851714
[Test build #27286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27286/consoleFull) for PR 4528 at commit [`46f5f95`](https://github.com/apache/spark/commit/46f5f955bb2dcba801afaf2b0f630769fc2e78d6).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4528#discussion_r24482138
--- Diff: python/pyspark/sql/types.py ---
@@ -188,6 +199,8 @@ class IntegerType(PrimitiveType):
The data type representing int values.
"""
+ def simpleString(self):
+ return 'int'
--- End diff --
how are Python data types connected to JVM types right now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4528#discussion_r24480165
--- Diff: python/pyspark/sql/context.py ---
@@ -336,19 +353,28 @@ def createDataFrame(self, rdd, schema=None, samplingRatio=None):
>>> df3.collect()
[Row(name=u'Alice', age=1)]
"""
- if isinstance(rdd, DataFrame):
+ if isinstance(data, DataFrame):
raise TypeError("rdd is already a DataFrame")
--- End diff --
rdd -> data
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73849200
[Test build #27283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27283/consoleFull) for PR 4528 at commit [`6cbc154`](https://github.com/apache/spark/commit/6cbc1540847bb04190713518dcf90cb7a4475ddc).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/4528#discussion_r24481115
--- Diff: python/pyspark/sql/context.py ---
@@ -336,19 +353,28 @@ def createDataFrame(self, rdd, schema=None, samplingRatio=None):
>>> df3.collect()
[Row(name=u'Alice', age=1)]
"""
- if isinstance(rdd, DataFrame):
+ if isinstance(data, DataFrame):
raise TypeError("rdd is already a DataFrame")
- if isinstance(schema, StructType):
- return self.applySchema(rdd, schema)
- else:
- if isinstance(schema, (list, tuple)):
- first = rdd.first()
- if not isinstance(first, (list, tuple)):
- raise ValueError("each row in `rdd` should be list or tuple")
- row_cls = Row(*schema)
- rdd = rdd.map(lambda r: row_cls(*r))
- return self.inferSchema(rdd, samplingRatio)
+ if has_pandas and isinstance(data, pandas.DataFrame):
--- End diff --
No, user cannot create a pandas.DataFrame object if pandas is not available.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/4528#discussion_r24481180
--- Diff: python/pyspark/sql/types.py ---
@@ -188,6 +199,8 @@ class IntegerType(PrimitiveType):
The data type representing int values.
"""
+ def simpleString(self):
+ return 'int'
--- End diff --
Not easy, then we may need another different implementation of DataTypes, we could do the refactor later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73955482
Merging in master. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4528#discussion_r24480407
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameImpl.scala ---
@@ -126,7 +126,10 @@ private[sql] class DataFrameImpl protected[sql](
logicalPlan.isInstanceOf[LocalRelation]
}
- override def show(): Unit = {
+ /**
+ * Internal API for Python
+ */
+ private[sql] def samples(): String = {
--- End diff --
samples is a bad name for this. since this is internal, we can create a longer name. maybe showString?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73858247
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27283/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73933395
@yhuai We're trying our best to have the same API between Scala, Java and Python, but sometimes we can't, because the difference between languages. For those out of scope of API, I think we should follow the conversation for the language, for example, we should follow PEP8 in Python.
If an argument is not keyword argument, the name is not part of the API (even user could still use it as keyword argument), we should feel free to choose the proper name for it. In Python, we prefer simple things, not the verbose or complicated one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73858234
[Test build #27283 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27283/consoleFull) for PR 4528 at commit [`6cbc154`](https://github.com/apache/spark/commit/6cbc1540847bb04190713518dcf90cb7a4475ddc).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73861215
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27286/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73921913
[Test build #27293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27293/consoleFull) for PR 4528 at commit [`6ba526e`](https://github.com/apache/spark/commit/6ba526ed82b6bab6a26cc295c88ead28079884cf).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/4528#discussion_r24515956
--- Diff: python/pyspark/sql/context.py ---
@@ -294,9 +303,9 @@ def applySchema(self, rdd, schema):
df = self._ssql_ctx.applySchemaToPythonRDD(jrdd.rdd(), schema.json())
return DataFrame(df, self)
- def createDataFrame(self, rdd, schema=None, samplingRatio=None):
+ def createDataFrame(self, data, schema=None, samplingRatio=None):
--- End diff --
If it is an RDD, why not just call it rdd (like what we do in Scala/Java)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73943042
[Test build #27293 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27293/consoleFull) for PR 4528 at commit [`6ba526e`](https://github.com/apache/spark/commit/6ba526ed82b6bab6a26cc295c88ead28079884cf).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73861205
[Test build #27286 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27286/consoleFull) for PR 4528 at commit [`46f5f95`](https://github.com/apache/spark/commit/46f5f955bb2dcba801afaf2b0f630769fc2e78d6).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class TaskCommitDenied(`
* `class CommitDeniedException(`
* ` class OutputCommitCoordinatorActor(outputCommitCoordinator: OutputCommitCoordinator)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/4528
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4528#discussion_r24482251
--- Diff: python/pyspark/sql/context.py ---
@@ -336,19 +353,32 @@ def createDataFrame(self, rdd, schema=None, samplingRatio=None):
>>> df3.collect()
[Row(name=u'Alice', age=1)]
"""
- if isinstance(rdd, DataFrame):
- raise TypeError("rdd is already a DataFrame")
+ if isinstance(data, DataFrame):
+ raise TypeError("data is already a DataFrame")
- if isinstance(schema, StructType):
- return self.applySchema(rdd, schema)
- else:
- if isinstance(schema, (list, tuple)):
- first = rdd.first()
- if not isinstance(first, (list, tuple)):
- raise ValueError("each row in `rdd` should be list or tuple")
- row_cls = Row(*schema)
- rdd = rdd.map(lambda r: row_cls(*r))
- return self.inferSchema(rdd, samplingRatio)
+ if has_pandas and isinstance(data, pandas.DataFrame):
+ data = self._sc.parallelize(data.to_records(index=False))
+ if schema is None:
+ schema = list(data.columns)
+
+ if not isinstance(data, RDD):
+ try:
+ # data could be list, tuple, generator ...
+ data = self._sc.parallelize(data)
+ except Exception:
+ raise ValueError("Cann't create an RDD from type: %s" % type(data))
--- End diff --
"cannot create an RDD from type: %s"
("cann't" is misspelled, but don't use "can't" in general)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5677] [SPARK-5734] [SQL] [PySpark] Pyth...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4528#issuecomment-73954122
[Test build #27295 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27295/consoleFull) for PR 4528 at commit [`014acea`](https://github.com/apache/spark/commit/014acea03b9945fcea72251e7749e9ba5949c4ba).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org