You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2016/03/01 14:39:35 UTC
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/11444
[SPARK-13593][SQL] add a `schema()` method to convert python RDD to DataFrame easily
## What changes were proposed in this pull request?
This PR adds a `schema()` method which can be used to convert python RDD to DataFrame easily, for example, `df = rdd.schema("a: int, b: string")`.
## How was this patch tested?
new tests in `test.py` and doc test in `types.py`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark pyrdd
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11444.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11444
----
commit 2d1c2e527414cc119eee82fe92d67fb5b0a753c0
Author: Wenchen Fan <we...@databricks.com>
Date: 2016-03-01T13:34:14Z
parse schema string
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r55467820
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,129 @@ def __eq__(self, other):
for v in [ArrayType, MapType, StructType])
+_FIXED_DECIMAL = re.compile("decimal\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)")
+
+
+_BRACKETS = {'(': ')', '[': ']', '{': '}'}
+
+
+def _parse_basic_datatype_string(s):
+ if s in _all_atomic_types.keys():
+ return _all_atomic_types[s]()
+ elif s == "int":
+ return IntegerType()
+ elif _FIXED_DECIMAL.match(s):
+ m = _FIXED_DECIMAL.match(s)
--- End diff --
I think so, the code is copied from https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L860-L864, we can improve them later
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11444
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r55316023
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,139 @@ def __eq__(self, other):
for v in [ArrayType, MapType, StructType])
+_FIXED_DECIMAL = re.compile("decimal\\((\\d+),(\\d+)\\)")
+
+
+def _parse_basic_datatype_string(s):
+ if s == "null":
+ return NullType()
+ elif s == "boolean":
+ return BooleanType()
+ elif s == "byte":
+ return ByteType()
+ elif s == "short":
+ return ShortType()
+ elif s == "int":
+ return IntegerType()
+ elif s == "long":
+ return LongType()
+ elif s == "float":
+ return FloatType()
+ elif s == "double":
+ return DoubleType()
+ elif s == "decimal":
+ return DecimalType()
+ elif _FIXED_DECIMAL.match(s):
+ m = _FIXED_DECIMAL.match(json_value)
+ return DecimalType(int(m.group(1)), int(m.group(2)))
+ elif s == "string":
+ return StringType()
+ elif s == "date":
+ return DateType()
+ elif s == "timestamp":
+ return TimestampType()
+ elif s == "binary":
+ return BinaryType()
+ else:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+
+
+def _ignore_brackets_split(s, separator):
+ parts = []
+ buf = ""
+ level = 0
+ for c in s:
+ if c == "<":
+ level += 1
+ buf += c
+ elif c == ">":
+ if level == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ level -= 1
+ buf += c
+ elif c == separator and level > 0:
+ buf += c
+ elif c == separator:
+ parts.append(buf)
+ buf = ""
+ else:
+ buf += c
+
+ if len(buf) == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ parts.append(buf)
+ return parts
+
+
+def _parse_struct_type_string(s):
+ parts = _ignore_brackets_split(s, ",")
+ fields = []
+ for part in parts:
+ name_and_type = _ignore_brackets_split(part, ":")
+ if len(name_and_type) != 2:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ field_name = name_and_type[0].strip()
+ field_type = _parse_datatype_string(name_and_type[1])
+ fields.append(StructField(field_name, field_type))
+ return StructType(fields)
+
+
+def _parse_datatype_string(s):
--- End diff --
not now, should we support it? It will make the parser more complicated though...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-190727887
cc @marmbrus @rxin @yhuai
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-194044560
follow up JIRA created: https://issues.apache.org/jira/browse/SPARK-13757
@davies I'm not quite understand "We still does not support a schema string with only column names", `createDataFrame` can accept schema of string list, and regard it as column names.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r55258804
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,139 @@ def __eq__(self, other):
for v in [ArrayType, MapType, StructType])
+_FIXED_DECIMAL = re.compile("decimal\\((\\d+),(\\d+)\\)")
+
+
+def _parse_basic_datatype_string(s):
+ if s == "null":
+ return NullType()
+ elif s == "boolean":
+ return BooleanType()
+ elif s == "byte":
+ return ByteType()
+ elif s == "short":
+ return ShortType()
+ elif s == "int":
+ return IntegerType()
+ elif s == "long":
+ return LongType()
+ elif s == "float":
+ return FloatType()
+ elif s == "double":
+ return DoubleType()
+ elif s == "decimal":
+ return DecimalType()
+ elif _FIXED_DECIMAL.match(s):
+ m = _FIXED_DECIMAL.match(json_value)
+ return DecimalType(int(m.group(1)), int(m.group(2)))
+ elif s == "string":
+ return StringType()
+ elif s == "date":
+ return DateType()
+ elif s == "timestamp":
+ return TimestampType()
+ elif s == "binary":
+ return BinaryType()
+ else:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+
+
+def _ignore_brackets_split(s, separator):
+ parts = []
+ buf = ""
+ level = 0
+ for c in s:
+ if c == "<":
+ level += 1
+ buf += c
+ elif c == ">":
+ if level == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ level -= 1
+ buf += c
+ elif c == separator and level > 0:
+ buf += c
+ elif c == separator:
+ parts.append(buf)
+ buf = ""
+ else:
+ buf += c
+
+ if len(buf) == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ parts.append(buf)
+ return parts
+
+
+def _parse_struct_type_string(s):
+ parts = _ignore_brackets_split(s, ",")
+ fields = []
+ for part in parts:
+ name_and_type = _ignore_brackets_split(part, ":")
+ if len(name_and_type) != 2:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ field_name = name_and_type[0].strip()
+ field_type = _parse_datatype_string(name_and_type[1])
+ fields.append(StructField(field_name, field_type))
+ return StructType(fields)
+
+
+def _parse_datatype_string(s):
--- End diff --
Does it support quoted column/field names?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191047560
**[Test build #52288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52288/consoleFull)** for PR 11444 at commit [`6aeab1e`](https://github.com/apache/spark/commit/6aeab1ed8439109e356b978e91480865a760c1b1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54666473
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,139 @@ def __eq__(self, other):
for v in [ArrayType, MapType, StructType])
+_FIXED_DECIMAL = re.compile("decimal\\((\\d+),(\\d+)\\)")
+
+
+def _parse_basic_datatype_string(s):
+ if s == "null":
+ return NullType()
+ elif s == "boolean":
+ return BooleanType()
+ elif s == "byte":
+ return ByteType()
+ elif s == "short":
+ return ShortType()
+ elif s == "int":
+ return IntegerType()
+ elif s == "long":
+ return LongType()
+ elif s == "float":
+ return FloatType()
+ elif s == "double":
+ return DoubleType()
+ elif s == "decimal":
+ return DecimalType()
+ elif _FIXED_DECIMAL.match(s):
+ m = _FIXED_DECIMAL.match(json_value)
+ return DecimalType(int(m.group(1)), int(m.group(2)))
+ elif s == "string":
+ return StringType()
+ elif s == "date":
+ return DateType()
+ elif s == "timestamp":
+ return TimestampType()
+ elif s == "binary":
+ return BinaryType()
+ else:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+
+
+def _ignore_brackets_split(s, separator):
+ parts = []
+ buf = ""
+ level = 0
+ for c in s:
+ if c == "<":
+ level += 1
+ buf += c
+ elif c == ">":
+ if level == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ level -= 1
+ buf += c
+ elif c == separator and level > 0:
+ buf += c
+ elif c == separator:
+ parts.append(buf)
+ buf = ""
+ else:
+ buf += c
+
+ if len(buf) == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ parts.append(buf)
+ return parts
+
+
+def _parse_struct_type_string(s):
+ parts = _ignore_brackets_split(s, ",")
+ fields = []
+ for part in parts:
+ name_and_type = _ignore_brackets_split(part, ":")
+ if len(name_and_type) != 2:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ field_name = name_and_type[0].strip()
+ field_type = _parse_datatype_string(name_and_type[1])
+ fields.append(StructField(field_name, field_type))
+ return StructType(fields)
+
+
+def _parse_datatype_string(s):
+ """
+ Parses the given data type string to a :class:`DataType`, the data type string format equals
+ to `DataType.simpleString`, except that top level struct type can omit the `struct<>`.
+
+ >>> _parse_datatype_string("int")
+ IntegerType
+ >>> _parse_datatype_string("a: int, b: string ")
+ StructType(List(StructField(a,IntegerType,true),StructField(b,StringType,true)))
+ >>> _parse_datatype_string("a: array<int>")
+ StructType(List(StructField(a,ArrayType(IntegerType,true),true)))
+ >>> _parse_datatype_string(" map<string , string > ")
+ MapType(StringType,StringType,true)
+
+ >>> # Error cases
+ >>> _parse_datatype_string("blabla") # doctest: +IGNORE_EXCEPTION_DETAIL
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+ >>> _parse_datatype_string("a: int,") # doctest: +IGNORE_EXCEPTION_DETAIL
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+ >>> _parse_datatype_string("array<int") # doctest: +IGNORE_EXCEPTION_DETAIL
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+ >>> _parse_datatype_string("map<int, boolean>>") # doctest: +IGNORE_EXCEPTION_DETAIL
--- End diff --
Could you docment the format somewhere?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191055445
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54661055
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,139 @@ def __eq__(self, other):
for v in [ArrayType, MapType, StructType])
+_FIXED_DECIMAL = re.compile("decimal\\((\\d+),(\\d+)\\)")
+
+
+def _parse_basic_datatype_string(s):
+ if s == "null":
+ return NullType()
+ elif s == "boolean":
--- End diff --
We could lookup these from `_all_atomic_types`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191577686
**[Test build #52366 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52366/consoleFull)** for PR 11444 at commit [`dd2c929`](https://github.com/apache/spark/commit/dd2c929d33a28bb130d992c9226d76020357e4b6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191632150
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191575690
**[Test build #52363 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52363/consoleFull)** for PR 11444 at commit [`30f4137`](https://github.com/apache/spark/commit/30f4137d410f8d5b32ac28f58521c74aadb9cbf4).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-193611603
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52629/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191580120
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52366/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191575732
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52363/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-194074866
@cloud-fan In Python, the list of names could be a single string, for example:
```
>>> from collections import namedtuple
>>> row = namedtuple("row", "a b c")
```
"a b c" is better than ["a", "b", "c"]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191025436
**[Test build #52283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52283/consoleFull)** for PR 11444 at commit [`9435b98`](https://github.com/apache/spark/commit/9435b9853a65b854172572455993063f0cef397e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r55290303
--- Diff: python/pyspark/sql/context.py ---
@@ -50,18 +50,60 @@
def _monkey_patch_RDD(sqlContext):
def toDF(self, schema=None, sampleRatio=None):
"""
- Converts current :class:`RDD` into a :class:`DataFrame`
+ Converts current :class:`RDD` into a :class:`DataFrame` according to the given schema.
+ If the given schema is None or just column names, the schema will be inferred by scanning
+ the data in this RDD according to the sampleRatio.
- This is a shorthand for ``sqlContext.createDataFrame(rdd, schema, sampleRatio)``
+ Note that, the given schema must match the real data, or exception will be thrown at
+ runtime. If the given schema is not StructType, it will be wrapped into a StructType as
+ its only field, and the field name will be "value", each record will also be wrapped into a
+ tuple, which can be converted to row later.
- :param schema: a StructType or list of names of columns
+ :param schema: a :class:`DataType` or a datatype string or list of names of columns.
+ The data type string format equals to `DataType.simpleString`, except that
+ top level struct type can omit the `struct<>` and numeric types use
+ `typeName()` as their format, e.g. use `byte` instead of `tinyint` for
+ ByteType. We can also use `int` as a short name for IntegerType.
:param samplingRatio: the sample ratio of rows used for inferring
:return: a DataFrame
- >>> rdd.toDF().collect()
+ .. versionchanged:: 2.0
+ The schema parameter can be a DataType or a datatype string after 2.0. If it's not a
+ StructType, it will be wrapped into a StructType and each record will also be wrapped
+ into a tuple.
+
+ >>> rdd.toDF().collect() # will scan the data and infer the real schema.
+ [Row(name=u'Alice', age=1)]
+ >>> rdd.toDF(["name", "age"]) # will scan the data and infer the real schema.
[Row(name=u'Alice', age=1)]
+ >>> rdd.toDF(["a", "b"]) # will scan the data and infer the real schema.
+ [Row(a=u'Alice', b=1)]
+ >>> rdd.toDF("a: string, b: int").collect()
+ [Row(a=u'Alice', b=1)]
+ >>> rdd.map(lambda row: row.age).toDF("int").collect()
+ [Row(value=1)]
"""
- return sqlContext.createDataFrame(self, schema, sampleRatio)
+ if schema is None or isinstance(schema, (list, tuple)):
+ # if schema is going to be inferred, no need to verify it.
+ return sqlContext.createDataFrame(self, schema, sampleRatio)
+
+ if isinstance(schema, basestring):
+ schema = _parse_datatype_string(schema)
+
+ if not isinstance(schema, DataType):
+ raise TypeError("schema should be DataType or string or list or tuple, " +
+ "but got: %s" % schema)
+ else:
+ def verify(obj):
+ _verify_type(obj, schema)
+ return obj
+ rdd = self.map(verify)
+
+ if isinstance(schema, StructType):
+ return sqlContext.createDataFrame(rdd, schema)
+ else:
+ return sqlContext.createDataFrame(rdd.map(lambda obj: (obj, )),
--- End diff --
Do we support arbitrary Python objects? I think we do not need to still limit the object types to Row, Tuple, Dict, and List, right? We can extract values as long as the object have the requested field.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54657123
--- Diff: python/pyspark/sql/context.py ---
@@ -63,7 +63,33 @@ def toDF(self, schema=None, sampleRatio=None):
"""
return sqlContext.createDataFrame(self, schema, sampleRatio)
+ def schema(self, datatype):
+ """
+ Converts current :class:`RDD` into a :class:`DataFrame` according to the given data type.
+
+ :param datatype:
+ a :class:`DataType` or a data type string.
+ """
+ if isinstance(datatype, StructType):
+ schema = datatype
+ def to_row(obj):
+ _verify_type(obj, datatype)
--- End diff --
Since Python and Scala/Java have different range of int/long numbers, does `_verify_type` check if the actual value is within the allowed range?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191564132
**[Test build #52356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52356/consoleFull)** for PR 11444 at commit [`469f99a`](https://github.com/apache/spark/commit/469f99aa0e666c216955337b6c8be296fd4cb458).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-193604622
**[Test build #52629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52629/consoleFull)** for PR 11444 at commit [`77ff36b`](https://github.com/apache/spark/commit/77ff36baa992f4350d7c2650bdda1d267cdc0e77).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191565111
**[Test build #52357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52357/consoleFull)** for PR 11444 at commit [`e907933`](https://github.com/apache/spark/commit/e90793324213d167a290258396d075645e1cca00).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54979311
--- Diff: python/pyspark/sql/context.py ---
@@ -50,18 +50,56 @@
def _monkey_patch_RDD(sqlContext):
def toDF(self, schema=None, sampleRatio=None):
"""
- Converts current :class:`RDD` into a :class:`DataFrame`
+ Converts current :class:`RDD` into a :class:`DataFrame` according to the given schema.
+ If the given schema is None or just column names, the schema will be inferred by scanning
+ the data in this RDD according to the sampleRatio.
- This is a shorthand for ``sqlContext.createDataFrame(rdd, schema, sampleRatio)``
+ Note that, the given schema must match the real data, or exception will be thrown at
+ runtime. If the given schema is not StructType, it will be wrapped into a StructType as
+ its only field, and the field name will be "value", each record will also be wrapped into a
+ tuple, which can be converted to row later.
- :param schema: a StructType or list of names of columns
+ :param schema: a :class:`DataType` or a datatype string or list of names of columns
--- End diff --
or should I move it here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191579982
**[Test build #52366 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52366/consoleFull)** for PR 11444 at commit [`dd2c929`](https://github.com/apache/spark/commit/dd2c929d33a28bb130d992c9226d76020357e4b6).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-193611469
**[Test build #52629 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52629/consoleFull)** for PR 11444 at commit [`77ff36b`](https://github.com/apache/spark/commit/77ff36baa992f4350d7c2650bdda1d267cdc0e77).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191571532
**[Test build #52362 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52362/consoleFull)** for PR 11444 at commit [`a29cbb5`](https://github.com/apache/spark/commit/a29cbb5541d52baad3acb3c238784ceb8c44ecdc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54979106
--- Diff: python/pyspark/sql/context.py ---
@@ -50,18 +50,56 @@
def _monkey_patch_RDD(sqlContext):
def toDF(self, schema=None, sampleRatio=None):
"""
- Converts current :class:`RDD` into a :class:`DataFrame`
+ Converts current :class:`RDD` into a :class:`DataFrame` according to the given schema.
+ If the given schema is None or just column names, the schema will be inferred by scanning
+ the data in this RDD according to the sampleRatio.
- This is a shorthand for ``sqlContext.createDataFrame(rdd, schema, sampleRatio)``
+ Note that, the given schema must match the real data, or exception will be thrown at
+ runtime. If the given schema is not StructType, it will be wrapped into a StructType as
+ its only field, and the field name will be "value", each record will also be wrapped into a
+ tuple, which can be converted to row later.
- :param schema: a StructType or list of names of columns
+ :param schema: a :class:`DataType` or a datatype string or list of names of columns
:param samplingRatio: the sample ratio of rows used for inferring
:return: a DataFrame
+ .. versionchanged:: 2.0
+ The schema parameter can be a DataType or a datatype string after 2.0. If it's not a
+ StructType, it will be wrapped into a StructType and each record will also be wrapped
+ into a tuple.
+ The data type string format equals to `DataType.simpleString`, except that top level
+ struct type can omit the `struct<>` and numeric types use `typeName()` as their format,
+ e.g. use `byte` instead of `tinyint` for ByteType. We can also use `int` as a short name
+ for IntegerType.
+
>>> rdd.toDF().collect()
[Row(name=u'Alice', age=1)]
+ >>> rdd.toDF("a: string, b: int").collect()
--- End diff --
Could you add another example to show that the schema is a list of column names?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191565428
**[Test build #52357 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52357/consoleFull)** for PR 11444 at commit [`e907933`](https://github.com/apache/spark/commit/e90793324213d167a290258396d075645e1cca00).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191564729
**[Test build #52356 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52356/consoleFull)** for PR 11444 at commit [`469f99a`](https://github.com/apache/spark/commit/469f99aa0e666c216955337b6c8be296fd4cb458).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191632158
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52380/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191566939
**[Test build #52358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52358/consoleFull)** for PR 11444 at commit [`ff65b19`](https://github.com/apache/spark/commit/ff65b199d85228a68e562822f7087a7d3ebf1ab8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r55322763
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,139 @@ def __eq__(self, other):
for v in [ArrayType, MapType, StructType])
+_FIXED_DECIMAL = re.compile("decimal\\((\\d+),(\\d+)\\)")
+
+
+def _parse_basic_datatype_string(s):
+ if s == "null":
+ return NullType()
+ elif s == "boolean":
+ return BooleanType()
+ elif s == "byte":
+ return ByteType()
+ elif s == "short":
+ return ShortType()
+ elif s == "int":
+ return IntegerType()
+ elif s == "long":
+ return LongType()
+ elif s == "float":
+ return FloatType()
+ elif s == "double":
+ return DoubleType()
+ elif s == "decimal":
+ return DecimalType()
+ elif _FIXED_DECIMAL.match(s):
+ m = _FIXED_DECIMAL.match(json_value)
+ return DecimalType(int(m.group(1)), int(m.group(2)))
+ elif s == "string":
+ return StringType()
+ elif s == "date":
+ return DateType()
+ elif s == "timestamp":
+ return TimestampType()
+ elif s == "binary":
+ return BinaryType()
+ else:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+
+
+def _ignore_brackets_split(s, separator):
+ parts = []
+ buf = ""
+ level = 0
+ for c in s:
+ if c == "<":
+ level += 1
+ buf += c
+ elif c == ">":
+ if level == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ level -= 1
+ buf += c
+ elif c == separator and level > 0:
+ buf += c
+ elif c == separator:
+ parts.append(buf)
+ buf = ""
+ else:
+ buf += c
+
+ if len(buf) == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ parts.append(buf)
+ return parts
+
+
+def _parse_struct_type_string(s):
+ parts = _ignore_brackets_split(s, ",")
+ fields = []
+ for part in parts:
+ name_and_type = _ignore_brackets_split(part, ":")
+ if len(name_and_type) != 2:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ field_name = name_and_type[0].strip()
+ field_type = _parse_datatype_string(name_and_type[1])
+ fields.append(StructField(field_name, field_type))
+ return StructType(fields)
+
+
+def _parse_datatype_string(s):
--- End diff --
We could doc it for now, (and create a JIRA for it)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191631690
**[Test build #52380 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52380/consoleFull)** for PR 11444 at commit [`57eea79`](https://github.com/apache/spark/commit/57eea790a9af4fa11b6bc7b1fbfb1e5ec93d9edc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-192075706
**[Test build #52437 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52437/consoleFull)** for PR 11444 at commit [`d739239`](https://github.com/apache/spark/commit/d7392399f0ba1b934a18d9aea9e0e4e70b6721e9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191575730
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-190728868
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191564739
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52356/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191565430
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191572121
**[Test build #52363 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52363/consoleFull)** for PR 11444 at commit [`30f4137`](https://github.com/apache/spark/commit/30f4137d410f8d5b32ac28f58521c74aadb9cbf4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191565432
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52357/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r55449167
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,129 @@ def __eq__(self, other):
for v in [ArrayType, MapType, StructType])
+_FIXED_DECIMAL = re.compile("decimal\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)")
+
+
+_BRACKETS = {'(': ')', '[': ']', '{': '}'}
+
+
+def _parse_basic_datatype_string(s):
+ if s in _all_atomic_types.keys():
+ return _all_atomic_types[s]()
+ elif s == "int":
+ return IntegerType()
+ elif _FIXED_DECIMAL.match(s):
+ m = _FIXED_DECIMAL.match(s)
--- End diff --
Is it possible to call _FIXED_DECIMAL.match(s) once ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191055447
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52288/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191031528
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54667527
--- Diff: python/pyspark/sql/context.py ---
@@ -63,7 +63,33 @@ def toDF(self, schema=None, sampleRatio=None):
"""
return sqlContext.createDataFrame(self, schema, sampleRatio)
+ def schema(self, datatype):
--- End diff --
The difference is: this `schema` method support parsing schema string, and guarantee no type inference. It also supports RDD of custom objects, and use a flat schema here, and it will wrap the custom object into a row automatically.
Is it worth to add this new method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191573386
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54672512
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,139 @@ def __eq__(self, other):
for v in [ArrayType, MapType, StructType])
+_FIXED_DECIMAL = re.compile("decimal\\((\\d+),(\\d+)\\)")
+
+
+def _parse_basic_datatype_string(s):
+ if s == "null":
+ return NullType()
+ elif s == "boolean":
+ return BooleanType()
+ elif s == "byte":
+ return ByteType()
+ elif s == "short":
+ return ShortType()
+ elif s == "int":
+ return IntegerType()
+ elif s == "long":
+ return LongType()
+ elif s == "float":
+ return FloatType()
+ elif s == "double":
+ return DoubleType()
+ elif s == "decimal":
+ return DecimalType()
+ elif _FIXED_DECIMAL.match(s):
+ m = _FIXED_DECIMAL.match(json_value)
+ return DecimalType(int(m.group(1)), int(m.group(2)))
+ elif s == "string":
+ return StringType()
+ elif s == "date":
+ return DateType()
+ elif s == "timestamp":
+ return TimestampType()
+ elif s == "binary":
+ return BinaryType()
+ else:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+
+
+def _ignore_brackets_split(s, separator):
+ parts = []
+ buf = ""
+ level = 0
+ for c in s:
+ if c == "<":
+ level += 1
+ buf += c
+ elif c == ">":
+ if level == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ level -= 1
+ buf += c
+ elif c == separator and level > 0:
+ buf += c
+ elif c == separator:
+ parts.append(buf)
+ buf = ""
+ else:
+ buf += c
+
+ if len(buf) == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ parts.append(buf)
+ return parts
+
+
+def _parse_struct_type_string(s):
+ parts = _ignore_brackets_split(s, ",")
+ fields = []
+ for part in parts:
+ name_and_type = _ignore_brackets_split(part, ":")
+ if len(name_and_type) != 2:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ field_name = name_and_type[0].strip()
+ field_type = _parse_datatype_string(name_and_type[1])
+ fields.append(StructField(field_name, field_type))
+ return StructType(fields)
+
+
+def _parse_datatype_string(s):
+ """
+ Parses the given data type string to a :class:`DataType`, the data type string format equals
+ to `DataType.simpleString`, except that top level struct type can omit the `struct<>`.
+
+ >>> _parse_datatype_string("int")
+ IntegerType
+ >>> _parse_datatype_string("a: int, b: string ")
+ StructType(List(StructField(a,IntegerType,true),StructField(b,StringType,true)))
+ >>> _parse_datatype_string("a: array<int>")
+ StructType(List(StructField(a,ArrayType(IntegerType,true),true)))
+ >>> _parse_datatype_string(" map<string , string > ")
+ MapType(StringType,StringType,true)
+
+ >>> # Error cases
+ >>> _parse_datatype_string("blabla") # doctest: +IGNORE_EXCEPTION_DETAIL
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+ >>> _parse_datatype_string("a: int,") # doctest: +IGNORE_EXCEPTION_DETAIL
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+ >>> _parse_datatype_string("array<int") # doctest: +IGNORE_EXCEPTION_DETAIL
+ Traceback (most recent call last):
+ ...
+ ValueError:...
+ >>> _parse_datatype_string("map<int, boolean>>") # doctest: +IGNORE_EXCEPTION_DETAIL
--- End diff --
I mentioned it in the doc above, the format is almost equal to `DataType.simpleString`, with some differences.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54979408
--- Diff: python/pyspark/sql/context.py ---
@@ -50,18 +50,56 @@
def _monkey_patch_RDD(sqlContext):
def toDF(self, schema=None, sampleRatio=None):
"""
- Converts current :class:`RDD` into a :class:`DataFrame`
+ Converts current :class:`RDD` into a :class:`DataFrame` according to the given schema.
+ If the given schema is None or just column names, the schema will be inferred by scanning
+ the data in this RDD according to the sampleRatio.
- This is a shorthand for ``sqlContext.createDataFrame(rdd, schema, sampleRatio)``
+ Note that, the given schema must match the real data, or exception will be thrown at
+ runtime. If the given schema is not StructType, it will be wrapped into a StructType as
+ its only field, and the field name will be "value", each record will also be wrapped into a
+ tuple, which can be converted to row later.
- :param schema: a StructType or list of names of columns
+ :param schema: a :class:`DataType` or a datatype string or list of names of columns
--- End diff --
Do you want to support a schema string like this? "name age"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-193384541
@cloud-fan What's the reason we do not want to improve createDataFrame() directly?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191573388
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52362/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191567153
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52358/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191567147
**[Test build #52358 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52358/consoleFull)** for PR 11444 at commit [`ff65b19`](https://github.com/apache/spark/commit/ff65b199d85228a68e562822f7087a7d3ebf1ab8).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191564735
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54661176
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,139 @@ def __eq__(self, other):
for v in [ArrayType, MapType, StructType])
+_FIXED_DECIMAL = re.compile("decimal\\((\\d+),(\\d+)\\)")
+
+
+def _parse_basic_datatype_string(s):
+ if s == "null":
+ return NullType()
+ elif s == "boolean":
--- End diff --
see _parse_datatype_json_string
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54822526
--- Diff: python/pyspark/sql/context.py ---
@@ -63,7 +63,42 @@ def toDF(self, schema=None, sampleRatio=None):
"""
return sqlContext.createDataFrame(self, schema, sampleRatio)
+ def schema(self, datatype):
+ """
+ Converts current :class:`RDD` into a :class:`DataFrame` according to the given data type.
+ Note that, the given datatype must match the real data, or exception will be thrown at
+ runtime. If the given datatype is not StructType, it will be wrapped into a StructType as
+ its only field, and the field name will be "value", each record will also be wrapped into a
+ list, which can be converted to row later.
+
+ :param datatype:
+ a :class:`DataType` or a data type string. The data type string format equals to
+ `DataType.simpleString`, except that top level struct type can omit the `struct<>` and
+ numeric types use `typeName()` as their format, e.g. use `byte` instead of `tinyint` for
+ ByteType. We can also use `int` as a short name for IntegerType.
+
+ """
+ if isinstance(datatype, basestring):
+ datatype = _parse_datatype_string(datatype)
+
+ if not isinstance(datatype, DataType):
+ raise TypeError("datatype should be DataType or string, but got: %s" % datatype)
+ else:
+ def verify(obj):
+ _verify_type(obj, datatype)
--- End diff --
Since Python and Scala/Java have different range of int/long numbers, does _verify_type check if the actual value is within the allowed range?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54980211
--- Diff: python/pyspark/sql/context.py ---
@@ -50,18 +50,56 @@
def _monkey_patch_RDD(sqlContext):
def toDF(self, schema=None, sampleRatio=None):
"""
- Converts current :class:`RDD` into a :class:`DataFrame`
+ Converts current :class:`RDD` into a :class:`DataFrame` according to the given schema.
+ If the given schema is None or just column names, the schema will be inferred by scanning
+ the data in this RDD according to the sampleRatio.
- This is a shorthand for ``sqlContext.createDataFrame(rdd, schema, sampleRatio)``
+ Note that, the given schema must match the real data, or exception will be thrown at
+ runtime. If the given schema is not StructType, it will be wrapped into a StructType as
+ its only field, and the field name will be "value", each record will also be wrapped into a
+ tuple, which can be converted to row later.
- :param schema: a StructType or list of names of columns
+ :param schema: a :class:`DataType` or a datatype string or list of names of columns
--- End diff --
Let me check the grammar of pandas. Currently I just followed the `DataType.simplyString` format
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-190728862
**[Test build #52242 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52242/consoleFull)** for PR 11444 at commit [`2d1c2e5`](https://github.com/apache/spark/commit/2d1c2e527414cc119eee82fe92d67fb5b0a753c0).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54979052
--- Diff: python/pyspark/sql/context.py ---
@@ -50,18 +50,56 @@
def _monkey_patch_RDD(sqlContext):
def toDF(self, schema=None, sampleRatio=None):
"""
- Converts current :class:`RDD` into a :class:`DataFrame`
+ Converts current :class:`RDD` into a :class:`DataFrame` according to the given schema.
+ If the given schema is None or just column names, the schema will be inferred by scanning
+ the data in this RDD according to the sampleRatio.
- This is a shorthand for ``sqlContext.createDataFrame(rdd, schema, sampleRatio)``
+ Note that, the given schema must match the real data, or exception will be thrown at
+ runtime. If the given schema is not StructType, it will be wrapped into a StructType as
+ its only field, and the field name will be "value", each record will also be wrapped into a
+ tuple, which can be converted to row later.
- :param schema: a StructType or list of names of columns
+ :param schema: a :class:`DataType` or a datatype string or list of names of columns
--- End diff --
What's the format of datatype string?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191031445
**[Test build #52283 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52283/consoleFull)** for PR 11444 at commit [`9435b98`](https://github.com/apache/spark/commit/9435b9853a65b854172572455993063f0cef397e).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54666521
--- Diff: python/pyspark/sql/context.py ---
@@ -63,7 +63,33 @@ def toDF(self, schema=None, sampleRatio=None):
"""
return sqlContext.createDataFrame(self, schema, sampleRatio)
+ def schema(self, datatype):
--- End diff --
How will this compared to `toDF` and `createDataFrame`, should we make them identical?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-192075897
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52437/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191621204
**[Test build #52380 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52380/consoleFull)** for PR 11444 at commit [`57eea79`](https://github.com/apache/spark/commit/57eea790a9af4fa11b6bc7b1fbfb1e5ec93d9edc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-193611600
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-193985934
LGTM, merging this into master.
We still does not support a schema string with only column names, and quoted name in schema string, could you create follow up JIRA for them?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-192075891
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54979899
--- Diff: python/pyspark/sql/context.py ---
@@ -50,18 +50,56 @@
def _monkey_patch_RDD(sqlContext):
def toDF(self, schema=None, sampleRatio=None):
"""
- Converts current :class:`RDD` into a :class:`DataFrame`
+ Converts current :class:`RDD` into a :class:`DataFrame` according to the given schema.
+ If the given schema is None or just column names, the schema will be inferred by scanning
+ the data in this RDD according to the sampleRatio.
- This is a shorthand for ``sqlContext.createDataFrame(rdd, schema, sampleRatio)``
+ Note that, the given schema must match the real data, or exception will be thrown at
+ runtime. If the given schema is not StructType, it will be wrapped into a StructType as
+ its only field, and the field name will be "value", each record will also be wrapped into a
+ tuple, which can be converted to row later.
- :param schema: a StructType or list of names of columns
+ :param schema: a :class:`DataType` or a datatype string or list of names of columns
--- End diff --
I think we should say more here.
versionchanged is used to track the changes, do not need all the details there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54979708
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,129 @@ def __eq__(self, other):
for v in [ArrayType, MapType, StructType])
+_FIXED_DECIMAL = re.compile("decimal\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)")
+
+
+_BRACKETS = {'(': ')', '[': ']', '{': '}'}
+
+
+def _parse_basic_datatype_string(s):
+ if s in _all_atomic_types.keys():
+ return _all_atomic_types[s]()
+ elif s == "int":
+ return IntegerType()
+ elif _FIXED_DECIMAL.match(s):
+ m = _FIXED_DECIMAL.match(s)
+ return DecimalType(int(m.group(1)), int(m.group(2)))
+ else:
+ raise ValueError("Could not parse datatype: %s" % s)
+
+
+def _ignore_brackets_split(s, separator):
+ """
+ Splits the given string by given separator, but ignore separators inside brackets pairs, e.g.
+ given "a,b" and separator ",", it will return ["a", "b"], but given "a<b,c>, d", it will return
+ ["a<b,c>", "d"].
+ """
+ parts = []
+ buf = ""
+ level = 0
+ for c in s:
+ if c in _BRACKETS.keys():
+ level += 1
+ buf += c
+ elif c in _BRACKETS.values():
+ if level == 0:
+ raise ValueError("Brackets are not correctly paired: %s" % s)
+ level -= 1
+ buf += c
+ elif c == separator and level > 0:
+ buf += c
+ elif c == separator:
+ parts.append(buf)
+ buf = ""
+ else:
+ buf += c
+
+ if len(buf) == 0:
+ raise ValueError("The %s cannot be the last char: %s" % (separator, s))
+ parts.append(buf)
+ return parts
+
+
+def _parse_struct_type_string(s):
--- End diff --
Does this support StructType?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-190728275
**[Test build #52242 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52242/consoleFull)** for PR 11444 at commit [`2d1c2e5`](https://github.com/apache/spark/commit/2d1c2e527414cc119eee82fe92d67fb5b0a753c0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191055384
**[Test build #52288 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52288/consoleFull)** for PR 11444 at commit [`6aeab1e`](https://github.com/apache/spark/commit/6aeab1ed8439109e356b978e91480865a760c1b1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54980428
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,129 @@ def __eq__(self, other):
for v in [ArrayType, MapType, StructType])
+_FIXED_DECIMAL = re.compile("decimal\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)")
+
+
+_BRACKETS = {'(': ')', '[': ']', '{': '}'}
+
+
+def _parse_basic_datatype_string(s):
+ if s in _all_atomic_types.keys():
+ return _all_atomic_types[s]()
+ elif s == "int":
+ return IntegerType()
+ elif _FIXED_DECIMAL.match(s):
+ m = _FIXED_DECIMAL.match(s)
+ return DecimalType(int(m.group(1)), int(m.group(2)))
+ else:
+ raise ValueError("Could not parse datatype: %s" % s)
+
+
+def _ignore_brackets_split(s, separator):
+ """
+ Splits the given string by given separator, but ignore separators inside brackets pairs, e.g.
+ given "a,b" and separator ",", it will return ["a", "b"], but given "a<b,c>, d", it will return
+ ["a<b,c>", "d"].
+ """
+ parts = []
+ buf = ""
+ level = 0
+ for c in s:
+ if c in _BRACKETS.keys():
+ level += 1
+ buf += c
+ elif c in _BRACKETS.values():
+ if level == 0:
+ raise ValueError("Brackets are not correctly paired: %s" % s)
+ level -= 1
+ buf += c
+ elif c == separator and level > 0:
+ buf += c
+ elif c == separator:
+ parts.append(buf)
+ buf = ""
+ else:
+ buf += c
+
+ if len(buf) == 0:
+ raise ValueError("The %s cannot be the last char: %s" % (separator, s))
+ parts.append(buf)
+ return parts
+
+
+def _parse_struct_type_string(s):
--- End diff --
oh sorry the name is confusing. It should be `_parse_struct_fields_string`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191573361
**[Test build #52362 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52362/consoleFull)** for PR 11444 at commit [`a29cbb5`](https://github.com/apache/spark/commit/a29cbb5541d52baad3acb3c238784ceb8c44ecdc).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54979292
--- Diff: python/pyspark/sql/context.py ---
@@ -50,18 +50,56 @@
def _monkey_patch_RDD(sqlContext):
def toDF(self, schema=None, sampleRatio=None):
"""
- Converts current :class:`RDD` into a :class:`DataFrame`
+ Converts current :class:`RDD` into a :class:`DataFrame` according to the given schema.
+ If the given schema is None or just column names, the schema will be inferred by scanning
+ the data in this RDD according to the sampleRatio.
- This is a shorthand for ``sqlContext.createDataFrame(rdd, schema, sampleRatio)``
+ Note that, the given schema must match the real data, or exception will be thrown at
+ runtime. If the given schema is not StructType, it will be wrapped into a StructType as
+ its only field, and the field name will be "value", each record will also be wrapped into a
+ tuple, which can be converted to row later.
- :param schema: a StructType or list of names of columns
+ :param schema: a :class:`DataType` or a datatype string or list of names of columns
--- End diff --
it's in the `versionchanged` section.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54980380
--- Diff: python/pyspark/sql/context.py ---
@@ -50,18 +50,56 @@
def _monkey_patch_RDD(sqlContext):
def toDF(self, schema=None, sampleRatio=None):
"""
- Converts current :class:`RDD` into a :class:`DataFrame`
+ Converts current :class:`RDD` into a :class:`DataFrame` according to the given schema.
+ If the given schema is None or just column names, the schema will be inferred by scanning
+ the data in this RDD according to the sampleRatio.
- This is a shorthand for ``sqlContext.createDataFrame(rdd, schema, sampleRatio)``
+ Note that, the given schema must match the real data, or exception will be thrown at
+ runtime. If the given schema is not StructType, it will be wrapped into a StructType as
+ its only field, and the field name will be "value", each record will also be wrapped into a
+ tuple, which can be converted to row later.
- :param schema: a StructType or list of names of columns
+ :param schema: a :class:`DataType` or a datatype string or list of names of columns
:param samplingRatio: the sample ratio of rows used for inferring
:return: a DataFrame
+ .. versionchanged:: 2.0
--- End diff --
`createDataFrame` hasn't changed. It still accepts `StructType` or list or tuple, all the new stuff is handled here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54979766
--- Diff: python/pyspark/sql/context.py ---
@@ -50,18 +50,56 @@
def _monkey_patch_RDD(sqlContext):
def toDF(self, schema=None, sampleRatio=None):
"""
- Converts current :class:`RDD` into a :class:`DataFrame`
+ Converts current :class:`RDD` into a :class:`DataFrame` according to the given schema.
+ If the given schema is None or just column names, the schema will be inferred by scanning
+ the data in this RDD according to the sampleRatio.
- This is a shorthand for ``sqlContext.createDataFrame(rdd, schema, sampleRatio)``
+ Note that, the given schema must match the real data, or exception will be thrown at
+ runtime. If the given schema is not StructType, it will be wrapped into a StructType as
+ its only field, and the field name will be "value", each record will also be wrapped into a
+ tuple, which can be converted to row later.
- :param schema: a StructType or list of names of columns
+ :param schema: a :class:`DataType` or a datatype string or list of names of columns
:param samplingRatio: the sample ratio of rows used for inferring
:return: a DataFrame
+ .. versionchanged:: 2.0
--- End diff --
Should we move these into createDataFrame() ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191567151
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-190728871
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52242/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191031530
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52283/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `createDataFram...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-194096618
ah i see, thanks! JIRA created: https://issues.apache.org/jira/browse/SPARK-13762
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54567086
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,139 @@ def __eq__(self, other):
for v in [ArrayType, MapType, StructType])
+_FIXED_DECIMAL = re.compile("decimal\\((\\d+),(\\d+)\\)")
+
+
+def _parse_basic_datatype_string(s):
+ if s == "null":
+ return NullType()
+ elif s == "boolean":
+ return BooleanType()
+ elif s == "byte":
+ return ByteType()
+ elif s == "short":
+ return ShortType()
+ elif s == "int":
+ return IntegerType()
+ elif s == "long":
+ return LongType()
+ elif s == "float":
+ return FloatType()
+ elif s == "double":
+ return DoubleType()
+ elif s == "decimal":
+ return DecimalType()
+ elif _FIXED_DECIMAL.match(s):
+ m = _FIXED_DECIMAL.match(json_value)
+ return DecimalType(int(m.group(1)), int(m.group(2)))
+ elif s == "string":
+ return StringType()
+ elif s == "date":
+ return DateType()
+ elif s == "timestamp":
+ return TimestampType()
+ elif s == "binary":
+ return BinaryType()
+ else:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+
+
+def _ignore_brackets_split(s, separator):
+ parts = []
+ buf = ""
+ level = 0
+ for c in s:
+ if c == "<":
+ level += 1
+ buf += c
+ elif c == ">":
+ if level == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ level -= 1
+ buf += c
+ elif c == separator and level > 0:
+ buf += c
+ elif c == separator:
+ parts.append(buf)
+ buf = ""
+ else:
+ buf += c
+
+ if len(buf) == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ parts.append(buf)
+ return parts
+
+
+def _parse_struct_type_string(s):
+ parts = _ignore_brackets_split(s, ",")
+ fields = []
+ for part in parts:
+ name_and_type = _ignore_brackets_split(part, ":")
+ if len(name_and_type) != 2:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ field_name = name_and_type[0].strip()
+ field_type = _parse_datatype_string(name_and_type[1])
+ fields.append(StructField(field_name, field_type))
+ return StructType(fields)
+
+
+def _parse_datatype_string(s):
--- End diff --
We can create a one-pass parser for it if performance matters here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-192070415
**[Test build #52437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52437/consoleFull)** for PR 11444 at commit [`d739239`](https://github.com/apache/spark/commit/d7392399f0ba1b934a18d9aea9e0e4e70b6721e9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] improve the `toDF()` method...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11444#issuecomment-191580114
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13593][SQL] add a `schema()` method to ...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11444#discussion_r54660587
--- Diff: python/pyspark/sql/types.py ---
@@ -681,6 +681,139 @@ def __eq__(self, other):
for v in [ArrayType, MapType, StructType])
+_FIXED_DECIMAL = re.compile("decimal\\((\\d+),(\\d+)\\)")
+
+
+def _parse_basic_datatype_string(s):
+ if s == "null":
+ return NullType()
+ elif s == "boolean":
+ return BooleanType()
+ elif s == "byte":
+ return ByteType()
+ elif s == "short":
+ return ShortType()
+ elif s == "int":
+ return IntegerType()
+ elif s == "long":
+ return LongType()
+ elif s == "float":
+ return FloatType()
+ elif s == "double":
+ return DoubleType()
+ elif s == "decimal":
+ return DecimalType()
+ elif _FIXED_DECIMAL.match(s):
+ m = _FIXED_DECIMAL.match(json_value)
+ return DecimalType(int(m.group(1)), int(m.group(2)))
+ elif s == "string":
+ return StringType()
+ elif s == "date":
+ return DateType()
+ elif s == "timestamp":
+ return TimestampType()
+ elif s == "binary":
+ return BinaryType()
+ else:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+
+
+def _ignore_brackets_split(s, separator):
+ parts = []
+ buf = ""
+ level = 0
+ for c in s:
+ if c == "<":
+ level += 1
+ buf += c
+ elif c == ">":
+ if level == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ level -= 1
+ buf += c
+ elif c == separator and level > 0:
+ buf += c
+ elif c == separator:
+ parts.append(buf)
+ buf = ""
+ else:
+ buf += c
+
+ if len(buf) == 0:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ parts.append(buf)
+ return parts
+
+
+def _parse_struct_type_string(s):
+ parts = _ignore_brackets_split(s, ",")
+ fields = []
+ for part in parts:
+ name_and_type = _ignore_brackets_split(part, ":")
+ if len(name_and_type) != 2:
+ raise ValueError("Cannot parse datatype string: %s" % s)
+ field_name = name_and_type[0].strip()
+ field_type = _parse_datatype_string(name_and_type[1])
+ fields.append(StructField(field_name, field_type))
+ return StructType(fields)
+
+
+def _parse_datatype_string(s):
--- End diff --
It does not matter, I think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org