You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/05/23 18:34:02 UTC
[GitHub] [spark] xinrong-databricks opened a new pull request, #36640: Correct error messages when creating DataFrame
xinrong-databricks opened a new pull request, #36640:
URL: https://github.com/apache/spark/pull/36640
### What changes were proposed in this pull request?
Correct error messages when creating DataFrame.
Previously, when the first element of RDD is 0, we raise a ValueError "The first row in RDD is empty, can not infer schema".
However, a TypeError "Can not infer schema for type: <class 'int'>" should be raised instead.
### Why are the changes needed?
Improve error messages and debugability.
### Does this PR introduce _any_ user-facing change?
Error message change.
Before:
```py
>>> spark.createDataFrame(spark._sc.parallelize([0, 1]))
Traceback (most recent call last):
...
ValueError: The first row in RDD is empty, can not infer schema
```
After:
```py
>>> spark.createDataFrame(spark._sc.parallelize([0, 1]))
Traceback (most recent call last):
...
TypeError: Can not infer schema for type: <class 'int'>
```
### How was this patch tested?
Existing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct error messages when creating DataFrame from an RDD with the first element `0`
Posted by GitBox <gi...@apache.org>.
itholic commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r880065419
##########
python/pyspark/sql/session.py:
##########
@@ -611,7 +611,7 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
+ if first != 0 and not first:
Review Comment:
> Oh got it. What about we check if it is `collections.abc.Sized`, and if the size is `0` here instead?
+1 for this comment.
Otherwise looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct error messages when creating DataFrame from an RDD with the first element `0`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r880021145
##########
python/pyspark/sql/session.py:
##########
@@ -611,7 +611,7 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
+ if first != 0 and not first:
Review Comment:
I think we can just ignore this condition. If RDD is empty, `RDD.first` seems throwing an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on PR #36640:
URL: https://github.com/apache/spark/pull/36640#issuecomment-1138893839
After further consideration, the behaviors below are expected:
```py
>>> spark.createDataFrame(spark._sc.parallelize([[1], [2]])).show()
+---+
| _1|
+---+
| 1|
| 2|
+---+
>>> spark.createDataFrame(spark._sc.parallelize([[],[]])).show()
++
||
++
||
||
++
```
that's because bolded **[]** in [**[]**,..] marks a row in PySpark.
So the PR is adjusted to enable `[[],[]]` input, by simply bypassing the check `if not first:`.
The input as `[None, None]` will raise the `ValueError` as before.
```py
>>> spark.createDataFrame(spark._sc.parallelize([None, None])).show()
Traceback (most recent call last):
...
ValueError: The first row in RDD is empty, can not infer schema
```
CC @ueshin @HyukjinKwon @itholic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct error messages when creating DataFrame from an RDD with the first element `0`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r879786294
##########
python/pyspark/sql/session.py:
##########
@@ -611,7 +611,7 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
+ if first != 0 and not first:
Review Comment:
When `first` is a nested list with the first element is an empty list, PySpark diverges from Spark SQL. A follow-up PR will be filed to adjust that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct error messages when creating DataFrame from an RDD with the first element `0`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r880031452
##########
python/pyspark/sql/session.py:
##########
@@ -611,7 +611,7 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
+ if first != 0 and not first:
Review Comment:
Oh got it. What about we check if it is `collections.abc.Sized`, and if the size is `0` here instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
ueshin commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r887134684
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +611,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if first is None:
Review Comment:
That's fine, but I feel the error messages are pretty inconsistent:
1. for the case the first value is `None`:
```py
>>> spark.createDataFrame(spark._sc.parallelize([None, None]))
...
TypeError: Can not infer schema for type: <class 'NoneType'>
```
should be:
```py
ValueError: The first row in RDD is empty, can not infer schema
```
?
2. if the first value is an int or a non-empty string:
```py
>>> spark.createDataFrame(spark._sc.parallelize([0, 1]))
...
TypeError: Can not infer schema for type: <class 'int'>
>>> spark.createDataFrame(spark._sc.parallelize(["a", "b"]))
...
TypeError: Can not infer schema for type: <class 'str'>
```
then I think the error message for the case of an empty string:
```py
>>> spark.createDataFrame(spark._sc.parallelize(["", "a"]))
...
ValueError: The first row in RDD is empty, can not infer schema
```
should be:
```py
TypeError: Can not infer schema for type: <class 'str'>
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r887601346
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +612,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if isinstance(first, Sized) and len(first) == 0:
Review Comment:
we can do .. but I started to think that this check isn't very useful ... especially given that it will trigger another Spark job that can potentially be very expensive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct error messages when creating DataFrame from an RDD with the first element `0`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r879786294
##########
python/pyspark/sql/session.py:
##########
@@ -611,7 +611,7 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
+ if first != 0 and not first:
Review Comment:
When `first` is a nested list with the first element is an empty list, PySpark diverges from scala DataFrame API. A follow-up PR will be filed to adjust that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct error messages when creating DataFrame from an RDD with the first element `0`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r880030149
##########
python/pyspark/sql/session.py:
##########
@@ -611,7 +611,7 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
+ if first != 0 and not first:
Review Comment:
Sorry I meant an RDD like `spark._sc.parallelize([[],[]])`, Spark SQL is able to create a DataFrame with an array column but PySpark raises an exception.
```py
>>> spark._sc.parallelize([[],[]]).first()
[]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on PR #36640:
URL: https://github.com/apache/spark/pull/36640#issuecomment-1139237103
CC @zhengruifeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on PR #36640:
URL: https://github.com/apache/spark/pull/36640#issuecomment-1139769583
Hmm, I can see why `[]` leads @zhengruifeng to think about `Seq` in Scala, since they seem to have the same semantics, that is, representing a row, regarding `createDataFrame`.
I have no strong opinion on supporting `[[], []]` in PySpark or not. What do you think about it?
@ueshin @HyukjinKwon @itholic @Yikun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
ueshin commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r887134684
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +611,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if first is None:
Review Comment:
That's fine, but I feel the error messages are pretty inconsistent:
1. for the case the first value is `None`:
```py
>>> spark.createDataFrame(spark._sc.parallelize([None, None]))
...
TypeError: Can not infer schema for type: <class 'NoneType'>
```
should be:
```py
ValueError: The first row in RDD is empty, can not infer schema
```
?
2. if the first value is int:
```py
>>> spark.createDataFrame(spark._sc.parallelize([0, 1]))
...
TypeError: Can not infer schema for type: <class 'int'>
```
then I think the error message for the case of string:
```py
>>> spark.createDataFrame(spark._sc.parallelize(["", "a"]))
...
ValueError: The first row in RDD is empty, can not infer schema
```
should be:
```py
TypeError: Can not infer schema for type: <class 'str'>
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
ueshin commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r886035398
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +611,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if first is None:
Review Comment:
What about `""`, btw?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r887367163
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +611,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if first is None:
Review Comment:
Maybe we can explicitly check if the type is dictionary, list or row instead of Sized ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r887439636
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +612,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if isinstance(first, Sized) and len(first) == 0:
Review Comment:
Or maybe shall we just remove this check away? We take 100 rows and infer so I don't think take 1 + failfast is super beneficial but it burdens more overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r887627360
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +612,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if isinstance(first, Sized) and len(first) == 0:
Review Comment:
+1, the `first` job seems unnecessary:
1, `if samplingRatio is None`, we should directly take first 100 non-None rows, check on the 100 rows and infer the schema;
2, `if samplingRatio is not None`, the `_merge_type` job should be enough. If we need to check the count, we can add a `count` into `_merge_type`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
itholic commented on PR #36640:
URL: https://github.com/apache/spark/pull/36640#issuecomment-1139193145
> After further consideration, the behaviors below are expected:
>
> ```python
> >>> spark.createDataFrame(spark._sc.parallelize([[1], [2]])).show()
> +---+
> | _1|
> +---+
> | 1|
> | 2|
> +---+
>
> >>> spark.createDataFrame(spark._sc.parallelize([[],[]])).show()
> ++
> ||
> ++
> ||
> ||
> ++
> ```
>
> that's because bolded **[]** in [**[]**,..] marks a row in PySpark.
>
> So the PR is adjusted to enable `[[],[]]` input, by simply bypassing the check `if not first:`.
>
> The input as `[None, None]` will raise the `ValueError` as before.
>
> ```python
> >>> spark.createDataFrame(spark._sc.parallelize([None, None])).show()
> Traceback (most recent call last):
> ...
> ValueError: The first row in RDD is empty, can not infer schema
> ```
>
> CC @ueshin @HyukjinKwon @itholic
Sounds good to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #36640:
URL: https://github.com/apache/spark/pull/36640#issuecomment-1139270710
> So the PR is adjusted to enable [[],[]] input,
seems that the scala side do not support it:
```
scala> val rdd=sc.parallelize(Seq(Seq(), Seq()))
rdd: org.apache.spark.rdd.RDD[Seq[Nothing]] = ParallelCollectionRDD[4] at parallelize at <console>:23
scala> val df = spark.createDataFrame(rdd=rdd)
<console>:23: error: inferred type arguments [Seq[Nothing]] do not conform to method createDataFrame's type parameter bounds [A <: Product]
val df = spark.createDataFrame(rdd=rdd)
^
<console>:23: error: type mismatch;
found : org.apache.spark.rdd.RDD[Seq[Nothing]]
required: org.apache.spark.rdd.RDD[A]
val df = spark.createDataFrame(rdd=rdd)
^
```
BTW, if we are to support ` [[],[]] ` in the python side, I think we need to update the PR title/descprition and add a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r886780576
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +611,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if first is None:
Review Comment:
It cannot infer the type from `None` or `""`. It has to be either a list, dictionary or `Row` instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
ueshin commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r885944686
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +611,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if first is None:
Review Comment:
If `rdd.first()` uses `if rs:` check, `first` will never be `[]`, `{}`, `0` or `""`, either?
I guess we should fix `rdd.first()` to properly return the first row, and `if first is None:` check here is vaild according to the error message `"The first row in RDD is empty, can not infer schema"`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r884368543
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +611,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if first is None:
+ raise ValueError("The first row in RDD is empty, can not infer schema")
Review Comment:
Sorry I was confused with `DataFrame.take`. `RDD.take` seems like always retuning something because it's `items: List[T] = []`. Can we just check if the length is 0 instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct error messages when creating DataFrame from an RDD with the first element `0`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r880030149
##########
python/pyspark/sql/session.py:
##########
@@ -611,7 +611,7 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
+ if first != 0 and not first:
Review Comment:
Sorry I meant an RDD like `spark._sc.parallelize([[],[]])`, Spark SQL is able to create a DataFrame with an array column but PySpark raises an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct error messages when creating DataFrame from an RDD with the first element `0`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r880035876
##########
python/pyspark/sql/session.py:
##########
@@ -611,7 +611,7 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
+ if first != 0 and not first:
Review Comment:
Even if we let it pass this if-else check, PySpark returns an empty DataFrame as below
```py
>>> spark.createDataFrame(spark._sc.parallelize([[],[]])).show()
++
||
++
||
||
++
```
whereas Spark SQL returns a DataFrame where each row is an empty array.
Let me check the code paths further and adjust that. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct error messages when creating DataFrame from an RDD with the first element `0`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r880035876
##########
python/pyspark/sql/session.py:
##########
@@ -611,7 +611,7 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
+ if first != 0 and not first:
Review Comment:
Even if we let it pass this if-else check, PySpark returns an empty DataFrame as below
```py
>>> spark.createDataFrame(spark._sc.parallelize([[],[]])).show()
++
||
++
||
||
++
```
whereas Spark SQL returns a DataFrame of empty arrays.
Let me check the code paths further and adjust that. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r884369130
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +611,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if first is None:
Review Comment:
Hm, can this be `None`? Seems like `if rs:` check `RDD.first` will always throw an exception in case this is `None`. Can we just check if this has a length and larger than `0`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on PR #36640:
URL: https://github.com/apache/spark/pull/36640#issuecomment-1139758806
Sorry I should've attached an example first.
`[[], []]` input in Scala side is as below
```scala
scala> val rdd=sc.parallelize(List(List(), List()))
rdd: org.apache.spark.rdd.RDD[List[Nothing]] = ParallelCollectionRDD[11] at parallelize at <console>:23
scala> val df = spark.createDataFrame(rdd=rdd)
df: org.apache.spark.sql.DataFrame = [value: array<void>]
scala> df.show()
+-----+
|value|
+-----+
| []|
| []|
+-----+
```
I may add a test. Let me know if you have other concerns. Thanks @zhengruifeng .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
ueshin commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r886035398
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +611,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if first is None:
Review Comment:
How about `""`, btw?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
ueshin commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r885944686
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +611,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if first is None:
Review Comment:
If `rdd.first()` uses `if rs:` check, `first` will never be `[]`, `{}`, `0` or `""`, either?
I guess we should fix `rdd.first()` to properly return the first row, and `if first is None:` check here is vaild according to the error message `"The first row in RDD is empty, can not infer schema"`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
URL: https://github.com/apache/spark/pull/36640
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #36640:
URL: https://github.com/apache/spark/pull/36640#issuecomment-1141576890
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
ueshin commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r885963200
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +611,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if first is None:
Review Comment:
The `first` can be `None`.
```py
>>> print(sc.parallelize([None, 2, 3, 4]).first())
None
```
We should also check if `first is None` because `None` is not `Sized`.
```py
>>> isinstance(None, Sized)
False
```
Also we need a test to check the first row is `None` and it fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r887596704
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +612,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if isinstance(first, Sized) and len(first) == 0:
Review Comment:
what about filtering out the `None` rows:
```
first = rdd.filter( no None cond ).first()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a diff in pull request #36640: [SPARK-39262][PYTHON] Correct the behavior of creating DataFrame from an RDD
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on code in PR #36640:
URL: https://github.com/apache/spark/pull/36640#discussion_r884941255
##########
python/pyspark/sql/session.py:
##########
@@ -611,8 +611,8 @@ def _inferSchema(
:class:`pyspark.sql.types.StructType`
"""
first = rdd.first()
- if not first:
- raise ValueError("The first row in RDD is empty, " "can not infer schema")
+ if first is None:
Review Comment:
Sounds good! Updated.
Just to confirm, we may not want to support ` spark.createDataFrame(spark._sc.parallelize([[], []]))` for now, is that right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org