[GitHub] [spark] itholic commented on pull request #40280: [SPARK-42671][CONNECT] Fix bug for createDataFrame from complex type schema

["itholic (via GitHub)" <gi...@apache.org>]

itholic commented on PR #40280:
URL: https://github.com/apache/spark/pull/40280#issuecomment-1455565581

   Thanks @panbingkun for the nice fix!
   Btw, think I found another `createDataFrame` bug which is not working properly with non-nullable schema as below:
   ```python
   >>> from pyspark.sql.types import *
   >>> schema_false = StructType([StructField("id", IntegerType(), False)])
   >>> spark.createDataFrame([[1]], schema=schema_false)
   Traceback (most recent call last):
   ...
   pyspark.errors.exceptions.connect.AnalysisException: [NULLABLE_COLUMN_OR_FIELD] Column or field `id` is nullable while it's required to be non-nullable.
   ```
   whereas working find with nullable schema as below:
   ```python
   >>> schema_true = StructType([StructField("id", IntegerType(), True)])
   >>> spark.createDataFrame([[1]], schema=schema_true)
   DataFrame[id: int]
   ```
   
   Do you have any idea what might be causing this? Could you take a look at it if you're interested in? I have filed an issue at SPARK-42679.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org