[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36545: [SPARK-39168][PYTHON] Use all values in a python list when inferring ArrayType schema

[GitBox <gi...@apache.org>]

HyukjinKwon commented on code in PR #36545:
URL: https://github.com/apache/spark/pull/36545#discussion_r875352636


##########
python/pyspark/sql/session.py:
##########
@@ -570,10 +570,20 @@ def _inferSchemaFromList(
         if not data:
             raise ValueError("can not infer schema from empty dataset")
         infer_dict_as_struct = self._jconf.inferDictAsStruct()
+        infer_array_from_first_element = self._jconf.legacyInferArrayTypeFromFirstElement()

Review Comment:
   Yeah, that's right that it causes a behaviour change. However, the (previous) string type coercion behaviour in an element of an array is actually a mistake I believe. For example, such type coercion is not supported in regular type inference:
   
   ```python
   >>> spark.createDataFrame([{"a": "1"}, {"a" :2}])
   Traceback (most recent call last):
     ...
   TypeError: field a: Can not merge type <class 'pyspark.sql.types.StringType'> and <class 'pyspark.sql.types.LongType'>
   ```
   
   So, what this PR actually does is to match the behaviour with non-nested type inference. The switch was added for users dependent on the previous behaviour.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org