[GitHub] [spark] sadhen commented on pull request #32320: [SPARK-35211][PYTHON] verify inferred schema for _create_dataframe

[GitBox <gi...@apache.org>]

sadhen commented on pull request #32320:
URL: https://github.com/apache/spark/pull/32320#issuecomment-826278011


   @HyukjinKwon There are little differences in `_createFromRDD` and `_createFromLocal`. If I do inferred type verification in a separate PR, I need to insert the following code snippet twice:
   
   ``` python
                   verify_func = _make_type_verifier(struct) if verifySchema else lambda _: True
   
                   def verified_converter(obj):
                       verify_func(obj)
                       return converter(obj)
                   data = inner_map(verified_converter, data)
   ```
   
   That's why I did a refactor.
   
   Let me create another PR for inferred type verification.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org