You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/07 01:35:22 UTC
[GitHub] [spark] HyukjinKwon commented on a change in pull request #26783:
[SPARK-30153][PYTHON][WIP] Extend data exchange options for vectorized UDF
functions with vanilla Arrow serialization
HyukjinKwon commented on a change in pull request #26783: [SPARK-30153][PYTHON][WIP] Extend data exchange options for vectorized UDF functions with vanilla Arrow serialization
URL: https://github.com/apache/spark/pull/26783#discussion_r355089647
##########
File path: python/pyspark/serializers.py
##########
@@ -427,6 +427,64 @@ def load_stream(self, stream):
'Invalid number of pandas.DataFrames in group {0}'.format(dataframes_in_group))
+class ArrowStreamArrowUDFSerializer(ArrowStreamSerializer):
Review comment:
Also, I don't think we should add this special serializer for each pandas UDF but rather we should just have a switch to each existing serializers so that it works for all types of pandas UDFs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org