You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhouyuan (via GitHub)" <gi...@apache.org> on 2025/04/03 20:57:03 UTC
[I] [VL] Story: Improve PySpark support [incubator-gluten]
zhouyuan opened a new issue, #9224:
URL: https://github.com/apache/incubator-gluten/issues/9224
### Description
Gluten has basic support on Arrow UDF
https://github.com/apache/incubator-gluten/pull/5462
We need to verify and improve below methods use in PySpark:
Python UDF
Pandas UDF
Arrow UDF
Python UDTF
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Story: Improve PySpark support [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #9224:
URL: https://github.com/apache/incubator-gluten/issues/9224#issuecomment-2805121870
Hi @zjuwangg
In-general there are two parts I learned so far. I will try to update in this issue once I have more progress
- PySpark itself - ideally gluten should be able to support all dataframe operator, and do fallback under not supported case. However we do see it will report failure on some corner cases. This is due to lack of such tests. Based on previous experience, it will benefit a lot if we port such tests to gluten, and the issues will occur then:
https://github.com/apache/spark/tree/master/python/pyspark/sql/tests
- The 2nd part is for the Pandas/Arrow support. It is commonly used in AI/ML related workloads - to do the data cleanup. As pandas is more commonly used. Gluten should have enabled the general support. The TODO list should be include more tests and cover the rest of the APIs.
Arrow on pandas
https://spark.apache.org/docs/latest/api/python/user_guide/sql/arrow_pandas.html
Using arrow as the transformer layer
Pandas vectorized UDFs
https://spark.apache.org/docs/latest/api/python/user_guide/sql/arrow_pandas.html#pandas-udfs-a-k-a-vectorized-udfs
Arrow python UDF
https://spark.apache.org/docs/latest/api/python/user_guide/sql/arrow_pandas.html#arrow-python-udfs
thanks, -yuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Story: Improve PySpark support [incubator-gluten]
Posted by "zjuwangg (via GitHub)" <gi...@apache.org>.
zjuwangg commented on issue #9224:
URL: https://github.com/apache/incubator-gluten/issues/9224#issuecomment-2804905460
Is there any more detailed plan or design document on how doing so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org