You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ueshin (via GitHub)" <gi...@apache.org> on 2025/07/18 02:05:55 UTC
[PR] [SPARK-52861][PYTHON] Skip Row object creation in Arrow-optimized UDTF execution [spark]
ueshin opened a new pull request, #51546:
URL: https://github.com/apache/spark/pull/51546
### What changes were proposed in this pull request?
Skips `Row` object creation in Arrow-optimized UDTF execution.
### Why are the changes needed?
The `Row` object creation is not necessary in Arrow-optimized UDTF execution, although it's expensive.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
The existing tests, and manual benchmarks.
```py
def profile(f, *args, _n=10, **kwargs):
import cProfile
import pstats
import gc
st = None
for _ in range(5):
f(*args, **kwargs)
for _ in range(_n):
gc.collect()
with cProfile.Profile() as pr:
ret = f(*args, **kwargs)
if st is None:
st = pstats.Stats(pr)
else:
st.add(pstats.Stats(pr))
st.sort_stats("time", "cumulative").print_stats()
return ret
from pyspark.sql.conversion import ArrowTableToRowsConversion, LocalDataToArrowConversion
from pyspark.sql.types import *
data = [
(i if i % 1000 else None, str(i))
for i in range(1000000)
]
schema = (
StructType()
.add("i", IntegerType(), nullable=True)
.add("s", StringType(), nullable=True)
)
def to_arrow():
return LocalDataToArrowConversion.convert(data, schema, use_large_var_types=False)
def from_arrow(tbl, return_as_tuples):
return ArrowTableToRowsConversion.convert(tbl, schema, return_as_tuples=return_as_tuples)
tbl = to_arrow()
profile(from_arrow, tbl, return_as_tuples=False)
profile(from_arrow, tbl, return_as_tuples=True)
```
- before (`return_as_tuples=False`)
```
60655810 function calls in 14.112 seconds
```
- after (`return_as_tuples=True`)
```
20328060 function calls in 5.613 seconds
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52861][PYTHON] Skip Row object creation in Arrow-optimized UDTF execution [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51546:
URL: https://github.com/apache/spark/pull/51546#issuecomment-3089690153
cc @peter-toth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52861][PYTHON] Skip Row object creation in Arrow-optimized UDTF execution [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin closed pull request #51546: [SPARK-52861][PYTHON] Skip Row object creation in Arrow-optimized UDTF execution
URL: https://github.com/apache/spark/pull/51546
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52861][PYTHON] Skip Row object creation in Arrow-optimized UDTF execution [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #51546:
URL: https://github.com/apache/spark/pull/51546#issuecomment-3089689830
Could you resolve the conflict, @ueshin ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52861][PYTHON] Skip Row object creation in Arrow-optimized UDTF execution [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #51546:
URL: https://github.com/apache/spark/pull/51546#issuecomment-3090627073
Thanks! merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org