You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2025/11/06 03:42:52 UTC
[PR] [SPARK-54211][PYTHON][DOCS] Update exmple of mapInArrow to use arrow builtin function [spark]
zhengruifeng opened a new pull request, #52909:
URL: https://github.com/apache/spark/pull/52909
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
Update exmple of mapInArrow to use arrow builtin function
### Why are the changes needed?
1, it is encouraged to use arrow builtin function in arrow-based UDFs, so that additional conversion between arrow and pandas can be saved;
2, mapInArrow doctest on longer depends on pandas, so that we can still test it when pyarrow is installed but pandas is missing
### Does this PR introduce _any_ user-facing change?
yes, doc-only change
### How was this patch tested?
ci
### Was this patch authored or co-authored using generative AI tooling?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54211][PYTHON][DOCS] Update exmple of mapInArrow to use arrow builtin function [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #52909:
URL: https://github.com/apache/spark/pull/52909#discussion_r2508838769
##########
python/pyspark/sql/dataframe.py:
##########
@@ -6524,8 +6524,7 @@ def mapInArrow(
>>> df = spark.createDataFrame([(1, 21), (2, 30)], ("id", "age"))
>>> def filter_func(iterator):
... for batch in iterator:
- ... pdf = batch.to_pandas()
- ... yield pa.RecordBatch.from_pandas(pdf[pdf.id == 1])
+ ... yield batch.filter(pa.compute.field("id") == 1)
>>> df.mapInArrow(filter_func, df.schema).show()
Review Comment:
yeah, I am fixing it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54211][PYTHON][DOCS] Update exmple of mapInArrow to use arrow builtin function [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52909:
URL: https://github.com/apache/spark/pull/52909#issuecomment-3495105486
Merged to master for Apache Spark 4.2.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54211][PYTHON][DOCS] Update exmple of mapInArrow to use arrow builtin function [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #52909:
URL: https://github.com/apache/spark/pull/52909#discussion_r2508838769
##########
python/pyspark/sql/dataframe.py:
##########
@@ -6524,8 +6524,7 @@ def mapInArrow(
>>> df = spark.createDataFrame([(1, 21), (2, 30)], ("id", "age"))
>>> def filter_func(iterator):
... for batch in iterator:
- ... pdf = batch.to_pandas()
- ... yield pa.RecordBatch.from_pandas(pdf[pdf.id == 1])
+ ... yield batch.filter(pa.compute.field("id") == 1)
>>> df.mapInArrow(filter_func, df.schema).show()
Review Comment:
yeah, I am fixing it in https://github.com/apache/spark/pull/52965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54211][PYTHON][DOCS] Update exmple of mapInArrow to use arrow builtin function [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #52909:
URL: https://github.com/apache/spark/pull/52909#discussion_r2505979202
##########
python/pyspark/sql/dataframe.py:
##########
@@ -6524,8 +6524,7 @@ def mapInArrow(
>>> df = spark.createDataFrame([(1, 21), (2, 30)], ("id", "age"))
>>> def filter_func(iterator):
... for batch in iterator:
- ... pdf = batch.to_pandas()
- ... yield pa.RecordBatch.from_pandas(pdf[pdf.id == 1])
+ ... yield batch.filter(pa.compute.field("id") == 1)
>>> df.mapInArrow(filter_func, df.schema).show()
Review Comment:
The failure happens here.
```
File "/__w/spark/spark/python/pyspark/sql/dataframe.py", line 409, in pyspark.sql.dataframe.DataFrame.mapInArrow
Failed example:
df.mapInArrow(filter_func, df.schema).show()
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.10/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest pyspark.sql.dataframe.DataFrame.mapInArrow[3]>", line 1, in <module>
df.mapInArrow(filter_func, df.schema).show()
File "/__w/spark/spark/python/pyspark/sql/classic/dataframe.py", line 285, in show
print(self._show_string(n, truncate, vertical))
File "/__w/spark/spark/python/pyspark/sql/classic/dataframe.py", line 303, in _show_string
return self._jdf.showString(n, 20, vertical)
File "/__w/spark/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/java_gateway.py", line 1362, in __call__
return_value = get_return_value(
File "/__w/spark/spark/python/pyspark/errors/exceptions/captured.py", line 269, in deco
raise converted from None
File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 3343, in main
process()
File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 3334, in process
serializer.dump_stream(out_iter, outfile)
File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 187, in dump_stream
return super(ArrowStreamUDFSerializer, self).dump_stream(wrap_and_init_stream(), stream)
File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 121, in dump_stream
for batch in iterator:
File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 167, in wrap_and_init_stream
for batch, _ in iterator:
File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 2857, in func
for result_batch, result_type in result_iter:
File "<doctest pyspark.sql.dataframe.DataFrame.mapInArrow[2]>", line 3, in filter_func
yield batch.filter(pa.compute.field("id") == 1)
File "pyarrow/table.pxi", line 2580, in pyarrow.lib.RecordBatch.filter
return _pc().filter(self, mask, null_selection_behavior)
File "/usr/local/lib/python3.10/dist-packages/pyarrow/compute.py", line 263, in wrapper
return func.call(args, options, memory_pool)
File "pyarrow/_compute.pyx", line 372, in pyarrow._compute.Function.call
_pack_compute_args(args, &c_batch.values)
File "pyarrow/_compute.pyx", line 505, in pyarrow._compute._pack_compute_args
raise TypeError(f"Got unexpected argument type {type(val)} "
pyspark.errors.exceptions.captured.PythonException:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54211][PYTHON][DOCS] Update exmple of mapInArrow to use arrow builtin function [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #52909: [SPARK-54211][PYTHON][DOCS] Update exmple of mapInArrow to use arrow builtin function
URL: https://github.com/apache/spark/pull/52909
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org