You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/04 16:37:15 UTC
[GitHub] [spark] lidavidm opened a new pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
lidavidm opened a new pull request #31738:
URL: https://github.com/apache/spark/pull/31738
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
As a followup for #29818, document caveats of using the Arrow selfDestruct option in toPandas, which include:
- toPandas() may be slower;
- the resulting dataframe may not support some Pandas operations due to immutable backing arrays.
### Why are the changes needed?
This will hopefully reduce user confusion as with SPARK-34463.
### Does this PR introduce _any_ user-facing change?
Yes - documentation is updated and a config setting description is updated to clearly indicate the config is experimental.
### How was this patch tested?
This is a documentation-only change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] lidavidm commented on a change in pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #31738:
URL: https://github.com/apache/spark/pull/31738#discussion_r588357765
##########
File path: python/docs/source/user_guide/arrow_pandas.rst
##########
@@ -410,3 +410,11 @@ described in `SPARK-29367 <https://issues.apache.org/jira/browse/SPARK-29367>`_
``pandas_udf``\s or :meth:`DataFrame.toPandas` with Arrow enabled. More information about the Arrow IPC change can
be read on the Arrow 0.15.0 release `blog <https://arrow.apache.org/blog/2019/10/06/0.15.0-release/#columnar-streaming-protocol-change-since-0140>`_.
+Setting Arrow self_destruct for memory savings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Spark 3.2, the Spark configuration ``spark.sql.execution.arrow.pyspark.selfDestruct.enabled`` can be used to enable PyArrow's ``self_destruct`` feature, which can save memory when creating a Pandas dataframe via ``toPandas`` by freeing Arrow-allocated memory while building the Pandas dataframe.
+This option is experimental, and some operations may fail on the resulting Pandas dataframe due to immutable backing arrays.
+Typically, you would see the error ``ValueError: buffer source array is read-only``.
+Newer versions of Pandas may fix these errors by improving support for such cases.
+Additionally, this conversion may be slower because it is single-threaded.
Review comment:
Unfortunately there's no real umbrella issue; it's an artifact of Pandas' implementation that's slowly being corrected as people find more cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-809311797
**[Test build #136650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136650/testReport)** for PR 31738 at commit [`b0115e5`](https://github.com/apache/spark/commit/b0115e576e8fc1133acee9286e067a42703b792e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] lidavidm commented on a change in pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #31738:
URL: https://github.com/apache/spark/pull/31738#discussion_r597751840
##########
File path: python/docs/source/user_guide/arrow_pandas.rst
##########
@@ -410,3 +410,11 @@ described in `SPARK-29367 <https://issues.apache.org/jira/browse/SPARK-29367>`_
``pandas_udf``\s or :meth:`DataFrame.toPandas` with Arrow enabled. More information about the Arrow IPC change can
be read on the Arrow 0.15.0 release `blog <https://arrow.apache.org/blog/2019/10/06/0.15.0-release/#columnar-streaming-protocol-change-since-0140>`_.
+Setting Arrow self_destruct for memory savings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Spark 3.2, the Spark configuration ``spark.sql.execution.arrow.pyspark.selfDestruct.enabled`` can be used to enable PyArrow's ``self_destruct`` feature, which can save memory when creating a Pandas dataframe via ``toPandas`` by freeing Arrow-allocated memory while building the Pandas dataframe.
+This option is experimental, and some operations may fail on the resulting Pandas dataframe due to immutable backing arrays.
+Typically, you would see the error ``ValueError: buffer source array is read-only``.
+Newer versions of Pandas may fix these errors by improving support for such cases.
+Additionally, this conversion may be slower because it is single-threaded.
Review comment:
We could link to https://github.com/pandas-dev/pandas/issues/33001 perhaps? It's not a blanket umbrella issue but it does cover a few hundred test cases and explains the issues facing a universal fix on their side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791177569
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40375/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791311354
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135793/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31738:
URL: https://github.com/apache/spark/pull/31738#discussion_r588032574
##########
File path: python/docs/source/user_guide/arrow_pandas.rst
##########
@@ -410,3 +410,11 @@ described in `SPARK-29367 <https://issues.apache.org/jira/browse/SPARK-29367>`_
``pandas_udf``\s or :meth:`DataFrame.toPandas` with Arrow enabled. More information about the Arrow IPC change can
be read on the Arrow 0.15.0 release `blog <https://arrow.apache.org/blog/2019/10/06/0.15.0-release/#columnar-streaming-protocol-change-since-0140>`_.
+Setting Arrow self_destruct for memory savings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Spark 3.2, the Spark configuration ``spark.sql.execution.arrow.pyspark.selfDestruct.enabled`` can be used to enable PyArrow's ``self_destruct`` feature, which can save memory when creating a Pandas dataframe via ``toPandas`` by freeing Arrow-allocated memory while building the Pandas dataframe.
+This option is experimental, and some operations may fail on the resulting Pandas dataframe due to immutable backing arrays.
+Typically, you would see the error ``ValueError: buffer source array is read-only``.
+Newer versions of Pandas may fix these errors by improving support for such cases.
+Additionally, this conversion may be slower because it is single-threaded.
Review comment:
@lidavidm it would be great if we link related issues (so we can remove these notes later when they are solved).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] lidavidm commented on a change in pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #31738:
URL: https://github.com/apache/spark/pull/31738#discussion_r603223180
##########
File path: python/docs/source/user_guide/arrow_pandas.rst
##########
@@ -410,3 +410,11 @@ described in `SPARK-29367 <https://issues.apache.org/jira/browse/SPARK-29367>`_
``pandas_udf``\s or :meth:`DataFrame.toPandas` with Arrow enabled. More information about the Arrow IPC change can
be read on the Arrow 0.15.0 release `blog <https://arrow.apache.org/blog/2019/10/06/0.15.0-release/#columnar-streaming-protocol-change-since-0140>`_.
+Setting Arrow self_destruct for memory savings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Spark 3.2, the Spark configuration ``spark.sql.execution.arrow.pyspark.selfDestruct.enabled`` can be used to enable PyArrow's ``self_destruct`` feature, which can save memory when creating a Pandas dataframe via ``toPandas`` by freeing Arrow-allocated memory while building the Pandas dataframe.
+This option is experimental, and some operations may fail on the resulting Pandas dataframe due to immutable backing arrays.
+Typically, you would see the error ``ValueError: buffer source array is read-only``.
+Newer versions of Pandas may fix these errors by improving support for such cases.
+Additionally, this conversion may be slower because it is single-threaded.
Review comment:
No, it's actually anytime that the underlying array is read-only. PyArrow is a common source of this but not the only source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-809173492
I am okay with this too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-809898936
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-802941127
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40843/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31738:
URL: https://github.com/apache/spark/pull/31738#discussion_r588032272
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2035,8 +2035,8 @@ object SQLConf {
val ARROW_PYSPARK_SELF_DESTRUCT_ENABLED =
buildConf("spark.sql.execution.arrow.pyspark.selfDestruct.enabled")
- .doc("When true, make use of Apache Arrow's self-destruct and split-blocks options " +
- "for columnar data transfers in PySpark, when converting from Arrow to Pandas. " +
+ .doc("EXPERIMENTAL. When true, make use of Apache Arrow's self-destruct and split-blocks " +
Review comment:
```suggestion
.doc("(Experimental) When true, make use of Apache Arrow's self-destruct and split-blocks " +
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31738:
URL: https://github.com/apache/spark/pull/31738#discussion_r588973127
##########
File path: python/docs/source/user_guide/arrow_pandas.rst
##########
@@ -410,3 +410,11 @@ described in `SPARK-29367 <https://issues.apache.org/jira/browse/SPARK-29367>`_
``pandas_udf``\s or :meth:`DataFrame.toPandas` with Arrow enabled. More information about the Arrow IPC change can
be read on the Arrow 0.15.0 release `blog <https://arrow.apache.org/blog/2019/10/06/0.15.0-release/#columnar-streaming-protocol-change-since-0140>`_.
+Setting Arrow self_destruct for memory savings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Spark 3.2, the Spark configuration ``spark.sql.execution.arrow.pyspark.selfDestruct.enabled`` can be used to enable PyArrow's ``self_destruct`` feature, which can save memory when creating a Pandas dataframe via ``toPandas`` by freeing Arrow-allocated memory while building the Pandas dataframe.
+This option is experimental, and some operations may fail on the resulting Pandas dataframe due to immutable backing arrays.
+Typically, you would see the error ``ValueError: buffer source array is read-only``.
+Newer versions of Pandas may fix these errors by improving support for such cases.
+Additionally, this conversion may be slower because it is single-threaded.
Review comment:
Hm, can we maybe file an issue in pandas side and link it here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31738:
URL: https://github.com/apache/spark/pull/31738#discussion_r588973127
##########
File path: python/docs/source/user_guide/arrow_pandas.rst
##########
@@ -410,3 +410,11 @@ described in `SPARK-29367 <https://issues.apache.org/jira/browse/SPARK-29367>`_
``pandas_udf``\s or :meth:`DataFrame.toPandas` with Arrow enabled. More information about the Arrow IPC change can
be read on the Arrow 0.15.0 release `blog <https://arrow.apache.org/blog/2019/10/06/0.15.0-release/#columnar-streaming-protocol-change-since-0140>`_.
+Setting Arrow self_destruct for memory savings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Spark 3.2, the Spark configuration ``spark.sql.execution.arrow.pyspark.selfDestruct.enabled`` can be used to enable PyArrow's ``self_destruct`` feature, which can save memory when creating a Pandas dataframe via ``toPandas`` by freeing Arrow-allocated memory while building the Pandas dataframe.
+This option is experimental, and some operations may fail on the resulting Pandas dataframe due to immutable backing arrays.
+Typically, you would see the error ``ValueError: buffer source array is read-only``.
+Newer versions of Pandas may fix these errors by improving support for such cases.
+Additionally, this conversion may be slower because it is single-threaded.
Review comment:
Hm, can we maybe file an issue in pandas side and like it here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-809615685
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136650/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791177569
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40375/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] lidavidm commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-797667254
> Just two questions.
>
> * When can we remove this `Experimental` tag?
It's hard to say, but once it sees some usage, we can see how many such cases in Pandas need fixing. It might be the case that most Pandas operations work; even the one in the linked issue is already fixed upstream.
> * Can we hold on this PR until we make a branch for Apache Spark 3.2.0?
No objections here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791549671
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40397/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on a change in pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
WeichenXu123 commented on a change in pull request #31738:
URL: https://github.com/apache/spark/pull/31738#discussion_r589288599
##########
File path: python/docs/source/user_guide/arrow_pandas.rst
##########
@@ -410,3 +410,11 @@ described in `SPARK-29367 <https://issues.apache.org/jira/browse/SPARK-29367>`_
``pandas_udf``\s or :meth:`DataFrame.toPandas` with Arrow enabled. More information about the Arrow IPC change can
be read on the Arrow 0.15.0 release `blog <https://arrow.apache.org/blog/2019/10/06/0.15.0-release/#columnar-streaming-protocol-change-since-0140>`_.
+Setting Arrow ``self_destruct`` for memory savings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Spark 3.2, the Spark configuration ``spark.sql.execution.arrow.pyspark.selfDestruct.enabled`` can be used to enable PyArrow's ``self_destruct`` feature, which can save memory when creating a Pandas dataframe via ``toPandas`` by freeing Arrow-allocated memory while building the Pandas dataframe.
+This option is experimental, and some operations may fail on the resulting Pandas dataframe due to immutable backing arrays.
+Typically, you would see the error ``ValueError: buffer source array is read-only``.
+Newer versions of Pandas may fix these errors by improving support for such cases.
+Additionally, this conversion may be slower because it is single-threaded.
Review comment:
Could we explicitly say which version pandas will trigger the bug ?
Currently my test show that pandas version > 1.0.5 will trigger the bug.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791519523
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40397/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791648533
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135815/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791490540
**[Test build #135815 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135815/testReport)** for PR 31738 at commit [`b231ac6`](https://github.com/apache/spark/commit/b231ac64b95d601c03dfcb5570917295700e8bdc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791154404
**[Test build #135793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135793/testReport)** for PR 31738 at commit [`af26e25`](https://github.com/apache/spark/commit/af26e25dbf937f34c8a98bec3b9a0a81afcf35ed).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791293979
**[Test build #135793 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135793/testReport)** for PR 31738 at commit [`af26e25`](https://github.com/apache/spark/commit/af26e25dbf937f34c8a98bec3b9a0a81afcf35ed).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31738:
URL: https://github.com/apache/spark/pull/31738#discussion_r603094298
##########
File path: python/docs/source/user_guide/arrow_pandas.rst
##########
@@ -410,3 +410,11 @@ described in `SPARK-29367 <https://issues.apache.org/jira/browse/SPARK-29367>`_
``pandas_udf``\s or :meth:`DataFrame.toPandas` with Arrow enabled. More information about the Arrow IPC change can
be read on the Arrow 0.15.0 release `blog <https://arrow.apache.org/blog/2019/10/06/0.15.0-release/#columnar-streaming-protocol-change-since-0140>`_.
+Setting Arrow self_destruct for memory savings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Spark 3.2, the Spark configuration ``spark.sql.execution.arrow.pyspark.selfDestruct.enabled`` can be used to enable PyArrow's ``self_destruct`` feature, which can save memory when creating a Pandas dataframe via ``toPandas`` by freeing Arrow-allocated memory while building the Pandas dataframe.
+This option is experimental, and some operations may fail on the resulting Pandas dataframe due to immutable backing arrays.
+Typically, you would see the error ``ValueError: buffer source array is read-only``.
+Newer versions of Pandas may fix these errors by improving support for such cases.
+Additionally, this conversion may be slower because it is single-threaded.
Review comment:
I doubt if that covers the current case here .. our issue happens specifically when the pandas DataFrame is from PyArrow, right?
I wonder if we can just show a reproducible example and file an issue in pandas dev side to track .. otherwise, this warning will likely stay forever ..
I am okay with as-is too .. assuming that we're sure on tracking the related issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31738:
URL: https://github.com/apache/spark/pull/31738#discussion_r588032796
##########
File path: python/docs/source/user_guide/arrow_pandas.rst
##########
@@ -410,3 +410,11 @@ described in `SPARK-29367 <https://issues.apache.org/jira/browse/SPARK-29367>`_
``pandas_udf``\s or :meth:`DataFrame.toPandas` with Arrow enabled. More information about the Arrow IPC change can
be read on the Arrow 0.15.0 release `blog <https://arrow.apache.org/blog/2019/10/06/0.15.0-release/#columnar-streaming-protocol-change-since-0140>`_.
+Setting Arrow self_destruct for memory savings
Review comment:
```suggestion
Setting Arrow `self_destruct` for Memory Savings
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-803081916
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136261/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791549671
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40397/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-802932524
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40843/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-802941127
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40843/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-809587108
**[Test build #136650 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136650/testReport)** for PR 31738 at commit [`b0115e5`](https://github.com/apache/spark/commit/b0115e576e8fc1133acee9286e067a42703b792e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-809615685
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136650/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-802941085
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40843/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791153094
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791644384
**[Test build #135815 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135815/testReport)** for PR 31738 at commit [`b231ac6`](https://github.com/apache/spark/commit/b231ac64b95d601c03dfcb5570917295700e8bdc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791177547
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40375/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31738:
URL: https://github.com/apache/spark/pull/31738#discussion_r603091063
##########
File path: python/docs/source/user_guide/arrow_pandas.rst
##########
@@ -410,3 +410,12 @@ described in `SPARK-29367 <https://issues.apache.org/jira/browse/SPARK-29367>`_
``pandas_udf``\s or :meth:`DataFrame.toPandas` with Arrow enabled. More information about the Arrow IPC change can
be read on the Arrow 0.15.0 release `blog <https://arrow.apache.org/blog/2019/10/06/0.15.0-release/#columnar-streaming-protocol-change-since-0140>`_.
+Setting Arrow ``self_destruct`` for memory savings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Spark 3.2, the Spark configuration ``spark.sql.execution.arrow.pyspark.selfDestruct.enabled`` can be used to enable PyArrow's ``self_destruct`` feature, which can save memory when creating a Pandas dataframe via ``toPandas`` by freeing Arrow-allocated memory while building the Pandas dataframe.
Review comment:
```suggestion
Since Spark 3.2, the Spark configuration ``spark.sql.execution.arrow.pyspark.selfDestruct.enabled`` can be used to enable PyArrow's ``self_destruct`` feature, which can save memory when creating a Pandas DataFrame via ``toPandas`` by freeing Arrow-allocated memory while building the Pandas DataFrame.
```
##########
File path: python/docs/source/user_guide/arrow_pandas.rst
##########
@@ -410,3 +410,12 @@ described in `SPARK-29367 <https://issues.apache.org/jira/browse/SPARK-29367>`_
``pandas_udf``\s or :meth:`DataFrame.toPandas` with Arrow enabled. More information about the Arrow IPC change can
be read on the Arrow 0.15.0 release `blog <https://arrow.apache.org/blog/2019/10/06/0.15.0-release/#columnar-streaming-protocol-change-since-0140>`_.
+Setting Arrow ``self_destruct`` for memory savings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Spark 3.2, the Spark configuration ``spark.sql.execution.arrow.pyspark.selfDestruct.enabled`` can be used to enable PyArrow's ``self_destruct`` feature, which can save memory when creating a Pandas dataframe via ``toPandas`` by freeing Arrow-allocated memory while building the Pandas dataframe.
+This option is experimental, and some operations may fail on the resulting Pandas dataframe due to immutable backing arrays.
Review comment:
```suggestion
This option is experimental, and some operations may fail on the resulting Pandas DataFrame due to immutable backing arrays.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-809494668
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41232/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791154404
**[Test build #135793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135793/testReport)** for PR 31738 at commit [`af26e25`](https://github.com/apache/spark/commit/af26e25dbf937f34c8a98bec3b9a0a81afcf35ed).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791648533
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135815/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] BryanCutler commented on a change in pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
BryanCutler commented on a change in pull request #31738:
URL: https://github.com/apache/spark/pull/31738#discussion_r597241868
##########
File path: python/docs/source/user_guide/arrow_pandas.rst
##########
@@ -410,3 +410,11 @@ described in `SPARK-29367 <https://issues.apache.org/jira/browse/SPARK-29367>`_
``pandas_udf``\s or :meth:`DataFrame.toPandas` with Arrow enabled. More information about the Arrow IPC change can
be read on the Arrow 0.15.0 release `blog <https://arrow.apache.org/blog/2019/10/06/0.15.0-release/#columnar-streaming-protocol-change-since-0140>`_.
+Setting Arrow ``self_destruct`` for memory savings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Spark 3.2, the Spark configuration ``spark.sql.execution.arrow.pyspark.selfDestruct.enabled`` can be used to enable PyArrow's ``self_destruct`` feature, which can save memory when creating a Pandas dataframe via ``toPandas`` by freeing Arrow-allocated memory while building the Pandas dataframe.
+This option is experimental, and some operations may fail on the resulting Pandas dataframe due to immutable backing arrays.
+Typically, you would see the error ``ValueError: buffer source array is read-only``.
Review comment:
Would it be good to say a workaround is to make a copy of the column(s) used in the operation? I suppose they could just disable the setting is most cases though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #31738:
URL: https://github.com/apache/spark/pull/31738
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791490540
**[Test build #135815 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135815/testReport)** for PR 31738 at commit [`b231ac6`](https://github.com/apache/spark/commit/b231ac64b95d601c03dfcb5570917295700e8bdc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-803081916
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136261/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791542215
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40397/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791174105
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40375/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-790766278
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] lidavidm commented on a change in pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #31738:
URL: https://github.com/apache/spark/pull/31738#discussion_r589571748
##########
File path: python/docs/source/user_guide/arrow_pandas.rst
##########
@@ -410,3 +410,11 @@ described in `SPARK-29367 <https://issues.apache.org/jira/browse/SPARK-29367>`_
``pandas_udf``\s or :meth:`DataFrame.toPandas` with Arrow enabled. More information about the Arrow IPC change can
be read on the Arrow 0.15.0 release `blog <https://arrow.apache.org/blog/2019/10/06/0.15.0-release/#columnar-streaming-protocol-change-since-0140>`_.
+Setting Arrow ``self_destruct`` for memory savings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since Spark 3.2, the Spark configuration ``spark.sql.execution.arrow.pyspark.selfDestruct.enabled`` can be used to enable PyArrow's ``self_destruct`` feature, which can save memory when creating a Pandas dataframe via ``toPandas`` by freeing Arrow-allocated memory while building the Pandas dataframe.
+This option is experimental, and some operations may fail on the resulting Pandas dataframe due to immutable backing arrays.
+Typically, you would see the error ``ValueError: buffer source array is read-only``.
+Newer versions of Pandas may fix these errors by improving support for such cases.
+Additionally, this conversion may be slower because it is single-threaded.
Review comment:
I think I haven't fully explained the nature of this - it's not any single issue in Pandas, nor is it specific to any particular version. Instead, it's just that depending on how each Pandas operation was implemented underneath, it may or may not have been _declared_ to accept an immutable backing array, independently of whether that operation _could be implemented_ on an immutable array. So whether you see this will depend on what exactly you do with the dataframe, and there's no one version range we can list or one issue we can link to. And indeed, you could see this error see this _without_ this Arrow option enabled; it's just much less likely, since there will be few cases that Arrow can perform a zero-copy conversion in that case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-790766278
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-809494668
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41232/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-809311797
**[Test build #136650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136650/testReport)** for PR 31738 at commit [`b0115e5`](https://github.com/apache/spark/commit/b0115e576e8fc1133acee9286e067a42703b792e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-791311354
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135793/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] lidavidm commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-790752450
CC @WeichenXu123 and @BryanCutler.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] lidavidm commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-810154304
Thank you both for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31738: [SPARK-34463][PYSPARK][DOCS] Document caveats of Arrow selfDestruct
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31738:
URL: https://github.com/apache/spark/pull/31738#issuecomment-802898545
**[Test build #136261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136261/testReport)** for PR 31738 at commit [`19183a0`](https://github.com/apache/spark/commit/19183a035a8c062793190caa4a1cde7564a13f8b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org