You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ueshin (via GitHub)" <gi...@apache.org> on 2023/08/03 22:33:48 UTC
[GitHub] [spark] ueshin opened a new pull request, #42328: [SPARK-43967][SQL][PYTHON] Add memory limits for Python UDTF analyzer
ueshin opened a new pull request, #42328:
URL: https://github.com/apache/spark/pull/42328
### What changes were proposed in this pull request?
Adds memory limits for Python UDTF analyzer.
- `spark.sql.analyzer.pythonUDTF.analyzeInPython.memory` (`None` by default)
> The amount of memory to be allocated to PySpark for Python UDTF analyzer, in MiB unless otherwise specified. If set, PySpark memory for Python UDTF analyzer will be limited to this amount. If not set, Spark will not limit Python's memory use and it is up to the application to avoid exceeding the overhead memory space shared with other non-JVM processes.\nNote: Windows does not support resource limiting and actual resource is not limited on MacOS.
### Why are the changes needed?
Python UDTF analyzer should be able to set a memory limit.
### Does this PR introduce _any_ user-facing change?
Users will be able to set the memory limit for Python UDTF analyzer.
### How was this patch tested?
Existing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #42328: [SPARK-43967][SQL][PYTHON] Add memory limits for Python UDTF analyzer
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #42328: [SPARK-43967][SQL][PYTHON] Add memory limits for Python UDTF analyzer
URL: https://github.com/apache/spark/pull/42328
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #42328: [SPARK-43967][SQL][PYTHON] Add memory limits for Python UDTF analyzer
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42328:
URL: https://github.com/apache/spark/pull/42328#issuecomment-1665108108
merged to master
there is merge conflict, seems need a separate PR for 3.5 (if needed) @ueshin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #42328: [SPARK-43967][SQL][PYTHON] Add memory limits for Python UDTF analyzer
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #42328:
URL: https://github.com/apache/spark/pull/42328#issuecomment-1664724038
cc @allisonwang-db @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #42328: [SPARK-43967][SQL][PYTHON] Add memory limits for Python UDTF analyzer
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42328:
URL: https://github.com/apache/spark/pull/42328#issuecomment-1665109686
oh, this config is added since version("4.0.0"), ignore ^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org