You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HeartSaVioR (via GitHub)" <gi...@apache.org> on 2025/06/09 02:28:46 UTC
[PR] [SPARK-50752][SQL][PYSPARK][FOLLOWUP] Respect PYTHON_UDF_MAX_RECORDS_PER_BATCH in Python worker [spark]
HeartSaVioR opened a new pull request, #51124:
URL: https://github.com/apache/spark/pull/51124
### What changes were proposed in this pull request?
This PR is a follow-up of SPARK-50752 (PR #49397), which introduced PYTHON_UDF_MAX_RECORDS_PER_BATCH into SQLConf to control the batch size of normal Python UDF. This PR is to fix the config to be effective for Python worker side as well.
### Why are the changes needed?
The original PR enabled the control of batch size for JVM side, but it wasn't properly propagated to Python worker since we missed to override the value for Python UDF. (Default value before overriding is a static one, 100)
### Does this PR introduce _any_ user-facing change?
Yes, but it's about tuning and does not impact any change on the output.
### How was this patch tested?
No automated test, since there is no good way to test this properly since missing this does not change the output itself.
This was manually tested with artificial debug logging.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50752][SQL][PYSPARK][FOLLOWUP] Respect PYTHON_UDF_MAX_RECORDS_PER_BATCH in Python worker [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #51124:
URL: https://github.com/apache/spark/pull/51124#issuecomment-2954694968
Thanks! Merging to master. (I'm merging this quickly since it's a follow-up and just 2 lines of changes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50752][SQL][PYSPARK][FOLLOWUP] Respect PYTHON_UDF_MAX_RECORDS_PER_BATCH in Python worker [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #51124: [SPARK-50752][SQL][PYSPARK][FOLLOWUP] Respect PYTHON_UDF_MAX_RECORDS_PER_BATCH in Python worker
URL: https://github.com/apache/spark/pull/51124
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org