You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/01/07 01:20:10 UTC
[PR] [SPARK-54930][PYTHON][MINOR] Remove redundant _accumulatorRegistry.clear() call in worker.py [spark]
Yicong-Huang opened a new pull request, #53708:
URL: https://github.com/apache/spark/pull/53708
### What changes were proposed in this pull request?
Remove a redundant `_accumulatorRegistry.clear()` call in `worker.py`.
Currently there are two consecutive `clear()` calls with no accumulator-modifying code in between:
```python
shuffle.MemoryBytesSpilled = 0
shuffle.DiskBytesSpilled = 0
_accumulatorRegistry.clear() # first call
setup_spark_files(infile)
setup_broadcasts(infile)
_accumulatorRegistry.clear() # second call
```
Neither `setup_spark_files` nor `setup_broadcasts` adds anything to `_accumulatorRegistry`, so the first `clear()` is redundant.
### Why are the changes needed?
This is dead code cleanup. The redundant call was introduced when:
- SPARK-3463 (2014) added the first `clear()` after shuffle initialization
- SPARK-3030 (2014) added the second `clear()` after broadcasts setup
- SPARK-44533 (2023) refactored to extract `setup_spark_files` and `setup_broadcasts`, but preserved both `clear()` calls
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests. This is a simple dead code removal with no functional change.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54930][PYTHON] Remove redundant _accumulatorRegistry.clear() call in worker.py [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #53708:
URL: https://github.com/apache/spark/pull/53708#discussion_r2667432960
##########
python/pyspark/worker.py:
##########
@@ -3493,7 +3493,6 @@ def main(infile, outfile):
shuffle.MemoryBytesSpilled = 0
shuffle.DiskBytesSpilled = 0
- _accumulatorRegistry.clear()
Review Comment:
I feel like it's a safeguard because accumulator server is running async, and when the worker is reused ... but not sure. just speculating it. Can we dig the commits and see if there are any hint related to it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54930][PYTHON] Remove redundant _accumulatorRegistry.clear() call in worker.py [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #53708:
URL: https://github.com/apache/spark/pull/53708#discussion_r2669727439
##########
python/pyspark/worker.py:
##########
@@ -3493,7 +3493,6 @@ def main(infile, outfile):
shuffle.MemoryBytesSpilled = 0
shuffle.DiskBytesSpilled = 0
- _accumulatorRegistry.clear()
Review Comment:
Yeah, I didn’t see a strong justification to keep it based on the previous PRs. Happy to revisit if we think there’s a concrete scenario we’re missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54930][PYTHON] Remove redundant _accumulatorRegistry.clear() call in worker.py [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #53708:
URL: https://github.com/apache/spark/pull/53708#issuecomment-3721236521
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54930][PYTHON] Remove redundant _accumulatorRegistry.clear() call in worker.py [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #53708:
URL: https://github.com/apache/spark/pull/53708#discussion_r2667434054
##########
python/pyspark/worker.py:
##########
@@ -3493,7 +3493,6 @@ def main(infile, outfile):
shuffle.MemoryBytesSpilled = 0
shuffle.DiskBytesSpilled = 0
- _accumulatorRegistry.clear()
Review Comment:
Ah you already did in the pr description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54930][PYTHON] Remove redundant _accumulatorRegistry.clear() call in worker.py [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #53708:
URL: https://github.com/apache/spark/pull/53708#discussion_r2667434600
##########
python/pyspark/worker.py:
##########
@@ -3493,7 +3493,6 @@ def main(infile, outfile):
shuffle.MemoryBytesSpilled = 0
shuffle.DiskBytesSpilled = 0
- _accumulatorRegistry.clear()
Review Comment:
ah ok seems a mistake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54930][PYTHON][MINOR] Remove redundant _accumulatorRegistry.clear() call in worker.py [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53708:
URL: https://github.com/apache/spark/pull/53708#issuecomment-3716945894
## JIRA Issue Information
=== Improvement SPARK-54930 ===
Summary: Remove redundant _accumulatorRegistry.clear() call in worker.py
Assignee: None
Status: Open
Affected: ["4.1.0"]
---
*This comment was automatically generated by GitHub Actions*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54930][PYTHON] Remove redundant _accumulatorRegistry.clear() call in worker.py [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #53708: [SPARK-54930][PYTHON] Remove redundant _accumulatorRegistry.clear() call in worker.py
URL: https://github.com/apache/spark/pull/53708
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org