You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kimjune01 (via GitHub)" <gi...@apache.org> on 2026/05/09 16:35:09 UTC
[PR] fix: remove double-counting of write_time in shuffle writer [datafusion-ballista]
kimjune01 opened a new pull request, #1673:
URL: https://github.com/apache/datafusion-ballista/pull/1673
# Which issue does this PR close?
Closes #631
# Rationale for this change
`write_time` in `ShuffleWriterExec` reports the total stream duration instead of actual write time. The `None` partitioning branch wraps the entire `write_stream_to_disk` call in a timer, but `write_stream_to_disk` already records `write_time` internally per batch. The outer timer double-counts every batch write and also includes time spent waiting for upstream data.
# What changes are included in this PR?
Removed the outer `write_time` timer in the `None` partitioning branch of `ShuffleWriterExec::execute_shuffle_write`. `write_stream_to_disk` handles its own `write_time` measurement per batch, so the outer timer was redundant.
# Are there any user-facing changes?
`write_time` now accurately reflects disk write duration rather than total stream processing time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: remove double-counting of write_time in shuffle writer [datafusion-ballista]
Posted by "kimjune01 (via GitHub)" <gi...@apache.org>.
kimjune01 closed pull request #1673: fix: remove double-counting of write_time in shuffle writer
URL: https://github.com/apache/datafusion-ballista/pull/1673
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: remove double-counting of write_time in shuffle writer [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1673:
URL: https://github.com/apache/datafusion-ballista/pull/1673#issuecomment-4413404585
Thanks @kimjune01. I think these changes may have already been made in main branch. Could you rebase this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: remove double-counting of write_time in shuffle writer [datafusion-ballista]
Posted by "kimjune01 (via GitHub)" <gi...@apache.org>.
kimjune01 commented on PR #1673:
URL: https://github.com/apache/datafusion-ballista/pull/1673#issuecomment-4417499145
You're right — these changes are already in main. Closing. Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org