You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2019/01/25 00:15:00 UTC
[jira] [Resolved] (SPARK-26697) ShuffleBlockFetcherIterator can log
block sizes in addition to num blocks
[ https://issues.apache.org/jira/browse/SPARK-26697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin resolved SPARK-26697.
------------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 23621
[https://github.com/apache/spark/pull/23621]
> ShuffleBlockFetcherIterator can log block sizes in addition to num blocks
> -------------------------------------------------------------------------
>
> Key: SPARK-26697
> URL: https://issues.apache.org/jira/browse/SPARK-26697
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, Spark Core
> Affects Versions: 2.4.0
> Reporter: Imran Rashid
> Assignee: Imran Rashid
> Priority: Major
> Fix For: 3.0.0
>
>
> Every so often I find myself looking at executor logs, wondering why something is going wrong (failed exec, or seems to be stuck etc) and I wish I had a bit more info about shuffle sizes. {{ShuffleBlockFetcherIterator}} logs the number of local & remote blocks, but not their sizes. It would be really easy to add in size info too.
> eg. instead of
> {noformat}
> 19/01/22 23:10:49.978 Executor task launch worker for task 8 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks including 1 local blocks and 1 remote blocks
> {noformat}
> it should be
> {noformat}
> 19/01/22 23:10:49.978 Executor task launch worker for task 8 INFO ShuffleBlockFetcherIterator: Getting 2 (194.0 B) non-empty blocks including 1 (97.0 B) local blocks and 1 (97.0 B) remote blocks
> {noformat}
> I know this is a really minor change, but I've wanted it multiple times, seems worth it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org