[jira] [Created] (SPARK-26697) ShuffleBlockFetcherIterator can log block sizes in addition to num blocks

["Imran Rashid (JIRA)" <ji...@apache.org>]

Imran Rashid created SPARK-26697:
------------------------------------

             Summary: ShuffleBlockFetcherIterator can log block sizes in addition to num blocks
                 Key: SPARK-26697
                 URL: https://issues.apache.org/jira/browse/SPARK-26697
             Project: Spark
          Issue Type: Improvement
          Components: Shuffle, Spark Core
    Affects Versions: 2.4.0
            Reporter: Imran Rashid


Every so often I find myself looking at executor logs, wondering why something is going wrong (failed exec, or seems to be stuck etc) and I wish I had a bit more info about shuffle sizes.  {{ShuffleBlockFetcherIterator}} logs the number of local & remote blocks, but not their sizes.  It would be really easy to add in size info too.

eg. instead of 

{noformat}
19/01/22 23:10:49.978 Executor task launch worker for task 8 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks including 1 local blocks and 1 remote blocks
{noformat}

it should be 

{noformat}
19/01/22 23:10:49.978 Executor task launch worker for task 8 INFO ShuffleBlockFetcherIterator: Getting 2 (194.0 B) non-empty blocks including 1 (97.0 B) local blocks and 1 (97.0 B) remote blocks
{noformat}

I know this is a really minor change, but I've wanted it multiple times, seems worth it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org