You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Chen Jin <ka...@gmail.com> on 2014/02/11 20:14:46 UTC

How to log Shuffle read/write size

Hi All,

Apart from getting the shuffle read/write size from application UI, is
there any way to enable the metrics in the log file?

Thank you very much,

-chen

Re: How to log Shuffle read/write size

Posted by Andrew Or <an...@gmail.com>.

Hi Chen,

Have you looked at JobLogger.scala? You can register it with SparkContext
and it will log task metrics onTaskEnd. Unfortunately, there is currently
not a configurable option to enable this. We are looking into a more
systematic way to persist the UI information in a more flexible format
(JSON).

Andrew


2014-02-11 11:14 GMT-08:00 Chen Jin <ka...@gmail.com>:

> Hi All,
>
> Apart from getting the shuffle read/write size from application UI, is
> there any way to enable the metrics in the log file?
>
> Thank you very much,
>
> -chen
>