You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Michael Mior <mm...@uwaterloo.ca> on 2017/07/20 15:44:14 UTC

Task partition ID in Spark event logs

I see there's a comment in the TaskInfo class that the index may not be the
same as the ID of the RDD partition the task is computing. Under what
circumstances *will* the ID by the same? If there are zero guarantees, any
suggestions on how to grab this info from the scheduler to populate a new
field inside TaskInfo?

Cheers,
--
Michael Mior
mmior@apache.org