You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/01/13 03:00:09 UTC

[GitHub] [spark] stczwd opened a new pull request #35185: [SPARK-37831][CORE][WIP] add task partition id in TaskInfo and Task Metrics

stczwd opened a new pull request #35185:
URL: https://github.com/apache/spark/pull/35185


   ### Why are the changes needed?
   There is no partition id in the current task metrics. It is difficult to track the task metrics of a specific partition or the stage metrics of processing data, especially when the stage was retried.
   ```
   class TaskData private[spark](
       val taskId: Long,
       val index: Int,
       val attempt: Int,
       val launchTime: Date,
       val resultFetchStart: Option[Date],
       @JsonDeserialize(contentAs = classOf[JLong])
       val duration: Option[Long],
       val executorId: String,
       val host: String,
       val status: String,
       val taskLocality: String,
       val speculative: Boolean,
       val accumulatorUpdates: Seq[AccumulableInfo],
       val errorMessage: Option[String] = None,
       val taskMetrics: Option[TaskMetrics] = None,
       val executorLogs: Map[String, String],
       val schedulerDelay: Long,
       val gettingResultTime: Long) 
   ```
   
   ### Does this PR introduce _any_ user-facing change?
   no
   
   ### How was this patch tested?
   add new tests
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r785993842



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       I have test it with event files, leveldb and rocksdb, and it is ok with rocksdb and event files, which retruns `"partitionId" : -1,`. But when testing leveldb, we get `"partitionId" : 0,`, which is different from what we expected.
   
   BTW, the test scenario is generating eventLog and leveldb/rocksdb with old spark-client and reading it on new spark-client which has this pr.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r786339392



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       Would be good to explore how we can default the value to -1 if missing.
   Else cleaning up all level-db would be a prerequisite to leveraging this.
   
   I am not sure what is the cleanest way to enforce incompatible leveldb format changes - IIRC you had made some changes recently which require regen, right @dongjoon-hyun ? Any suggestions ? (IIRC we did not bump up the store version)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm edited a comment on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1014966382


   > > Took an initial pass through the PR and added some comments - overall looks good. We would need to make sure that skew join and partition coalescing in SQL interact well with this change.
   > 
   > Thanks for you reply. I have test partition coalescing in SQL interact, it works well with this change.
   
   What I want @cloud-fan, @dongjoon-hyun, etc who are more familiar with SQL to look at is - given a single partition gets computed by multiple tasks (for skew), or multiple partitions are getting computed by single task (for coalascing) what is the expectation (between the 'original' reducer stage, and executed reducer stage) Also whether this is compatible with future evolution of sql.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r816434674



##########
File path: core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
##########
@@ -599,7 +599,8 @@ class EventLoggingListenerSuite extends SparkFunSuite with LocalSparkContext wit
       stageId: Int,
       taskType: String,
       executorMetrics: ExecutorMetrics): SparkListenerTaskEnd = {
-    val taskInfo = new TaskInfo(taskId, taskIndex, 0, 1553291556000L, executorId, "executor",
+    val taskInfo = new TaskInfo(
+      taskId, taskIndex, 0, taskIndex, 1553291556000L, executorId, "executor",

Review comment:
       nit: We should rename `taskIndex` as `partitionId` in this method - taskIndex happens to be partition id here.

##########
File path: core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
##########
@@ -1860,15 +1860,16 @@ abstract class AppStatusListenerSuite extends SparkFunSuite with BeforeAndAfter
 
   private def newAttempt(orig: TaskInfo, nextId: Long): TaskInfo = {
     // Task reattempts have a different ID, but the same index as the original.
-    new TaskInfo(nextId, orig.index, orig.attemptNumber + 1, time, orig.executorId,
+    new TaskInfo(
+      nextId, orig.index, orig.attemptNumber + 1, orig.partitionId, time, orig.executorId,
       s"${orig.executorId}.example.com", TaskLocality.PROCESS_LOCAL, orig.speculative)
   }
 
   private def createTasks(count: Int, execs: Array[String]): Seq[TaskInfo] = {
     (1 to count).map { id =>
       val exec = execs(id.toInt % execs.length)
       val taskId = nextTaskId()
-      new TaskInfo(taskId, taskId.toInt, 1, time, exec, s"$exec.example.com",
+      new TaskInfo(taskId, taskId.toInt, 1, taskId.toInt, time, exec, s"$exec.example.com",
         TaskLocality.PROCESS_LOCAL, id % 2 == 0)

Review comment:
       Looks like this method is incorrect to begin with.
   task index should be < number of tasks in stage and partition id should be < number of partitions.
   
   We should change both to `id` instead.
   Thoughts @stczwd, @Ngone51, @cloud-fan ?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r817516613



##########
File path: core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
##########
@@ -1860,15 +1860,16 @@ abstract class AppStatusListenerSuite extends SparkFunSuite with BeforeAndAfter
 
   private def newAttempt(orig: TaskInfo, nextId: Long): TaskInfo = {
     // Task reattempts have a different ID, but the same index as the original.
-    new TaskInfo(nextId, orig.index, orig.attemptNumber + 1, time, orig.executorId,
+    new TaskInfo(
+      nextId, orig.index, orig.attemptNumber + 1, orig.partitionId, time, orig.executorId,
       s"${orig.executorId}.example.com", TaskLocality.PROCESS_LOCAL, orig.speculative)
   }
 
   private def createTasks(count: Int, execs: Array[String]): Seq[TaskInfo] = {
     (1 to count).map { id =>
       val exec = execs(id.toInt % execs.length)
       val taskId = nextTaskId()
-      new TaskInfo(taskId, taskId.toInt, 1, time, exec, s"$exec.example.com",
+      new TaskInfo(taskId, taskId.toInt, 1, taskId.toInt, time, exec, s"$exec.example.com",
         TaskLocality.PROCESS_LOCAL, id % 2 == 0)

Review comment:
       I change it to index, which is actually id - 1, and run AppStatusListenerSuite, no error found.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r830732415



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       Could you give the reproduce steps? @jackylee-ch 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm edited a comment on pull request #35185: [SPARK-37831][CORE][WIP] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1011843060


   ~~For the specific purpose detailed, use `index` ?~~
   Scratch that - misread the description.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm edited a comment on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1012496679


   @tgravescs Agree, for the first stage attempt this should be fine.
   But for any stage attempt which is computing a subset of tasks (retries, specific partition computation via toLocalIterator, etc), this is a concern.
   
   An interesting question would be how to handle this for skewed joins - thoughts @cloud-fan ?
   
   This does look like an important missing functionality, thanks for working on it @stczwd - tagged others since I wanted to make sure I was not missing something here :-)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1014966382


   > > Took an initial pass through the PR and added some comments - overall looks good. We would need to make sure that skew join and partition coalescing in SQL interact well with this change.
   > 
   > Thanks for you reply. I have test partition coalescing in SQL interact, it works well with this change.
   
   What I want @cloud-fan, @dongjoon-hyun, etc who are more familiar with SQL to look at is - given a single partition gets computed by multiple tasks, what is the expectation ?
   Do multiple tasks end up with the same partition-id ? If yes, how do we differentiate between them in case of failures/recompute - if not, how do we identify them ? (or, if I am missing something - would love to understand how this pr is compatible with sql in partition coalascing/skew join scenarios).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1012497670


   This does look like an important missing functionality, thanks for identifying and working on it @stczwd ! Given size of the PR, I will try to circle back to it over the weekend.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r787439464



##########
File path: project/MimaExcludes.scala
##########
@@ -48,7 +48,10 @@ object MimaExcludes {
     // [SPARK-37780][SQL] QueryExecutionListener support SQLConf as constructor parameter
     ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.util.ExecutionListenerManager.this"),
     // [SPARK-37786][SQL] StreamingQueryListener support use SQLConf.get to get corresponding SessionState's SQLConf
-    ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StreamingQueryManager.this")
+    ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StreamingQueryManager.this"),
+
+    // [SPARK-37831][CORE] TaskInfo add partitionId as a constructor parameter

Review comment:
       nit: be consistent with the title?

##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
##########
@@ -35,12 +35,34 @@ class TaskInfo(
      */
     val index: Int,
     val attemptNumber: Int,
+    /**
+     * The actual RDD partition ID in this task.
+     * The ID of the RDD partition is always same across task or stage retries.
+     * This will be -1 for historical data, and available for all spark applications since 3.3.

Review comment:
       nit: `...for all applications since Spark 3.3.`

##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
##########
@@ -35,12 +35,34 @@ class TaskInfo(
      */
     val index: Int,
     val attemptNumber: Int,
+    /**
+     * The actual RDD partition ID in this task.
+     * The ID of the RDD partition is always same across task or stage retries.

Review comment:
       nit: `...across task attempts...`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r829144804



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       @jackylee-ch @mridulm  is this issue addressed? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1022002197


   kindly remind @cloud-fan @HyukjinKwon @dongjoon-hyun
   Any more question about this pr?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r830455612



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       leveldb is used by default right? If so, I did see partitionId is -1 actually:
   <img width="792" alt="WeChat05d412b13735d2fe389b4c1199307fe1" src="https://user-images.githubusercontent.com/16397174/159113195-ac7a00f8-27bb-4c95-883a-49a108c0318e.png">
   
   Could you double check again? @jackylee-ch 
   .




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r786339891



##########
File path: core/src/test/resources/HistoryServerExpectations/excludeOnFailure_for_stage_expectation.json
##########
@@ -631,6 +642,7 @@
       "taskId" : 4,
       "index" : 2,
       "attempt" : 1,
+      "partitionId": -1,
       "launchTime" : "2018-01-09T10:21:18.943GMT",
       "duration" : 16,

Review comment:
       Sounds good to keeping it as-is.
   We need to validate that missing value defaults to -1 (for both json and level db) - to test upgrade case (this is dependent on Dongjoon's suggestions above on how to mitigate the issue though).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1016127328


   kindly ping @cloud-fan @HyukjinKwon @dongjoon-hyun @rdblue 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1012022711


   cc @dongjoon-hyun @srowen @cloud-fan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r785399113



##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
##########
@@ -35,12 +35,33 @@ class TaskInfo(
      */
     val index: Int,
     val attemptNumber: Int,
+    /**
+     * The actual RDD partition ID in this task.
+     * The ID of the RDD partition is always same even task retries.

Review comment:
       `The ID of the RDD partition is always same across task or stage retries.`
   Also, add that this will be `-1` for historical data, and available for all spark applications since `3.3`.

##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
##########
@@ -35,12 +35,33 @@ class TaskInfo(
      */
     val index: Int,
     val attemptNumber: Int,
+    /**
+     * The actual RDD partition ID in this task.
+     * The ID of the RDD partition is always same even task retries.
+     */
+    val partitionId: Int,
     val launchTime: Long,
     val executorId: String,
     val host: String,
     val taskLocality: TaskLocality.TaskLocality,
     val speculative: Boolean) {
 
+  def this(
+      taskId: Long,
+      /**
+       * The index of this task within its task set. Not necessarily the same as the ID of the RDD
+       * partition that the task is computing.
+       */
+      index: Int,
+      attemptNumber: Int,
+      launchTime: Long,
+      executorId: String,
+      host: String,
+      taskLocality: TaskLocality.TaskLocality,
+      speculative: Boolean) = {
+    this(taskId, index, attemptNumber, -1, launchTime, executorId, host, taskLocality, speculative)
+  }

Review comment:
       All use of `TaskInfo` in spark should pass the `partitionId` - leaving this only for backward compatibility for end users.

##########
File path: core/src/test/resources/HistoryServerExpectations/excludeOnFailure_for_stage_expectation.json
##########
@@ -631,6 +642,7 @@
       "taskId" : 4,
       "index" : 2,
       "attempt" : 1,
+      "partitionId": -1,
       "launchTime" : "2018-01-09T10:21:18.943GMT",
       "duration" : 16,

Review comment:
       Since we expect this to default to `-1`, revert changes to this (and similar other) file ?
   Similarly for Partition Id = -1 from json files as well.

##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       +CC @thejdeep, please take a look to make sure there are no backwardly incompatible changes.
   Existing event files and/or level db's should be readable after this change.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r816749409



##########
File path: core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala
##########
@@ -599,7 +599,8 @@ class EventLoggingListenerSuite extends SparkFunSuite with LocalSparkContext wit
       stageId: Int,
       taskType: String,
       executorMetrics: ExecutorMetrics): SparkListenerTaskEnd = {
-    val taskInfo = new TaskInfo(taskId, taskIndex, 0, 1553291556000L, executorId, "executor",
+    val taskInfo = new TaskInfo(
+      taskId, taskIndex, 0, taskIndex, 1553291556000L, executorId, "executor",

Review comment:
       change `taskId, taskIndex, 0, taskIndex` to `taskId, taskIndex, 0, partitionId = taskIndex` is sound good to me.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r830827582



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       Thanks, I have reproduced the issue with the above steps. And now I see where the problem is. Basically, the fix in `JsonProtocol` only solves the compatible issue when reading event files. But, when reading from LeveDB, there's no compatible conversion for it. I tried to set the `partitionId` of `TaskDataWrapper` with the default value -1 and seems to work. Could you double-check? @jackylee-ch 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r829290863



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       Yet to hear from @dongjoon-hyun on it ... do you have any thoughts @Ngone51  ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r787534484



##########
File path: project/MimaExcludes.scala
##########
@@ -48,7 +48,10 @@ object MimaExcludes {
     // [SPARK-37780][SQL] QueryExecutionListener support SQLConf as constructor parameter
     ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.util.ExecutionListenerManager.this"),
     // [SPARK-37786][SQL] StreamingQueryListener support use SQLConf.get to get corresponding SessionState's SQLConf
-    ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StreamingQueryManager.this")
+    ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StreamingQueryManager.this"),
+
+    // [SPARK-37831][CORE] TaskInfo add partitionId as a constructor parameter

Review comment:
       ok




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] jackylee-ch commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
jackylee-ch commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r830735879



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       Firstly, use old spark-client to generate eventLog and leveldb with these configs;
   ```
   spark.history.fs.logDirectory       /tmp/spark-client/spark-old/eventLog
   spark.history.store.path            /tmp/spark-client/spark-old/kvstore
   spark.history.store.hybridStore.diskBackend     LEVELDB
   ```
   Then, use new spark-client to read eventLog and level db with the configs in step 1;
   
   These are the reproduce steps and configs that I have used. @Ngone51 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r829144804



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       @jackylee-ch @mridulm  is this issue addressed? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1084594939


   thanks, merging to master/3.3!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] jackylee-ch commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
jackylee-ch commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1084453784


   cc @cloud-fan @mridulm @Ngone51 @HyukjinKwon @MaxGekk GA has passed, can we merge this pr to master/3.3.0?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1055414369


   >  I wanted to confirm whether we have tests where task index != partitionId - for example stage resubmission case.
   
   Yes, we have already test this. Here ia a simple example for stage recomputation with FetchFailed error.
   
   ```
   {
     "status" : "FAILED",
     "stageId" : 39,
     "attemptId" : 1,
     "numTasks" : 407,
     "numActiveTasks" : 0,
     "numCompleteTasks" : 155,
     "numFailedTasks" : 77,
     "numKilledTasks" : 0,
     "numCompletedIndices" : 155,
     "executorRunTime" : 821532,
     "executorCpuTime" : 806253615510,
     "submissionTime" : "2022-03-01T12:36:32.018GMT",
     "firstTaskLaunchedTime" : "2022-03-01T12:36:32.028GMT",
     "completionTime" : "2022-03-01T12:37:33.643GMT",
     "failureReason" : "org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 1\n",
     "inputBytes" : 0,
     "inputRecords" : 0,
     "outputBytes" : 0,
     "outputRecords" : 0,
     "shuffleReadBytes" : 22927763615,
     "shuffleReadRecords" : 943893085,
     "shuffleWriteBytes" : 0,
     "shuffleWriteRecords" : 0,
     "memoryBytesSpilled" : 0,
     "diskBytesSpilled" : 0,
     "name" : "run at ThreadPoolExecutor.java:1149",
     "description" : "benchmark q24a-v2.4",
     "details" : "scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\njava.lang.Thread.run(Thread.java:748)",
     "schedulingPool" : "default",
     "rddIds" : [ 93, 92, 91, 90, 89 ],
     "accumulatorUpdates" : [ ],
     "tasks" : {
       "14343" : {
         "taskId" : 14343,
         "index" : 65,
         "attempt" : 0,
         "partitionId" : 157,
         "launchTime" : "2022-03-01T12:36:32.031GMT",
         "duration" : 5475,
         "executorId" : "5",
         "host" : "",
         "status" : "SUCCESS",
         "taskLocality" : "PROCESS_LOCAL",
         "speculative" : false,
         "accumulatorUpdates" : [ ],
         "taskMetrics" : {
         }
       },
       "14367" : {
         "taskId" : 14367,
         "index" : 89,
         "attempt" : 0,
         "partitionId" : 182,
         "launchTime" : "2022-03-01T12:36:37.415GMT",
         "duration" : 5322,
         "executorId" : "8",
         "host" : "",
         "status" : "SUCCESS",
         "taskLocality" : "PROCESS_LOCAL",
         "speculative" : false,
         "accumulatorUpdates" : [ ],
         "taskMetrics" : {
         }
       },
       "14404" : {
         "taskId" : 14404,
         "index" : 126,
         "attempt" : 0,
         "partitionId" : 219,
         "launchTime" : "2022-03-01T12:36:37.900GMT",
         "duration" : 5246,
         "executorId" : "7",
         "host" : "",
         "status" : "SUCCESS",
         "taskLocality" : "PROCESS_LOCAL",
         "speculative" : false,
         "accumulatorUpdates" : [ ],
         "taskMetrics" : {
         }
       },
       "14463" : {
         "taskId" : 14463,
         "index" : 185,
         "attempt" : 0,
         "partitionId" : 278,
         "launchTime" : "2022-03-01T12:36:42.840GMT",
         "duration" : 103059,
         "executorId" : "6",
         "host" : "",
         "status" : "FAILED",
         "taskLocality" : "PROCESS_LOCAL",
         "speculative" : false,
         "accumulatorUpdates" : [ ],
         "errorMessage" : "FetchFailed",
         "taskMetrics" : {
         }
       },
       "14305" : {
         "taskId" : 14305,
         "index" : 27,
         "attempt" : 0,
         "partitionId" : 98,
         "launchTime" : "2022-03-01T12:36:32.030GMT",
         "duration" : 4910,
         "executorId" : "10",
         "host" : "",
         "status" : "SUCCESS",
         "taskLocality" : "PROCESS_LOCAL",
         "speculative" : false,
         "accumulatorUpdates" : [ ],
         "taskMetrics" : {
         }
       },
       "14358" : {
         "taskId" : 14358,
         "index" : 80,
         "attempt" : 0,
         "partitionId" : 173,
         "launchTime" : "2022-03-01T12:36:37.257GMT",
         "duration" : 5062,
         "executorId" : "9",
         "host" : "",
         "status" : "SUCCESS",
         "taskLocality" : "PROCESS_LOCAL",
         "speculative" : false,
         "accumulatorUpdates" : [ ],
         "taskMetrics" : {
         }
       },
       "14419" : {
         "taskId" : 14419,
         "index" : 141,
         "attempt" : 0,
         "partitionId" : 234,
         "launchTime" : "2022-03-01T12:36:41.673GMT",
         "duration" : 4798,
         "executorId" : "10",
         "host" : "",
         "status" : "SUCCESS",
         "taskLocality" : "PROCESS_LOCAL",
         "speculative" : false,
         "accumulatorUpdates" : [ ],
         "taskMetrics" : {
         }
       },
       "14399" : {
         "taskId" : 14399,
         "index" : 121,
         "attempt" : 0,
         "partitionId" : 214,
         "launchTime" : "2022-03-01T12:36:37.797GMT",
         "duration" : 5337,
         "executorId" : "7",
         "host" : "",
         "status" : "SUCCESS",
         "taskLocality" : "PROCESS_LOCAL",
         "speculative" : false,
         "accumulatorUpdates" : [ ],
         "taskMetrics" : {
         }
       },
       "14296" : {
         "taskId" : 14296,
         "index" : 18,
         "attempt" : 0,
         "partitionId" : 83,
         "launchTime" : "2022-03-01T12:36:32.029GMT",
         "duration" : 4944,
         "executorId" : "10",
         "host" : "",
         "status" : "SUCCESS",
         "taskLocality" : "PROCESS_LOCAL",
         "speculative" : false,
         "accumulatorUpdates" : [ ],
         "taskMetrics" : {
         }
       },
       "14468" : {
         "taskId" : 14468,
         "index" : 190,
         "attempt" : 0,
         "partitionId" : 283,
         "launchTime" : "2022-03-01T12:36:43.134GMT",
         "duration" : 103319,
         "executorId" : "7",
         "host" : "",
         "status" : "FAILED",
         "taskLocality" : "PROCESS_LOCAL",
         "speculative" : false,
         "accumulatorUpdates" : [ ],
         "errorMessage" : "FetchFailed",
         "taskMetrics" : {
         }
       }
   ....
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r817404993



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       @dongjoon-hyun Any point bout this?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r811645567



##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
##########
@@ -35,12 +35,34 @@ class TaskInfo(
      */
     val index: Int,
     val attemptNumber: Int,
+    /**
+     * The actual RDD partition ID in this task.
+     * The ID of the RDD partition is always same across task attempts.
+     * This will be -1 for historical data, and available for all applications since 3.3.

Review comment:
       ```suggestion
        * This will be -1 for historical data, and available for all applications since Spark 3.3.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] jackylee-ch commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
jackylee-ch commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r830097111



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       @Ngone51 Thanks for your attension. You can get the actual `partitionId` from [rest-api](https://spark.apache.org/docs/latest/monitoring.html#rest-api), such as `/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskList`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on pull request #35185: [SPARK-37831][CORE][WIP] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1011843060


   For the specific purpose detailed, use `index` ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r785427783



##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
##########
@@ -35,12 +35,33 @@ class TaskInfo(
      */
     val index: Int,
     val attemptNumber: Int,
+    /**
+     * The actual RDD partition ID in this task.
+     * The ID of the RDD partition is always same even task retries.

Review comment:
       Sure, thanks for your suggestion.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r786339392



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       Would be good to see if we can default the value to -1 if missing.
   Else cleaning up all level-db would be a prerequisite to leveraging this.
   
   I am not sure what is the cleanest way to enforce incompatible leveldb format changes - IIRC you had made some changes recently which require regen, right @dongjoon-hyun ? Any suggestions ? (IIRC we did not bump up the store version)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1012496679


   @tgravescs Agree, for the first stage attempt this should be fine.
   But for any stage attempt which is computing a subset of tasks (retries, specific partition computation via toLocalIterator, etc), this is a concern.
   
   This does look like an important missing functionality, thanks for working on it @stczwd - tagged others since I wanted to make sure I was not missing something here :-)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #35185: [SPARK-37831][CORE][WIP] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1011941728


   FWIW, there's a bug on updating the status in the check (https://github.com/apache/spark/pull/35179#issuecomment-1011881076). The tests in this PR looks passed (https://github.com/stczwd/spark/runs/4799251925).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] jackylee-ch commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
jackylee-ch commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r830477531



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       > Could you double check again? @jackylee-ch .
   
   @Ngone51 Thanks for your double check. It works fun while reading old eventLog json files with new spark-client. The recurring scenario of this problem is that using the new spark-client read old leveldb data.
   ```
   "7" : {
         "taskId" : 7,
         "index" : 1,
         "attempt" : 0,
         "partitionId" : 0,
         ...
   }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r829290863



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       Yet to hear from @dongjoon-hyun on it ... do you have any thoughts @Ngone51  ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r816759429



##########
File path: core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala
##########
@@ -1860,15 +1860,16 @@ abstract class AppStatusListenerSuite extends SparkFunSuite with BeforeAndAfter
 
   private def newAttempt(orig: TaskInfo, nextId: Long): TaskInfo = {
     // Task reattempts have a different ID, but the same index as the original.
-    new TaskInfo(nextId, orig.index, orig.attemptNumber + 1, time, orig.executorId,
+    new TaskInfo(
+      nextId, orig.index, orig.attemptNumber + 1, orig.partitionId, time, orig.executorId,
       s"${orig.executorId}.example.com", TaskLocality.PROCESS_LOCAL, orig.speculative)
   }
 
   private def createTasks(count: Int, execs: Array[String]): Seq[TaskInfo] = {
     (1 to count).map { id =>
       val exec = execs(id.toInt % execs.length)
       val taskId = nextTaskId()
-      new TaskInfo(taskId, taskId.toInt, 1, time, exec, s"$exec.example.com",
+      new TaskInfo(taskId, taskId.toInt, 1, taskId.toInt, time, exec, s"$exec.example.com",
         TaskLocality.PROCESS_LOCAL, id % 2 == 0)

Review comment:
       [here](https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala#L1127) use createTasks to genenrate different tasks. If we change `taskId` to `id`, this should be changed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #35185:
URL: https://github.com/apache/spark/pull/35185


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] jackylee-ch commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
jackylee-ch commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1084712535


   Thanks all.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] jackylee-ch commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
jackylee-ch commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1078578053


   Any more question about this pr? Can this pr merge now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1056435644


   > Would be good to explore how we can default the value to -1 if missing.
   >Else cleaning up all level-db would be a prerequisite to deploying this.
   >
   >I am not sure what is the cleanest way to enforce incompatible leveldb format changes - IIRC you had made some >changes recently which require regen, right @dongjoon-hyun ? Any suggestions ? (IIRC we did not bump up the store >version)
   
   ping @dongjoon-hyun. Could you take a look at the question above?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on pull request #35185: [SPARK-37831][CORE][WIP] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1011845827


   +CC @tgravescs, @Ngone51 
   I am surprised this is an issue ... want to make sure I am not missing something here !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tgravescs commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1012201870


   yeah generally index = partition id on the first execution but doesn't work on retries. I remember running into this a few times but was never a huge issue, seems like a good thing to add


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1038576606


   Hi guys, the long holiday is over, can we move on?
   @cloud-fan @HyukjinKwon @dongjoon-hyun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1048380050


   > It would be good to make sure we test backwards compatibility on the history server. ie use the old history server to read newer event log with this change.
   
   Thanks for your attension. Yeap, we have already test it with old history server and new server while using a server generate eventLog and read it from another server, they all works fine with this change. Besides, we have already used it in our environment, which has different versions of servers, especially spark2.x.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tgravescs commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1047941793


   It would be good to make sure we test backwards compatibility on the history server. ie use the old history server to read newer event log with this change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r811646227



##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
##########
@@ -35,12 +35,34 @@ class TaskInfo(
      */
     val index: Int,
     val attemptNumber: Int,
+    /**
+     * The actual RDD partition ID in this task.
+     * The ID of the RDD partition is always same across task attempts.
+     * This will be -1 for historical data, and available for all applications since 3.3.
+     */
+    val partitionId: Int,
     val launchTime: Long,
     val executorId: String,
     val host: String,
     val taskLocality: TaskLocality.TaskLocality,
     val speculative: Boolean) {
 
+  def this(
+      taskId: Long,
+      /**
+       * The index of this task within its task set. Not necessarily the same as the ID of the RDD
+       * partition that the task is computing.

Review comment:
       do we need comments here?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r811646064



##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
##########
@@ -35,12 +35,34 @@ class TaskInfo(
      */
     val index: Int,
     val attemptNumber: Int,
+    /**
+     * The actual RDD partition ID in this task.
+     * The ID of the RDD partition is always same across task attempts.
+     * This will be -1 for historical data, and available for all applications since 3.3.
+     */
+    val partitionId: Int,
     val launchTime: Long,
     val executorId: String,
     val host: String,
     val taskLocality: TaskLocality.TaskLocality,
     val speculative: Boolean) {
 
+  def this(

Review comment:
       can we add some comments to say this is for backward compatibility?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r786339392



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       Would be good to explore how we can default the value to -1 if missing.
   Else cleaning up all level-db would be a prerequisite to deploying this.
   
   I am not sure what is the cleanest way to enforce incompatible leveldb format changes - IIRC you had made some changes recently which require regen, right @dongjoon-hyun ? Any suggestions ? (IIRC we did not bump up the store version)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Ngone51 commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r830074187



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       I tried to reproduce this issue...but I just realize that this change doesn't reflect in UI..So how could I verify the "partitionId" value that is read by the history server? (I tried to hard code it by printing the "partitionId" but somehow it didn't print anything as expected...) cc @jackylee-ch 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] jackylee-ch commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
jackylee-ch commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r830999606



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       @Ngone51 Thanks for your advise. It works fun with default value. I will add the default value -1 and some comments to it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] jackylee-ch commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
jackylee-ch commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1073888536


   > Changes look good to me. Would like to understand how to handle the format change impact based on @dongjoon-hyun's experiences though.
   
   @Ngone51 @mridulm Since the default value can be set, the previous problem does not exist now. Can this pr be merged?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on pull request #35185: [SPARK-37831][CORE][WIP] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1011938660


   The main purpose of this issue is to track the task metrics of a specific RDD partition, espetially for input/shuffle reads or output/shuffle writes.
   It is very useful when shuffle data diff is found, we can use this to check data computing.
   Also, it will be helpful to show the actual computing data size or count in SQL UI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r811647794



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       ping @dongjoon-hyun 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r811685761



##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
##########
@@ -35,12 +35,34 @@ class TaskInfo(
      */
     val index: Int,
     val attemptNumber: Int,
+    /**
+     * The actual RDD partition ID in this task.
+     * The ID of the RDD partition is always same across task attempts.
+     * This will be -1 for historical data, and available for all applications since 3.3.

Review comment:
       sure

##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
##########
@@ -35,12 +35,34 @@ class TaskInfo(
      */
     val index: Int,
     val attemptNumber: Int,
+    /**
+     * The actual RDD partition ID in this task.
+     * The ID of the RDD partition is always same across task attempts.
+     * This will be -1 for historical data, and available for all applications since 3.3.
+     */
+    val partitionId: Int,
     val launchTime: Long,
     val executorId: String,
     val host: String,
     val taskLocality: TaskLocality.TaskLocality,
     val speculative: Boolean) {
 
+  def this(

Review comment:
       Sure

##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
##########
@@ -35,12 +35,34 @@ class TaskInfo(
      */
     val index: Int,
     val attemptNumber: Int,
+    /**
+     * The actual RDD partition ID in this task.
+     * The ID of the RDD partition is always same across task attempts.
+     * This will be -1 for historical data, and available for all applications since 3.3.
+     */
+    val partitionId: Int,
     val launchTime: Long,
     val executorId: String,
     val host: String,
     val taskLocality: TaskLocality.TaskLocality,
     val speculative: Boolean) {
 
+  def this(
+      taskId: Long,
+      /**
+       * The index of this task within its task set. Not necessarily the same as the ID of the RDD
+       * partition that the task is computing.

Review comment:
       I will remove this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r785993842



##########
File path: core/src/main/scala/org/apache/spark/status/storeTypes.scala
##########
@@ -286,6 +289,7 @@ private[spark] class TaskDataWrapper(
       taskId,
       index,
       attempt,
+      partitionId,
       new Date(launchTime),

Review comment:
       I have test it with event files, leveldb and rocksdb, and it is ok with rocksdb and event files, which retruns `"partitionId" : -1,`. But when testing leveldb, we get `"partitionId" : 0,`, which is different from what we expected.
   
   BTW, the test scenario is generating eventLog and leveldb/rocksdb with old spark-client and read it on new spark-client which has this pr.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1014537402


   > Took an initial pass through the PR and added some comments - overall looks good. We would need to make sure that skew join and partition coalescing in SQL interact well with this change.
   
   Thanks for you reply. I have test partition coalescing in SQL interact, it works well with this change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r785427811



##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
##########
@@ -35,12 +35,33 @@ class TaskInfo(
      */
     val index: Int,
     val attemptNumber: Int,
+    /**
+     * The actual RDD partition ID in this task.
+     * The ID of the RDD partition is always same even task retries.
+     */
+    val partitionId: Int,
     val launchTime: Long,
     val executorId: String,
     val host: String,
     val taskLocality: TaskLocality.TaskLocality,
     val speculative: Boolean) {
 
+  def this(
+      taskId: Long,
+      /**
+       * The index of this task within its task set. Not necessarily the same as the ID of the RDD
+       * partition that the task is computing.
+       */
+      index: Int,
+      attemptNumber: Int,
+      launchTime: Long,
+      executorId: String,
+      host: String,
+      taskLocality: TaskLocality.TaskLocality,
+      speculative: Boolean) = {
+    this(taskId, index, attemptNumber, -1, launchTime, executorId, host, taskLocality, speculative)
+  }

Review comment:
       Sure, I will change them.

##########
File path: core/src/test/resources/HistoryServerExpectations/excludeOnFailure_for_stage_expectation.json
##########
@@ -631,6 +642,7 @@
       "taskId" : 4,
       "index" : 2,
       "attempt" : 1,
+      "partitionId": -1,
       "launchTime" : "2018-01-09T10:21:18.943GMT",
       "duration" : 16,

Review comment:
       These json file is generated by rest api. Unless we do not want to show  partitionId with history data, this still needs to be modified.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm edited a comment on pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #35185:
URL: https://github.com/apache/spark/pull/35185#issuecomment-1012496679


   @tgravescs Agree, for the first stage attempt this should be fine.
   But for any stage attempt which is computing a subset of tasks (retries, specific partition computation via toLocalIterator, etc), this is a concern.
   
   An interesting question would be how to handle this for skewed joins - thoughts @cloud-fan ?
   
   Tagged folks since I wanted to make sure I was not missing something here :-)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] stczwd commented on a change in pull request #35185: [SPARK-37831][CORE] add task partition id in TaskInfo and Task Metrics

Posted by GitBox <gi...@apache.org>.
stczwd commented on a change in pull request #35185:
URL: https://github.com/apache/spark/pull/35185#discussion_r787534020



##########
File path: core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala
##########
@@ -35,12 +35,34 @@ class TaskInfo(
      */
     val index: Int,
     val attemptNumber: Int,
+    /**
+     * The actual RDD partition ID in this task.
+     * The ID of the RDD partition is always same across task or stage retries.

Review comment:
       ok




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org