[GitHub] [spark] xuanyuanking commented on a change in pull request #28965: [SPARK-32124][CORE][FOLLOW-UP] Use the invalid value -2 to fill the map index when the event logs from the old Spark version

[GitBox <gi...@apache.org>]

xuanyuanking commented on a change in pull request #28965:
URL: https://github.com/apache/spark/pull/28965#discussion_r448830595



##########
File path: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala
##########
@@ -1078,8 +1078,12 @@ private[spark] object JsonProtocol {
         val blockManagerAddress = blockManagerIdFromJson(json \ "Block Manager Address")
         val shuffleId = (json \ "Shuffle ID").extract[Int]
         val mapId = (json \ "Map ID").extract[Long]
-        val mapIndex = (json \ "Map Index") match {
-          case JNothing => 0
+        val mapIndex = json \ "Map Index" match {
+          case JNothing =>
+            // Note, we use the invalid value -2 here to fill the map index for backward
+            // compatibility. Otherwise, the fetch failed event will be dropped when the history
+            // server loads the event log written by the Spark version before 3.0.
+            -2

Review comment:
       Thanks Gengliang for confirming!
   ```
   Any special reasons using -2 instead of -1?
   ```
   mapIndex = -1 has been used in `MetadataFetchFailedException`.
   
   Yep, I think `Int.MinValue` is a better choice.
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org