[GitHub] [spark] otterc commented on a change in pull request #32287: [SPARK-27991][CORE] Defer the fetch request on Netty OOM

[GitBox <gi...@apache.org>]

otterc commented on a change in pull request #32287:
URL: https://github.com/apache/spark/pull/32287#discussion_r618789195



##########
File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
##########
@@ -613,6 +618,12 @@ final class ShuffleBlockFetcherIterator(
           }
           if (isNetworkReqDone) {
             reqsInFlight -= 1
+            if (!buf.isInstanceOf[NettyManagedBuffer]) {
+              // Non-`NettyManagedBuffer` doesn't occupy Netty's memory so we can unset the flag
+              // directly once the request succeeds. But for the `NettyManagedBuffer`, we'll only
+              // unset the flag when the data is fully consumed (see `BufferReleasingInputStream`).
+              NettyUtils.isNettyOOMOnShuffle = false

Review comment:
       This flag is in `NettyUtils` so I am assuming that if there are multiple tasks with their own iterators, then they all will be setting/unsetting the same flag. Is that correct? 
   Wouldn't this decision to unset the flag here, when it is a non-netty_managed_buffer, by a single iterator cause a problem?

##########
File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
##########
@@ -683,7 +694,28 @@ final class ShuffleBlockFetcherIterator(
             }
           }
 
-        case FailureFetchResult(blockId, mapIndex, address, e) =>
+        // Catching OOM and do something based on it is only a workaround for handling the
+        // Netty OOM issue, which is not the best way towards memory management. We can
+        // get rid of it when we find a way to manage Netty's memory precisely.
+        case FailureFetchResult(blockId, mapIndex, address, size, isNetworkReqDone, e)
+            if e.isInstanceOf[OutOfDirectMemoryError] || e.isInstanceOf[NettyOutOfMemoryError] =>
+          assert(address != blockManager.blockManagerId &&
+            !hostLocalBlocks.contains(blockId -> mapIndex),
+            "Netty OOM error should only happen on remote fetch requests")
+          logWarning(s"Failed to fetch block $blockId due to Netty OOM, will retry", e)
+          NettyUtils.isNettyOOMOnShuffle = true
+          numBlocksInFlightPerAddress(address) = numBlocksInFlightPerAddress(address) - 1
+          bytesInFlight -= size
+          if (isNetworkReqDone) {
+            reqsInFlight -= 1
+            logDebug("Number of requests in flight " + reqsInFlight)
+          }
+          val defReqQueue =
+            deferredFetchRequests.getOrElseUpdate(address, new Queue[FetchRequest]())
+          defReqQueue.enqueue(FetchRequest(address, Array(FetchBlockInfo(blockId, size, mapIndex))))

Review comment:
       If an executor is not assigned enough off-heap memory. Is it possible in this case that it keeps retrying forever? Let's say there is some skew and one of the blocks is large and the executor isn't assigned enough memory. Whenever this block is fetched Netty OOMs. In this case will this keep retrying? Maybe I am missing something.

##########
File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
##########
@@ -268,7 +269,10 @@ final class ShuffleBlockFetcherIterator(
 
       override def onBlockFetchFailure(blockId: String, e: Throwable): Unit = {
         logError(s"Failed to get block(s) from ${req.address.host}:${req.address.port}", e)
-        results.put(new FailureFetchResult(BlockId(blockId), infoMap(blockId)._2, address, e))
+        remainingBlocks -= blockId

Review comment:
       If error is instance of `OutOfDirectMemoryError` or `NettyOutOfMemoryError` should we set `isNettyOOMOnShuffle` set to `true` here? The reason is that we would want to stop sending more remote fetch requests as soon as possible. Right now it is being set when this particular instance of FailureFetchResult would be picked up from the `result` queue and processed.
   I am just thinking it would be better to check the error type here and set `NettyUtils.isNettyOOMOnShuffle=true`. 
   
   To add the request to the deferred queue, I think it would better to create a new type of FetchResult, let's say, `DeferFetchResult` and add that to results. The reason for that is we are change `FailureFetchResult` but it seems that this fix is just a workaround. So once we get rid of the workaround, it will be much simpler to get rid of `DeferFetchResult` and not modify `FailureFetchResult` back. Also an added benefit of this is that other places where `FailureFetchResult` is created, that will not change.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org