[GitHub] [spark] attilapiros opened a new pull request #26869: [SPARK-30235][CORE] Switching off host local disk reading of shuffle blocks in case of useOldFetchProtocol

[GitBox <gi...@apache.org>]

attilapiros opened a new pull request #26869: [SPARK-30235][CORE] Switching off host local disk reading of shuffle blocks in case of useOldFetchProtocol
URL: https://github.com/apache/spark/pull/26869
 
 
   ### Why are the changes needed?
   
   In `[SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host` a new block transfer message is introduced, `GetLocalDirsForExecutors`. This new message could be sent to the external shuffle service and as it is not supported by the previous version of external shuffle service it should be avoided when `spark.shuffle.useOldFetchProtocol` is true.
   
   In the migration guide I changed the exception type as `org.apache.spark.network.shuffle.protocol.BlockTransferMessage.Decoder#fromByteBuffer`
   throws a IllegalArgumentException with the given text and uses the message type which is just a simple number (byte). I have checked and this is true for version 2.4.4 too.
   
   ### How was this patch tested?
   
   This specific case (considering one extra boolean to switch off host local disk reading feature) is not tested but existing tests were run.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org