[GitHub] [spark] cloud-fan edited a comment on issue #24565: [SPARK-27665][Core] Split fetch shuffle blocks protocol from OpenBlocks

[GitBox <gi...@apache.org>]

cloud-fan edited a comment on issue #24565: [SPARK-27665][Core] Split fetch shuffle blocks protocol from OpenBlocks
URL: https://github.com/apache/spark/pull/24565#issuecomment-493523580
 
 
   The thing I'm not sure is how important it is to support shuffle service 2.x + spark application 3.x. I feel it's good enough to have a shuffle service 3.x that support all versions of spark applications, but I might be wrong as I'm far away from real deploy environment.
   
   If we do want to support that, I think a reasonable choice can be: create a config to revert to the old shuffle protocol, and ask users to set this config if they see shuffle protocol related errors.
   
   We can also extend the shuffle protocol: client asks the server for its version first, and then pick the correct shuffle message to send. This might cause perf regression as it adds an extra RPC call, and I'm not sure it's worth if old shuffle service + new spark application is not a common use case.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org