[GitHub] [spark] attilapiros commented on pull request #31876: [SPARK-34942][API][CORE] Abstract Location in MapStatus to enable support for custom storage

[GitBox <gi...@apache.org>]

attilapiros commented on pull request #31876:
URL: https://github.com/apache/spark/pull/31876#issuecomment-1020193485


   Hi @Ngone51! 
   
   I am working on the adding of metadata to the map status to support 3rd party shuffle solutions better. My prototype available at https://github.com/attilapiros/spark/pull/4. 
   But I think your PR (this one) can be used to solve the same problem and this seems to be more elegant for the problem.
   In my case the location is untouched and the metadata is an extra addition. As blocks was retrieved grouped by the location in my case that adds extra complexity (and redundant code) meanwhile t I think in real situations we do need the metadata to support custom block coordinates. And even when this is not the case a custom location can be used to store the extra info, too. So one customization in the MapStatus will be definitely enough. WDYT?
   
   I think it would nice to continue this PR so may I ask you to update this with the current master and reopen the PR?
   I would be happy to help by reviewing it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org