You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/11 03:17:14 UTC

[GitHub] [spark] HeartSaVioR commented on pull request #29066: [WIP][SPARK-23889] DataSourceV2: required sorting and clustering for writes

HeartSaVioR commented on pull request #29066:
URL: https://github.com/apache/spark/pull/29066#issuecomment-690848690


   This is definitely a major missing piece on DSv2 compared to DSv1, as DSv1 writer can deal with Dataframe directly, hence able to do arbitrary changes (including repartition/sort) before doing actual write, like I did for state data source -  https://github.com/HeartSaVioR/spark-state-tools/blob/8a74bdb1bc7911a6f71785cf68b784b0a331a1d9/src/main/scala/net/heartsavior/spark/sql/state/StateStoreWriter.scala#L67-L74
   
   Lots of data sources are blocked to migrate to DSv2. Shall we consider prioritizing this?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org