[GitHub] [spark] rdblue commented on pull request #29066: [WIP][SPARK-23889] DataSourceV2: required sorting and clustering for writes

[GitBox <gi...@apache.org>]

rdblue commented on pull request #29066:
URL: https://github.com/apache/spark/pull/29066#issuecomment-706324579


   I'm testing this out and I hit a problem. There is no way to create Spark's internal SortOrder implementation that is public. I created a class that implements SortOrder, but that fails in `toCatalyst`: `Caused by: java.lang.RuntimeException: com.netflix.spark.OrderField@7e1d244 is not currently supported`.
   
   Although the interface is defined, this doesn't accept any implementation of that interface. For the public `Expression` API, we added extractors that can match any implementation of the public interfaces. I think we need to do that for the `SortOrder` as well. And, we should expose a way to create a `SortOrder` expression. Here's an example extractor: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/expressions/expressions.scala#L192-L199


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org