[GitHub] [iceberg] RussellSpitzer edited a comment on pull request #3375: Spark: CALL procedure for rewrite_data_files

[GitBox <gi...@apache.org>]

RussellSpitzer edited a comment on pull request #3375:
URL: https://github.com/apache/iceberg/pull/3375#issuecomment-972115074


   In the Datasource code in Spark3 it uses this little bit
   ```          val filters = DataSourceStrategy.normalizeExprs(condition.toSeq, output)
                 .flatMap(splitConjunctivePredicates(_).map {
                   f => DataSourceStrategy.translateFilter(f, true).getOrElse(
                     throw QueryCompilationErrors.cannotTranslateExpressionToSourceFilterError(f))
                 }).toArray
   ```
   To convert to Source Filters, I think this is a little simpler than reimplementing the Spark Expressions -> Iceberg Expressions directly. I just figure this way, we will have less direct dependencies on Spark code.
   
   This is protected but it may make sense for us to expose it in our extensions for this purpose.
   
   The relevant sections are here
   https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala#L567-L570


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org