[I] [PIPELINES] Support partition transforms in SDP partition_cols [spark]

["wilhelmagren (via GitHub)" <gi...@apache.org>]

wilhelmagren opened a new issue, #57908:
URL: https://github.com/apache/spark/issues/57908

   spark=4.1.3
   iceberg=1.11.0
   
   ### Problem
   
   SDP currently treats every entry in `partition_cols` as an identity partition transform.
   
   for example
   
   ```python
   @dp.materialized_view(partition_cols=["months(event_time)"])
   def events():
       ...
   ```
   
   causes `months(event_time)` to be interpreted as a column name instead of a partition transform.
   
   Behaviour comes from `DatasetManager.materializeTable` where all partition cols are converted using identity
   
   ```scala
   table.partitionCols.toSeq.flatten.map(Expressions.identity)
   ```
   
   and as a result using the above partition col it just becomes `identity('months(event_time)')`.
   
   ### Expected behaviour
   
   SDP should support the same partition transforms that Spark SQL already supports
   
   Bare col-names (`partition_col=["event_date"]`) should continue producing `identity(event_date)`
   but partition transforms should be respected, and combining partitions like
   
   ```python
   partition_cols=["country", "bucket(16, user_id)"]
   ```
   
   should produce its corresponding V2 partition transforms.
   
   ### Existing SQL support
   
   Catalyst parses partition transform syntax for SQL `PARTITIONED BY` and creates corresponding V2 `Transform`
   
   SDP should just reuse the existing Catalyst parsing logic and not short-circuit everything to identity transform.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org