[GitHub] [hudi] nikoshet commented on issue #8372: [SUPPORT] Config conflict with Deltastreamer CustomKeyGenerator - PartitionPath

["nikoshet (via GitHub)" <gi...@apache.org>]

nikoshet commented on issue #8372:
URL: https://github.com/apache/hudi/issues/8372#issuecomment-1497193253

   I also faced this yesterday. In order to overcome this without modifying the Hudi source code, I used the `SqlQueryBasedTransformer` and the `ComplexKeyGenerator` to extract the `year`,`month` and `day` values from the `inserted_at` column, like this:
   
   ```
   --hoodie-conf hoodie.datasource.write.recordkey.field=id \
   --hoodie-conf hoodie.datasource.write.hive_style_partitioning=true \
   --hoodie-conf hoodie.datasource.write.partitionpath.field=year,month,day \
   --hoodie-conf hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.ComplexKeyGenerator \
   --transformer-class org.apache.hudi.utilities.transform.SqlQueryBasedTransformer \
   --hoodie-conf hoodie.deltastreamer.transformer.sql="SELECT * , extract(year from inserted_at) as year, extract(month from inserted_at) as month, extract(day from inserted_at) as day FROM <SRC> a;"
   
   ```
   Hudi version: 0.13.0, Storage: S3, Deployment: Kubernetes
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org