You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/18 12:24:20 UTC

[GitHub] [hudi] Priyanka128 opened a new issue, #7242: [SUPPORT] Partition field value lost in table column

Priyanka128 opened a new issue, #7242:
URL: https://github.com/apache/hudi/issues/7242

   I am running spark-submit job to populate data into hudi tables from kafka topics. I have below properties set in my table-config.properties file:
   hoodie.datasource.write.partitionpath.field=partitionFieldColumn
   hoodie.datasource.hive_sync.table=tabledata
   hoodie.datasource.hive_sync.partition_fields=partitionFieldColumn
   hoodie.deltastreamer.keygen.timebased.timestamp.type=EPOCHMILLISECONDS
   hoodie.deltastreamer.keygen.timebased.output.dateformat=yyyy/MM/dd
   
   I am using "partitionFieldColumn" which is of datetime type. Using this, I want to have 3 level of partitioning (year -> month -> date). To avoid using time in the partitioning, "hoodie.deltastreamer.keygen.timebased.output.dateformat" property has the date format value. This results in correct partitioning levels but the "partitionFieldColumn" column created in the "tabledata" table also has the time field truncated, which is data loss.
   
   Is there any way to retain the complete value of the "partitionFieldColumn" without truncating the time field?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] ROOBALJINDAL commented on issue #7242: [SUPPORT] Partition field value lost in table column

Posted by GitBox <gi...@apache.org>.
ROOBALJINDAL commented on issue #7242:
URL: https://github.com/apache/hudi/issues/7242#issuecomment-1321930398

   @nsivabalan @xushiyan Is this expected behaviour or there is some configuration for it? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] ROOBALJINDAL commented on issue #7242: [SUPPORT] Partition field value lost in table column

Posted by "ROOBALJINDAL (via GitHub)" <gi...@apache.org>.
ROOBALJINDAL commented on issue #7242:
URL: https://github.com/apache/hudi/issues/7242#issuecomment-1592996448

   @ad1happy2go @nsivabalan @xushiyan 
   Currently we are duplicating the date column with required date format before writing to hudi but we dont want it that way. Is there any configuration so that we can have date value in column different than partitioning format?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] ad1happy2go commented on issue #7242: [SUPPORT] Partition field value lost in table column

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #7242:
URL: https://github.com/apache/hudi/issues/7242#issuecomment-1529365385

   @Priyanka128 @ROOBALJINDAL The correct way is to add a new date column before writing to hudi. For the same you can use "SqlQueryBasedTransformer".


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] ROOBALJINDAL commented on issue #7242: [SUPPORT] Partition field value lost in table column

Posted by "ROOBALJINDAL (via GitHub)" <gi...@apache.org>.
ROOBALJINDAL commented on issue #7242:
URL: https://github.com/apache/hudi/issues/7242#issuecomment-1594459909

   I have tried passing following too but didn't work
   ```
     --hoodie-conf write.datetime.partitioning=true \
     --hoodie-conf write.partition.format=yyyy \
   ```
   
   Should these configuration solve our purpose? @nsivabalan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org