[I] CustomKeyGenerator can not be created with flink [hudi]

["hudi-bot (via GitHub)" <gi...@apache.org>]

hudi-bot opened a new issue, #16621:
URL: https://github.com/apache/hudi/issues/16621

   If we create a table in flink with custom key generator it fails with error message.
   {code:java}
   [ERROR] Could not execute SQL statement. Reason:
   org.apache.hudi.exception.HoodieValidationException: Field ts:timestamp does not exist{code}
   {code:java}
   CREATE TABLE hudi_table(
       ts BIGINT,
       uuid VARCHAR(40) PRIMARY KEY NOT ENFORCED,
       rider VARCHAR(20),
       driver VARCHAR(20),
       fare DOUBLE,
       city VARCHAR(20)
   )
   WITH (
     'connector' = 'hudi',
     'path' = 'file:///tmp/hudi_table',
     'table.type' = 'MERGE_ON_READ',
     'hoodie.keygen.timebased.timestamp.type' = 'SCALAR',
     'hoodie.keygen.timebased.output.dateformat' = 'yyyy/MM/dd',
     'hoodie.keygen.timebased.timestamp.scalar.time.unit' = 'seconds',
     'hoodie.datasource.write.keygenerator.class' = 'org.apache.hudi.keygen.CustomAvroKeyGenerator',
     'hoodie.datasource.write.partitionpath.field' = 'ts:timestamp'
   ); {code}
   We need to support using custom keygen with flink engine.
   
    
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-8220
   - Type: Bug
   - Fix version(s):
     - 1.1.0
   
   
   ---
   
   
   ## Comments
   
   20/Sep/24 09:34;danny0405;Let's revisit this option value, it looks like a beaking change, you can not force user to always declare the very specific partition data type, there should be a default value, why not introduce a new option just for the custom ken gen partition data type?
   
   
   {code:java}
   'hoodie.datasource.write.partitionpath.field' = 'ts:timestamp' {code};;;
   
   ---
   
   23/Sep/24 04:46;ljain;Updated the exception message which is being seen.
   [~danny0405] The problem here is that the user would not be able to specify a partition type even if they want to. The partition type is only required for custom keygen and not required for other keygens.;;;
   
   ---
   
   27/Sep/24 04:21;danny0405;Moved it to 1.1.0 release because it looks like we never support the custom avro key gen for Flink before, so it is not a blocker. And I do think the special value for partition path field is quite hacky, I'm conservative about the modification.;;;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] CustomKeyGenerator can not be created with flink [hudi]

["rangareddy (via GitHub)" <gi...@apache.org>]

rangareddy commented on issue #16621:
URL: https://github.com/apache/hudi/issues/16621#issuecomment-5434691371

   This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8220).
   
   **Findings: still open.**
   
   No commit on `master` references HUDI-8220, so nothing has landed under this ticket's key.
   
   The description is substantive and self-describing, so this is workable as written - what is missing is an owner rather than a specification.
   
   Keeping this open.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org