You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "BruceKellan (via GitHub)" <gi...@apache.org> on 2023/03/16 10:04:30 UTC

[GitHub] [hudi] BruceKellan opened a new issue, #8203: [SUPPORT] Global index upsert when using flink and FLINK_STATE index type.

BruceKellan opened a new issue, #8203:
URL: https://github.com/apache/hudi/issues/8203

   **_Tips before filing an issue_**
   
   Now when partitioning upsert, according to `precombine.field` parameter, keep the record with the largest value after upserting.
   
   This is widely used to solve the case of out-of-order data, by setting the `precombine.field` to the event time to keep records with the largest event time.
   
   However, when using the FLINK_STATE index type, if cross-partition occurs, the precombine.field parameter will not fully take effect.
   
   In the case of cross-partitioning, the master's logic uses data that arrives later, even if the event time is smaller.
   
   https://github.com/apache/hudi/blob/b79ce80f709ddffb5ad7aead10711222c086db24/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/BucketAssignFunction.java#L181-L184
   
   I'm wondering if there is any particular reason for implementing it this way?
   If not, I'll open a PR to fix it.
   
   It may be necessary to unify the logic of intra-partition upsert and cross-partition upsert, which is convenient for users to understand and use.
   
   **Environment Description**
   
   * Hudi version : 0.13.0
   
   * Hive version : 2.3.9
   
   * Hadoop version : 2.8.5
   
   * Storage (HDFS/S3/GCS..) : OSS
   
   * Running on Docker? (yes/no) : no


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on issue #8203: [SUPPORT] Global index upsert when using flink and FLINK_STATE index type.

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8203:
URL: https://github.com/apache/hudi/issues/8203#issuecomment-1475577566

   It's kind of a trade-off, we can implemet the coss-partition update only if we record ther version field for each index item, that can increate the index storage significantly, but I think it is feasible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] BruceKellan commented on issue #8203: [SUPPORT] Global index upsert when using flink and FLINK_STATE index type.

Posted by "BruceKellan (via GitHub)" <gi...@apache.org>.
BruceKellan commented on issue #8203:
URL: https://github.com/apache/hudi/issues/8203#issuecomment-1475589705

   Thanks for your reply, danny. I will open a ticket to do it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org