You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/02/28 13:10:11 UTC
[GitHub] [hudi] abhishekshenoy edited a comment on issue #3313: [SUPPORT] CoW: Hudi Upsert not working when there is a timestamp field in the composite key
abhishekshenoy edited a comment on issue #3313:
URL: https://github.com/apache/hudi/issues/3313#issuecomment-1054225531
@nsivabalan i see the issue is closed . But in 0.10.1 i still face the duplicate issue when i provide a timestamp column as part of composite key.
```
hoodiConfigs.put("hoodie.insert.shuffle.parallelism", "1")
hoodiConfigs.put("hoodie.upsert.shuffle.parallelism", "1")
hoodiConfigs.put("hoodie.bulkinsert.shuffle.parallelism", "1")
hoodiConfigs.put("hoodie.delete.shuffle.parallelism", "1")
hoodiConfigs.put("hoodie.datasource.write.row.writer.enable", "true")
hoodiConfigs.put("hoodie.table.keygenerator.class", classOf[ComplexKeyGenerator].getName)
hoodiConfigs.put("hoodie.datasource.write.keygenerator.class", classOf[ComplexKeyGenerator].getName)
hoodiConfigs.put("hoodie.datasource.write.recordkey.field", "transactionId,storeNbr,transactionTs")
hoodiConfigs.put("hoodie.datasource.write.precombine.field", "messageMetadata.srcLoadTs")
hoodiConfigs.put("hoodie.table.precombine.field", "messageMetadata.srcLoadTs")
hoodiConfigs.put("hoodie.datasource.write.partitionpath.field", "transactionDt")
hoodiConfigs.put("hoodie.datasource.write.payload.class",classOf[DefaultHoodieRecordPayload].getName)
hoodiConfigs.put("hoodie.datasource.write.hive_style_partitioning", "true")
hoodiConfigs.put("hoodie.datasource.write.table.type",COW_TABLE_TYPE_OPT_VAL)
hoodiConfigs.put("hoodie.combine.before.upsert","true")
hoodiConfigs.put("hoodie.table.name","huditransaction")
hoodiConfigs.put("hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled","true")
```
Tried with both BULK_INSERT_OPERATION_OPT_VAL and UPSERT_OPERATION_OPT_VAL
Output dataset after first Insert , If you see Combine before insert did not work
Key (abc, 4162 , 2022-02-25T05:08:10.73-05:00)
```
+-------------------+---------------------+---------------------------------------------------------------------+------------------------+-----------------------------------------------------------------------+-------------+--------+-----------------------+--------------------------------------------------+----------+--------+---------+----------------+-------------+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key |_hoodie_partition_path |_hoodie_file_name |transactionId|storeNbr|transactionTs |messageMetadata |prefixes |dummyInt|dummyLong|dummyObjects |transactionDt|
+-------------------+---------------------+---------------------------------------------------------------------+------------------------+-----------------------------------------------------------------------+-------------+--------+-----------------------+--------------------------------------------------+----------+--------+---------+----------------+-------------+
|20220228183206147 |20220228183206147_0_1|transactionId:abc,storeNbr:4162,transactionTs:2022-02-25 05:08:10.073|transactionDt=2022-02-25|f86d9a60-8465-410d-bca6-c478bf3a48e9-0_0-10-0_20220228183206147.parquet|abc |4162 |2022-02-25 05:08:10.073|{key, value, 1, 2, 2022-02-25 05:09:10, 1, { -> }}|[abc, def]|1 |1 |[{a, 1}, {a, 1}]|2022-02-25 |
|20220228183206147 |20220228183206147_0_2|transactionId:abc,storeNbr:4162,transactionTs:2022-02-25 05:08:10.073|transactionDt=2022-02-25|f86d9a60-8465-410d-bca6-c478bf3a48e9-0_0-10-0_20220228183206147.parquet|abc |4162 |2022-02-25 05:08:10.073|{key, value, 1, 2, 2022-02-25 05:09:10, 1, { -> }}|[abc, def]|1 |1 |[{a, 1}, {a, 1}]|2022-02-25 |
|20220228183206147 |20220228183206147_0_3|transactionId:bcd,storeNbr:4162,transactionTs:2022-02-25 05:08:10.073|transactionDt=2022-02-25|f86d9a60-8465-410d-bca6-c478bf3a48e9-0_0-10-0_20220228183206147.parquet|bcd |4162 |2022-02-25 05:08:10.073|{key, value, 1, 2, 2022-02-25 05:09:10, 1, { -> }}|[abc, def]|1 |1 |[{a, 1}, {a, 1}]|2022-02-25 |
|20220228183206147 |20220228183206147_0_4|transactionId:cde,storeNbr:4163,transactionTs:2022-02-25 05:08:10.073|transactionDt=2022-02-25|f86d9a60-8465-410d-bca6-c478bf3a48e9-0_0-10-0_20220228183206147.parquet|cde |4163 |2022-02-25 05:08:10.073|{key, value, 1, 2, 2022-02-25 05:09:10, 1, { -> }}|[abc, def]|1 |1 |[{a, 1}, {a, 1}]|2022-02-25 |
|20220228183206147 |20220228183206147_0_5|transactionId:def,storeNbr:4163,transactionTs:2022-02-25 05:08:10.073|transactionDt=2022-02-25|f86d9a60-8465-410d-bca6-c478bf3a48e9-0_0-10-0_20220228183206147.parquet|def |4163 |2022-02-25 05:08:10.073|{key, value, 1, 2, 2022-02-25 05:09:10, 1, { -> }}|[abc, def]|1 |1 |[{a, 1}, {a, 1}]|2022-02-25 |
+-------------------+---------------------+---------------------------------------------------------------------+------------------------+-----------------------------------------------------------------------+-------------+--------+-----------------------+--------------------------------------------------+----------+--------+---------+----------------+-------------+
```
Republishing with an addition record for the same key (abc, 4162 , 2022-02-25T05:08:10.73-05:00) does not get deduped
```
+-------------------+---------------------+---------------------------------------------------------------------+------------------------+-----------------------------------------------------------------------+-------------+--------+-----------------------+--------------------------------------------------+----------+--------+---------+----------------+-------------+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key |_hoodie_partition_path |_hoodie_file_name |transactionId|storeNbr|transactionTs |messageMetadata |prefixes |dummyInt|dummyLong|dummyObjects |transactionDt|
+-------------------+---------------------+---------------------------------------------------------------------+------------------------+-----------------------------------------------------------------------+-------------+--------+-----------------------+--------------------------------------------------+----------+--------+---------+----------------+-------------+
|20220228183206147 |20220228183206147_0_1|transactionId:abc,storeNbr:4162,transactionTs:2022-02-25 05:08:10.073|transactionDt=2022-02-25|f86d9a60-8465-410d-bca6-c478bf3a48e9-0_0-10-0_20220228183206147.parquet|abc |4162 |2022-02-25 05:08:10.073|{key, value, 1, 2, 2022-02-25 05:09:10, 1, { -> }}|[abc, def]|1 |1 |[{a, 1}, {a, 1}]|2022-02-25 |
|20220228183206147 |20220228183206147_0_2|transactionId:abc,storeNbr:4162,transactionTs:2022-02-25 05:08:10.073|transactionDt=2022-02-25|f86d9a60-8465-410d-bca6-c478bf3a48e9-0_0-10-0_20220228183206147.parquet|abc |4162 |2022-02-25 05:08:10.073|{key, value, 1, 2, 2022-02-25 05:09:10, 1, { -> }}|[abc, def]|1 |1 |[{a, 1}, {a, 1}]|2022-02-25 |
|20220228183206147 |20220228183206147_0_3|transactionId:bcd,storeNbr:4162,transactionTs:2022-02-25 05:08:10.073|transactionDt=2022-02-25|f86d9a60-8465-410d-bca6-c478bf3a48e9-0_0-10-0_20220228183206147.parquet|bcd |4162 |2022-02-25 05:08:10.073|{key, value, 1, 2, 2022-02-25 05:09:10, 1, { -> }}|[abc, def]|1 |1 |[{a, 1}, {a, 1}]|2022-02-25 |
|20220228183206147 |20220228183206147_0_4|transactionId:cde,storeNbr:4163,transactionTs:2022-02-25 05:08:10.073|transactionDt=2022-02-25|f86d9a60-8465-410d-bca6-c478bf3a48e9-0_0-10-0_20220228183206147.parquet|cde |4163 |2022-02-25 05:08:10.073|{key, value, 1, 2, 2022-02-25 05:09:10, 1, { -> }}|[abc, def]|1 |1 |[{a, 1}, {a, 1}]|2022-02-25 |
|20220228183206147 |20220228183206147_0_5|transactionId:def,storeNbr:4163,transactionTs:2022-02-25 05:08:10.073|transactionDt=2022-02-25|f86d9a60-8465-410d-bca6-c478bf3a48e9-0_0-10-0_20220228183206147.parquet|def |4163 |2022-02-25 05:08:10.073|{key, value, 1, 2, 2022-02-25 05:09:10, 1, { -> }}|[abc, def]|1 |1 |[{a, 1}, {a, 1}]|2022-02-25 |
|20220228183821299 |20220228183821299_0_1|transactionId:abc,storeNbr:4162,transactionTs:2022-02-25 05:08:10.073|transactionDt=2022-02-25|66ee158e-93f3-4ccc-8b2a-1712c3cdf5cf-0_0-2-0_20220228183821299.parquet |abc |4162 |2022-02-25 05:08:10.073|{key, value, 1, 2, 2022-02-25 05:09:10, 1, { -> }}|[abc, def]|1 |1 |[{a, 1}, {a, 1}]|2022-02-25 |
+-------------------+---------------------+---------------------------------------------------------------------+------------------------+-----------------------------------------------------------------------+-------------+--------+-----------------------+--------------------------------------------------+----------+--------+---------+----------------+-------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org