You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "soumilshah1995 (via GitHub)" <gi...@apache.org> on 2023/07/22 12:49:32 UTC
[GitHub] [hudi] soumilshah1995 opened a new issue, #9263: [SUPPORT] Hard deletes in Hudi to remove Duplicates
soumilshah1995 opened a new issue, #9263:
URL: https://github.com/apache/hudi/issues/9263
TO @nadine farah @shivNarayan
Subject : Hard deletes in Hudi
Explanation of problem
i had a conversation with @Howard Cho yesterday on google meet and we had some conversation on hard delete to remove duplicates and i decided to give a shot.here is scenarios
INGESTED Batch 1 using BULK_INSERT
```
+---------+----+-------+-------+-------------------+----------------------+
|partition|uuid|precomb|message|_hoodie_commit_time|_hoodie_commit_seqno |
+---------+----+-------+-------+-------------------+----------------------+
|2 |2 |222 |mes 2 |20230722082644003 |20230722082644003_15_2|
|1 |1 |111 |mess 1 |20230722082644003 |20230722082644003_7_1 |
+---------+----+-------+-------+-------------------+----------------------+
```
Ingested BATCH 2 Using BULK_INSERT
```
+---------+----+-------+--------+-------------------+----------------------+
|partition|uuid|precomb|message |_hoodie_commit_time|_hoodie_commit_seqno |
+---------+----+-------+--------+-------------------+----------------------+
|3 |3 |333 |insert 3|20230722082703914 |20230722082703914_15_4|
|2 |2 |222 |mes 2 |20230722082644003 |20230722082644003_15_2|
|1 |1 |111 |mess 1 |20230722082644003 |20230722082644003_7_1 |
|2 |2 |222 |upt 2 |20230722082703914 |20230722082703914_7_3 |
+---------+----+-------+--------+-------------------+----------------------+
```
Ask was lets use hard delete in Hudi to remove UUID 2 and where Commit is 20230722082644003
Great
```
df = spark.read.format("hudi").load(path).createOrReplaceTempView("hudi_snapshot")
query = """
SELECT *
FROM hudi_snapshot
WHERE (uuid, _hoodie_commit_time) IN (
SELECT uuid, MIN(_hoodie_commit_time) AS min_commit_time
FROM hudi_snapshot
GROUP BY uuid
HAVING COUNT(*) > 1
);
"""
delete_df = spark.sql(query)
delete_df.select(["partition", "uuid", "precomb", "message", "_hoodie_commit_time", "_hoodie_commit_seqno"]).show(truncate=False)
```
OUTPUT
```
+---------+----+-------+-------+-------------------+----------------------+
|partition|uuid|precomb|message|_hoodie_commit_time|_hoodie_commit_seqno |
+---------+----+-------+-------+-------------------+----------------------+
|2 |2 |222 |mes 2 |20230722082644003 |20230722082644003_15_2|
+---------+----+-------+-------+-------------------+----------------------+
```
Great we identified item to delete
After Hard delete
```
+---------+----+-------+--------+-------------------+----------------------+
|partition|uuid|precomb|message |_hoodie_commit_time|_hoodie_commit_seqno |
+---------+----+-------+--------+-------------------+----------------------+
|3 |3 |333 |insert 3|20230722082703914 |20230722082703914_15_4|
|1 |1 |111 |mess 1 |20230722082644003 |20230722082644003_7_1 |
+---------+----+-------+--------+-------------------+----------------------+
```
But it removed
```
|2 |2 |222 |upt 2 |20230722082703914 |20230722082703914_7_3 |
```
Which it should not
Any advice | Feedback or suggestion how to handle hard delete
Thanking you TIme in advance
Shah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] soumilshah1995 closed issue #9263: [SUPPORT] Hard deletes in Hudi to remove Duplicates
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 closed issue #9263: [SUPPORT] Hard deletes in Hudi to remove Duplicates
URL: https://github.com/apache/hudi/issues/9263
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] soumilshah1995 commented on issue #9263: [SUPPORT] Hard deletes in Hudi to remove Duplicates
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #9263:
URL: https://github.com/apache/hudi/issues/9263#issuecomment-1647909246
Great
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9263: [SUPPORT] Hard deletes in Hudi to remove Duplicates
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9263:
URL: https://github.com/apache/hudi/issues/9263#issuecomment-1647170050
@soumilshah1995 This is expected. Hudi uses record keys to identify records to be upserted or deleted. It acts like the primary key for hudi table regardless of other columns(except partition columns) present in data frame to delete.
So it deleted both the records with record key - 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nfarah86 commented on issue #9263: [SUPPORT] Hard deletes in Hudi to remove Duplicates
Posted by "nfarah86 (via GitHub)" <gi...@apache.org>.
nfarah86 commented on issue #9263:
URL: https://github.com/apache/hudi/issues/9263#issuecomment-1647095535
tagging @ad1happy2go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] howardcho commented on issue #9263: [SUPPORT] Hard deletes in Hudi to remove Duplicates
Posted by "howardcho (via GitHub)" <gi...@apache.org>.
howardcho commented on issue #9263:
URL: https://github.com/apache/hudi/issues/9263#issuecomment-1697528578
> i had a conversation with @howard Cho yesterday on google meet and we had some conversation on hard delete to remove duplicates and i decided to give a shot.here is scenarios
The user link above links to the wrong person. Adding a comment so I can reference this in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org