You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "asija-varun92 (via GitHub)" <gi...@apache.org> on 2025/09/23 11:36:12 UTC
[I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
asija-varun92 opened a new issue, #13973:
URL: https://github.com/apache/hudi/issues/13973
### Bug Description
**What happened:** I am experiencing an issue where delete operations in a Hudi table do not propagate to another table when using DeltaStreamer.
**What you expected:** I have one main hudi table in which I am setting '_hoodie_is_deleted' while upsert for deletion purpose. This is actually deleting the row from here. I have another pipeline which is a deltaStreamer job and that syncs the changes in this table to another hudi table. The mode of write in both is COW. Inserts and updates are working fine but deletion is not propagating to downstream table.
**Steps to reproduce:**
1. Create hudi table using [options](https://github.com/asija-varun92/cdc-hudi-lakehouse/tree/main/src/hudi_config)
2. Run deltaStreamer job to propagate same changes into another hudi table. [my job](https://github.com/asija-varun92/hudi-data-sync)
3. Try inserting new row into 1st hudi table and run deltaS
treamer job. it will work fine. Then try deleting the same row. It will be deleted from first table bit after running deltaStreamer job the delete operation will not propagate on the downstream table
### Environment
**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
Posted by "varun-asija (via GitHub)" <gi...@apache.org>.
varun-asija commented on issue #13973:
URL: https://github.com/apache/hudi/issues/13973#issuecomment-3334293545
In my source table, i am doing hard delete on row and therefore row is gone from latest parquet file in cow table, that’s why deltaStreamer is not able to detect which row to delete and so not propagating the delete operation. Is this correct?
I am using deltaStreamer class. This is from previous version of hudi but still valid in latest version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
Posted by "vaibhavk1992 (via GitHub)" <gi...@apache.org>.
vaibhavk1992 commented on issue #13973:
URL: https://github.com/apache/hudi/issues/13973#issuecomment-3334924555
@asija-varun92 correct . if you need the record you can use soft delete. Please check this doc https://hudi.apache.org/docs/faq_reading_tables/
Let me know if I can close the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on issue #13973:
URL: https://github.com/apache/hudi/issues/13973#issuecomment-3325988567
Thanks @vaibhavk1992 , you have it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
Posted by "asija-varun92 (via GitHub)" <gi...@apache.org>.
asija-varun92 commented on issue #13973:
URL: https://github.com/apache/hudi/issues/13973#issuecomment-3327757673
Hi @vinothchandar,
Were you able to reproduce the issue? Do you need any further details?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
Posted by "vaibhavk1992 (via GitHub)" <gi...@apache.org>.
vaibhavk1992 commented on issue #13973:
URL: https://github.com/apache/hudi/issues/13973#issuecomment-3344994959
btw, @asija-varun92 can we do a insert overwrite to your target table in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
Posted by "asija-varun92 (via GitHub)" <gi...@apache.org>.
asija-varun92 commented on issue #13973:
URL: https://github.com/apache/hudi/issues/13973#issuecomment-3335180474
but for GDPR compliance we need to do hard delete so soft delete is not an option for us
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
Posted by "vaibhavk1992 (via GitHub)" <gi...@apache.org>.
vaibhavk1992 commented on issue #13973:
URL: https://github.com/apache/hudi/issues/13973#issuecomment-3335104380
@asija-varun92 Soft Deletes retains the record key and just nulling out the values for all the other fields
Please check this doc once it has all the details around deletes https://hudi.apache.org/docs/writing_data#deletes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
Posted by "vaibhavk1992 (via GitHub)" <gi...@apache.org>.
vaibhavk1992 commented on issue #13973:
URL: https://github.com/apache/hudi/issues/13973#issuecomment-3323688995
Hi @rangareddy
Please assign this issue to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
Posted by "asija-varun92 (via GitHub)" <gi...@apache.org>.
asija-varun92 commented on issue #13973:
URL: https://github.com/apache/hudi/issues/13973#issuecomment-3335034910
Thanks for confirmation @vaibhavk1992 . If we enable cdc on hudi table, will it be possible to propagate delete operation, then?
I tried with cdc enabled but that also giving cdstats for update only. i found cdstats = null for delete operation. Is there any config which can enable delete logs using cdc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #13973:
URL: https://github.com/apache/hudi/issues/13973#issuecomment-3345175617
@asija-varun92 Yes insert overwrite one time should be the right way to do to handle this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
Posted by "vaibhavk1992 (via GitHub)" <gi...@apache.org>.
vaibhavk1992 commented on issue #13973:
URL: https://github.com/apache/hudi/issues/13973#issuecomment-3352250443
@vinothchandar Can we please close this issue as user query has been addressed with the possible solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
Posted by "vaibhavk1992 (via GitHub)" <gi...@apache.org>.
vaibhavk1992 commented on issue #13973:
URL: https://github.com/apache/hudi/issues/13973#issuecomment-3334014335
@asija-varun92 If I refer the doc as per the environment mentioned above I am following this one https://hudi.apache.org/docs/1.0.0/hoodie_streaming_ingestion#using-hudi-utilities-slim-bundle-bundle-jar.
And if I compare it to the spark-submit provided above I see a different --class is being used. The doc suggest to use `org.apache.hudi.utilities.streamer.HoodieStreamer `.
Can you please check on this and also let me know any doc you are using to follow this migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Delete operations not propagating to downstream Hudi table via DeltaStreamer [hudi]
Posted by "vaibhavk1992 (via GitHub)" <gi...@apache.org>.
vaibhavk1992 commented on issue #13973:
URL: https://github.com/apache/hudi/issues/13973#issuecomment-3337767843
True but with hard deletes you can't sync that record.
@rangareddy @ad1happy2go Any suggestions here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org