You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/05/25 18:41:32 UTC
[GitHub] [hudi] andaag opened a new issue #2135: [SUPPORT] GDPR safe deletes is complex
andaag opened a new issue #2135:
URL: https://github.com/apache/hudi/issues/2135
**Describe the problem you faced**
I'm trying to come up with a consistent and understandable way to deal with gdpr deletes.
What I'd like to do:
1. Stream realtime data into bucket A
2. Collect GDPR delete signals into B (or delete one and one record in A?)
3. Every 24h delete from A where userid in B
4. And after that ack back to the gdpr controller, saying records 1,2,3 are 100% deleted.
My problem is step 4. I'd like to ack only the records that are deleted on disk, aka after compaction. I understand that I need to set 'hoodie.cleaner.commits.retained': 1 and 'hoodie.cleaner.fileversions.retained': 1 to try to reduce the time window for point in time queries, and I also saw some code references to manually running compaction, but this is complex.
I really like delta.io's "vacuum 7 days" command, as it makes it very understandable and explicit that I can ack any records older than 7 days. How can I know which records have been compacted, and which haven't? Can I pull out the last compaction time from the table in an easy way? (and are these time ordered, so I can know that older records than X are compacted?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2135: [SUPPORT] GDPR safe deletes is complex
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2135:
URL: https://github.com/apache/hudi/issues/2135#issuecomment-766439085
@andaag : I have created a Hudi ticket for this. Feel free to update the desc of the ticket with more details
https://issues.apache.org/jira/browse/HUDI-1549
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash closed issue #2135: [SUPPORT] GDPR safe deletes is complex
Posted by GitBox <gi...@apache.org>.
n3nash closed issue #2135:
URL: https://github.com/apache/hudi/issues/2135
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash closed issue #2135: [SUPPORT] GDPR safe deletes is complex
Posted by GitBox <gi...@apache.org>.
n3nash closed issue #2135:
URL: https://github.com/apache/hudi/issues/2135
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2135: [SUPPORT] GDPR safe deletes is complex
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2135:
URL: https://github.com/apache/hudi/issues/2135#issuecomment-702799057
@andaag : I think you are referring to "cleaner" here. We expose cleaner metadata through CLI. "cleans show" (Source code : CleansCommand.java).
Cleaner track what is the earliest commit that was retained (still present). If this value is higher than (say previous day), then all old data is cleaned earlier than that. Is this something that would work for you ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] andaag commented on issue #2135: [SUPPORT] GDPR safe deletes is complex
Posted by GitBox <gi...@apache.org>.
andaag commented on issue #2135:
URL: https://github.com/apache/hudi/issues/2135#issuecomment-702821679
Yes, but it'd be quite critical to be able to access this from the apis/programatically.
If we have a job running the gdpr deletion, we'd need that job to know the times involved and ack. Not have to deal with the cli.
We can grab the logic from CleansCommand though, but it's not optimal. It'd be very nice to access this as part of the standard api.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2135: [SUPPORT] GDPR safe deletes is complex
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2135:
URL: https://github.com/apache/hudi/issues/2135#issuecomment-766439085
@andaag : I have created a Hudi ticket for this. Feel free to update the desc of the ticket with more details
https://issues.apache.org/jira/browse/HUDI-1549
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2135: [SUPPORT] GDPR safe deletes is complex
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2135:
URL: https://github.com/apache/hudi/issues/2135#issuecomment-824540132
Closing this ticket due to inactivity and existing JIRA to follow up on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash closed issue #2135: [SUPPORT] GDPR safe deletes is complex
Posted by GitBox <gi...@apache.org>.
n3nash closed issue #2135:
URL: https://github.com/apache/hudi/issues/2135
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #2135: [SUPPORT] GDPR safe deletes is complex
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #2135:
URL: https://github.com/apache/hudi/issues/2135#issuecomment-702868560
@andaag, we would be happy to take a feature request do you mind raising a jira, with more detailed requirements
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org