You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/07/16 21:45:07 UTC

[GitHub] [hudi] reenarosid opened a new issue #1840: [SUPPORT

reenarosid opened a new issue #1840:
URL: https://github.com/apache/hudi/issues/1840


   **Describe the problem you faced**
   
   I am trying to delete a record, yes the delete operation was reflected, but I still find a hard copy of the files in s3. 
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   1.I tried inserting a few records, then update a few and delete a few, but in my s3 folder i only saw few more files being created for the partition i updated or delted. So if i read through any other service then, i see duplicates.
   
   **Expected behavior**
   
   A clear and concise description of what you expected to happen.
   When i update a record I ideally want only one file(the latest version) and when i delete i want only the latest version (minus the deleted record)
   
   **Environment Description**
   
   * Hudi version :0.53
   
   * Spark version :2.44
   
   * Storage (HDFS/S3/GCS..) :s3
   
   * Running on Docker? (yes/no) : no
   
   
   **Additional context**
   
   For GDPR purpose , once a delete is performed we shouldn't be able to read/trace back the record again.
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bvaradar commented on issue #1840: HUDI DELETE

Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1840:
URL: https://github.com/apache/hudi/issues/1840#issuecomment-659804569


   @nsivabalan : Would you be able to take this up ? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #1840: HUDI DELETE

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #1840:
URL: https://github.com/apache/hudi/issues/1840#issuecomment-660335554


   I didn't mean it that way. if you are ok giving up older commits, then setting the two config params you posted would suffice. But incase you want to retain older commits and you also want deletes to delete all older copies, I don't think its doable as of now. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bvaradar commented on issue #1840: HUDI DELETE

Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1840:
URL: https://github.com/apache/hudi/issues/1840#issuecomment-662536376


   @reenarosid @nsivabalan : Can we close this issue ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #1840: HUDI DELETE

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #1840:
URL: https://github.com/apache/hudi/issues/1840#issuecomment-662806446


   it is already closed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] reenarosid commented on issue #1840: HUDI DELETE

Posted by GitBox <gi...@apache.org>.
reenarosid commented on issue #1840:
URL: https://github.com/apache/hudi/issues/1840#issuecomment-660013270


   Adding 
   
   "hoodie.cleaner.commits.retained": "1",
   "hoodie.cleaner.fileversions.retained": "1"
   
   in my option has helped me, can someone verify if i am doing the right operation?
   As mentioned above we are doing this for GDPR compliance and we won't revert it back so having just the latest file will do right?
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #1840: HUDI DELETE

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #1840:
URL: https://github.com/apache/hudi/issues/1840#issuecomment-660119391


   yes, if you want all older copies to be deleted, guess you have to set the configs you have listed. For incremental pulling, point in time queries, etc older commits will still have the records. only the commit in which delete was issued will have the record deleted. So if you query the table in general with latest data, you will not see the deleted records. But if you query with a older commit time, you could see the deleted record as per that time instant, since these records are not deleted as of that time instant. 
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] reenarosid closed issue #1840: HUDI DELETE

Posted by GitBox <gi...@apache.org>.
reenarosid closed issue #1840:
URL: https://github.com/apache/hudi/issues/1840


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] reenarosid commented on issue #1840: HUDI DELETE

Posted by GitBox <gi...@apache.org>.
reenarosid commented on issue #1840:
URL: https://github.com/apache/hudi/issues/1840#issuecomment-660262865


   > yes, if you want all older copies to be deleted, guess you have to set the configs you have listed. For incremental pulling, point in time queries, etc older commits will still have the records. only the commit in which delete was issued will have the record deleted. So if you query the table in general with latest data, you will not see the deleted records. But if you query with a older commit time, you could see the deleted record as per that time instant, since these records are not deleted as of that time instant.
   
   Thanks, it was helpful. But i wonder how querying an older commit time will show the deleted record. Because file versions retained is set 1, then how will it retrive it?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org