You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/03/30 08:32:03 UTC
[GitHub] [hudi] aditiwari01 opened a new issue #2743: Do we have any TTL mechanism in Hudi?
aditiwari01 opened a new issue #2743:
URL: https://github.com/apache/hudi/issues/2743
Do we have any TTL mechanism in Hudi? We have streaming jobs writinng to Hudi and we do not want the data to grow indefinitely. Lets say we are only interested in rows updated in last 1 month. Is there any way to specify row level TTL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-810670533
Have opened a feature request for this : https://issues.apache.org/jira/browse/HUDI-1741
@nsivabalan @n3nash : FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aditiwari01 commented on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-813779228
Thanks @lw309637554 Will look into this deletePartition in depth.
As for my use case, the ideal situation would be to have some kind of row level TTL taken care by cleaner/compactor. In absence of any such feature, I was wondering if I could partition on commit time and regularly delete older partitions. Our use case includes updates and for a given key the updates can exist on different partitions. We want to avoid such duplicates. I think global indexing can help in here. (haven't tested yet).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-813194180
Yeah, we don't recommend deleting partitions outside of hudi. Hudi support insert_override operation. But not sure if it can just delete partition w/o ingesting any data.
@satishkotha : Is there a way to trigger deletion of a partition w/o any ingestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aditiwari01 commented on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-815462565
@n3nash Thanks for the clarificatio. Can we create a jira for the same. I can't pick this right away but would try to conntribute as and when I get time.
Meanwhile I will try to directly use the low level api to unblock myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aditiwari01 commented on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-813450681
I think the ideal way should be around compactor and cleaner.
Time based cleaner and filtering records with older commit time while compacting base file should solve the issue.
If there's a workaround using any delete option, please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-815473655
@aditiwari01 Here is the ticket and is assigned to you for now :) BTW, there is some relevant work happening here https://github.com/apache/hudi/pull/2452. Please comment on the PR for further changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] lw309637554 commented on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
lw309637554 commented on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-813765155
> @nsivabalan @aditiwari01 We actually introduced a delete_partition operation. See https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/WriteOperationType.java#L46
>
> I _think_ this can be called on empty dataframe (I havent tested it).
> Looks like documentation is missing for this. @nsivabalan Could you open a ticket.
>
> cc: @lw309637554 who implemented this API.
@aditiwari01
hello
1. now you can batch delete partitions use hudiTable.deletePartitions() just like in https://issues.apache.org/jira/browse/HUDI-1350. Also a pr is doing more easy tool to delete partitions https://github.com/apache/hudi/pull/2452
2. @aditiwari01 why you need to duplicate the data? In my scene, if the data is user behaviors log from kafka , not need to duplicate.
3. Now have a issue to add the doc https://issues.apache.org/jira/browse/HUDI-1674 @satishkotha @nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-815015923
@lw309637554 @satishkotha : fyi we are yet to add spark ds support for this "delete_partition" operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha edited a comment on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
satishkotha edited a comment on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-813568836
@nsivabalan @aditiwari01 We actually introduced a delete_partition operation. See https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/WriteOperationType.java#L46
I *think* this can be called on empty dataframe (I havent tested it).
Looks like documentation is missing for this. @nsivabalan Could you open a ticket.
cc: @lw309637554 who implemented this API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash closed issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
n3nash closed issue #2743:
URL: https://github.com/apache/hudi/issues/2743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
satishkotha commented on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-813568836
@nsivabalan @aditiwari01 We actually introduce a delete_partition operation. See https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/WriteOperationType.java#L46
I *think* this can be called on empty dataframe (I havent tested it).
Looks like documentation is missing for this. @nsivabalan Could you open a ticket.
cc: @lw309637554 who implemented this API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-815456351
@aditiwari01 I think you mentioned 2 issues here
1. Record level TTL -> We don't have such a feature in Hudi. Like others have pointed out, using the `hudiTable.deletePartitions()` API is a way to manage older partitions. Yes, you could partition based on _hoodie_commit_time or any other date based partitioning that structures your table to be eligible for deleting older partitions completely.
2. Duplicates across partitions -> If you have an update workload and are using the `upsert` API, yes, using a GlobalIndex will help eliminate duplicates for your table.
As @nsivabalan pointed out, we don't have such support out of the spark datasource but have a low level API as pointed above. We welcome contributions and would be good to add this support in spark datasource - let me know if you want to contribute this feature and we can guide you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-810738337
@aditiwari01 : if you are not running a continuous job (deltastreamer), is it possible to trigger a job at regular cadence to fetch records from hudi that are > 1 month old and issue deletes back to hudi. I know this may not be efficient, but atleast will unblock you for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] aditiwari01 commented on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
aditiwari01 commented on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-810782992
I was thinking around similar lines but we do have continuous jobs (not deltastreamer, but spark streaming jobs with 5/10 mins minni batches). We can't have a separate job for deletion since we do not support concurrent writers.
Another possible solution can be to have our table partitioned at commit time and then have a manual cleanup schedule jobs which deletes older partitions. The challenges here are that I am not sure if deleting some partition from outside can messup hoodie meta in some way? Also this will require us to have global indexing to avoid duplicates, which in turn can result in increased latencies.
What are your thoughts on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2743: Do we have any TTL mechanism in Hudi?
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2743:
URL: https://github.com/apache/hudi/issues/2743#issuecomment-814143042
@n3nash @vinothchandar @bvaradar : any thoughts here or workarounds until we have per record support for TTL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org