You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/08/04 04:18:09 UTC
[GitHub] [hudi] sannidhiteredesai opened a new issue #3399: [SUPPORT] How to purge older partitions in hudi partitioned tables ?
sannidhiteredesai opened a new issue #3399:
URL: https://github.com/apache/hudi/issues/3399
**Question:** How to purge older partitions in hudi partitioned tables ?
**Scenario:**
- I have a partitioned table that will be created on daily basis.
- If everyday a new partition is created the number of partitions will keep on increasing.
- We might have to purge older partitions (Eg: Older than 1 month).
**What is the best way to achieve this in hudi ?**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on issue #3399: [SUPPORT] How to purge older partitions in hudi partitioned tables ?
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on issue #3399:
URL: https://github.com/apache/hudi/issues/3399#issuecomment-900765748
The latest master already supports,
@sannidhiteredesai
https://github.com/apache/hudi/pull/3488 Here is an example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 commented on issue #3399: [SUPPORT] How to purge older partitions in hudi partitioned tables ?
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on issue #3399:
URL: https://github.com/apache/hudi/issues/3399#issuecomment-900765748
The latest master already supports,
@sannidhiteredesai
https://github.com/apache/hudi/pull/3488 Here is an example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3399: [SUPPORT] How to purge older partitions in hudi partitioned tables ?
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3399:
URL: https://github.com/apache/hudi/issues/3399#issuecomment-905184393
Closing this out as latest master already has it. Feel free to re-open if you are looking for something else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #3399: [SUPPORT] How to purge older partitions in hudi partitioned tables ?
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #3399:
URL: https://github.com/apache/hudi/issues/3399
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3399: [SUPPORT] How to purge older partitions in hudi partitioned tables ?
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3399:
URL: https://github.com/apache/hudi/issues/3399#issuecomment-893138598
WriteClient has support for [deletePartitions](https://github.com/apache/hudi/blob/1df5ded433eced3bbb66137bc741e616beab9a70/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java#L257). But looks like we have not added the support to spark data source yet.
Are you using spark datasource?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] sannidhiteredesai commented on issue #3399: [SUPPORT] How to purge older partitions in hudi partitioned tables ?
Posted by GitBox <gi...@apache.org>.
sannidhiteredesai commented on issue #3399:
URL: https://github.com/apache/hudi/issues/3399#issuecomment-895457054
Hi @nsivabalan,
Please suggest if any alternate way to do this with spark datasource.
In which upcoming version of hudi can we find this feature ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] sannidhiteredesai commented on issue #3399: [SUPPORT] How to purge older partitions in hudi partitioned tables ?
Posted by GitBox <gi...@apache.org>.
sannidhiteredesai commented on issue #3399:
URL: https://github.com/apache/hudi/issues/3399#issuecomment-893214335
Hi Siva,
Yes I am using spark datasource.
So, I use this for table upserts: df.write.format("hudi").....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] sannidhiteredesai commented on issue #3399: [SUPPORT] How to purge older partitions in hudi partitioned tables ?
Posted by GitBox <gi...@apache.org>.
sannidhiteredesai commented on issue #3399:
URL: https://github.com/apache/hudi/issues/3399#issuecomment-893214335
Hi Siva,
Yes I am using spark datasource.
So, I use this for table upserts: df.write.format("hudi").....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] sannidhiteredesai edited a comment on issue #3399: [SUPPORT] How to purge older partitions in hudi partitioned tables ?
Posted by GitBox <gi...@apache.org>.
sannidhiteredesai edited a comment on issue #3399:
URL: https://github.com/apache/hudi/issues/3399#issuecomment-893214335
Hi @nsivabalan ,
Yes I am using spark datasource.
So, I use this for table upserts: df.write.format("hudi").....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #3399: [SUPPORT] How to purge older partitions in hudi partitioned tables ?
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3399:
URL: https://github.com/apache/hudi/issues/3399#issuecomment-893138598
WriteClient has support for [deletePartitions](https://github.com/apache/hudi/blob/1df5ded433eced3bbb66137bc741e616beab9a70/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java#L257). But looks like we have not added the support to spark data source yet.
Are you using spark datasource?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org