You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bk-mz (via GitHub)" <gi...@apache.org> on 2024/03/27 15:18:34 UTC
[I] [SUPPORT] No way to clean archived/ folder [hudi]
bk-mz opened a new issue, #10930:
URL: https://github.com/apache/hudi/issues/10930
**Describe the problem you faced**
There's no way to control `archived/` folder size and no way to trigger its cleaning.
We have a long running table which accumulated a lot of archives (~100 GB) which now damages cleaner performance and overall performance of ingestion process.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to All Configuration in Hudi Site
2. Check for all settings that control archived/ folder of hudi
3. Ensure there is none
**Expected behavior**
There should be a description somewhere in documentation of stating how to upkeep `archived/` folder.
Upkeep of archived/ folder should be delegated to cleaner.
**Environment Description**
* Hudi version : 0.14.0
* Spark version : 3.4.1
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) :
**Additional context**
Related slack thread: https://apache-hudi.slack.com/archives/C4D716NPQ/p1711531654297129
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] No way to clean `archived/` folder [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #10930:
URL: https://github.com/apache/hudi/issues/10930#issuecomment-2043988319
may be we should introduce a ArchivalClean table service to auto clean files older than say 2 months. Not many users are going to inspect archival timeline after 2+ months. and it will avoid accumulating entire history. Interested users can still choose to not clean it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] No way to clean `archived/` folder [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10930:
URL: https://github.com/apache/hudi/issues/10930#issuecomment-2024212538
We have some option such as `hoodie.archive.merge.enable` for archival log merging, the cleaing is introduced only after 1.0 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] No way to clean `archived/` folder [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10930:
URL: https://github.com/apache/hudi/issues/10930#issuecomment-2713521438
yes, it is supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] No way to clean `archived/` folder [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10930:
URL: https://github.com/apache/hudi/issues/10930#issuecomment-2024972572
Pointing to slack thread also here - https://apache-hudi.slack.com/archives/C4D716NPQ/p1711531654297129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] No way to clean `archived/` folder [hudi]
Posted by "bk-mz (via GitHub)" <gi...@apache.org>.
bk-mz commented on issue #10930:
URL: https://github.com/apache/hudi/issues/10930#issuecomment-2023082472
just a side note, we were re-creating metadata for that table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] No way to clean `archived/` folder [hudi]
Posted by "SGITLOGIN (via GitHub)" <gi...@apache.org>.
SGITLOGIN commented on issue #10930:
URL: https://github.com/apache/hudi/issues/10930#issuecomment-2713384430
> We have some option such as `hoodie.archive.merge.enable` for archival log merging, the cleaing is introduced only after 1.0 release.
@danny0405 Hi, Does hudi 1.0.1 support cleaning up archived directories? I don't see any parameters for cleaning up archived directories on the official website.
<img width="976" alt="Image" src="https://github.com/user-attachments/assets/8678207d-2417-438c-9301-a73e0ba63357" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org