You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "TheR1sing3un (via GitHub)" <gi...@apache.org> on 2026/01/23 11:17:05 UTC
[PR] feat: introduce timeline manifest retained version conf [hudi]
TheR1sing3un opened a new pull request, #17996:
URL: https://github.com/apache/hudi/pull/17996
### Describe the issue this Pull Request addresses
closes #17995
### Summary and Changelog
1. introduce timeline manifest retained version conf
### Impact
introduce a conf to control manifest retained version
### Risk Level
none
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: introduce timeline manifest retained version conf [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #17996:
URL: https://github.com/apache/hudi/pull/17996#discussion_r2726359539
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieArchivalConfig.java:
##########
@@ -107,6 +107,13 @@ public class HoodieArchivalConfig extends HoodieConfig {
.defaultValue(1000L * 1024 * 1024)
.markAdvanced()
.withDocumentation("Max size (in bytes) for each archived timeline file.");
+
+ public static final ConfigProperty<Integer> TIMELINE_MANIFEST_VERSION_RETAINED = ConfigProperty
+ .key("hoodie.timeline.manifest.version.retained")
Review Comment:
> hoodie.timeline.manifest.retained.versions
done
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/versioning/v2/LSMTimelineWriter.java:
##########
@@ -329,9 +329,11 @@ public void compactFiles(List<String> candidateFiles, String compactedFileName)
* @param context HoodieEngineContext used for parallelize to delete obsolete files if necessary.
*/
public void clean(HoodieEngineContext context, int compactedVersions) throws IOException {
- // if there are more than 3 version of snapshots, clean the oldest files.
+ int versionRetained = config.getTimelineManifestVersionRetained();
+ log.info("Starting cleaning obsolete files, retaining last ({} + {}) manifest versions.", versionRetained, compactedVersions);
Review Comment:
> retaining last -> retaining latest
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: introduce timeline manifest retained version conf [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #17996:
URL: https://github.com/apache/hudi/pull/17996
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: introduce timeline manifest retained version conf [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #17996:
URL: https://github.com/apache/hudi/pull/17996#discussion_r2726351930
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieArchivalConfig.java:
##########
@@ -107,6 +107,13 @@ public class HoodieArchivalConfig extends HoodieConfig {
.defaultValue(1000L * 1024 * 1024)
.markAdvanced()
.withDocumentation("Max size (in bytes) for each archived timeline file.");
+
+ public static final ConfigProperty<Integer> TIMELINE_MANIFEST_VERSION_RETAINED = ConfigProperty
+ .key("hoodie.timeline.manifest.version.retained")
Review Comment:
hoodie.timeline.manifest.retained.versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: introduce timeline manifest retained version conf [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17996:
URL: https://github.com/apache/hudi/pull/17996#issuecomment-3797924838
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "444c6b74fa078f30142ba1a3b5f13f1d47282b5d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11311",
"triggerID" : "444c6b74fa078f30142ba1a3b5f13f1d47282b5d",
"triggerType" : "PUSH"
}, {
"hash" : "7815b26a4c989fbeec56fece12fe0a660be3b1d5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11347",
"triggerID" : "7815b26a4c989fbeec56fece12fe0a660be3b1d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 444c6b74fa078f30142ba1a3b5f13f1d47282b5d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11311)
* 7815b26a4c989fbeec56fece12fe0a660be3b1d5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11347)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: introduce timeline manifest retained version conf [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17996:
URL: https://github.com/apache/hudi/pull/17996#issuecomment-3790056818
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "444c6b74fa078f30142ba1a3b5f13f1d47282b5d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11311",
"triggerID" : "444c6b74fa078f30142ba1a3b5f13f1d47282b5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 444c6b74fa078f30142ba1a3b5f13f1d47282b5d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11311)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: introduce timeline manifest retained version conf [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17996:
URL: https://github.com/apache/hudi/pull/17996#issuecomment-3789792082
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "444c6b74fa078f30142ba1a3b5f13f1d47282b5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "444c6b74fa078f30142ba1a3b5f13f1d47282b5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 444c6b74fa078f30142ba1a3b5f13f1d47282b5d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: introduce timeline manifest retained version conf [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17996:
URL: https://github.com/apache/hudi/pull/17996#issuecomment-3798079625
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "444c6b74fa078f30142ba1a3b5f13f1d47282b5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11311",
"triggerID" : "444c6b74fa078f30142ba1a3b5f13f1d47282b5d",
"triggerType" : "PUSH"
}, {
"hash" : "7815b26a4c989fbeec56fece12fe0a660be3b1d5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11347",
"triggerID" : "7815b26a4c989fbeec56fece12fe0a660be3b1d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7815b26a4c989fbeec56fece12fe0a660be3b1d5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11347)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: introduce timeline manifest retained version conf [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #17996:
URL: https://github.com/apache/hudi/pull/17996#discussion_r2726352471
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/versioning/v2/LSMTimelineWriter.java:
##########
@@ -329,9 +329,11 @@ public void compactFiles(List<String> candidateFiles, String compactedFileName)
* @param context HoodieEngineContext used for parallelize to delete obsolete files if necessary.
*/
public void clean(HoodieEngineContext context, int compactedVersions) throws IOException {
- // if there are more than 3 version of snapshots, clean the oldest files.
+ int versionRetained = config.getTimelineManifestVersionRetained();
+ log.info("Starting cleaning obsolete files, retaining last ({} + {}) manifest versions.", versionRetained, compactedVersions);
Review Comment:
retaining last -> retaining latest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: introduce timeline manifest retained version conf [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17996:
URL: https://github.com/apache/hudi/pull/17996#issuecomment-3789802751
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "444c6b74fa078f30142ba1a3b5f13f1d47282b5d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11311",
"triggerID" : "444c6b74fa078f30142ba1a3b5f13f1d47282b5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 444c6b74fa078f30142ba1a3b5f13f1d47282b5d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11311)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: introduce timeline manifest retained version conf [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17996:
URL: https://github.com/apache/hudi/pull/17996#issuecomment-3797920371
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "444c6b74fa078f30142ba1a3b5f13f1d47282b5d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11311",
"triggerID" : "444c6b74fa078f30142ba1a3b5f13f1d47282b5d",
"triggerType" : "PUSH"
}, {
"hash" : "7815b26a4c989fbeec56fece12fe0a660be3b1d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7815b26a4c989fbeec56fece12fe0a660be3b1d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 444c6b74fa078f30142ba1a3b5f13f1d47282b5d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11311)
* 7815b26a4c989fbeec56fece12fe0a660be3b1d5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org