You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/27 17:50:52 UTC
[GitHub] [hudi] jonvex opened a new pull request, #7076: [HUDI-5032] Add archive to cli
jonvex opened a new pull request, #7076:
URL: https://github.com/apache/hudi/pull/7076
### Change Logs
Added archive command to cli with 3 args --minCommits, --maxCommits, and --commitsRetained
### Impact
Gives users the ability to run archiving through the command line
### Risk level (write none, low medium or high below)
low
### Documentation Update
No updates required
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7076:
URL: https://github.com/apache/hudi/pull/7076#issuecomment-1301254355
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12620",
"triggerID" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"triggerType" : "PUSH"
}, {
"hash" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12625",
"triggerID" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"triggerType" : "PUSH"
}, {
"hash" : "565b83f6ba744fd62312d9e2dff8dbf6956d6b80",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12694",
"triggerID" : "565b83f6ba744fd62312d9e2dff8dbf6956d6b80",
"triggerType" : "PUSH"
}, {
"hash" : "d4f5107ca1bd9bba7bda422609072e971f89f12c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12736",
"triggerID" : "d4f5107ca1bd9bba7bda422609072e971f89f12c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 565b83f6ba744fd62312d9e2dff8dbf6956d6b80 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12694)
* d4f5107ca1bd9bba7bda422609072e971f89f12c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12736)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7076:
URL: https://github.com/apache/hudi/pull/7076#issuecomment-1293908873
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82cc7354ee41a3983583dc8a5455110c9b66f0f6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7076:
URL: https://github.com/apache/hudi/pull/7076#issuecomment-1293915461
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12620",
"triggerID" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82cc7354ee41a3983583dc8a5455110c9b66f0f6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12620)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #7076:
URL: https://github.com/apache/hudi/pull/7076#discussion_r1009644201
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java:
##########
@@ -52,6 +67,35 @@
*/
@ShellComponent
public class ArchivedCommitsCommand {
+ private static final Logger LOG = LogManager.getLogger(ArchivedCommitsCommand.class);
+ private JavaSparkContext jsc;
+ @ShellMethod(key = "trigger archival", value = "trigger archival")
+ public void triggerArchival(
+ @ShellOption(value = {"--minCommits"},
+ help = "Minimum number of instants to retain in the active timeline. See hoodie.keep.min.commits",
+ defaultValue = "20") int minCommits,
+ @ShellOption(value = {"--maxCommits"},
+ help = "Maximum number of instants to retain in the active timeline. See hoodie.keep.max.commits",
+ defaultValue = "30") int maxCommits,
+ @ShellOption(value = {"--commitsRetained"}, help = "Number of commits to retain, without cleaning",
+ defaultValue = "10") int retained) {
+
+ initJavaSparkContext();
+ HoodieWriteConfig config = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+ .withArchivalConfig(HoodieArchivalConfig.newBuilder().archiveCommitsWith(minCommits,maxCommits).build())
+ .withCleanConfig(HoodieCleanConfig.newBuilder().retainCommits(retained).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder().enable(false).build())
Review Comment:
actually, lets take enableMetadata as an argument as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on a diff in pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
jonvex commented on code in PR #7076:
URL: https://github.com/apache/hudi/pull/7076#discussion_r1009716374
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java:
##########
@@ -206,4 +250,22 @@ private Comparable[] readCommit(GenericRecord record, boolean skipMetadata) {
return new Comparable[] {};
}
}
+
+ private void initJavaSparkContext() {
Review Comment:
I pretty much copied from MetadataCommand.java, but got rid of userDefinedMaster arg:
```
private void initJavaSparkContext(Option<String> userDefinedMaster) {
if (jsc == null) {
jsc = SparkUtil.initJavaSparkContext(SparkUtil.getDefaultConf("HoodieCLI", userDefinedMaster));
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #7076:
URL: https://github.com/apache/hudi/pull/7076#discussion_r1011089909
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java:
##########
@@ -206,4 +250,22 @@ private Comparable[] readCommit(GenericRecord record, boolean skipMetadata) {
return new Comparable[] {};
}
}
+
+ private void initJavaSparkContext() {
Review Comment:
not sure I understand. I only see
```
private void initJavaSparkContext(Option<String> userDefinedMaster) {
if (jsc == null) {
jsc = SparkUtil.initJavaSparkContext(SparkUtil.getDefaultConf("HoodieCLI", userDefinedMaster));
}
}
```
in MetadataCommand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7076:
URL: https://github.com/apache/hudi/pull/7076#issuecomment-1297718110
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12620",
"triggerID" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"triggerType" : "PUSH"
}, {
"hash" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12625",
"triggerID" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"triggerType" : "PUSH"
}, {
"hash" : "565b83f6ba744fd62312d9e2dff8dbf6956d6b80",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12694",
"triggerID" : "565b83f6ba744fd62312d9e2dff8dbf6956d6b80",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 565b83f6ba744fd62312d9e2dff8dbf6956d6b80 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12694)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7076:
URL: https://github.com/apache/hudi/pull/7076#issuecomment-1301490604
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12620",
"triggerID" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"triggerType" : "PUSH"
}, {
"hash" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12625",
"triggerID" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"triggerType" : "PUSH"
}, {
"hash" : "565b83f6ba744fd62312d9e2dff8dbf6956d6b80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12694",
"triggerID" : "565b83f6ba744fd62312d9e2dff8dbf6956d6b80",
"triggerType" : "PUSH"
}, {
"hash" : "d4f5107ca1bd9bba7bda422609072e971f89f12c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12736",
"triggerID" : "d4f5107ca1bd9bba7bda422609072e971f89f12c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d4f5107ca1bd9bba7bda422609072e971f89f12c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12736)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7076:
URL: https://github.com/apache/hudi/pull/7076#issuecomment-1293987119
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12620",
"triggerID" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"triggerType" : "PUSH"
}, {
"hash" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82cc7354ee41a3983583dc8a5455110c9b66f0f6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12620)
* a2d3672965a7bce164434b6cf64a7ec422db71da UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7076:
URL: https://github.com/apache/hudi/pull/7076#issuecomment-1294283414
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12620",
"triggerID" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"triggerType" : "PUSH"
}, {
"hash" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12625",
"triggerID" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a2d3672965a7bce164434b6cf64a7ec422db71da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12625)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #7076:
URL: https://github.com/apache/hudi/pull/7076#discussion_r1009645417
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java:
##########
@@ -206,4 +250,22 @@ private Comparable[] readCommit(GenericRecord record, boolean skipMetadata) {
return new Comparable[] {};
}
}
+
+ private void initJavaSparkContext() {
Review Comment:
we should have existing util methods to instantiate right. can we try re-using it? how other commands are initializing jsc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7076:
URL: https://github.com/apache/hudi/pull/7076#issuecomment-1293992790
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12620",
"triggerID" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"triggerType" : "PUSH"
}, {
"hash" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12625",
"triggerID" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82cc7354ee41a3983583dc8a5455110c9b66f0f6 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12620)
* a2d3672965a7bce164434b6cf64a7ec422db71da Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12625)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
nsivabalan merged PR #7076:
URL: https://github.com/apache/hudi/pull/7076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7076:
URL: https://github.com/apache/hudi/pull/7076#issuecomment-1297541392
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12620",
"triggerID" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"triggerType" : "PUSH"
}, {
"hash" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12625",
"triggerID" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"triggerType" : "PUSH"
}, {
"hash" : "565b83f6ba744fd62312d9e2dff8dbf6956d6b80",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "565b83f6ba744fd62312d9e2dff8dbf6956d6b80",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a2d3672965a7bce164434b6cf64a7ec422db71da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12625)
* 565b83f6ba744fd62312d9e2dff8dbf6956d6b80 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #7076:
URL: https://github.com/apache/hudi/pull/7076#discussion_r1012271387
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java:
##########
@@ -52,6 +60,37 @@
*/
@ShellComponent
public class ArchivedCommitsCommand {
+ private static final Logger LOG = LogManager.getLogger(ArchivedCommitsCommand.class);
+ @ShellMethod(key = "trigger archival", value = "trigger archival")
+ public String triggerArchival(
+ @ShellOption(value = {"--minCommits"},
+ help = "Minimum number of instants to retain in the active timeline. See hoodie.keep.min.commits",
+ defaultValue = "20") int minCommits,
+ @ShellOption(value = {"--maxCommits"},
+ help = "Maximum number of instants to retain in the active timeline. See hoodie.keep.max.commits",
+ defaultValue = "30") int maxCommits,
+ @ShellOption(value = {"--commitsRetained"}, help = "Number of commits to retain, without cleaning",
Review Comment:
lets rename to "commitsRetainedByCleaner" so its apparent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on a diff in pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
jonvex commented on code in PR #7076:
URL: https://github.com/apache/hudi/pull/7076#discussion_r1009726121
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ArchivedCommitsCommand.java:
##########
@@ -52,6 +67,35 @@
*/
@ShellComponent
public class ArchivedCommitsCommand {
+ private static final Logger LOG = LogManager.getLogger(ArchivedCommitsCommand.class);
+ private JavaSparkContext jsc;
+ @ShellMethod(key = "trigger archival", value = "trigger archival")
+ public void triggerArchival(
+ @ShellOption(value = {"--minCommits"},
+ help = "Minimum number of instants to retain in the active timeline. See hoodie.keep.min.commits",
+ defaultValue = "20") int minCommits,
+ @ShellOption(value = {"--maxCommits"},
+ help = "Maximum number of instants to retain in the active timeline. See hoodie.keep.max.commits",
+ defaultValue = "30") int maxCommits,
+ @ShellOption(value = {"--commitsRetained"}, help = "Number of commits to retain, without cleaning",
+ defaultValue = "10") int retained) {
+
+ initJavaSparkContext();
+ HoodieWriteConfig config = HoodieWriteConfig.newBuilder().withPath(HoodieCLI.basePath)
+ .withArchivalConfig(HoodieArchivalConfig.newBuilder().archiveCommitsWith(minCommits,maxCommits).build())
+ .withCleanConfig(HoodieCleanConfig.newBuilder().retainCommits(retained).build())
+ .withEmbeddedTimelineServerEnabled(false)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder().enable(false).build())
Review Comment:
Ok added. What about timelineserver?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7076:
URL: https://github.com/apache/hudi/pull/7076#issuecomment-1297547727
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12620",
"triggerID" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"triggerType" : "PUSH"
}, {
"hash" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12625",
"triggerID" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"triggerType" : "PUSH"
}, {
"hash" : "565b83f6ba744fd62312d9e2dff8dbf6956d6b80",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12694",
"triggerID" : "565b83f6ba744fd62312d9e2dff8dbf6956d6b80",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a2d3672965a7bce164434b6cf64a7ec422db71da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12625)
* 565b83f6ba744fd62312d9e2dff8dbf6956d6b80 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12694)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7076: [HUDI-5032] Add archive to cli
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7076:
URL: https://github.com/apache/hudi/pull/7076#issuecomment-1301242135
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12620",
"triggerID" : "82cc7354ee41a3983583dc8a5455110c9b66f0f6",
"triggerType" : "PUSH"
}, {
"hash" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12625",
"triggerID" : "a2d3672965a7bce164434b6cf64a7ec422db71da",
"triggerType" : "PUSH"
}, {
"hash" : "565b83f6ba744fd62312d9e2dff8dbf6956d6b80",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12694",
"triggerID" : "565b83f6ba744fd62312d9e2dff8dbf6956d6b80",
"triggerType" : "PUSH"
}, {
"hash" : "d4f5107ca1bd9bba7bda422609072e971f89f12c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d4f5107ca1bd9bba7bda422609072e971f89f12c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 565b83f6ba744fd62312d9e2dff8dbf6956d6b80 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12694)
* d4f5107ca1bd9bba7bda422609072e971f89f12c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org