You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/12/22 07:33:47 UTC
[GitHub] [hudi] nsivabalan opened a new pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
nsivabalan opened a new pull request #4420:
URL: https://github.com/apache/hudi/pull/4420
## What is the purpose of the pull request
At the write client layer, inline and async has a very tighter meaning. enabling inline compaction means, both scheduling and execution is inline. Same applies to clustering as well. If async config is enabled, both scheduling and execution has to be done async and regular writer will not do anything. But any scheduling has to be done in a coordinated manner or in other words, there should not be any other operation inflight. For eg, if someone has a separate process to schedule and execute compaction, they have to ensure no other writers are in progress while scheduling. If not, it might lead to data loss. Which means, this puts an extra burden for users to configure lock providers. But Hudi has the intelligence where scheduling can be done inline, but execution can be done async. Just that the configs aren't lined up well. So, adding two new configs named `hoodie.compact.schedule.async` and `hoodie.clustering.schedule.async`. When enabled, scheduling will happen inline by regula
r writers. And the expectation is that, users will have a separate job to execute the already scheduled ones. By this, users don't have to configure lock providers. With metadata table, this constraint might change, but if not for metadata table, users should have a way to exploit async execution of table services.
to be discussed:
I have not fixed the deltastreamer part and not added tests yet.
- Wanted to hear opinions if the approach is good.
-Deltastreamer current state of things
a. by default compaction is async for MOR table unless explicitly disabled via configs.
b. if async clustering is enabled, scheduling will happen explicitly(inline) and execution will happen in a different thread.
So, with this patch, wondering if we should leave the deltastreamer flow untouched.
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r801254435
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
if (config.inlineCompactionEnabled()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
- inlineCompact(extraMetadata);
+ inlineScheduleCompactAndOptionallyExecute(extraMetadata, !config.scheduleInlineCompaction());
Review comment:
I think the reason that i feel confusing here is that there is assumption:
`config.scheduleInlineCompaction()` === `only schedule and no execute compaction`, that means either we only schedule the compaction or we schedule and execute it. These two config options have responsibility overlap.
But from the config option itself, we can not infer such assumption.
What about we config both option inline compaction as true but inline schedule as false ? Should we throw exception there ?
Actually, in very early Flink options, we already split the `inline compact` option into 2: one for inline scheduling and one for inline execution. They are both default as true, people can disable one of them or two, and in general ,these 2 options are mutually exclusive.
> Infact I did had it that way, on my first attempt. but did not feel the need and later removed it.
I still feel that a separate method make things more clear, one method only have single responsibility, WDYT ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r801258717
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
if (config.inlineCompactionEnabled()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
- inlineCompact(extraMetadata);
+ inlineScheduleCompactAndOptionallyExecute(extraMetadata, !config.scheduleInlineCompaction());
} else {
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "false");
}
+ // if just inline schedule is enabled
+ if (!config.inlineCompactionEnabled() && config.scheduleInlineCompaction()
+ && !table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().getInstants().findAny().isPresent()) {
+ // proceed only if there are no pending compactions
Review comment:
No strong opinions here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r797294083
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
Review comment:
Do you think we should returns true for only scheduling(non-execute) table service in `areAnyTableServicesInline`, personally i think there are responsibilities similarity for these two methods here and they are confusing.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
if (config.inlineCompactionEnabled()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
- inlineCompact(extraMetadata);
+ inlineScheduleCompactAndOptionallyExecute(extraMetadata, !config.scheduleInlineCompaction());
} else {
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "false");
}
+ // if just inline schedule is enabled
+ if (!config.inlineCompactionEnabled() && config.scheduleInlineCompaction()
+ && !table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().getInstants().findAny().isPresent()) {
+ // proceed only if there are no pending compactions
Review comment:
Can we just add a tool method in `HoodieActiveTimeline` for decision `table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().getInstants().findAny().isPresent()`
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieClusteringConfig.java
##########
@@ -177,6 +178,16 @@
.withDocumentation("Determines how to handle updates, deletes to file groups that are under clustering."
+ " Default strategy just rejects the update");
+ public static final ConfigProperty<String> SCHEDULE_INLINE_CLUSTERING = ConfigProperty
+ .key("hoodie.clustering.schedule.inline")
Review comment:
`SCHEDULE_INLINE_CLUSTERING` -> `CLUSTERING_SCHEDULE_INLINE`
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
if (config.inlineCompactionEnabled()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
- inlineCompact(extraMetadata);
+ inlineScheduleCompactAndOptionallyExecute(extraMetadata, !config.scheduleInlineCompaction());
} else {
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "false");
}
+ // if just inline schedule is enabled
+ if (!config.inlineCompactionEnabled() && config.scheduleInlineCompaction()
+ && !table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().getInstants().findAny().isPresent()) {
+ // proceed only if there are no pending compactions
Review comment:
Can we also add a tool method for decision `config.inlineCompactionEnabled() && config.scheduleInlineCompaction()` in config.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -1005,13 +1021,16 @@ protected void rollbackFailedWrites(Map<String, Option<HoodiePendingRollbackInfo
/**
* Performs a compaction operation on a table, serially before or after an insert/upsert action.
+ * Scheduling is done inline, and optionally execution as well.
*/
- protected Option<String> inlineCompact(Option<Map<String, String>> extraMetadata) {
+ protected Option<String> inlineScheduleCompactAndOptionallyExecute(Option<Map<String, String>> extraMetadata, boolean executeInline) {
Option<String> compactionInstantTimeOpt = scheduleCompaction(extraMetadata);
Review comment:
Personally i prefer to keep the old method and add a new one called `inlineScheduleCompact` which is only for scheduling purposes.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
if (config.inlineCompactionEnabled()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
- inlineCompact(extraMetadata);
+ inlineScheduleCompactAndOptionallyExecute(extraMetadata, !config.scheduleInlineCompaction());
Review comment:
The decision already indicate deterministic incline `schedule and execute` table services here, so can we just keep the existing method `inlineCompact` here and made a new method for only scheduling purpose.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java
##########
@@ -90,6 +90,12 @@
.withDocumentation("When set to true, compaction service is triggered after each write. While being "
+ " simpler operationally, this adds extra latency on the write path.");
+ public static final ConfigProperty<String> SCHEDULE_ASYNC_COMPACT = ConfigProperty
+ .key("hoodie.compact.schedule.async")
+ .defaultValue("false")
Review comment:
`COMPACT_SCHEDULE_INLINE` please to keep inline with the string options sequence.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -1116,13 +1135,16 @@ protected boolean scheduleCleaningAtInstant(String instantTime, Option<Map<Strin
/**
* Executes a clustering plan on a table, serially before or after an insert/upsert action.
+ * Schedules clustering inline and may be optionally execute.
*/
- protected Option<String> inlineCluster(Option<Map<String, String>> extraMetadata) {
+ protected Option<String> inlineScheduleClusterAndOptionallyExecute(Option<Map<String, String>> extraMetadata, boolean executeInline) {
Option<String> clusteringInstantOpt = scheduleClustering(extraMetadata);
- clusteringInstantOpt.ifPresent(clusteringInstant -> {
- // inline cluster should auto commit as the user is never given control
- cluster(clusteringInstant, true);
- });
+ if (executeInline) {
+ clusteringInstantOpt.ifPresent(clusteringInstant -> {
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-999969285
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 066bfb55adff0afd0e0b0e86c3c42d88f842e37c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660)
* 4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1028987325
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
}, {
"hash" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5703",
"triggerID" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48fd6ddd0a36975f7e065ae3c4c4635422def062 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5703)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r773688261
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java
##########
@@ -90,6 +90,12 @@
.withDocumentation("When set to true, compaction service is triggered after each write. While being "
+ " simpler operationally, this adds extra latency on the write path.");
+ public static final ConfigProperty<String> SCHEDULE_ASYNC_COMPACT = ConfigProperty
+ .key("hoodie.compact.schedule.async")
+ .defaultValue("false")
Review comment:
`COMPACT_SCHEDULE_ASYNC` seems better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1017809926
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4283c94401cdda02247db57ea74338ae5b2315d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249)
* 1108c4219c9ae2149a60facdce7e54f4ae1b7e1f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1013225969
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4283c94401cdda02247db57ea74338ae5b2315d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-999987258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019596412
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a153e69c2d10ccd591cbc292553c1c25c20c7046 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1017837233
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1108c4219c9ae2149a60facdce7e54f4ae1b7e1f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019648127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a153e69c2d10ccd591cbc292553c1c25c20c7046 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450)
* 8a951644c0ab76f28e89aa0f2be248bbdcdb0601 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1026873141
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a951644c0ab76f28e89aa0f2be248bbdcdb0601 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451)
* d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-999970237
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 066bfb55adff0afd0e0b0e86c3c42d88f842e37c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660)
* 4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r801254435
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
if (config.inlineCompactionEnabled()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
- inlineCompact(extraMetadata);
+ inlineScheduleCompactAndOptionallyExecute(extraMetadata, !config.scheduleInlineCompaction());
Review comment:
I think the reason that i feel confusing here is that there is assumption:
`config.scheduleInlineCompaction()` === `only schedule and no execute compaction`, that means either we only schedule the compaction or we schedule and execute it. These two config options have responsibility overlap.
What about we config both option inline compaction as true but inline schedule as false ? Should we throw exception there ?
> Infact I did had it that way, on my first attempt. but did not feel the need and later removed it.
I still feel that a separate method make things more clear, one method only have single responsibility, WDYT ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r801254435
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
if (config.inlineCompactionEnabled()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
- inlineCompact(extraMetadata);
+ inlineScheduleCompactAndOptionallyExecute(extraMetadata, !config.scheduleInlineCompaction());
Review comment:
No strong options here, it is code style preference from my side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1032315323
@danny0405 : addressed your feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1032651666
@jaxonzhang : nope. not required. Lock provider is required if scheduling is also done async. Scheduling is the crux or critical section here where no concurrent writers should be operating. So, if scheduling is done inline by regular writer, and execution is done by async process, no lock provider is required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019542976
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60ff7726eb3d929ad300f690cba45c9d62397310 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425)
* d5ec0d04b00156ea53fdbb735f545a6ef3f7c118 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1018935914
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60ff7726eb3d929ad300f690cba45c9d62397310 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019582309
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5ec0d04b00156ea53fdbb735f545a6ef3f7c118 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447)
* a153e69c2d10ccd591cbc292553c1c25c20c7046 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-999350618
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 066bfb55adff0afd0e0b0e86c3c42d88f842e37c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1032316246
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
}, {
"hash" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5703",
"triggerID" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"triggerType" : "PUSH"
}, {
"hash" : "09a94bc5e7d1a5d5e84edc76a861eaf3be2369a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "09a94bc5e7d1a5d5e84edc76a861eaf3be2369a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48fd6ddd0a36975f7e065ae3c4c4635422def062 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5703)
* 09a94bc5e7d1a5d5e84edc76a861eaf3be2369a7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-999987258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1013150576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678)
* c4283c94401cdda02247db57ea74338ae5b2315d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1013150576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678)
* c4283c94401cdda02247db57ea74338ae5b2315d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019651470
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a951644c0ab76f28e89aa0f2be248bbdcdb0601 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019581829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5ec0d04b00156ea53fdbb735f545a6ef3f7c118 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447)
* a153e69c2d10ccd591cbc292553c1c25c20c7046 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1026975832
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1026873141
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a951644c0ab76f28e89aa0f2be248bbdcdb0601 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451)
* d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1020463031
@danny0405 : let me know if you have any more feedback. Sagar has approved it already. If you are good, we can go ahead and land it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019648127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a153e69c2d10ccd591cbc292553c1c25c20c7046 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450)
* 8a951644c0ab76f28e89aa0f2be248bbdcdb0601 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1026877064
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a951644c0ab76f28e89aa0f2be248bbdcdb0601 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451)
* d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1026877064
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a951644c0ab76f28e89aa0f2be248bbdcdb0601 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451)
* d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1013148089
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678)
* c4283c94401cdda02247db57ea74338ae5b2315d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1018886439
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1108c4219c9ae2149a60facdce7e54f4ae1b7e1f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388)
* 60ff7726eb3d929ad300f690cba45c9d62397310 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r773685897
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -460,24 +460,24 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.inlineTableServices()) {
+ if (config.inlineTableServices() || config.scheduleAsyncTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
- if (config.inlineCompactionEnabled()) {
+ if (config.inlineCompactionEnabled() || config.scheduleAsyncCompaction()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
Review comment:
Why we still need to run the pending compaction and clustering plans ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-999970237
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 066bfb55adff0afd0e0b0e86c3c42d88f842e37c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660)
* 4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r801254435
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
if (config.inlineCompactionEnabled()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
- inlineCompact(extraMetadata);
+ inlineScheduleCompactAndOptionallyExecute(extraMetadata, !config.scheduleInlineCompaction());
Review comment:
I think the reason that i feel confusing here is that there is assumption:
`config.scheduleInlineCompaction()` === `only schedule and no execute compaction`, that means either we only schedule the compaction or we schedule and execute it. These two config options have responsibility overlap.
But from the config option itself, we can not infer such assumption.
What about we config both option inline compaction as true but inline schedule as false ? Should we throw exception there ?
Actually, in very early Flink options, we already split the `inline compact` option into 2: one for inline scheduling and one for inline execution. They are both default as true, people can disable one of them of two, and in general ,these 2 options are mutually exclusive.
> Infact I did had it that way, on my first attempt. but did not feel the need and later removed it.
I still feel that a separate method make things more clear, one method only have single responsibility, WDYT ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1028929692
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
}, {
"hash" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651)
* 48fd6ddd0a36975f7e065ae3c4c4635422def062 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1026975832
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #4420:
URL: https://github.com/apache/hudi/pull/4420
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019542540
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60ff7726eb3d929ad300f690cba45c9d62397310 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425)
* d5ec0d04b00156ea53fdbb735f545a6ef3f7c118 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019582309
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5ec0d04b00156ea53fdbb735f545a6ef3f7c118 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447)
* a153e69c2d10ccd591cbc292553c1c25c20c7046 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1020463031
@danny0405 : let me know if you have any more feedback. Sagar has approved it already. If you are good, we can go ahead and land it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-999351864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 066bfb55adff0afd0e0b0e86c3c42d88f842e37c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019542540
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60ff7726eb3d929ad300f690cba45c9d62397310 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425)
* d5ec0d04b00156ea53fdbb735f545a6ef3f7c118 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1018888342
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1108c4219c9ae2149a60facdce7e54f4ae1b7e1f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388)
* 60ff7726eb3d929ad300f690cba45c9d62397310 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1018888342
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1108c4219c9ae2149a60facdce7e54f4ae1b7e1f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388)
* 60ff7726eb3d929ad300f690cba45c9d62397310 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1028932141
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
}, {
"hash" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5703",
"triggerID" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651)
* 48fd6ddd0a36975f7e065ae3c4c4635422def062 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5703)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1028929692
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
}, {
"hash" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651)
* 48fd6ddd0a36975f7e065ae3c4c4635422def062 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-999394613
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 066bfb55adff0afd0e0b0e86c3c42d88f842e37c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-999969285
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 066bfb55adff0afd0e0b0e86c3c42d88f842e37c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660)
* 4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1013225969
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4283c94401cdda02247db57ea74338ae5b2315d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jaxonzhang commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
jaxonzhang commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1032327356
hi @nsivabalan, if users have a separate job to execute the already scheduled compaction plans, do the job have to configure OPTIMISTIC CONCURRENCY (lock provider)? If not, what if one writer append records to the same log file that is being compact concurrently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1033223649
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
}, {
"hash" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5703",
"triggerID" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"triggerType" : "PUSH"
}, {
"hash" : "09a94bc5e7d1a5d5e84edc76a861eaf3be2369a7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5792",
"triggerID" : "09a94bc5e7d1a5d5e84edc76a861eaf3be2369a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 09a94bc5e7d1a5d5e84edc76a861eaf3be2369a7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5792)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1018886439
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1108c4219c9ae2149a60facdce7e54f4ae1b7e1f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388)
* 60ff7726eb3d929ad300f690cba45c9d62397310 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1016033994
1.inline scheduling enabled, and async execution enabled
I will add a validation to ensure that both inline execution and inline scheduling can't be enabled. (INLINE_COMPACT is true and SCHEDULE_INLINE_COMPACT is also true) Or are you asking something diff?
2. inline scheduling enabled, and async as well as inline execution disabled. Here, what if users do not have a separate async job, then we will always have a pending instant in the timeline. Do we need some validation to ensure this does not happen?
We can call it out in the config documentation that users should ensure execution happens via a diff process. We are not going to enable this new config by default. So, expectation is that users who enable knows the purpose of it and expectations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019542976
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60ff7726eb3d929ad300f690cba45c9d62397310 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425)
* d5ec0d04b00156ea53fdbb735f545a6ef3f7c118 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019596412
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a153e69c2d10ccd591cbc292553c1c25c20c7046 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1017812581
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4283c94401cdda02247db57ea74338ae5b2315d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249)
* 1108c4219c9ae2149a60facdce7e54f4ae1b7e1f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1017809926
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4283c94401cdda02247db57ea74338ae5b2315d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249)
* 1108c4219c9ae2149a60facdce7e54f4ae1b7e1f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1017812581
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4283c94401cdda02247db57ea74338ae5b2315d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249)
* 1108c4219c9ae2149a60facdce7e54f4ae1b7e1f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1028932141
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
}, {
"hash" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5703",
"triggerID" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651)
* 48fd6ddd0a36975f7e065ae3c4c4635422def062 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5703)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r773687455
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieClusteringConfig.java
##########
@@ -143,6 +143,12 @@
.withDocumentation("Determines how to handle updates, deletes to file groups that are under clustering."
+ " Default strategy just rejects the update");
+ public static final ConfigProperty<String> ASYNC_CLUSTERING_SCHEDULE = ConfigProperty
+ .key("hoodie.clustering.schedule.async")
+ .defaultValue("false")
+ .withDocumentation("When set to true, clustering service will be attempted for scheduling after each write. Users have to ensure "
+ + "they have a separate job to run async clustering(execution) for the one scheduled by this writer");
Review comment:
`CLUSTERING_SCHEDULE_ASYNC` seems better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-999351864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 066bfb55adff0afd0e0b0e86c3c42d88f842e37c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-999350618
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 066bfb55adff0afd0e0b0e86c3c42d88f842e37c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1028987325
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
}, {
"hash" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5703",
"triggerID" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48fd6ddd0a36975f7e065ae3c4c4635422def062 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5703)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1032316246
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
}, {
"hash" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5651",
"triggerID" : "d9e9f98b3025b2245d2e0ca6f54eca86b5286e3e",
"triggerType" : "PUSH"
}, {
"hash" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5703",
"triggerID" : "48fd6ddd0a36975f7e065ae3c4c4635422def062",
"triggerType" : "PUSH"
}, {
"hash" : "09a94bc5e7d1a5d5e84edc76a861eaf3be2369a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "09a94bc5e7d1a5d5e84edc76a861eaf3be2369a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48fd6ddd0a36975f7e065ae3c4c4635422def062 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5703)
* 09a94bc5e7d1a5d5e84edc76a861eaf3be2369a7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r785765597
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
if (config.inlineCompactionEnabled()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
- inlineCompact(extraMetadata);
+ inlineScheduleCompactAndOptionallyExecute(extraMetadata, !config.scheduleInlineCompaction());
Review comment:
Let's say INLINE_COMPACT is true and SCHEDULE_INLINE_COMPACT is also true. Then `inlineScheduleCompactAndOptionallyExecute` will schedule compaction but never execute right? This sounds counter-intuitive from user point of view. Don't you think?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieClusteringConfig.java
##########
@@ -177,6 +177,12 @@
.withDocumentation("Determines how to handle updates, deletes to file groups that are under clustering."
+ " Default strategy just rejects the update");
+ public static final ConfigProperty<String> SCHEDULE_INLINE_CLUSTERING = ConfigProperty
+ .key("hoodie.clustering.schedule.inline")
+ .defaultValue("false")
+ .withDocumentation("When set to true, clustering service will be attempted for inline scheduling after each write. Users have to ensure "
Review comment:
I think we should add more details regarding how it interacts with (or differs from) other inline/async configs otherwise it might confuse users why we have separate configs. First, we could mention what will happen if this config is enabled together with the existing inline/async compaction/clustering configs. Secondly, we can use the corresponding config keys in the doc to be more explicit.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
if (config.inlineCompactionEnabled()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
Review comment:
Should we move this as well inside `inlineScheduleCompactAndOptionallyExecute` and set only when execution happens?
##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestHoodieSparkMergeOnReadTableInsertUpdateDelete.java
##########
@@ -133,6 +134,56 @@ public void testSimpleInsertAndUpdate(HoodieFileFormat fileFormat, boolean popul
}
}
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testInlineScheduleCompaction(boolean scheduleInlineCompaction) throws Exception {
+ HoodieFileFormat fileFormat = HoodieFileFormat.PARQUET;
+ Properties properties = new Properties();
+ properties.setProperty(HoodieTableConfig.BASE_FILE_FORMAT.key(), fileFormat.toString());
+ HoodieTableMetaClient metaClient = getHoodieMetaClient(HoodieTableType.MERGE_ON_READ, properties);
+
+ HoodieWriteConfig cfg = getConfigBuilder(false)
+ .withCompactionConfig(HoodieCompactionConfig.newBuilder().compactionSmallFileSize(1024 * 1024 * 1024)
+ .withInlineCompaction(false).withMaxNumDeltaCommitsBeforeCompaction(2).withPreserveCommitMetadata(true).withScheduleInlineCompaction(scheduleInlineCompaction).build())
+ .build();
+ try (SparkRDDWriteClient client = getHoodieWriteClient(cfg)) {
+
+ HoodieTestDataGenerator dataGen = new HoodieTestDataGenerator();
+ /*
+ * Write 1 (only inserts)
+ */
+ String newCommitTime = "001";
+ client.startCommitWithTime(newCommitTime);
+
+ List<HoodieRecord> records = dataGen.generateInserts(newCommitTime, 200);
+ Stream<HoodieBaseFile> dataFiles = insertRecordsToMORTable(metaClient, records, client, cfg, newCommitTime, true);
+ assertTrue(dataFiles.findAny().isPresent(), "should list the base files we wrote in the delta commit");
+
+ /*
+ * Write 2 (updates)
+ */
+ newCommitTime = "004";
+ client.startCommitWithTime(newCommitTime);
+ records = dataGen.generateUpdates(newCommitTime, 100);
+ updateRecordsInMORTable(metaClient, records, client, cfg, newCommitTime, true);
+
+ // validate compaction has been scheduled inline
+ /*HoodieTable hoodieTable = HoodieSparkTable.create(cfg, context(), metaClient);
+ hoodieTable.getHoodieView().sync();
+ FileStatus[] allFiles = listAllBaseFilesInPath(hoodieTable);
+ HoodieTableFileSystemView tableView = getHoodieTableFileSystemView(metaClient, hoodieTable.getCompletedCommitsTimeline(), allFiles);
+ Stream<HoodieBaseFile> dataFilesToRead = tableView.getLatestBaseFiles();
+ assertTrue(dataFilesToRead.findAny().isPresent());*/
Review comment:
Let's remove this if not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1013148089
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678)
* c4283c94401cdda02247db57ea74338ae5b2315d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1013147552
@codope @danny0405 : patch is good to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019581829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5ec0d04b00156ea53fdbb735f545a6ef3f7c118 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447)
* a153e69c2d10ccd591cbc292553c1c25c20c7046 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019621343
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a153e69c2d10ccd591cbc292553c1c25c20c7046 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450)
* 8a951644c0ab76f28e89aa0f2be248bbdcdb0601 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019553901
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5ec0d04b00156ea53fdbb735f545a6ef3f7c118 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1018935914
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60ff7726eb3d929ad300f690cba45c9d62397310 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1017837233
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1108c4219c9ae2149a60facdce7e54f4ae1b7e1f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r801348606
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
if (config.inlineCompactionEnabled()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
- inlineCompact(extraMetadata);
+ inlineScheduleCompactAndOptionallyExecute(extraMetadata, !config.scheduleInlineCompaction());
Review comment:
ok, sounds good. will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1034897754
thanks @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019621343
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a153e69c2d10ccd591cbc292553c1c25c20c7046 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450)
* 8a951644c0ab76f28e89aa0f2be248bbdcdb0601 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019553901
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5ec0d04b00156ea53fdbb735f545a6ef3f7c118 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1019651470
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
}, {
"hash" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4678",
"triggerID" : "4fa49c9e59b4ae02cef190dfbc7b9d5707d6d049",
"triggerType" : "PUSH"
}, {
"hash" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5249",
"triggerID" : "c4283c94401cdda02247db57ea74338ae5b2315d",
"triggerType" : "PUSH"
}, {
"hash" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5388",
"triggerID" : "1108c4219c9ae2149a60facdce7e54f4ae1b7e1f",
"triggerType" : "PUSH"
}, {
"hash" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5425",
"triggerID" : "60ff7726eb3d929ad300f690cba45c9d62397310",
"triggerType" : "PUSH"
}, {
"hash" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5447",
"triggerID" : "d5ec0d04b00156ea53fdbb735f545a6ef3f7c118",
"triggerType" : "PUSH"
}, {
"hash" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5450",
"triggerID" : "a153e69c2d10ccd591cbc292553c1c25c20c7046",
"triggerType" : "PUSH"
}, {
"hash" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451",
"triggerID" : "8a951644c0ab76f28e89aa0f2be248bbdcdb0601",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a951644c0ab76f28e89aa0f2be248bbdcdb0601 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5451)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-1028912096
wrt variables names, I kept it in sync w/ existing ones. for eg, config for inline compaction is called "hoodie.compact.inline", but variable is named as INLINE_COMPACT.
Guess it goes well with the getter method inlineCompactionEnabled. So, will keep the existing configs and variable names for scheduleInlineCompaction as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4420: [HUDI-1847] Adding inline scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r784786957
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieClusteringConfig.java
##########
@@ -143,6 +143,12 @@
.withDocumentation("Determines how to handle updates, deletes to file groups that are under clustering."
+ " Default strategy just rejects the update");
+ public static final ConfigProperty<String> ASYNC_CLUSTERING_SCHEDULE = ConfigProperty
+ .key("hoodie.clustering.schedule.async")
+ .defaultValue("false")
+ .withDocumentation("When set to true, clustering service will be attempted for scheduling after each write. Users have to ensure "
+ + "they have a separate job to run async clustering(execution) for the one scheduled by this writer");
Review comment:
I have fixed the config to `hoodie.clustering.schedule.inline` and var name to SCHEDULE_INLINE_CLUSTERING since scheduling is done INLINE with this config and only execution is async.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java
##########
@@ -90,6 +90,12 @@
.withDocumentation("When set to true, compaction service is triggered after each write. While being "
+ " simpler operationally, this adds extra latency on the write path.");
+ public static final ConfigProperty<String> SCHEDULE_ASYNC_COMPACT = ConfigProperty
+ .key("hoodie.compact.schedule.async")
+ .defaultValue("false")
Review comment:
have responded else where. prefer to keep it as is to be in sync w/ existing inline compaction config.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieClusteringConfig.java
##########
@@ -177,6 +178,16 @@
.withDocumentation("Determines how to handle updates, deletes to file groups that are under clustering."
+ " Default strategy just rejects the update");
+ public static final ConfigProperty<String> SCHEDULE_INLINE_CLUSTERING = ConfigProperty
+ .key("hoodie.clustering.schedule.inline")
Review comment:
responded else where. I wanted to keep it in sync w/ existing inline clustering config.
"hoodie.clustering.inline", and the variable is named as INLINE_CLUSTERING. Guess it goes well w/ how the getter method is named (inlineClusteringEnabled). So, prefer to leave it as is.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieCompactionConfig.java
##########
@@ -90,6 +90,12 @@
.withDocumentation("When set to true, compaction service is triggered after each write. While being "
+ " simpler operationally, this adds extra latency on the write path.");
+ public static final ConfigProperty<String> SCHEDULE_ASYNC_COMPACT = ConfigProperty
+ .key("hoodie.compact.schedule.async")
+ .defaultValue("false")
Review comment:
same here. have fixed the naming.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -1116,13 +1135,16 @@ protected boolean scheduleCleaningAtInstant(String instantTime, Option<Map<Strin
/**
* Executes a clustering plan on a table, serially before or after an insert/upsert action.
+ * Schedules clustering inline and may be optionally execute.
*/
- protected Option<String> inlineCluster(Option<Map<String, String>> extraMetadata) {
+ protected Option<String> inlineScheduleClusterAndOptionallyExecute(Option<Map<String, String>> extraMetadata, boolean executeInline) {
Option<String> clusteringInstantOpt = scheduleClustering(extraMetadata);
- clusteringInstantOpt.ifPresent(clusteringInstant -> {
- // inline cluster should auto commit as the user is never given control
- cluster(clusteringInstant, true);
- });
+ if (executeInline) {
+ clusteringInstantOpt.ifPresent(clusteringInstant -> {
Review comment:
same response as above.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
Review comment:
have renamed the methods to be explicit.
areAnyTableServicesInline -> areAnyTableServicesExecutedInline
scheduleInlineTableServices -> areAnyTableServicesScheduledInline
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
if (config.inlineCompactionEnabled()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
- inlineCompact(extraMetadata);
+ inlineScheduleCompactAndOptionallyExecute(extraMetadata, !config.scheduleInlineCompaction());
Review comment:
just scheduling is only one line.
```
Option<String> compactionInstantTimeOpt = scheduleCompaction(extraMetadata);
```
thats the reason did not find a need to create a new method just for scheduling inline. Infact I did had it that way, on my first attempt. but did not feel the need and later removed it.
With new naming, I guess its clear that execution is optional. So, I feel we should be ok here.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -1005,13 +1021,16 @@ protected void rollbackFailedWrites(Map<String, Option<HoodiePendingRollbackInfo
/**
* Performs a compaction operation on a table, serially before or after an insert/upsert action.
+ * Scheduling is done inline, and optionally execution as well.
*/
- protected Option<String> inlineCompact(Option<Map<String, String>> extraMetadata) {
+ protected Option<String> inlineScheduleCompactAndOptionallyExecute(Option<Map<String, String>> extraMetadata, boolean executeInline) {
Option<String> compactionInstantTimeOpt = scheduleCompaction(extraMetadata);
Review comment:
have responded else where. scheduling is just oneline. does not warrant a new method. after renaming the method, I feel we are ok.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
##########
@@ -464,27 +464,43 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.areAnyTableServicesInline()) {
+ if (config.areAnyTableServicesInline() || config.scheduleInlineTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
if (config.inlineCompactionEnabled()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
- inlineCompact(extraMetadata);
+ inlineScheduleCompactAndOptionallyExecute(extraMetadata, !config.scheduleInlineCompaction());
} else {
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "false");
}
+ // if just inline schedule is enabled
+ if (!config.inlineCompactionEnabled() && config.scheduleInlineCompaction()
+ && !table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().getInstants().findAny().isPresent()) {
+ // proceed only if there are no pending compactions
Review comment:
1. Usually we try to have methods in HoodieActivetimeline that are used by many callers so that we can avoid duplication or errors. For one off cases like this, we keep it away from HoodieActivetimeline.
2. somehow I feel we are over populating HoodieWriteConfig. This is a very specific use-case and probably we can leave it here. condition here is very specific in the sense that, we need to check if inline is not enabled, but schedule inline is enabled. and I don't see this being used anywhere else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-999965020
On second thoughts, guess the new configs should be named with "inline" and not "async"
SCHEDULE_INLINE_COMPACT and SCHEDULE_INLINE_CLUSTER. Implicitly this means, execution will happen async.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r773684448
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -989,12 +989,14 @@ protected void rollbackFailedWrites(Map<String, Option<HoodiePendingRollbackInfo
/**
* Performs a compaction operation on a table, serially before or after an insert/upsert action.
*/
- protected Option<String> inlineCompact(Option<Map<String, String>> extraMetadata) {
+ protected Option<String> inlineCompact(Option<Map<String, String>> extraMetadata, boolean executeInline) {
Option<String> compactionInstantTimeOpt = scheduleCompaction(extraMetadata);
- compactionInstantTimeOpt.ifPresent(compactInstantTime -> {
Review comment:
The method name is `inlineCompact` and why is still a flag `executeInline` there ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#discussion_r773979454
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -460,24 +460,24 @@ protected void postCommit(HoodieTable<T, I, K, O> table, HoodieCommitMetadata me
}
protected void runTableServicesInline(HoodieTable<T, I, K, O> table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
- if (config.inlineTableServices()) {
+ if (config.inlineTableServices() || config.scheduleAsyncTableServices()) {
if (config.isMetadataTableEnabled()) {
table.getHoodieView().sync();
}
// Do an inline compaction if enabled
- if (config.inlineCompactionEnabled()) {
+ if (config.inlineCompactionEnabled() || config.scheduleAsyncCompaction()) {
runAnyPendingCompactions(table);
metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
Review comment:
I had some local changes which I missed to update here. essentially, we can schedule only if there are no other pending jobs. Will push my changes along with addressing other feedback as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4420: [HUDI-1847] [UNDER_DISCUSSION] Adding async scheduling support for spark datasource path for compaction and clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4420:
URL: https://github.com/apache/hudi/pull/4420#issuecomment-999394613
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660",
"triggerID" : "066bfb55adff0afd0e0b0e86c3c42d88f842e37c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 066bfb55adff0afd0e0b0e86c3c42d88f842e37c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4660)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org