You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/20 15:21:44 UTC
[GitHub] [hudi] prashantwason opened a new pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
prashantwason opened a new pull request #3836:
URL: https://github.com/apache/hudi/pull/3836
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
Fixes double bootstrap of metadata table.
## Brief change log
Check if upgrade/downgrade is required before bootstrapping metadata table.
## Verify this pull request
This pull request is a trivial rework / code cleanup without any test coverage.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-961588627
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#discussion_r744036617
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -95,10 +95,19 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeC
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ bootstrapMetadataTable();
+ }
+
+ private void bootstrapMetadataTable() {
if (config.isMetadataTableEnabled()) {
- // If the metadata table does not exist, it should be bootstrapped here
- // TODO: Check if we can remove this requirement - auto bootstrap on commit
- SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config, context);
+ // Defer bootstrap if upgrade / downgrade is pending
+ HoodieTableMetaClient metaClient = createMetaClient(true);
+ UpgradeDowngrade upgradeDowngrade = new UpgradeDowngrade(
+ metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance());
+ if (!upgradeDowngrade.needsUpgradeOrDowngrade(HoodieTableVersion.current())) {
Review comment:
thanks for the quick turn around. yes. Let's go with 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-961588627
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-962231751
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748)
* e916c158969b8b855cbffffde8bceac94e80b98d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-963349825
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
}, {
"hash" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e916c158969b8b855cbffffde8bceac94e80b98d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170)
* 9a4f8faa55d8605c7222dbfae302815c58efbd06 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-963352122
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
}, {
"hash" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220",
"triggerID" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e916c158969b8b855cbffffde8bceac94e80b98d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170)
* 9a4f8faa55d8605c7222dbfae302815c58efbd06 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-963400355
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
}, {
"hash" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220",
"triggerID" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9a4f8faa55d8605c7222dbfae302815c58efbd06 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#discussion_r743670241
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -95,10 +95,19 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeC
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ bootstrapMetadataTable();
+ }
+
+ private void bootstrapMetadataTable() {
if (config.isMetadataTableEnabled()) {
- // If the metadata table does not exist, it should be bootstrapped here
- // TODO: Check if we can remove this requirement - auto bootstrap on commit
- SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config, context);
+ // Defer bootstrap if upgrade / downgrade is pending
+ HoodieTableMetaClient metaClient = createMetaClient(true);
+ UpgradeDowngrade upgradeDowngrade = new UpgradeDowngrade(
+ metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance());
+ if (!upgradeDowngrade.needsUpgradeOrDowngrade(HoodieTableVersion.current())) {
Review comment:
@prashantwason : can you help me understand something. if upgrade is required, we defer the bootstrapping. So, when exactly the upgrade happens? in a subsequent commit? so IIUC, on first commit where upgrade happens, metadata won't be bootstrapped and with 2 -> 3, we will delete the metadata table. and on 2nd commit, will trigger the actual bootstrap of metadata table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-962230403
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748)
* e916c158969b8b855cbffffde8bceac94e80b98d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#discussion_r745360466
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -83,22 +83,31 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig client
@Deprecated
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig, boolean rollbackPending) {
- super(context, writeConfig);
+ this(context, writeConfig, Option.empty());
}
@Deprecated
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig, boolean rollbackPending,
Option<EmbeddedTimelineService> timelineService) {
- super(context, writeConfig, timelineService);
+ this(context, writeConfig, timelineService);
}
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ bootstrapMetadataTable(Option.empty());
+ }
+
+ private void bootstrapMetadataTable(Option<String> instantInProgressTimestamp) {
Review comment:
renamed to initializeMetadataTable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #3836:
URL: https://github.com/apache/hudi/pull/3836
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-947803091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#discussion_r745261515
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -83,22 +83,31 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig client
@Deprecated
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig, boolean rollbackPending) {
- super(context, writeConfig);
+ this(context, writeConfig, Option.empty());
}
@Deprecated
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig, boolean rollbackPending,
Option<EmbeddedTimelineService> timelineService) {
- super(context, writeConfig, timelineService);
+ this(context, writeConfig, timelineService);
}
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ bootstrapMetadataTable(Option.empty());
+ }
+
+ private void bootstrapMetadataTable(Option<String> instantInProgressTimestamp) {
Review comment:
not sure if we can call it as bootstrap metadata table. this is essentially initializing metadata table. internal impl could do bootstrap if applicable.
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -95,10 +95,19 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeC
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ bootstrapMetadataTable();
+ }
+
+ private void bootstrapMetadataTable() {
if (config.isMetadataTableEnabled()) {
- // If the metadata table does not exist, it should be bootstrapped here
- // TODO: Check if we can remove this requirement - auto bootstrap on commit
- SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config, context);
+ // Defer bootstrap if upgrade / downgrade is pending
+ HoodieTableMetaClient metaClient = createMetaClient(true);
+ UpgradeDowngrade upgradeDowngrade = new UpgradeDowngrade(
+ metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance());
+ if (!upgradeDowngrade.needsUpgradeOrDowngrade(HoodieTableVersion.current())) {
Review comment:
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#discussion_r744011894
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -95,10 +95,19 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeC
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ bootstrapMetadataTable();
+ }
+
+ private void bootstrapMetadataTable() {
if (config.isMetadataTableEnabled()) {
- // If the metadata table does not exist, it should be bootstrapped here
- // TODO: Check if we can remove this requirement - auto bootstrap on commit
- SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config, context);
+ // Defer bootstrap if upgrade / downgrade is pending
+ HoodieTableMetaClient metaClient = createMetaClient(true);
+ UpgradeDowngrade upgradeDowngrade = new UpgradeDowngrade(
+ metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance());
+ if (!upgradeDowngrade.needsUpgradeOrDowngrade(HoodieTableVersion.current())) {
Review comment:
gotcha. wondering if we can move the instantiation of metadata table
```
SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config, context);
```
just after upgrade downgrade step within getTableAndInitCtx.
So, that we may not miss to bootstrap the table even if upgrade is required.
Here is what I am thinking. I would assume typically users might stop all processes and then do an upgrade for next hudi version. Once first commit goes through, users might want to get started w/ multi writers. so, would prefer to do the bootstrap on the first write operation where upgrade is triggered.
Open to hear your thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#discussion_r743670241
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -95,10 +95,19 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeC
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ bootstrapMetadataTable();
+ }
+
+ private void bootstrapMetadataTable() {
if (config.isMetadataTableEnabled()) {
- // If the metadata table does not exist, it should be bootstrapped here
- // TODO: Check if we can remove this requirement - auto bootstrap on commit
- SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config, context);
+ // Defer bootstrap if upgrade / downgrade is pending
+ HoodieTableMetaClient metaClient = createMetaClient(true);
+ UpgradeDowngrade upgradeDowngrade = new UpgradeDowngrade(
+ metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance());
+ if (!upgradeDowngrade.needsUpgradeOrDowngrade(HoodieTableVersion.current())) {
Review comment:
@prashantwason : can you help me understand something. if upgrade is required, we defer the bootstrapping. So, when exactly the upgrade happens? in a subsequent commit? so IIUC, on first commit where upgrade happens, metadata won't be bootstrapped and with 2 -> 3, we will delete the metadata table. and on 2nd commit, will trigger the actual bootstrap of metadata table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-963928138
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
}, {
"hash" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220",
"triggerID" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"triggerType" : "PUSH"
}, {
"hash" : "efc802dc91182396b87ebba70baaa503b6955054",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efc802dc91182396b87ebba70baaa503b6955054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9a4f8faa55d8605c7222dbfae302815c58efbd06 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220)
* efc802dc91182396b87ebba70baaa503b6955054 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#discussion_r743981870
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -95,10 +95,19 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeC
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ bootstrapMetadataTable();
+ }
+
+ private void bootstrapMetadataTable() {
if (config.isMetadataTableEnabled()) {
- // If the metadata table does not exist, it should be bootstrapped here
- // TODO: Check if we can remove this requirement - auto bootstrap on commit
- SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config, context);
+ // Defer bootstrap if upgrade / downgrade is pending
+ HoodieTableMetaClient metaClient = createMetaClient(true);
+ UpgradeDowngrade upgradeDowngrade = new UpgradeDowngrade(
+ metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance());
+ if (!upgradeDowngrade.needsUpgradeOrDowngrade(HoodieTableVersion.current())) {
Review comment:
Yes, the bootstrap will happen the next time the SparkRDDWRiteCLient is created (probably in the next clean).
Currently this is what happens (assuming an existing Table with version 2 and using 0.10 code which has version 3):
1. SparkRDDWriteClient constructor - finds no table so bootstrap it (wasted bootstrap)
2. SparkRDDWriteClient.insert() - runs upgrade code in getTableAndXXX() and there the metadata table is deleted.
Next run:
1. SparkRDDWriteClient constructor - finds no table so bootstrap it (second bootstrap)
For file listing this wasted bootstrap is kinda ok but other indexes if enabled together (e.g. record-level-index enabled with metadata table), then this is a lot of wasted time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-962230403
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748)
* e916c158969b8b855cbffffde8bceac94e80b98d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-961588627
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-962472963
ping me here once you have addressed the feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-964257216
Thanks for the contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-947803091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-963930118
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
}, {
"hash" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220",
"triggerID" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"triggerType" : "PUSH"
}, {
"hash" : "efc802dc91182396b87ebba70baaa503b6955054",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3241",
"triggerID" : "efc802dc91182396b87ebba70baaa503b6955054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9a4f8faa55d8605c7222dbfae302815c58efbd06 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220)
* efc802dc91182396b87ebba70baaa503b6955054 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3241)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
prashantwason commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-963925818
@nsivabalan Addressed all the reviews. Ready for final look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-963930118
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
}, {
"hash" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220",
"triggerID" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"triggerType" : "PUSH"
}, {
"hash" : "efc802dc91182396b87ebba70baaa503b6955054",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3241",
"triggerID" : "efc802dc91182396b87ebba70baaa503b6955054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9a4f8faa55d8605c7222dbfae302815c58efbd06 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220)
* efc802dc91182396b87ebba70baaa503b6955054 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3241)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#discussion_r744011894
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -95,10 +95,19 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeC
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ bootstrapMetadataTable();
+ }
+
+ private void bootstrapMetadataTable() {
if (config.isMetadataTableEnabled()) {
- // If the metadata table does not exist, it should be bootstrapped here
- // TODO: Check if we can remove this requirement - auto bootstrap on commit
- SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config, context);
+ // Defer bootstrap if upgrade / downgrade is pending
+ HoodieTableMetaClient metaClient = createMetaClient(true);
+ UpgradeDowngrade upgradeDowngrade = new UpgradeDowngrade(
+ metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance());
+ if (!upgradeDowngrade.needsUpgradeOrDowngrade(HoodieTableVersion.current())) {
Review comment:
gotcha. wondering if we can move the instantiation of metadata table
```
SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config, context);
```
just after upgrade downgrade step within getTableAndInitCtx. (instead of in the constructor or SparkRDDWriteClient)
So, that we may not miss to bootstrap the table even if upgrade is required.
Here is what I am thinking. I would assume typically users might stop all processes and then do an upgrade for next hudi version. Once first commit goes through, users might want to get started w/ multi writers. so, would prefer to do the bootstrap on the first write operation where upgrade is triggered.
Open to hear your thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-947803091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-962231751
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748)
* e916c158969b8b855cbffffde8bceac94e80b98d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-962254215
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e916c158969b8b855cbffffde8bceac94e80b98d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#discussion_r744019234
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -95,10 +95,19 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeC
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ bootstrapMetadataTable();
+ }
+
+ private void bootstrapMetadataTable() {
if (config.isMetadataTableEnabled()) {
- // If the metadata table does not exist, it should be bootstrapped here
- // TODO: Check if we can remove this requirement - auto bootstrap on commit
- SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config, context);
+ // Defer bootstrap if upgrade / downgrade is pending
+ HoodieTableMetaClient metaClient = createMetaClient(true);
+ UpgradeDowngrade upgradeDowngrade = new UpgradeDowngrade(
+ metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance());
+ if (!upgradeDowngrade.needsUpgradeOrDowngrade(HoodieTableVersion.current())) {
Review comment:
I tried that step but it did not work because:
1. When getTableAndInitCtx is called an action is already started on the table
2. Metadata bootstrap does not happen because it detects an in-progress action
Bootstrap in the constructor is surely not ideal.
Possible ways:
1. Make the bootstrap aware of the "current" operation so it can neglect it. Then we can bootstrap right after upgrade/downgrade step (as you suggested).
2. Bootstrap automatically before write-to-metadata step.
I prefer #1 too as it is cleaner and metadata table will be available before any actions on the dataset start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-962254215
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e916c158969b8b855cbffffde8bceac94e80b98d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-963352122
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
}, {
"hash" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220",
"triggerID" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e916c158969b8b855cbffffde8bceac94e80b98d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170)
* 9a4f8faa55d8605c7222dbfae302815c58efbd06 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-963928138
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
}, {
"hash" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220",
"triggerID" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"triggerType" : "PUSH"
}, {
"hash" : "efc802dc91182396b87ebba70baaa503b6955054",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efc802dc91182396b87ebba70baaa503b6955054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9a4f8faa55d8605c7222dbfae302815c58efbd06 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220)
* efc802dc91182396b87ebba70baaa503b6955054 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-947803091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
prashantwason commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-951668308
@nsivabalan Description updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#discussion_r745263375
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -217,8 +222,10 @@ public HoodieBackedTableMetadata metadata() {
* Initialize the metadata table if it does not exist.
*
* If the metadata table did not exist, then file and partition listing is used to bootstrap the table.
+ * @param instantInProgressTimestamp Timestap of an instant in progress on the dataset. This instant is ignored
+ * while deciding to bootstrap the metadata table.
*/
- protected abstract void initialize(HoodieEngineContext engineContext);
+ protected abstract void initialize(HoodieEngineContext engineContext, Option<String> instantInProgressTimestamp);
Review comment:
minor: may be we can rename to inflightInstantTimestamp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-947803091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#discussion_r744903007
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -95,10 +95,19 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeC
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ bootstrapMetadataTable();
+ }
+
+ private void bootstrapMetadataTable() {
if (config.isMetadataTableEnabled()) {
- // If the metadata table does not exist, it should be bootstrapped here
- // TODO: Check if we can remove this requirement - auto bootstrap on commit
- SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config, context);
+ // Defer bootstrap if upgrade / downgrade is pending
+ HoodieTableMetaClient metaClient = createMetaClient(true);
+ UpgradeDowngrade upgradeDowngrade = new UpgradeDowngrade(
+ metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance());
+ if (!upgradeDowngrade.needsUpgradeOrDowngrade(HoodieTableVersion.current())) {
Review comment:
I have implemented #1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-963400355
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
}, {
"hash" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220",
"triggerID" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9a4f8faa55d8605c7222dbfae302815c58efbd06 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-947803091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#discussion_r745363631
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -217,8 +222,10 @@ public HoodieBackedTableMetadata metadata() {
* Initialize the metadata table if it does not exist.
*
* If the metadata table did not exist, then file and partition listing is used to bootstrap the table.
+ * @param instantInProgressTimestamp Timestap of an instant in progress on the dataset. This instant is ignored
+ * while deciding to bootstrap the metadata table.
*/
- protected abstract void initialize(HoodieEngineContext engineContext);
+ protected abstract void initialize(HoodieEngineContext engineContext, Option<String> instantInProgressTimestamp);
Review comment:
sure. renamed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-963349825
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
}, {
"hash" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e916c158969b8b855cbffffde8bceac94e80b98d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170)
* 9a4f8faa55d8605c7222dbfae302815c58efbd06 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-961588627
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3cb57760c2ba7769d9f9892fa73486c9c7ee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#discussion_r743670241
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -95,10 +95,19 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeC
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ bootstrapMetadataTable();
+ }
+
+ private void bootstrapMetadataTable() {
if (config.isMetadataTableEnabled()) {
- // If the metadata table does not exist, it should be bootstrapped here
- // TODO: Check if we can remove this requirement - auto bootstrap on commit
- SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config, context);
+ // Defer bootstrap if upgrade / downgrade is pending
+ HoodieTableMetaClient metaClient = createMetaClient(true);
+ UpgradeDowngrade upgradeDowngrade = new UpgradeDowngrade(
+ metaClient, config, context, SparkUpgradeDowngradeHelper.getInstance());
+ if (!upgradeDowngrade.needsUpgradeOrDowngrade(HoodieTableVersion.current())) {
Review comment:
@prashantwason : can you help me understand something. if upgrade is required, we defer the bootstrapping. So, when exactly the upgrade happens? in a subsequent commit? so IIUC, on first commit where upgrade happens, metadata won't be bootstrapped and with 2 -> 3, we will delete the metadata table. and on 2nd commit, will trigger the actual bootstrap of metadata table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-963971206
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2748",
"triggerID" : "2e3cb57760c2ba7769d9f9892fa73486c9c7ee74",
"triggerType" : "PUSH"
}, {
"hash" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3170",
"triggerID" : "e916c158969b8b855cbffffde8bceac94e80b98d",
"triggerType" : "PUSH"
}, {
"hash" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3220",
"triggerID" : "9a4f8faa55d8605c7222dbfae302815c58efbd06",
"triggerType" : "PUSH"
}, {
"hash" : "efc802dc91182396b87ebba70baaa503b6955054",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3241",
"triggerID" : "efc802dc91182396b87ebba70baaa503b6955054",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efc802dc91182396b87ebba70baaa503b6955054 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3241)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3836: [HUDI-2591] Bootstrap metadata table only if upgrade / downgrade is not required.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3836:
URL: https://github.com/apache/hudi/pull/3836#issuecomment-950954830
@prashantwason : Can you update the description with the purpose of the patch. It's not very apparent why we need this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org