You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Mulavar (via GitHub)" <gi...@apache.org> on 2023/04/05 07:48:54 UTC
[GitHub] [hudi] Mulavar opened a new pull request, #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Mulavar opened a new pull request, #8385:
URL: https://github.com/apache/hudi/pull/8385
### Change Logs
Stop writing and reading compaction plans from .aux folder
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
Reduce I/O when schedule compaction from 2 to 1.
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
None.
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
None
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1503083226
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8075b51a7422ab15ef1608e19cf1cc90fa930ff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158)
* 19ff36f7635f289b752b46cf692014b22f4b9ab8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1510607484
@bvaradar Yeah bvaradar, I agreed with you that we need to add some test cases to verify the upgrader/downgrader, but I'm not sure if there's a need to run subsequent compaction, cause we do not change anything about the compaction plan content, so maybe we should better verify if we can still read the compaction plan from .hoodie folder when after deleting compaction.requested from .aux folder. Since the upgrade verification has been covered by org.apache.hudi.common.table.timeline.TestHoodieActiveTimeline#testLoadingInstantsFromFiles and we could add some tests to verify SixToFiveDowngradeHandler. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1183240534
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.table.HoodieTable;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
+ * To ensure compatibility, we need recreate the compaction requested file to
+ * .aux folder.
+ */
+public class SixToFiveDowngradeHandler implements DowngradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> downgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // sync compaction requested file to .aux
+ HoodieTimeline compactionTimeline = new HoodieActiveTimeline(metaClient, false).filterPendingCompactionTimeline()
+ .filter(instant -> instant.getState() == HoodieInstant.State.REQUESTED);
Review Comment:
did we follow up on this. i.e. if we have a compaction instant that is inflight, does filterPendingCompactionTimeline() return both inflight and requested or just the inflight?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1164912220
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/FiveToSixUpgradeHandler.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieUpgradeDowngradeException;
+import org.apache.hudi.table.HoodieTable;
+
+import java.io.IOException;
+import java.util.Map;
+
+/**
+ * Upgrade handle to assist in upgrading hoodie table from version 5 to 6.
+ */
+public class FiveToSixUpgradeHandler implements UpgradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> upgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // delete compaction requested file from .aux
+ HoodieTimeline compactionTimeline = metaClient.getActiveTimeline().filterPendingCompactionTimeline()
+ .filter(instant -> instant.getState() == HoodieInstant.State.REQUESTED);
+ compactionTimeline.getInstants().stream().forEach(instant -> {
+ String fileName = instant.getFileName();
+ try {
+ metaClient.getFs().delete(new Path(metaClient.getMetaAuxiliaryPath(), fileName));
Review Comment:
I found that some other feature like bootstrap index also use the .aux folder, refer to org.apache.hudi.common.table.HoodieTableMetaClient#BOOTSTRAP_INDEX_ROOT_FOLDER_PATH and we can not just remove the creation of .aux folder. Maybe we should remove the caller of metaClient.getMetaAuxiliaryPath() related to compaction logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1506281377
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 768ffaabf5934199e1afa1c0b6b37f9bb665b989 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276)
* 6fd073a2061145c3800023590c50f55837a59171 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1497293239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 980b3f30e07c909b04078b86ed4dc793a9fcce19 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1515758278
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476",
"triggerID" : "1515660755",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* bfe8f905ffd99f983173a653481915c34a0c4fbe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1512343133
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1175955522
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/FiveToSixUpgradeHandler.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieUpgradeDowngradeException;
+import org.apache.hudi.table.HoodieTable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Upgrade handle to assist in upgrading hoodie table from version 5 to 6.
+ */
+public class FiveToSixUpgradeHandler implements UpgradeHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FiveToSixUpgradeHandler.class);
+
+ @Override
+ public Map<ConfigProperty, String> upgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // delete compaction file from .aux
+ HoodieTimeline compactionTimeline = metaClient.getActiveTimeline().filterPendingCompactionTimeline()
+ .filter(instant -> instant.getState() == HoodieInstant.State.REQUESTED);
Review Comment:
"I found that some other features like bootstrap index also use the .aux folder, refer to org.apache.hudi.common.table.HoodieTableMetaClient#BOOTSTRAP_INDEX_ROOT_FOLDER_PATH and we can not just remove the creation of .aux folder. Maybe we should remove the caller of metaClient.getMetaAuxiliaryPath() related to compaction logic?"
This patch only cares about the compaction, and there is some other file/folder in the .aux folder, we should only remove the files about compaction. If we want to confirm if the entire .aux folder could be deleted, maybe better if we should open another issue to discuss it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1521084798
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476",
"triggerID" : "1515660755",
"triggerType" : "MANUAL"
}, {
"hash" : "9659d420b2386c90428afc88954adecd70afa7e6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16586",
"triggerID" : "9659d420b2386c90428afc88954adecd70afa7e6",
"triggerType" : "PUSH"
}, {
"hash" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 9659d420b2386c90428afc88954adecd70afa7e6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16586)
* 8f14d7af57b04b77c42eea20d7e2ef9810d23210 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1167981665
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngrade.java:
##########
@@ -87,7 +87,10 @@ public boolean needsUpgradeOrDowngrade(HoodieTableVersion toVersion) {
* pre 0.6.0 -> v0
* 0.6.0 to 0.8.0 -> v1
* 0.9.0 -> v2
- * 0.10.0 to current -> v3
+ * 0.10.0 -> v3
Review Comment:
thank for updating this.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/FiveToSixUpgradeHandler.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieUpgradeDowngradeException;
+import org.apache.hudi.table.HoodieTable;
+
+import java.io.IOException;
+import java.util.Map;
+
+/**
+ * Upgrade handle to assist in upgrading hoodie table from version 5 to 6.
+ */
+public class FiveToSixUpgradeHandler implements UpgradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> upgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // delete compaction requested file from .aux
+ HoodieTimeline compactionTimeline = metaClient.getActiveTimeline().filterPendingCompactionTimeline()
+ .filter(instant -> instant.getState() == HoodieInstant.State.REQUESTED);
+ compactionTimeline.getInstants().stream().forEach(instant -> {
+ String fileName = instant.getFileName();
+ try {
+ metaClient.getFs().delete(new Path(metaClient.getMetaAuxiliaryPath(), fileName));
Review Comment:
Yes, Good point. We should only cleanup compaction plans in .aux folder and remove compaction only callers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1503144145
@bvaradar hi bvaradar, could u take some time to review the pr again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1510616319
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 6fd073a2061145c3800023590c50f55837a59171 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299)
* 1cd0db680780d02ff786121f394dccfcd621d37d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1519103009
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476",
"triggerID" : "1515660755",
"triggerType" : "MANUAL"
}, {
"hash" : "9659d420b2386c90428afc88954adecd70afa7e6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16586",
"triggerID" : "9659d420b2386c90428afc88954adecd70afa7e6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* bfe8f905ffd99f983173a653481915c34a0c4fbe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476)
* 9659d420b2386c90428afc88954adecd70afa7e6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16586)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1519134912
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476",
"triggerID" : "1515660755",
"triggerType" : "MANUAL"
}, {
"hash" : "9659d420b2386c90428afc88954adecd70afa7e6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16586",
"triggerID" : "9659d420b2386c90428afc88954adecd70afa7e6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 9659d420b2386c90428afc88954adecd70afa7e6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16586)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1521276587
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476",
"triggerID" : "1515660755",
"triggerType" : "MANUAL"
}, {
"hash" : "9659d420b2386c90428afc88954adecd70afa7e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16586",
"triggerID" : "9659d420b2386c90428afc88954adecd70afa7e6",
"triggerType" : "PUSH"
}, {
"hash" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16623",
"triggerID" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"triggerType" : "PUSH"
}, {
"hash" : "83a903bc42bed17039281352a762d61c60b56ab1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16624",
"triggerID" : "83a903bc42bed17039281352a762d61c60b56ab1",
"triggerType" : "PUSH"
}, {
"hash" : "0f1555e44261ded36b9f4b930f973ebea6625cfa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16636",
"triggerID" : "0f1555e44261ded36b9f4b930f973ebea6625cfa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 83a903bc42bed17039281352a762d61c60b56ab1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16624)
* 0f1555e44261ded36b9f4b930f973ebea6625cfa Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16636)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1498461775
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 980b3f30e07c909b04078b86ed4dc793a9fcce19 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136)
* 2f99dbd9ffb42d979dfa753aef509f90ec557e66 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155)
* d8075b51a7422ab15ef1608e19cf1cc90fa930ff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1498466346
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f99dbd9ffb42d979dfa753aef509f90ec557e66 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155)
* d8075b51a7422ab15ef1608e19cf1cc90fa930ff Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1175940922
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/FiveToSixUpgradeHandler.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieUpgradeDowngradeException;
+import org.apache.hudi.table.HoodieTable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Upgrade handle to assist in upgrading hoodie table from version 5 to 6.
Review Comment:
can we add java docs to call out whats being done as part of upgrade from 5 to 6 ?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/FiveToSixUpgradeHandler.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieUpgradeDowngradeException;
+import org.apache.hudi.table.HoodieTable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Upgrade handle to assist in upgrading hoodie table from version 5 to 6.
+ */
+public class FiveToSixUpgradeHandler implements UpgradeHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FiveToSixUpgradeHandler.class);
+
+ @Override
+ public Map<ConfigProperty, String> upgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // delete compaction file from .aux
+ HoodieTimeline compactionTimeline = metaClient.getActiveTimeline().filterPendingCompactionTimeline()
+ .filter(instant -> instant.getState() == HoodieInstant.State.REQUESTED);
Review Comment:
I checked the usages of aux folder and its used only incase of compaction plans and may be flink clustering.
why can't we delete entire aux folder ?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.table.HoodieTable;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
+ */
+public class SixToFiveDowngradeHandler implements DowngradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> downgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // sync compaction requested file to .aux
+ HoodieTimeline compactionTimeline = metaClient.getActiveTimeline().filterPendingCompactionTimeline()
Review Comment:
this could return instants either in inflight or in requested. of its inflight, we need to fetch the filename by constructing the requested instant as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1176018366
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.table.HoodieTable;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
+ */
+public class SixToFiveDowngradeHandler implements DowngradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> downgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // sync compaction requested file to .aux
+ HoodieTimeline compactionTimeline = metaClient.getActiveTimeline().filterPendingCompactionTimeline()
Review Comment:
what I meant is, for a compaction that has both requested and inflight in ".hoodie", only INFLIGHT will be returned with this call (metaClient.getActiveTimeline().filterPendingCompactionTimeline()) and in subsequent like you are filtering only for requested and hence we might miss.
can you check please.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/FiveToSixUpgradeHandler.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieUpgradeDowngradeException;
+import org.apache.hudi.table.HoodieTable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Upgrade handle to assist in upgrading hoodie table from version 5 to 6.
+ */
+public class FiveToSixUpgradeHandler implements UpgradeHandler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FiveToSixUpgradeHandler.class);
+
+ @Override
+ public Map<ConfigProperty, String> upgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // delete compaction file from .aux
+ HoodieTimeline compactionTimeline = metaClient.getActiveTimeline().filterPendingCompactionTimeline()
+ .filter(instant -> instant.getState() == HoodieInstant.State.REQUESTED);
Review Comment:
sg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1504942140
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 768ffaabf5934199e1afa1c0b6b37f9bb665b989 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1512354643
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 1cd0db680780d02ff786121f394dccfcd621d37d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1510695513
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 1cd0db680780d02ff786121f394dccfcd621d37d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1504537773
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19ff36f7635f289b752b46cf692014b22f4b9ab8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254)
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 768ffaabf5934199e1afa1c0b6b37f9bb665b989 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1504603719
> @Mulavar : This requires a table version change also as we need to create .aux files when we downgrade to older version. Can you add relevant UpgradeDowngrade handlers for this.
@bvaradar thanks, done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1502746067
@bvaradar Thanks for your review. I think your suggestion is very valuable, and I need to read that piece of code first because I'm not very familiar with the code of table version downgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1164912220
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/FiveToSixUpgradeHandler.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieUpgradeDowngradeException;
+import org.apache.hudi.table.HoodieTable;
+
+import java.io.IOException;
+import java.util.Map;
+
+/**
+ * Upgrade handle to assist in upgrading hoodie table from version 5 to 6.
+ */
+public class FiveToSixUpgradeHandler implements UpgradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> upgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // delete compaction requested file from .aux
+ HoodieTimeline compactionTimeline = metaClient.getActiveTimeline().filterPendingCompactionTimeline()
+ .filter(instant -> instant.getState() == HoodieInstant.State.REQUESTED);
+ compactionTimeline.getInstants().stream().forEach(instant -> {
+ String fileName = instant.getFileName();
+ try {
+ metaClient.getFs().delete(new Path(metaClient.getMetaAuxiliaryPath(), fileName));
Review Comment:
I found that some other feature like bootstrap index also use the .aux directory, refer to org.apache.hudi.common.table.HoodieTableMetaClient#BOOTSTRAP_INDEX_ROOT_FOLDER_PATH. Maybe we should remove the caller of metaClient.getMetaAuxiliaryPath() related to compaction logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1515660755
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1521057380
we should also https://issues.apache.org/jira/browse/HUDI-546 along with 6040.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1185903815
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.table.HoodieTable;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
+ * To ensure compatibility, we need recreate the compaction requested file to
+ * .aux folder.
+ */
+public class SixToFiveDowngradeHandler implements DowngradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> downgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // sync compaction requested file to .aux
+ HoodieTimeline compactionTimeline = new HoodieActiveTimeline(metaClient, false).filterPendingCompactionTimeline()
+ .filter(instant -> instant.getState() == HoodieInstant.State.REQUESTED);
Review Comment:
@nsivabalan Hi, do you have any other questions about the discussion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1164549351
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/FiveToSixUpgradeHandler.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieUpgradeDowngradeException;
+import org.apache.hudi.table.HoodieTable;
+
+import java.io.IOException;
+import java.util.Map;
+
+/**
+ * Upgrade handle to assist in upgrading hoodie table from version 5 to 6.
+ */
+public class FiveToSixUpgradeHandler implements UpgradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> upgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // delete compaction requested file from .aux
+ HoodieTimeline compactionTimeline = metaClient.getActiveTimeline().filterPendingCompactionTimeline()
Review Comment:
As compaction plan is the only file type present in .aux folder, we should cleanup the all the compaction files there. You can use the following code found in HoodieTimelineArchiver.deleteAllInstantsOlderOrEqualsInAuxMetaFolder to collect all the instants to be deleted.
```
instantsToBeDeleted =
metaClient.scanHoodieInstantsFromFileSystem(
new Path(metaClient.getMetaAuxiliaryPath()),
HoodieActiveTimeline.VALID_EXTENSIONS_IN_ACTIVE_TIMELINE,
false);
for (HoodieInstant deleteInstant : instantsToBeDeleted) {
LOG.info("Deleting instant " + deleteInstant + " in auxiliary meta path " + metaClient.getMetaAuxiliaryPath());
Path metaFile = new Path(metaClient.getMetaAuxiliaryPath(), deleteInstant.getFileName());
if (metaClient.getFs().exists(metaFile)) {
success &= metaClient.getFs().delete(metaFile, false);
LOG.info("Deleted instant file in auxiliary meta path : " + metaFile);
}
}
```
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/FiveToSixUpgradeHandler.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieUpgradeDowngradeException;
+import org.apache.hudi.table.HoodieTable;
+
+import java.io.IOException;
+import java.util.Map;
+
+/**
+ * Upgrade handle to assist in upgrading hoodie table from version 5 to 6.
+ */
+public class FiveToSixUpgradeHandler implements UpgradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> upgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // delete compaction requested file from .aux
+ HoodieTimeline compactionTimeline = metaClient.getActiveTimeline().filterPendingCompactionTimeline()
+ .filter(instant -> instant.getState() == HoodieInstant.State.REQUESTED);
+ compactionTimeline.getInstants().stream().forEach(instant -> {
+ String fileName = instant.getFileName();
+ try {
+ metaClient.getFs().delete(new Path(metaClient.getMetaAuxiliaryPath(), fileName));
Review Comment:
@Mulavar : We need to basically look at all caller of metaClient.getMetaAuxiliaryPath() and remove the usage of metaAuxiliaryPath since we should not no longer write to them. Also in HoodieTableMetaClient.initTableAndGetMetaClient, can you remove the creation of .aux folder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1498421815
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 980b3f30e07c909b04078b86ed4dc793a9fcce19 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136)
* 2f99dbd9ffb42d979dfa753aef509f90ec557e66 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1506286678
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 768ffaabf5934199e1afa1c0b6b37f9bb665b989 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276)
* 6fd073a2061145c3800023590c50f55837a59171 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1521159303
> Should we completely remove all usages of "aux" folder with this patch? <img alt="image" width="669" src="https://user-images.githubusercontent.com/513218/234157269-4e18e48c-2060-41af-8991-1ed9d15c5592.png">
>
> I don't see any other usages after this patch.
>
> Also, @bvaradar : can you help me understand what will happen in this case. if a compaction plan was written to aux but failed to add to ".hoodie" just before upgrade. after upgrade, if we delete the entire "aux" folder, we should be good right. there won't be any other traces of that failed compaction schedule ?
@nsivabalan : It is still used for bootstrap. Bootstrap folder is under aux
Regarding the partial write case, it should be fine since we rely on timeline (.hoodie/) to figure out pending compactions. In both upgrade and normal case, this would be as if no compaction is pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1521980266
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476",
"triggerID" : "1515660755",
"triggerType" : "MANUAL"
}, {
"hash" : "9659d420b2386c90428afc88954adecd70afa7e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16586",
"triggerID" : "9659d420b2386c90428afc88954adecd70afa7e6",
"triggerType" : "PUSH"
}, {
"hash" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16623",
"triggerID" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"triggerType" : "PUSH"
}, {
"hash" : "83a903bc42bed17039281352a762d61c60b56ab1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16624",
"triggerID" : "83a903bc42bed17039281352a762d61c60b56ab1",
"triggerType" : "PUSH"
}, {
"hash" : "0f1555e44261ded36b9f4b930f973ebea6625cfa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16636",
"triggerID" : "0f1555e44261ded36b9f4b930f973ebea6625cfa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 0f1555e44261ded36b9f4b930f973ebea6625cfa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16636)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1174716920
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/compact/ITTestHoodieFlinkCompactor.java:
##########
@@ -169,6 +172,85 @@ public void testHoodieFlinkCompactor(boolean enableChangelog) throws Exception {
TestData.checkWrittenDataCOW(tempFile, EXPECTED1);
}
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testHoodieFlinkCompactorWithUpgradeAndDowngrade(boolean upgrade) throws Exception {
+ // Create hoodie table and insert into data.
+ EnvironmentSettings settings = EnvironmentSettings.newInstance().inBatchMode().build();
+ TableEnvironment tableEnv = TableEnvironmentImpl.create(settings);
+ tableEnv.getConfig().getConfiguration()
+ .setInteger(ExecutionConfigOptions.TABLE_EXEC_RESOURCE_DEFAULT_PARALLELISM, 4);
+ Map<String, String> options = new HashMap<>();
+ options.put(FlinkOptions.COMPACTION_SCHEDULE_ENABLED.key(), "false");
+ options.put(FlinkOptions.COMPACTION_ASYNC_ENABLED.key(), "false");
+ options.put(FlinkOptions.PATH.key(), tempFile.getAbsolutePath());
+ options.put(FlinkOptions.TABLE_TYPE.key(), "MERGE_ON_READ");
+ String hoodieTableDDL = TestConfigurations.getCreateHoodieTableDDL("t1", options);
+ tableEnv.executeSql(hoodieTableDDL);
+ tableEnv.executeSql(TestSQL.INSERT_T1).await();
+
+ // wait for the asynchronous commit to finish
+ TimeUnit.SECONDS.sleep(3);
+
+ // Make configuration and setAvroSchema.
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ FlinkCompactionConfig cfg = new FlinkCompactionConfig();
+ cfg.path = tempFile.getAbsolutePath();
+ Configuration conf = FlinkCompactionConfig.toFlinkConfig(cfg);
+ conf.setString(FlinkOptions.TABLE_TYPE.key(), "MERGE_ON_READ");
+
+ // create metaClient
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(conf);
+
+ // set the table name
+ conf.setString(FlinkOptions.TABLE_NAME, metaClient.getTableConfig().getTableName());
+
+ // set table schema
+ CompactionUtil.setAvroSchema(conf, metaClient);
+
+ // infer changelog mode
+ CompactionUtil.inferChangelogMode(conf, metaClient);
+
+ HoodieFlinkWriteClient writeClient = FlinkWriteClients.createWriteClient(conf);
+
+ String compactionInstantTime = scheduleCompactionPlan(metaClient, writeClient);
+
+ HoodieFlinkTable<?> table = writeClient.getHoodieTable();
+
+ // try to upgrade or downgrade
+ if (upgrade) {
+ new FiveToSixUpgradeHandler().upgrade(writeClient.getConfig(), writeClient.getEngineContext(), "none", FlinkUpgradeDowngradeHelper.getInstance());
+ } else {
+ new SixToFiveDowngradeHandler().downgrade(writeClient.getConfig(), writeClient.getEngineContext(), "none", FlinkUpgradeDowngradeHelper.getInstance());
+ }
Review Comment:
@bvaradar Hi, I've added test about compaction after upgrading/downgrading operation, could u take a look at thie pr again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1519101523
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476",
"triggerID" : "1515660755",
"triggerType" : "MANUAL"
}, {
"hash" : "9659d420b2386c90428afc88954adecd70afa7e6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9659d420b2386c90428afc88954adecd70afa7e6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* bfe8f905ffd99f983173a653481915c34a0c4fbe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476)
* 9659d420b2386c90428afc88954adecd70afa7e6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1515639862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 1cd0db680780d02ff786121f394dccfcd621d37d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409)
* bfe8f905ffd99f983173a653481915c34a0c4fbe UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1512468237
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 1cd0db680780d02ff786121f394dccfcd621d37d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1176061911
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.table.HoodieTable;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
+ */
+public class SixToFiveDowngradeHandler implements DowngradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> downgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // sync compaction requested file to .aux
+ HoodieTimeline compactionTimeline = metaClient.getActiveTimeline().filterPendingCompactionTimeline()
Review Comment:
Thanks, nice catch! fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1521169606
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476",
"triggerID" : "1515660755",
"triggerType" : "MANUAL"
}, {
"hash" : "9659d420b2386c90428afc88954adecd70afa7e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16586",
"triggerID" : "9659d420b2386c90428afc88954adecd70afa7e6",
"triggerType" : "PUSH"
}, {
"hash" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16623",
"triggerID" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"triggerType" : "PUSH"
}, {
"hash" : "83a903bc42bed17039281352a762d61c60b56ab1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16624",
"triggerID" : "83a903bc42bed17039281352a762d61c60b56ab1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 83a903bc42bed17039281352a762d61c60b56ab1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16624)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1183246091
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.table.HoodieTable;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
+ * To ensure compatibility, we need recreate the compaction requested file to
+ * .aux folder.
+ */
+public class SixToFiveDowngradeHandler implements DowngradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> downgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // sync compaction requested file to .aux
+ HoodieTimeline compactionTimeline = new HoodieActiveTimeline(metaClient, false).filterPendingCompactionTimeline()
+ .filter(instant -> instant.getState() == HoodieInstant.State.REQUESTED);
Review Comment:
We have fixed this problem by setting `applyLayoutFilters` to false when creating the HoodieActiveTimeline, filterPendingCompactionTimeline() would return both inflight and requested instants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1498398896
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 980b3f30e07c909b04078b86ed4dc793a9fcce19 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136)
* 2f99dbd9ffb42d979dfa753aef509f90ec557e66 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1498718043
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8075b51a7422ab15ef1608e19cf1cc90fa930ff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1497089764
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 980b3f30e07c909b04078b86ed4dc793a9fcce19 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1497079666
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 980b3f30e07c909b04078b86ed4dc793a9fcce19 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1175661779
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/compact/ITTestHoodieFlinkCompactor.java:
##########
@@ -169,6 +172,85 @@ public void testHoodieFlinkCompactor(boolean enableChangelog) throws Exception {
TestData.checkWrittenDataCOW(tempFile, EXPECTED1);
}
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testHoodieFlinkCompactorWithUpgradeAndDowngrade(boolean upgrade) throws Exception {
+ // Create hoodie table and insert into data.
+ EnvironmentSettings settings = EnvironmentSettings.newInstance().inBatchMode().build();
+ TableEnvironment tableEnv = TableEnvironmentImpl.create(settings);
+ tableEnv.getConfig().getConfiguration()
+ .setInteger(ExecutionConfigOptions.TABLE_EXEC_RESOURCE_DEFAULT_PARALLELISM, 4);
+ Map<String, String> options = new HashMap<>();
+ options.put(FlinkOptions.COMPACTION_SCHEDULE_ENABLED.key(), "false");
+ options.put(FlinkOptions.COMPACTION_ASYNC_ENABLED.key(), "false");
+ options.put(FlinkOptions.PATH.key(), tempFile.getAbsolutePath());
+ options.put(FlinkOptions.TABLE_TYPE.key(), "MERGE_ON_READ");
+ String hoodieTableDDL = TestConfigurations.getCreateHoodieTableDDL("t1", options);
+ tableEnv.executeSql(hoodieTableDDL);
+ tableEnv.executeSql(TestSQL.INSERT_T1).await();
+
+ // wait for the asynchronous commit to finish
+ TimeUnit.SECONDS.sleep(3);
+
+ // Make configuration and setAvroSchema.
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ FlinkCompactionConfig cfg = new FlinkCompactionConfig();
+ cfg.path = tempFile.getAbsolutePath();
+ Configuration conf = FlinkCompactionConfig.toFlinkConfig(cfg);
+ conf.setString(FlinkOptions.TABLE_TYPE.key(), "MERGE_ON_READ");
+
+ // create metaClient
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(conf);
+
+ // set the table name
+ conf.setString(FlinkOptions.TABLE_NAME, metaClient.getTableConfig().getTableName());
+
+ // set table schema
+ CompactionUtil.setAvroSchema(conf, metaClient);
+
+ // infer changelog mode
+ CompactionUtil.inferChangelogMode(conf, metaClient);
+
+ HoodieFlinkWriteClient writeClient = FlinkWriteClients.createWriteClient(conf);
+
+ String compactionInstantTime = scheduleCompactionPlan(metaClient, writeClient);
+
+ HoodieFlinkTable<?> table = writeClient.getHoodieTable();
+
+ // try to upgrade or downgrade
+ if (upgrade) {
+ new FiveToSixUpgradeHandler().upgrade(writeClient.getConfig(), writeClient.getEngineContext(), "none", FlinkUpgradeDowngradeHelper.getInstance());
Review Comment:
Thanks @Mulavar for adding the testcase. Instead of directly calling FiveToSix and SixToFive upgrade/downgrade, can you call UpgradeDowngrade.run(). It will make the test generic and not specific to the version. We can make the test's goal be to ensure compaction succeeds after upgrading and downgrading. Let me know your thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1175942283
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.table.HoodieTable;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
+ */
+public class SixToFiveDowngradeHandler implements DowngradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> downgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // sync compaction requested file to .aux
+ HoodieTimeline compactionTimeline = metaClient.getActiveTimeline().filterPendingCompactionTimeline()
Review Comment:
this could return instants either in inflight or in requested. if its inflight, we need to fetch the filename by constructing the requested instant as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1521230954
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476",
"triggerID" : "1515660755",
"triggerType" : "MANUAL"
}, {
"hash" : "9659d420b2386c90428afc88954adecd70afa7e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16586",
"triggerID" : "9659d420b2386c90428afc88954adecd70afa7e6",
"triggerType" : "PUSH"
}, {
"hash" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16623",
"triggerID" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"triggerType" : "PUSH"
}, {
"hash" : "83a903bc42bed17039281352a762d61c60b56ab1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16624",
"triggerID" : "83a903bc42bed17039281352a762d61c60b56ab1",
"triggerType" : "PUSH"
}, {
"hash" : "0f1555e44261ded36b9f4b930f973ebea6625cfa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0f1555e44261ded36b9f4b930f973ebea6625cfa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 83a903bc42bed17039281352a762d61c60b56ab1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16624)
* 0f1555e44261ded36b9f4b930f973ebea6625cfa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1504389808
@Mulavar : We need to create a higher version 6 and write upgrade/downgrade to handle transition from current version (5). You can look at https://github.com/apache/hudi/pull/6248 as example of how to do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1506365981
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 6fd073a2061145c3800023590c50f55837a59171 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1504528832
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19ff36f7635f289b752b46cf692014b22f4b9ab8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254)
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1504582669
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19ff36f7635f289b752b46cf692014b22f4b9ab8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254)
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 768ffaabf5934199e1afa1c0b6b37f9bb665b989 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1515659350
@bvaradar Yeah, but I'm not very familiar with the compaction test, could you give me some tips on which example should I look at?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1515676161
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476",
"triggerID" : "1515660755",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* bfe8f905ffd99f983173a653481915c34a0c4fbe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1183246091
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.table.HoodieTable;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
+ * To ensure compatibility, we need recreate the compaction requested file to
+ * .aux folder.
+ */
+public class SixToFiveDowngradeHandler implements DowngradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> downgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // sync compaction requested file to .aux
+ HoodieTimeline compactionTimeline = new HoodieActiveTimeline(metaClient, false).filterPendingCompactionTimeline()
+ .filter(instant -> instant.getState() == HoodieInstant.State.REQUESTED);
Review Comment:
> Thanks, nice catch! fixed.
We have fixed this problem by setting `applyLayoutFilters` to false, filterPendingCompactionTimeline() would return both inflight and requested instants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1527181265
@nsivabalan : Are you ok with this change ? If so, can you approve it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1510649769
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 6fd073a2061145c3800023590c50f55837a59171 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299)
* 1cd0db680780d02ff786121f394dccfcd621d37d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1515651124
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* bfe8f905ffd99f983173a653481915c34a0c4fbe Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1515644714
@Mulavar : W.r.t running compaction after upgrade/downgrade, this will help prove that the upgrade will be safer if we can ensure compaction did not fail in any place after upgrade/downgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1499876221
@danny0405 Hi danny, could u take the time to review this pr, Is there anything I need to modify?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1159889774
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java:
##########
@@ -704,8 +687,6 @@ public void saveToCompactionRequested(HoodieInstant instant, Option<byte[]> cont
public void saveToCompactionRequested(HoodieInstant instant, Option<byte[]> content, boolean overwrite) {
ValidationUtils.checkArgument(instant.getAction().equals(HoodieTimeline.COMPACTION_ACTION));
- // Write workload to auxiliary folder
- createFileInAuxiliaryFolder(instant, content);
Review Comment:
cc @bvaradar for taking over this, generally looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1503071558
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8075b51a7422ab15ef1608e19cf1cc90fa930ff Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158)
* 19ff36f7635f289b752b46cf692014b22f4b9ab8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1503455188
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19ff36f7635f289b752b46cf692014b22f4b9ab8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1521161987
> It is still used for bootstrap. Bootstrap folder is under aux
> Regarding the partial write case, it should be fine since we rely on timeline (.hoodie/) to figure out pending compactions. In both upgrade and normal case, this would be as if no compaction is pending.
Flink actually uses the .aux folder for some configurations like 1. fs view storage conf and 2. checkpoint metadata, so we may need to keep this folder, just drop the file creation for compaction should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on a diff in pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on code in PR #8385:
URL: https://github.com/apache/hudi/pull/8385#discussion_r1175951603
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.upgrade;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.FileIOUtils;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.table.HoodieTable;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
+ */
+public class SixToFiveDowngradeHandler implements DowngradeHandler {
+
+ @Override
+ public Map<ConfigProperty, String> downgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ // sync compaction requested file to .aux
+ HoodieTimeline compactionTimeline = metaClient.getActiveTimeline().filterPendingCompactionTimeline()
Review Comment:
the code snippet has been followed by the filter which does the thing you said.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1521092134
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476",
"triggerID" : "1515660755",
"triggerType" : "MANUAL"
}, {
"hash" : "9659d420b2386c90428afc88954adecd70afa7e6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16586",
"triggerID" : "9659d420b2386c90428afc88954adecd70afa7e6",
"triggerType" : "PUSH"
}, {
"hash" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16623",
"triggerID" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"triggerType" : "PUSH"
}, {
"hash" : "83a903bc42bed17039281352a762d61c60b56ab1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "83a903bc42bed17039281352a762d61c60b56ab1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 9659d420b2386c90428afc88954adecd70afa7e6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16586)
* 8f14d7af57b04b77c42eea20d7e2ef9810d23210 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16623)
* 83a903bc42bed17039281352a762d61c60b56ab1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1521100735
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16136",
"triggerID" : "980b3f30e07c909b04078b86ed4dc793a9fcce19",
"triggerType" : "PUSH"
}, {
"hash" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16155",
"triggerID" : "2f99dbd9ffb42d979dfa753aef509f90ec557e66",
"triggerType" : "PUSH"
}, {
"hash" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16158",
"triggerID" : "d8075b51a7422ab15ef1608e19cf1cc90fa930ff",
"triggerType" : "PUSH"
}, {
"hash" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16254",
"triggerID" : "19ff36f7635f289b752b46cf692014b22f4b9ab8",
"triggerType" : "PUSH"
}, {
"hash" : "3874447e48c21cb336f28625e1682b8f229f623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3874447e48c21cb336f28625e1682b8f229f623c",
"triggerType" : "PUSH"
}, {
"hash" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16276",
"triggerID" : "768ffaabf5934199e1afa1c0b6b37f9bb665b989",
"triggerType" : "PUSH"
}, {
"hash" : "6fd073a2061145c3800023590c50f55837a59171",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16299",
"triggerID" : "6fd073a2061145c3800023590c50f55837a59171",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16378",
"triggerID" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"triggerType" : "PUSH"
}, {
"hash" : "1cd0db680780d02ff786121f394dccfcd621d37d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16409",
"triggerID" : "1512343133",
"triggerType" : "MANUAL"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16473",
"triggerID" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"triggerType" : "PUSH"
}, {
"hash" : "bfe8f905ffd99f983173a653481915c34a0c4fbe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16476",
"triggerID" : "1515660755",
"triggerType" : "MANUAL"
}, {
"hash" : "9659d420b2386c90428afc88954adecd70afa7e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16586",
"triggerID" : "9659d420b2386c90428afc88954adecd70afa7e6",
"triggerType" : "PUSH"
}, {
"hash" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16623",
"triggerID" : "8f14d7af57b04b77c42eea20d7e2ef9810d23210",
"triggerType" : "PUSH"
}, {
"hash" : "83a903bc42bed17039281352a762d61c60b56ab1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16624",
"triggerID" : "83a903bc42bed17039281352a762d61c60b56ab1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3874447e48c21cb336f28625e1682b8f229f623c UNKNOWN
* 8f14d7af57b04b77c42eea20d7e2ef9810d23210 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16623)
* 83a903bc42bed17039281352a762d61c60b56ab1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16624)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Mulavar commented on pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "Mulavar (via GitHub)" <gi...@apache.org>.
Mulavar commented on PR #8385:
URL: https://github.com/apache/hudi/pull/8385#issuecomment-1521788645
@nsivabalan Hi nsivabalan, I've fixed the comment, could u check this patch again~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #8385: [HUDI-6040]Stop writing and reading compaction plans from .aux folder
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #8385:
URL: https://github.com/apache/hudi/pull/8385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org