You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2023/03/01 01:46:45 UTC
[GitHub] [hudi] yihua opened a new pull request, #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
yihua opened a new pull request, #8079:
URL: https://github.com/apache/hudi/pull/8079
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1454556422
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
}, {
"hash" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564",
"triggerID" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"triggerType" : "PUSH"
}, {
"hash" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15571",
"triggerID" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* 9c8cfc00357c28e105e3fdeb33e3fbee3f384103 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564)
* c162956f9f418b4603328c37f9e2babf59613d4b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15571)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1454712975
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
}, {
"hash" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564",
"triggerID" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"triggerType" : "PUSH"
}, {
"hash" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15571",
"triggerID" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"triggerType" : "PUSH"
}, {
"hash" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15572",
"triggerID" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"triggerType" : "PUSH"
}, {
"hash" : "d660deb903eed17560554a0145464598089fb3ea",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15574",
"triggerID" : "d660deb903eed17560554a0145464598089fb3ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* d660deb903eed17560554a0145464598089fb3ea Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15574)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1454564278
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
}, {
"hash" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564",
"triggerID" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"triggerType" : "PUSH"
}, {
"hash" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15571",
"triggerID" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"triggerType" : "PUSH"
}, {
"hash" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* 9c8cfc00357c28e105e3fdeb33e3fbee3f384103 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564)
* c162956f9f418b4603328c37f9e2babf59613d4b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15571)
* 7fff406e74cdf3faf047634a2d596399fa49f059 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1125389847
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -151,30 +156,38 @@ public void stop() {
* Determines if local view of table's timeline is behind that of client's view.
*/
private boolean isLocalViewBehind(Context ctx) {
- String basePath = ctx.queryParam(RemoteHoodieTableFileSystemView.BASEPATH_PARAM);
- String lastKnownInstantFromClient =
- ctx.queryParamAsClass(RemoteHoodieTableFileSystemView.LAST_INSTANT_TS, String.class).getOrDefault(HoodieTimeline.INVALID_INSTANT_TS);
- String timelineHashFromClient = ctx.queryParamAsClass(RemoteHoodieTableFileSystemView.TIMELINE_HASH, String.class).getOrDefault("");
- HoodieTimeline localTimeline =
- viewManager.getFileSystemView(basePath).getTimeline().filterCompletedOrMajorOrMinorCompactionInstants();
- if (LOG.isDebugEnabled()) {
- LOG.debug("Client [ LastTs=" + lastKnownInstantFromClient + ", TimelineHash=" + timelineHashFromClient
- + "], localTimeline=" + localTimeline.getInstants());
- }
+ try {
+ // This read lock makes sure that if the local view of the table is being synced,
+ // no timeline server requests should be processed or handled until the sync process
Review Comment:
@danny0405 This is simplified now. You can also check my updated PR description for how the race condition can happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1125390087
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -78,6 +79,10 @@ public class RequestHandler {
private final BaseFileHandler dataFileHandler;
private final MarkerHandler markerHandler;
private final Registry metricsRegistry = Registry.getRegistry("TimelineService");
+ // This read-write lock is used for syncing the file system view if it is behind client's view
Review Comment:
This read-write lock is removed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1454499436
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
}, {
"hash" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564",
"triggerID" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* 9c8cfc00357c28e105e3fdeb33e3fbee3f384103 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1449692187
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* ed1183d4b566427d6c24934fe5193c6e0e759f67 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498)
* 8a753b0470ea16af842cf0f274054c031847a753 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1449714290
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* ed1183d4b566427d6c24934fe5193c6e0e759f67 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498)
* 8a753b0470ea16af842cf0f274054c031847a753 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1122642714
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -78,6 +79,10 @@ public class RequestHandler {
private final BaseFileHandler dataFileHandler;
private final MarkerHandler markerHandler;
private final Registry metricsRegistry = Registry.getRegistry("TimelineService");
+ // This read-write lock is used for syncing the file system view if it is behind client's view
Review Comment:
We have locking at two levels now. Here. and then inside AbstractTableFileSystemView. Could we simplify this.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -269,19 +269,35 @@ public void close() {
/**
* Clears the partition Map and reset view states.
+ * <p>
+ * NOTE: This method SHOULD NOT BE OVERRIDDEN which may cause stale file system view
+ * to be served. Instead, override {@link AbstractTableFileSystemView#runReset} to
+ * add custom logic.
*/
@Override
public void reset() {
Review Comment:
make this `final`? so it cant be overridden? why add comment :)
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -269,19 +269,35 @@ public void close() {
/**
* Clears the partition Map and reset view states.
+ * <p>
+ * NOTE: This method SHOULD NOT BE OVERRIDDEN which may cause stale file system view
+ * to be served. Instead, override {@link AbstractTableFileSystemView#runReset} to
+ * add custom logic.
*/
@Override
public void reset() {
try {
writeLock.lock();
- clear();
- // Initialize with new Hoodie timeline.
- init(metaClient, getTimeline());
+ runReset();
} finally {
writeLock.unlock();
}
}
+ /**
+ * Resets the view states, which can be overridden by subclasses. This reset logic is guarded
+ * by the write lock.
+ * <p>
+ * NOTE: This method SHOULD BE OVERRIDDEN for any custom logic. DO NOT OVERRIDE
Review Comment:
then make it abstract? the interface design here should force the implementations to do the right thing.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataFileSystemView.java:
##########
@@ -90,14 +90,14 @@ protected Map<Pair<String, Path>, FileStatus[]> listPartitions(List<Pair<String,
}
@Override
- public void reset() {
- super.reset();
Review Comment:
instead of all this. Can we make this impl just use the writelock here as well? and move on? i.e make the `writeLock` protected. and call the code in `super.reset()` here inline.
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -187,12 +200,20 @@ private boolean syncIfLocalViewBehind(Context ctx) {
SyncableFileSystemView view = viewManager.getFileSystemView(basePath);
synchronized (view) {
if (isLocalViewBehind(ctx)) {
- HoodieTimeline localTimeline = viewManager.getFileSystemView(basePath).getTimeline();
- LOG.info("Syncing view as client passed last known instant " + lastKnownInstantFromClient
- + " as last known instant but server has the following last instant on timeline :"
- + localTimeline.lastInstant());
- view.sync();
- return true;
+ try {
Review Comment:
yeah this seems like a good use of read/write locks.. but then, the whole thing can be inside a synchronized block for the qps levels we deal with here, I feel. Still - don't think this is an anti pattern per se.
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -187,12 +200,20 @@ private boolean syncIfLocalViewBehind(Context ctx) {
SyncableFileSystemView view = viewManager.getFileSystemView(basePath);
synchronized (view) {
if (isLocalViewBehind(ctx)) {
- HoodieTimeline localTimeline = viewManager.getFileSystemView(basePath).getTimeline();
- LOG.info("Syncing view as client passed last known instant " + lastKnownInstantFromClient
- + " as last known instant but server has the following last instant on timeline :"
- + localTimeline.lastInstant());
- view.sync();
- return true;
+ try {
Review Comment:
I wonder how much perf is actually saved by checking for `isLocalViewBehind(ctx)` outside the synchronized block? We can just do a simple, single synchronized block
```
synchronized (view) {
if (isLocalViewBehind(ctx)) {
HoodieTimeline localTimeline = viewManager.getFileSystemView(basePath).getTimeline();
LOG.info("Syncing view as client passed last known instant " + lastKnownInstantFromClient
+ " as last known instant but server has the following last instant on timeline :"
+ localTimeline.lastInstant());
view.sync();
return true;
}
}
```
read-write locks are going to add cognitive overhead everytime we read this :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1121420324
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -187,12 +200,20 @@ private boolean syncIfLocalViewBehind(Context ctx) {
SyncableFileSystemView view = viewManager.getFileSystemView(basePath);
synchronized (view) {
if (isLocalViewBehind(ctx)) {
- HoodieTimeline localTimeline = viewManager.getFileSystemView(basePath).getTimeline();
- LOG.info("Syncing view as client passed last known instant " + lastKnownInstantFromClient
- + " as last known instant but server has the following last instant on timeline :"
- + localTimeline.lastInstant());
- view.sync();
- return true;
+ try {
Review Comment:
cc @nbalajee Do you have interest for the reviewing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1450832068
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1122164561
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -151,30 +156,38 @@ public void stop() {
* Determines if local view of table's timeline is behind that of client's view.
*/
private boolean isLocalViewBehind(Context ctx) {
- String basePath = ctx.queryParam(RemoteHoodieTableFileSystemView.BASEPATH_PARAM);
- String lastKnownInstantFromClient =
- ctx.queryParamAsClass(RemoteHoodieTableFileSystemView.LAST_INSTANT_TS, String.class).getOrDefault(HoodieTimeline.INVALID_INSTANT_TS);
- String timelineHashFromClient = ctx.queryParamAsClass(RemoteHoodieTableFileSystemView.TIMELINE_HASH, String.class).getOrDefault("");
- HoodieTimeline localTimeline =
- viewManager.getFileSystemView(basePath).getTimeline().filterCompletedOrMajorOrMinorCompactionInstants();
- if (LOG.isDebugEnabled()) {
- LOG.debug("Client [ LastTs=" + lastKnownInstantFromClient + ", TimelineHash=" + timelineHashFromClient
- + "], localTimeline=" + localTimeline.getInstants());
- }
+ try {
+ // This read lock makes sure that if the local view of the table is being synced,
+ // no timeline server requests should be processed or handled until the sync process
Review Comment:
The `synchronized` won't be executed if the timeline is already refreshed and this method returns `false` to skip the synchronized block, but in such a case the metadata file system view might not have been fully updated, causing the issue.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -282,6 +286,20 @@ public void reset() {
}
}
+ /**
+ * Resets the view states, which can be overridden by subclasses. This reset logic is guarded
+ * by the write lock.
+ * <p>
+ * NOTE: This method SHOULD BE OVERRIDDEN for any custom logic. DO NOT OVERRIDE
+ * {@link AbstractTableFileSystemView#reset} directly, which may cause stale file system view
+ * to be served.
+ */
+ protected void runReset() {
Review Comment:
Yes, `reset()` should call `runReset()`. Not sure why the change is reverted somehow.
I don't think we should change the naming of `sync` and `reset` in the interface `SyncableFileSystemView` as they can also have a different implementation. I've added documentation to make sure how the custom logic should be overridden.
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -187,12 +200,20 @@ private boolean syncIfLocalViewBehind(Context ctx) {
SyncableFileSystemView view = viewManager.getFileSystemView(basePath);
synchronized (view) {
if (isLocalViewBehind(ctx)) {
- HoodieTimeline localTimeline = viewManager.getFileSystemView(basePath).getTimeline();
- LOG.info("Syncing view as client passed last known instant " + lastKnownInstantFromClient
- + " as last known instant but server has the following last instant on timeline :"
- + localTimeline.lastInstant());
- view.sync();
- return true;
+ try {
Review Comment:
As @nsivabalan mentioned, this is not thread-safe before, as some requests may completely skip the synchronized block as `isLocalViewBehind` returns false while the `view.sync()` is still in progress, and the read-write lock does not prevent the stale file system view to be served.
##########
hudi-common/src/test/java/org/apache/hudi/metadata/HoodieBackedTestDelayedTableMetadata.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Table metadata provided by an internal DFS backed Hudi metadata table,
+ * with an intentional delay in `reset()` to test concurrent reads and writes.
+ */
+public class HoodieBackedTestDelayedTableMetadata extends HoodieBackedTableMetadata {
Review Comment:
Generally a good idea to have a separate class for a notable test logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1454555052
> Is this a regression? what version? Can I look at the offending commit to understand how it was before.
I updated the PR description to provide more detailed information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1454506895
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
}, {
"hash" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564",
"triggerID" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"triggerType" : "PUSH"
}, {
"hash" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* 9c8cfc00357c28e105e3fdeb33e3fbee3f384103 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564)
* c162956f9f418b4603328c37f9e2babf59613d4b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1454572272
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
}, {
"hash" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564",
"triggerID" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"triggerType" : "PUSH"
}, {
"hash" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15571",
"triggerID" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"triggerType" : "PUSH"
}, {
"hash" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15572",
"triggerID" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* c162956f9f418b4603328c37f9e2babf59613d4b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15571)
* 7fff406e74cdf3faf047634a2d596399fa49f059 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15572)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1125388754
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -282,6 +286,20 @@ public void reset() {
}
}
+ /**
+ * Resets the view states, which can be overridden by subclasses. This reset logic is guarded
+ * by the write lock.
+ * <p>
+ * NOTE: This method SHOULD BE OVERRIDDEN for any custom logic. DO NOT OVERRIDE
+ * {@link AbstractTableFileSystemView#reset} directly, which may cause stale file system view
+ * to be served.
+ */
+ protected void runReset() {
Review Comment:
Removed `runSync` and `runReset` methods to avoid confusion and make every implementation explicitly use write lock except remote FSV. If new file system view needs to be added, the author should look at existing implementation for reference. Renaming won't prevent them the author doing the wrong thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1125388754
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -282,6 +286,20 @@ public void reset() {
}
}
+ /**
+ * Resets the view states, which can be overridden by subclasses. This reset logic is guarded
+ * by the write lock.
+ * <p>
+ * NOTE: This method SHOULD BE OVERRIDDEN for any custom logic. DO NOT OVERRIDE
+ * {@link AbstractTableFileSystemView#reset} directly, which may cause stale file system view
+ * to be served.
+ */
+ protected void runReset() {
Review Comment:
Removed `runSync` and `runReset` methods to avoid confusion and make every implementation explicitly use write lock except remote FSV. If new file system view needs to be added, the author should look at existing implementation for reference. Renaming won't prevent the author doing the wrong thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1450694928
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* 8a753b0470ea16af842cf0f274054c031847a753 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512)
* a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1450712694
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* 8a753b0470ea16af842cf0f274054c031847a753 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512)
* a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1454346850
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
}, {
"hash" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564",
"triggerID" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514)
* 9c8cfc00357c28e105e3fdeb33e3fbee3f384103 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #8079:
URL: https://github.com/apache/hudi/pull/8079
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1454667392
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
}, {
"hash" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564",
"triggerID" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"triggerType" : "PUSH"
}, {
"hash" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15571",
"triggerID" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"triggerType" : "PUSH"
}, {
"hash" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15572",
"triggerID" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"triggerType" : "PUSH"
}, {
"hash" : "d660deb903eed17560554a0145464598089fb3ea",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15574",
"triggerID" : "d660deb903eed17560554a0145464598089fb3ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* 7fff406e74cdf3faf047634a2d596399fa49f059 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15572)
* d660deb903eed17560554a0145464598089fb3ea Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15574)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1454730347
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
}, {
"hash" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564",
"triggerID" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"triggerType" : "PUSH"
}, {
"hash" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15571",
"triggerID" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"triggerType" : "PUSH"
}, {
"hash" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15572",
"triggerID" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"triggerType" : "PUSH"
}, {
"hash" : "d660deb903eed17560554a0145464598089fb3ea",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15574",
"triggerID" : "d660deb903eed17560554a0145464598089fb3ea",
"triggerType" : "PUSH"
}, {
"hash" : "d2b11c5747266e7f3cb77dfa19193bdb89548e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2b11c5747266e7f3cb77dfa19193bdb89548e50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* d660deb903eed17560554a0145464598089fb3ea Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15574)
* d2b11c5747266e7f3cb77dfa19193bdb89548e50 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1454732191
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
}, {
"hash" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564",
"triggerID" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"triggerType" : "PUSH"
}, {
"hash" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15571",
"triggerID" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"triggerType" : "PUSH"
}, {
"hash" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15572",
"triggerID" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"triggerType" : "PUSH"
}, {
"hash" : "d660deb903eed17560554a0145464598089fb3ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15574",
"triggerID" : "d660deb903eed17560554a0145464598089fb3ea",
"triggerType" : "PUSH"
}, {
"hash" : "d2b11c5747266e7f3cb77dfa19193bdb89548e50",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15562",
"triggerID" : "d2b11c5747266e7f3cb77dfa19193bdb89548e50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* d2b11c5747266e7f3cb77dfa19193bdb89548e50 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15562)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1122369457
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -187,12 +200,20 @@ private boolean syncIfLocalViewBehind(Context ctx) {
SyncableFileSystemView view = viewManager.getFileSystemView(basePath);
synchronized (view) {
if (isLocalViewBehind(ctx)) {
- HoodieTimeline localTimeline = viewManager.getFileSystemView(basePath).getTimeline();
- LOG.info("Syncing view as client passed last known instant " + lastKnownInstantFromClient
- + " as last known instant but server has the following last instant on timeline :"
- + localTimeline.lastInstant());
- view.sync();
- return true;
+ try {
Review Comment:
Did you mean when the local view is not behind, it still needs to wait for the next refresh? Sorry I didn’t think that is the correct direction to go.
And there is another question you didn’t answer, why it is okey when MDT is disabled? They share the same lock guard right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1449244307
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* ed1183d4b566427d6c24934fe5193c6e0e759f67 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1449237384
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1125390520
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -269,19 +269,35 @@ public void close() {
/**
* Clears the partition Map and reset view states.
+ * <p>
+ * NOTE: This method SHOULD NOT BE OVERRIDDEN which may cause stale file system view
+ * to be served. Instead, override {@link AbstractTableFileSystemView#runReset} to
+ * add custom logic.
*/
@Override
public void reset() {
Review Comment:
no longer needed as discussed. We directly use the write lock in each overriding implementation instead of indirect usage.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataFileSystemView.java:
##########
@@ -90,14 +90,14 @@ protected Map<Pair<String, Path>, FileStatus[]> listPartitions(List<Pair<String,
}
@Override
- public void reset() {
- super.reset();
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1125390671
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -269,19 +269,35 @@ public void close() {
/**
* Clears the partition Map and reset view states.
+ * <p>
+ * NOTE: This method SHOULD NOT BE OVERRIDDEN which may cause stale file system view
+ * to be served. Instead, override {@link AbstractTableFileSystemView#runReset} to
+ * add custom logic.
*/
@Override
public void reset() {
try {
writeLock.lock();
- clear();
- // Initialize with new Hoodie timeline.
- init(metaClient, getTimeline());
+ runReset();
} finally {
writeLock.unlock();
}
}
+ /**
+ * Resets the view states, which can be overridden by subclasses. This reset logic is guarded
+ * by the write lock.
+ * <p>
+ * NOTE: This method SHOULD BE OVERRIDDEN for any custom logic. DO NOT OVERRIDE
Review Comment:
No longer needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1125389071
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -187,12 +200,20 @@ private boolean syncIfLocalViewBehind(Context ctx) {
SyncableFileSystemView view = viewManager.getFileSystemView(basePath);
synchronized (view) {
if (isLocalViewBehind(ctx)) {
- HoodieTimeline localTimeline = viewManager.getFileSystemView(basePath).getTimeline();
- LOG.info("Syncing view as client passed last known instant " + lastKnownInstantFromClient
- + " as last known instant but server has the following last instant on timeline :"
- + localTimeline.lastInstant());
- view.sync();
- return true;
+ try {
Review Comment:
As synced offline, only keeping the synchronized block now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1450573773
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* 8a753b0470ea16af842cf0f274054c031847a753 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1122010419
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -282,6 +286,20 @@ public void reset() {
}
}
+ /**
+ * Resets the view states, which can be overridden by subclasses. This reset logic is guarded
+ * by the write lock.
+ * <p>
+ * NOTE: This method SHOULD BE OVERRIDDEN for any custom logic. DO NOT OVERRIDE
+ * {@link AbstractTableFileSystemView#reset} directly, which may cause stale file system view
+ * to be served.
+ */
+ protected void runReset() {
Review Comment:
it is guarded Danny.
As per master.
Non metadata :
```
sync() {
lock
runSync {
clean and init timeline.
}
release lock
}
```
w/ Metadata file system view
sync() in MFSV {
super.sync {
lock
runSync {
clean and init timeline.
}
release lock
}
tableMetadata.reset();
}
With this patch, here is how it is changing:
w/o metadata.
```
sync() {
lock
runSync {
clean and init timeline.
}
release lock
}
```
no changes
w/ MFSV
```
sync() in ATFSV (not overridden in MFSV) {
lock
runSync { // overridden in MFSV {
super.runSync {
clean and init timeline.
}
tableMetadata.reset();
}
release lock
}
}
```
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -187,12 +200,20 @@ private boolean syncIfLocalViewBehind(Context ctx) {
SyncableFileSystemView view = viewManager.getFileSystemView(basePath);
synchronized (view) {
if (isLocalViewBehind(ctx)) {
- HoodieTimeline localTimeline = viewManager.getFileSystemView(basePath).getTimeline();
- LOG.info("Syncing view as client passed last known instant " + lastKnownInstantFromClient
- + " as last known instant but server has the following last instant on timeline :"
- + localTimeline.lastInstant());
- view.sync();
- return true;
+ try {
Review Comment:
nope. L202 is not within synchronized Danny. hence we are introducing a read write lock.
what ethan is doing in this is patch is:
when thread1 is updating the view (view.sync()), no other threads should be able to read isLocalViewBehind(). If there is no other writers (view.sync()), multiple readers can read concurrenlty (i.e isLocalViewBehind)
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -151,30 +156,38 @@ public void stop() {
* Determines if local view of table's timeline is behind that of client's view.
*/
private boolean isLocalViewBehind(Context ctx) {
- String basePath = ctx.queryParam(RemoteHoodieTableFileSystemView.BASEPATH_PARAM);
- String lastKnownInstantFromClient =
- ctx.queryParamAsClass(RemoteHoodieTableFileSystemView.LAST_INSTANT_TS, String.class).getOrDefault(HoodieTimeline.INVALID_INSTANT_TS);
- String timelineHashFromClient = ctx.queryParamAsClass(RemoteHoodieTableFileSystemView.TIMELINE_HASH, String.class).getOrDefault("");
- HoodieTimeline localTimeline =
- viewManager.getFileSystemView(basePath).getTimeline().filterCompletedOrMajorOrMinorCompactionInstants();
- if (LOG.isDebugEnabled()) {
- LOG.debug("Client [ LastTs=" + lastKnownInstantFromClient + ", TimelineHash=" + timelineHashFromClient
- + "], localTimeline=" + localTimeline.getInstants());
- }
+ try {
+ // This read lock makes sure that if the local view of the table is being synced,
+ // no timeline server requests should be processed or handled until the sync process
Review Comment:
we call this method twice. only one caller is within synchronized
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -282,6 +286,20 @@ public void reset() {
}
}
+ /**
+ * Resets the view states, which can be overridden by subclasses. This reset logic is guarded
+ * by the write lock.
+ * <p>
+ * NOTE: This method SHOULD BE OVERRIDDEN for any custom logic. DO NOT OVERRIDE
+ * {@link AbstractTableFileSystemView#reset} directly, which may cause stale file system view
+ * to be served.
+ */
+ protected void runReset() {
Review Comment:
but agree w/ naming. we can call runSyncThreadSafely or syncThreadSafely instead of runSync.
##########
hudi-common/src/test/java/org/apache/hudi/metadata/HoodieBackedTestDelayedTableMetadata.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Table metadata provided by an internal DFS backed Hudi metadata table,
+ * with an intentional delay in `reset()` to test concurrent reads and writes.
+ */
+public class HoodieBackedTestDelayedTableMetadata extends HoodieBackedTableMetadata {
Review Comment:
does this really have to be in a separate file of its own. can we not embed within another test class where its used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1454666283
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
}, {
"hash" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15564",
"triggerID" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"triggerType" : "PUSH"
}, {
"hash" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15571",
"triggerID" : "c162956f9f418b4603328c37f9e2babf59613d4b",
"triggerType" : "PUSH"
}, {
"hash" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15572",
"triggerID" : "7fff406e74cdf3faf047634a2d596399fa49f059",
"triggerType" : "PUSH"
}, {
"hash" : "d660deb903eed17560554a0145464598089fb3ea",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d660deb903eed17560554a0145464598089fb3ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* c162956f9f418b4603328c37f9e2babf59613d4b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15571)
* 7fff406e74cdf3faf047634a2d596399fa49f059 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15572)
* d660deb903eed17560554a0145464598089fb3ea UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1449484773
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* ed1183d4b566427d6c24934fe5193c6e0e759f67 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1121418704
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -151,30 +156,38 @@ public void stop() {
* Determines if local view of table's timeline is behind that of client's view.
*/
private boolean isLocalViewBehind(Context ctx) {
- String basePath = ctx.queryParam(RemoteHoodieTableFileSystemView.BASEPATH_PARAM);
- String lastKnownInstantFromClient =
- ctx.queryParamAsClass(RemoteHoodieTableFileSystemView.LAST_INSTANT_TS, String.class).getOrDefault(HoodieTimeline.INVALID_INSTANT_TS);
- String timelineHashFromClient = ctx.queryParamAsClass(RemoteHoodieTableFileSystemView.TIMELINE_HASH, String.class).getOrDefault("");
- HoodieTimeline localTimeline =
- viewManager.getFileSystemView(basePath).getTimeline().filterCompletedOrMajorOrMinorCompactionInstants();
- if (LOG.isDebugEnabled()) {
- LOG.debug("Client [ LastTs=" + lastKnownInstantFromClient + ", TimelineHash=" + timelineHashFromClient
- + "], localTimeline=" + localTimeline.getInstants());
- }
+ try {
+ // This read lock makes sure that if the local view of the table is being synced,
+ // no timeline server requests should be processed or handled until the sync process
Review Comment:
Don't think we need a explicit lock obj again, the whole code block is guarded by a `synchronized` object lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1451146223
Is this a regression? what version? Can I look at the offending commit to understand how it was before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1125407040
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -181,22 +181,24 @@ private boolean isLocalViewBehind(Context ctx) {
* Syncs data-set view if local view is behind.
*/
private boolean syncIfLocalViewBehind(Context ctx) {
- if (isLocalViewBehind(ctx)) {
- String basePath = ctx.queryParam(RemoteHoodieTableFileSystemView.BASEPATH_PARAM);
- String lastKnownInstantFromClient = ctx.queryParamAsClass(RemoteHoodieTableFileSystemView.LAST_INSTANT_TS, String.class).getOrDefault(HoodieTimeline.INVALID_INSTANT_TS);
- SyncableFileSystemView view = viewManager.getFileSystemView(basePath);
- synchronized (view) {
- if (isLocalViewBehind(ctx)) {
- HoodieTimeline localTimeline = viewManager.getFileSystemView(basePath).getTimeline();
- LOG.info("Syncing view as client passed last known instant " + lastKnownInstantFromClient
- + " as last known instant but server has the following last instant on timeline :"
- + localTimeline.lastInstant());
- view.sync();
- return true;
- }
+ boolean result = false;
+ String basePath = ctx.queryParam(RemoteHoodieTableFileSystemView.BASEPATH_PARAM);
+ SyncableFileSystemView view = viewManager.getFileSystemView(basePath);
+ synchronized (view) {
+ if (isLocalViewBehind(ctx)) {
+
+ String lastKnownInstantFromClient = ctx.queryParamAsClass(
+ RemoteHoodieTableFileSystemView.LAST_INSTANT_TS, String.class)
+ .getOrDefault(HoodieTimeline.INVALID_INSTANT_TS);
+ HoodieTimeline localTimeline = viewManager.getFileSystemView(basePath).getTimeline();
+ LOG.info("Syncing view as client passed last known instant " + lastKnownInstantFromClient
+ + " as last known instant but server has the following last instant on timeline :"
+ + localTimeline.lastInstant());
+ view.sync();
+ result = true;
Review Comment:
Can we return directly from this line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1121416847
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -282,6 +286,20 @@ public void reset() {
}
}
+ /**
+ * Resets the view states, which can be overridden by subclasses. This reset logic is guarded
+ * by the write lock.
+ * <p>
+ * NOTE: This method SHOULD BE OVERRIDDEN for any custom logic. DO NOT OVERRIDE
+ * {@link AbstractTableFileSystemView#reset} directly, which may cause stale file system view
+ * to be served.
+ */
+ protected void runReset() {
Review Comment:
Guess the method should be guarded by reset `writeLock` ? `#reset` should invoke `#runReset` instead I guess.
And can we give these methods more straight-forward names?
Like sync -> syncThreadSafely, runSync -> sync, reset->resetThreadSafely, runReset -> reset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1122370993
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -282,6 +286,20 @@ public void reset() {
}
}
+ /**
+ * Resets the view states, which can be overridden by subclasses. This reset logic is guarded
+ * by the write lock.
+ * <p>
+ * NOTE: This method SHOULD BE OVERRIDDEN for any custom logic. DO NOT OVERRIDE
+ * {@link AbstractTableFileSystemView#reset} directly, which may cause stale file system view
+ * to be served.
+ */
+ protected void runReset() {
Review Comment:
Don’t think the doc makes any sense, let’s give them better name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1454344932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
}, {
"hash" : "8a753b0470ea16af842cf0f274054c031847a753",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15512",
"triggerID" : "8a753b0470ea16af842cf0f274054c031847a753",
"triggerType" : "PUSH"
}, {
"hash" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514",
"triggerID" : "a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d",
"triggerType" : "PUSH"
}, {
"hash" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c8cfc00357c28e105e3fdeb33e3fbee3f384103",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* a3062bb83dc4bdbdc39bb3ff4a5c612b2cb5401d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15514)
* 9c8cfc00357c28e105e3fdeb33e3fbee3f384103 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1121419697
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -187,12 +200,20 @@ private boolean syncIfLocalViewBehind(Context ctx) {
SyncableFileSystemView view = viewManager.getFileSystemView(basePath);
synchronized (view) {
if (isLocalViewBehind(ctx)) {
- HoodieTimeline localTimeline = viewManager.getFileSystemView(basePath).getTimeline();
- LOG.info("Syncing view as client passed last known instant " + lastKnownInstantFromClient
- + " as last known instant but server has the following last instant on timeline :"
- + localTimeline.lastInstant());
- view.sync();
- return true;
+ try {
Review Comment:
All the code block is thread safe already because of the `synchronized (view)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1125415237
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -181,22 +181,24 @@ private boolean isLocalViewBehind(Context ctx) {
* Syncs data-set view if local view is behind.
*/
private boolean syncIfLocalViewBehind(Context ctx) {
- if (isLocalViewBehind(ctx)) {
- String basePath = ctx.queryParam(RemoteHoodieTableFileSystemView.BASEPATH_PARAM);
- String lastKnownInstantFromClient = ctx.queryParamAsClass(RemoteHoodieTableFileSystemView.LAST_INSTANT_TS, String.class).getOrDefault(HoodieTimeline.INVALID_INSTANT_TS);
- SyncableFileSystemView view = viewManager.getFileSystemView(basePath);
- synchronized (view) {
- if (isLocalViewBehind(ctx)) {
- HoodieTimeline localTimeline = viewManager.getFileSystemView(basePath).getTimeline();
- LOG.info("Syncing view as client passed last known instant " + lastKnownInstantFromClient
- + " as last known instant but server has the following last instant on timeline :"
- + localTimeline.lastInstant());
- view.sync();
- return true;
- }
+ boolean result = false;
+ String basePath = ctx.queryParam(RemoteHoodieTableFileSystemView.BASEPATH_PARAM);
+ SyncableFileSystemView view = viewManager.getFileSystemView(basePath);
+ synchronized (view) {
+ if (isLocalViewBehind(ctx)) {
+
+ String lastKnownInstantFromClient = ctx.queryParamAsClass(
+ RemoteHoodieTableFileSystemView.LAST_INSTANT_TS, String.class)
+ .getOrDefault(HoodieTimeline.INVALID_INSTANT_TS);
+ HoodieTimeline localTimeline = viewManager.getFileSystemView(basePath).getTimeline();
+ LOG.info("Syncing view as client passed last known instant " + lastKnownInstantFromClient
+ + " as last known instant but server has the following last instant on timeline :"
+ + localTimeline.lastInstant());
+ view.sync();
+ result = true;
Review Comment:
Good catch! The variable is not needed. Fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8079:
URL: https://github.com/apache/hudi/pull/8079#discussion_r1122010419
##########
hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java:
##########
@@ -282,6 +286,20 @@ public void reset() {
}
}
+ /**
+ * Resets the view states, which can be overridden by subclasses. This reset logic is guarded
+ * by the write lock.
+ * <p>
+ * NOTE: This method SHOULD BE OVERRIDDEN for any custom logic. DO NOT OVERRIDE
+ * {@link AbstractTableFileSystemView#reset} directly, which may cause stale file system view
+ * to be served.
+ */
+ protected void runReset() {
Review Comment:
it is guarded Danny.
As per master.
Non metadata :
```
sync() {
lock
runSync {
clean and init timeline.
}
release lock
}
```
w/ Metadata file system view
```
sync() in MFSV {
super.sync {
lock
runSync {
clean and init timeline.
}
release lock
}
tableMetadata.reset();
}
```
With this patch, here is how it is changing:
w/o metadata.
```
sync() {
lock
runSync {
clean and init timeline.
}
release lock
}
```
no changes
w/ MFSV
```
sync() in ATFSV (not overridden in MFSV) {
lock
runSync { // overridden in MFSV {
super.runSync {
clean and init timeline.
}
tableMetadata.reset();
}
release lock
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8079: [HUDI-5863] Fix HoodieMetadataFileSystemView serving stale view at the timeline server
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8079:
URL: https://github.com/apache/hudi/pull/8079#issuecomment-1449248325
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "103f3efa119c4de262544fd1ee412c5375bf55cf",
"triggerType" : "PUSH"
}, {
"hash" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498",
"triggerID" : "ed1183d4b566427d6c24934fe5193c6e0e759f67",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 103f3efa119c4de262544fd1ee412c5375bf55cf UNKNOWN
* ed1183d4b566427d6c24934fe5193c6e0e759f67 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15498)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org