You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bvaradar (via GitHub)" <gi...@apache.org> on 2024/05/22 23:37:19 UTC
[PR] [MINOR] LSMTimeline needs to handle case for tables which has not performed first archived yet [hudi]
bvaradar opened a new pull request, #11271:
URL: https://github.com/apache/hudi/pull/11271
Found during backwards compatibility testing. Sometimes, Archiving would not have run for a Hudi table. In that case, LSMTimeline must gracefully handle instead of throwing File not found exception.
### Change Logs
Found during backwards compatibility testing. Sometimes, Archiving would not have run for a Hudi table. In that case, LSMTimeline must gracefully handle instead of throwing File not found exception.
### Impact
Backwards Compatibility
### Risk level (write none, low medium or high below)
None
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] LSMTimeline needs to handle case for tables which has not performed first archived yet [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on code in PR #11271:
URL: https://github.com/apache/hudi/pull/11271#discussion_r1610840996
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/LSMTimeline.java:
##########
@@ -158,13 +159,18 @@ public static int latestSnapshotVersion(HoodieTableMetaClient metaClient) throws
LOG.warn("Error reading version file {}", versionFilePath, e);
}
}
+
return allSnapshotVersions(metaClient).stream().max(Integer::compareTo).orElse(-1);
}
/**
* Returns all the valid snapshot versions.
*/
public static List<Integer> allSnapshotVersions(HoodieTableMetaClient metaClient) throws IOException {
+ StoragePath archivedFolderPath = new StoragePath(metaClient.getArchivePath());
Review Comment:
Yes, found during testing the 0.14.x writer with 1.x reader testing for a newly created 0.14.x table. LSMTimeline#latestSnapshotVersion was calling LSMTimeline#allSnapshotVersions due to the absence of latest snapshot version file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] LSMTimeline needs to handle case for tables which has not performed first archived yet [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11271:
URL: https://github.com/apache/hudi/pull/11271#discussion_r1611302439
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/LSMTimeline.java:
##########
@@ -158,13 +159,18 @@ public static int latestSnapshotVersion(HoodieTableMetaClient metaClient) throws
LOG.warn("Error reading version file {}", versionFilePath, e);
}
}
+
return allSnapshotVersions(metaClient).stream().max(Integer::compareTo).orElse(-1);
}
/**
* Returns all the valid snapshot versions.
*/
public static List<Integer> allSnapshotVersions(HoodieTableMetaClient metaClient) throws IOException {
+ StoragePath archivedFolderPath = new StoragePath(metaClient.getArchivePath());
Review Comment:
Got it, thanks for thee explaination.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] LSMTimeline needs to handle case for tables which has not performed first archived yet [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #11271:
URL: https://github.com/apache/hudi/pull/11271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] LSMTimeline needs to handle case for tables which has not performed first archived yet [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11271:
URL: https://github.com/apache/hudi/pull/11271#issuecomment-2125988423
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "818e29cf9483b5b3a17724a490e5c602f5b408c4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "818e29cf9483b5b3a17724a490e5c602f5b408c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 818e29cf9483b5b3a17724a490e5c602f5b408c4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] LSMTimeline needs to handle case for tables which has not performed first archived yet [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11271:
URL: https://github.com/apache/hudi/pull/11271#discussion_r1610818043
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/LSMTimeline.java:
##########
@@ -158,13 +159,18 @@ public static int latestSnapshotVersion(HoodieTableMetaClient metaClient) throws
LOG.warn("Error reading version file {}", versionFilePath, e);
}
}
+
return allSnapshotVersions(metaClient).stream().max(Integer::compareTo).orElse(-1);
}
/**
* Returns all the valid snapshot versions.
*/
public static List<Integer> allSnapshotVersions(HoodieTableMetaClient metaClient) throws IOException {
+ StoragePath archivedFolderPath = new StoragePath(metaClient.getArchivePath());
Review Comment:
I search around the invokers, one is archive `LSMTimelineWriter#clean` and another is `LSMTimeline#`latestSnapshotVersion, both would ensure the archving already happens, is this a case specific for backward compatibility?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/LSMTimeline.java:
##########
@@ -158,13 +159,18 @@ public static int latestSnapshotVersion(HoodieTableMetaClient metaClient) throws
LOG.warn("Error reading version file {}", versionFilePath, e);
}
}
+
return allSnapshotVersions(metaClient).stream().max(Integer::compareTo).orElse(-1);
}
/**
* Returns all the valid snapshot versions.
*/
public static List<Integer> allSnapshotVersions(HoodieTableMetaClient metaClient) throws IOException {
+ StoragePath archivedFolderPath = new StoragePath(metaClient.getArchivePath());
Review Comment:
I search around the invokers, one is archive `LSMTimelineWriter#clean` and another is `LSMTimeline#latestSnapshotVersion`, both would ensure the archving already happens, is this a case specific for backward compatibility?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org