You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sadanand48 (via GitHub)" <gi...@apache.org> on 2025/07/31 07:32:35 UTC
[PR] HDDS-13227. Integration test for inode based bootstrap flow. [ozone]
sadanand48 opened a new pull request, #8884:
URL: https://github.com/apache/ozone/pull/8884
## What changes were proposed in this pull request?
- Makes the current test testContentsOfTarballWithSnapshot parameterized to cover both snapshot + non snapshot cases.
- Test for batching logic in the transfer
Existing tests in TestOMDBCheckpointServletInodeBasedXfer and TestOMRatisSnapshots cover this flow too.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13227
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13227. Integration test for inode based bootstrap flow. [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #8884:
URL: https://github.com/apache/ozone/pull/8884
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13227. Integration test for inode based bootstrap flow. [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8884:
URL: https://github.com/apache/ozone/pull/8884#discussion_r2246153253
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServletInodeBasedXfer.java:
##########
@@ -224,12 +225,44 @@ public void write(int b) throws IOException {
doCallRealMethod().when(omDbCheckpointServletMock).getSstBackupDir();
}
- @Test
- void testContentsOfTarballWithSnapshot() throws Exception {
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testTarballBatching(boolean includeSnapshots) throws Exception {
String volumeName = "vol" + RandomStringUtils.secure().nextNumeric(5);
String bucketName = "buck" + RandomStringUtils.secure().nextNumeric(5);
AtomicReference<DBCheckpoint> realCheckpoint = new AtomicReference<>();
- setupClusterAndMocks(volumeName, bucketName, realCheckpoint);
+ setupClusterAndMocks(volumeName, bucketName, realCheckpoint, includeSnapshots);
+ long maxFileSizeLimit = 4096;
+ om.getConfiguration().setLong(OZONE_OM_RATIS_SNAPSHOT_MAX_TOTAL_SST_SIZE_KEY, maxFileSizeLimit);
+ // Get the tarball.
+ omDbCheckpointServletMock.doGet(requestMock, responseMock);
+ String testDirName = folder.resolve("testDir").toString();
+ String newDbDirName = testDirName + OM_KEY_PREFIX + OM_DB_NAME;
+ File newDbDir = new File(newDbDirName);
+ assertTrue(newDbDir.mkdirs());
+ FileUtil.unTar(tempFile, newDbDir);
+ long totalSize;
+ try (Stream<Path> list = Files.list(newDbDir.toPath())) {
+ totalSize = list.mapToLong(path -> path.toFile().length()).sum();
+ }
+ boolean obtainedFilesUnderMaxLimit = totalSize < maxFileSizeLimit;
+ if (!includeSnapshots) {
+ // If includeSnapshotData flag is set to false , it always sends all data
+ // in one batch and doesn't respect the max size config. This is how Recon
+ // uses it today.
+ assertFalse(obtainedFilesUnderMaxLimit);
+ } else {
+ assertTrue(obtainedFilesUnderMaxLimit);
+ }
Review Comment:
Can simplify as:
```suggestion
assertEquals(includeSnapshots, totalSize < maxFileSizeLimit);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13227. Integration test for inode based bootstrap flow. [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8884:
URL: https://github.com/apache/ozone/pull/8884#discussion_r2246153253
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServletInodeBasedXfer.java:
##########
@@ -224,12 +225,44 @@ public void write(int b) throws IOException {
doCallRealMethod().when(omDbCheckpointServletMock).getSstBackupDir();
}
- @Test
- void testContentsOfTarballWithSnapshot() throws Exception {
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testTarballBatching(boolean includeSnapshots) throws Exception {
String volumeName = "vol" + RandomStringUtils.secure().nextNumeric(5);
String bucketName = "buck" + RandomStringUtils.secure().nextNumeric(5);
AtomicReference<DBCheckpoint> realCheckpoint = new AtomicReference<>();
- setupClusterAndMocks(volumeName, bucketName, realCheckpoint);
+ setupClusterAndMocks(volumeName, bucketName, realCheckpoint, includeSnapshots);
+ long maxFileSizeLimit = 4096;
+ om.getConfiguration().setLong(OZONE_OM_RATIS_SNAPSHOT_MAX_TOTAL_SST_SIZE_KEY, maxFileSizeLimit);
+ // Get the tarball.
+ omDbCheckpointServletMock.doGet(requestMock, responseMock);
+ String testDirName = folder.resolve("testDir").toString();
+ String newDbDirName = testDirName + OM_KEY_PREFIX + OM_DB_NAME;
+ File newDbDir = new File(newDbDirName);
+ assertTrue(newDbDir.mkdirs());
+ FileUtil.unTar(tempFile, newDbDir);
+ long totalSize;
+ try (Stream<Path> list = Files.list(newDbDir.toPath())) {
+ totalSize = list.mapToLong(path -> path.toFile().length()).sum();
+ }
+ boolean obtainedFilesUnderMaxLimit = totalSize < maxFileSizeLimit;
+ if (!includeSnapshots) {
+ // If includeSnapshotData flag is set to false , it always sends all data
+ // in one batch and doesn't respect the max size config. This is how Recon
+ // uses it today.
+ assertFalse(obtainedFilesUnderMaxLimit);
+ } else {
+ assertTrue(obtainedFilesUnderMaxLimit);
+ }
Review Comment:
Can simplify as:
```suggestion
// If includeSnapshotData flag is set to false , it always sends all data
// in one batch and doesn't respect the max size config. This is how Recon
// uses it today.
assertEquals(includeSnapshots, totalSize < maxFileSizeLimit);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org