You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/10/19 13:02:56 UTC
[GitHub] [ozone] ayushtkn commented on pull request #3853: HDDS-7290. provide a config to increase the list batch size in OzoneFileSystem
ayushtkn commented on PR #3853:
URL: https://github.com/apache/ozone/pull/3853#issuecomment-1283981164
>So, lets investigate in a separate JIRA item as it is highly unlikely and unrealistic that a client would set a value less than equal to 3 and also this may be our test case problem as well and not the actual problem, I have created a new JIRA to track this: https://issues.apache.org/jira/browse/HDDS-7360
Makes sense, we can investigate further on a different Jira,
Just to clear it isn't a `Test-Only` issue. It is bug in the prod code. The listStatusRoot behaves differently than the regular listStatus. So that is most probably the reason for the failure of the 2nd case.
Moreover `3` isn't as much a magic number, it is because of the number of volumes, If you create more volumes the tests will fail with higher page size as well.
ex:
```
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystem.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystem.java
index 96ac47f06..1946e60fb 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystem.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystem.java
@@ -101,6 +101,8 @@
import static org.apache.hadoop.fs.FileSystem.TRASH_PREFIX;
import static org.apache.hadoop.fs.ozone.Constants.LISTING_PAGE_SIZE;
import static org.apache.hadoop.ozone.OzoneAcl.AclScope.ACCESS;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_CLIENT_FS_LISTING_PAGE_SIZE;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_CLIENT_FS_LISTING_PAGE_SIZE_DEFAULT;
import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_FS_ITERATE_BATCH_SIZE;
import static org.apache.hadoop.ozone.OzoneConsts.OZONE_URI_DELIMITER;
import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_ADDRESS_KEY;
@@ -184,8 +186,6 @@ public static Path getBucketPath() {
return bucketPath;
}
- @Rule
- public Timeout globalTimeout = Timeout.seconds(300);
private static boolean enabledFileSystemPaths;
private static boolean omRatisEnabled;
@@ -217,6 +217,7 @@ public static Path getBucketPath() {
public static void initClusterAndEnv() throws IOException,
InterruptedException, TimeoutException {
conf = new OzoneConfiguration();
+ conf.setInt(OZONE_CLIENT_FS_LISTING_PAGE_SIZE, 50);
conf.setFloat(OMConfigKeys.OZONE_FS_TRASH_INTERVAL_KEY, TRASH_INTERVAL);
conf.setFloat(FS_TRASH_INTERVAL_KEY, TRASH_INTERVAL);
conf.setFloat(FS_TRASH_CHECKPOINT_INTERVAL_KEY, TRASH_INTERVAL / 2);
@@ -1084,6 +1085,9 @@ private void listStatusCheckHelper(Path path) throws IOException {
public void testListStatusRootAndVolumeRecursive() throws IOException {
Path bucketPath1 = createRandomVolumeBucketWithDirs();
Path bucketPath2 = createRandomVolumeBucketWithDirs();
+ for(int i=0;i<55;i++) {
+ createRandomVolumeBucketWithDirs();
+ }
// listStatus("/volume/bucket")
listStatusCheckHelper(bucketPath1);
// listStatus("/volume")
```
Run this and it will fail with 50 as well.
Regarding the 1st one the Client getting Hanged, yep may be nobody will do that, but better when you chase the new jira have a validation to not allow a value which looks genuine but leads to issues...
The 2nd issue I feel is independent of this PR, 1st one is exposed by this PR but can say not a prod use case & can be chased separately in the new Jira.
Rest changes LGTM. Leaving for others to have the final look & commit. Thanx Everyone!!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org