You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "swamirishi (via GitHub)" <gi...@apache.org> on 2024/11/16 03:18:40 UTC
[PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
swamirishi opened a new pull request, #7440:
URL: https://github.com/apache/ozone/pull/7440
## What changes were proposed in this pull request?
Currently we have listKeyLight API & listStatus light api implemented ozone manager & rpc client. But currently Ozone filesystem implementations doesn't call the listStatusLight API for FSO buckets for listStatus & listStatusIterator function.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11697
## How was this patch tested?
Adding Integration tests for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1846156095
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -985,8 +985,23 @@ public OzoneDataStreamOutput createStreamFile(String keyName, long size,
*/
public List<OzoneFileStatus> listStatus(String keyName, boolean recursive,
String startKey, long numEntries) throws IOException {
- return proxy
- .listStatus(volumeName, name, keyName, recursive, startKey, numEntries);
+ return proxy.listStatus(volumeName, name, keyName, recursive, startKey, numEntries);
+ }
+
+ /**
+ * List the lightweight status for a file or a directory and its contents.
+ *
+ * @param keyName Absolute path of the entry to be listed
+ * @param recursive For a directory if true all the descendants of a
+ * particular directory are listed
+ * @param startKey Key from which listing needs to start. If startKey exists
+ * its status is included in the final list.
+ * @param numEntries Number of entries to list from the start key
+ * @return list of file status
+ */
+ public List<OzoneFileStatusLight> listStatusLight(String keyName, boolean recursive,
Review Comment:
If OzoneFileStatus extended OzoneFileStatusLight we could have done with a single method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1846081635
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -2311,6 +2311,22 @@ public List<OzoneFileStatus> listStatus(String volumeName, String bucketName,
allowPartialPrefixes);
}
+ @Override
+ public List<OzoneFileStatusLight> listStatusLight(String volumeName, String bucketName, String keyName,
+ boolean recursive, String startKey, long numEntries) throws IOException {
+ OmKeyArgs keyArgs = prepareOmKeyArgs(volumeName, bucketName, keyName);
+ if (omVersion.compareTo(OzoneManagerVersion.LIGHTWEIGHT_LIST_STATUS) >= 0) {
+ return ozoneManagerClient.listStatusLight(keyArgs, recursive, startKey,
+ numEntries);
Review Comment:
> We have implemented the same thing even for listStatus API. It would be better to have some uniformity there.
For `listStatus` methods, new method with additional parameter was introduced later:
```
2019-05-21 14:47:48 +0530 968) List<OzoneFileStatus> listStatus(OmKeyArgs keyArgs, boolean recursive,
2019-05-21 14:47:48 +0530 969) String startKey, long numEntries) throws IOException;
2022-06-03 08:41:29 +0530 984) List<OzoneFileStatus> listStatus(OmKeyArgs keyArgs, boolean recursive,
2022-06-03 08:41:29 +0530 985) String startKey, long numEntries,
2022-06-03 08:41:29 +0530 986) boolean allowPartialPrefixes)
```
Since the existing `listStatusLight` covers both cases, overloading the method is not necessary now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1846073580
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneClientAdapterImpl.java:
##########
@@ -549,6 +558,31 @@ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatus status,
);
}
+ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatusLight status,
+ String owner, URI defaultUri, Path workingDir) {
+ BasicOmKeyInfo keyInfo = status.getKeyInfo();
+ short replication = (short) keyInfo.getReplicationConfig()
+ .getRequiredNodes();
+ return new FileStatusAdapter(
+ keyInfo.getDataSize(),
+ keyInfo.getReplicatedSize(),
+ new Path(OZONE_URI_DELIMITER + keyInfo.getKeyName())
+ .makeQualified(defaultUri, workingDir),
+ status.isDirectory(),
+ replication,
+ status.getBlockSize(),
+ keyInfo.getModificationTime(),
+ keyInfo.getModificationTime(),
+ status.isDirectory() ? (short) 00777 : (short) 00666,
Review Comment:
I also don't like the way BasicKeyInfo class has been definied. Ideally we should have OzoneFileStatus extend OzoneFileStatusLight. It would make our code much more modular and avoid such unnecessary duplication of code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1845606959
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -2311,6 +2311,22 @@ public List<OzoneFileStatus> listStatus(String volumeName, String bucketName,
allowPartialPrefixes);
}
+ @Override
+ public List<OzoneFileStatusLight> listStatusLight(String volumeName, String bucketName, String keyName,
+ boolean recursive, String startKey, long numEntries) throws IOException {
+ OmKeyArgs keyArgs = prepareOmKeyArgs(volumeName, bucketName, keyName);
+ if (omVersion.compareTo(OzoneManagerVersion.LIGHTWEIGHT_LIST_STATUS) >= 0) {
+ return ozoneManagerClient.listStatusLight(keyArgs, recursive, startKey,
+ numEntries);
Review Comment:
Sorry, I was wrong, it works, because client never calls the new method on the server. `OzoneManagerProtocolClientSideTranslatorPB` does the trick and calls the old method, somewhat like I suggested.
Still, I think removing the unused method would improve clarity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1846068283
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -2311,6 +2311,22 @@ public List<OzoneFileStatus> listStatus(String volumeName, String bucketName,
allowPartialPrefixes);
}
+ @Override
+ public List<OzoneFileStatusLight> listStatusLight(String volumeName, String bucketName, String keyName,
+ boolean recursive, String startKey, long numEntries) throws IOException {
+ OmKeyArgs keyArgs = prepareOmKeyArgs(volumeName, bucketName, keyName);
+ if (omVersion.compareTo(OzoneManagerVersion.LIGHTWEIGHT_LIST_STATUS) >= 0) {
+ return ozoneManagerClient.listStatusLight(keyArgs, recursive, startKey,
+ numEntries);
Review Comment:
We have implemented the same thing even for listStatus API. It would be better to have some uniformity there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "tanvipenumudy (via GitHub)" <gi...@apache.org>.
tanvipenumudy commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1846057449
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneClientAdapterImpl.java:
##########
@@ -549,6 +558,31 @@ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatus status,
);
}
+ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatusLight status,
+ String owner, URI defaultUri, Path workingDir) {
+ BasicOmKeyInfo keyInfo = status.getKeyInfo();
+ short replication = (short) keyInfo.getReplicationConfig()
+ .getRequiredNodes();
+ return new FileStatusAdapter(
+ keyInfo.getDataSize(),
+ keyInfo.getReplicatedSize(),
+ new Path(OZONE_URI_DELIMITER + keyInfo.getKeyName())
+ .makeQualified(defaultUri, workingDir),
+ status.isDirectory(),
+ replication,
+ status.getBlockSize(),
+ keyInfo.getModificationTime(),
+ keyInfo.getModificationTime(),
+ status.isDirectory() ? (short) 00777 : (short) 00666,
Review Comment:
I agree, this can be addressed as a follow-up task!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1845993219
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneClientAdapterImpl.java:
##########
@@ -549,6 +558,31 @@ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatus status,
);
}
+ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatusLight status,
+ String owner, URI defaultUri, Path workingDir) {
+ BasicOmKeyInfo keyInfo = status.getKeyInfo();
+ short replication = (short) keyInfo.getReplicationConfig()
+ .getRequiredNodes();
+ return new FileStatusAdapter(
+ keyInfo.getDataSize(),
+ keyInfo.getReplicatedSize(),
+ new Path(OZONE_URI_DELIMITER + keyInfo.getKeyName())
+ .makeQualified(defaultUri, workingDir),
+ status.isDirectory(),
+ replication,
+ status.getBlockSize(),
+ keyInfo.getModificationTime(),
+ keyInfo.getModificationTime(),
+ status.isDirectory() ? (short) 00777 : (short) 00666,
Review Comment:
Existing `toFileStatusAdapter(OzoneFileStatus, ...)` is duplicated here almost exactly, so I think we should make improvements like this in a follow-up task:
- create constants
- reduce duplication if possible (two methods within same class, as well as other FS adapter implementation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1845559434
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -2311,6 +2311,22 @@ public List<OzoneFileStatus> listStatus(String volumeName, String bucketName,
allowPartialPrefixes);
}
+ @Override
+ public List<OzoneFileStatusLight> listStatusLight(String volumeName, String bucketName, String keyName,
+ boolean recursive, String startKey, long numEntries) throws IOException {
+ OmKeyArgs keyArgs = prepareOmKeyArgs(volumeName, bucketName, keyName);
+ if (omVersion.compareTo(OzoneManagerVersion.LIGHTWEIGHT_LIST_STATUS) >= 0) {
+ return ozoneManagerClient.listStatusLight(keyArgs, recursive, startKey,
+ numEntries);
Review Comment:
Why would it be incompatible? These are just interfaces right? If a server doesn't have the interface method it will anyways call the method with allowPartialPrefixes as false on the server side. There should be no incompatibilty in my opinion. Since one could have 2 different interfaces one server side and client side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "tanvipenumudy (via GitHub)" <gi...@apache.org>.
tanvipenumudy commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1845904156
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneClientAdapterImpl.java:
##########
@@ -549,6 +558,31 @@ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatus status,
);
}
+ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatusLight status,
+ String owner, URI defaultUri, Path workingDir) {
+ BasicOmKeyInfo keyInfo = status.getKeyInfo();
+ short replication = (short) keyInfo.getReplicationConfig()
+ .getRequiredNodes();
+ return new FileStatusAdapter(
+ keyInfo.getDataSize(),
+ keyInfo.getReplicatedSize(),
+ new Path(OZONE_URI_DELIMITER + keyInfo.getKeyName())
+ .makeQualified(defaultUri, workingDir),
+ status.isDirectory(),
+ replication,
+ status.getBlockSize(),
+ keyInfo.getModificationTime(),
+ keyInfo.getModificationTime(),
+ status.isDirectory() ? (short) 00777 : (short) 00666,
Review Comment:
I think we may replace the hardcoded permission magic numbers (`00777` and `00666`) with descriptive constants:
```
private static final short DIRECTORY_PERMISSIONS = (short) 00777;
private static final short FILE_PERMISSIONS = (short) 00666;
return new FileStatusAdapter(
...,
status.isDirectory() ? DIRECTORY_PERMISSIONS : FILE_PERMISSIONS,
...
);
```
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1007,10 +1011,12 @@ private final class OzoneFileStatusIterator<T extends FileStatus>
* @param p path to file/directory.
* @throws IOException
*/
- private OzoneFileStatusIterator(Path p) throws IOException {
+ private OzoneFileStatusIterator(Path p, Function<FileStatus, T> transformFunc, boolean lite) throws IOException {
Review Comment:
We may update the Javadoc to incorporate the newly added parameters.
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java:
##########
@@ -1039,6 +1053,31 @@ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatus status,
);
}
+ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatusLight status,
+ String owner, URI defaultUri, Path workingDir, String ofsPathPrefix) {
+ BasicOmKeyInfo keyInfo = status.getKeyInfo();
+ short replication = (short) keyInfo.getReplicationConfig()
+ .getRequiredNodes();
+ return new FileStatusAdapter(
+ keyInfo.getDataSize(),
+ keyInfo.getReplicatedSize(),
+ new Path(ofsPathPrefix + OZONE_URI_DELIMITER + keyInfo.getKeyName())
+ .makeQualified(defaultUri, workingDir),
+ status.isDirectory(),
+ replication,
+ status.getBlockSize(),
+ keyInfo.getModificationTime(),
+ keyInfo.getModificationTime(),
+ status.isDirectory() ? (short) 00777 : (short) 00666,
Review Comment:
Same as the above comment for the hardcoded permission values.
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1088,15 +1094,15 @@ public T next() throws IOException {
* @return list of file status.
* @throws IOException
*/
- private List<FileStatus> listFileStatus(Path f, String startPath)
+ private List<FileStatus> listFileStatus(Path f, String startPath, boolean lite)
Review Comment:
We may update the Javadoc to include the `lite` parameter here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1845545421
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataReader.java:
##########
Review Comment:
nit: This file has only whitespace changes.
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -2311,6 +2311,22 @@ public List<OzoneFileStatus> listStatus(String volumeName, String bucketName,
allowPartialPrefixes);
}
+ @Override
+ public List<OzoneFileStatusLight> listStatusLight(String volumeName, String bucketName, String keyName,
+ boolean recursive, String startKey, long numEntries) throws IOException {
+ OmKeyArgs keyArgs = prepareOmKeyArgs(volumeName, bucketName, keyName);
+ if (omVersion.compareTo(OzoneManagerVersion.LIGHTWEIGHT_LIST_STATUS) >= 0) {
+ return ozoneManagerClient.listStatusLight(keyArgs, recursive, startKey,
+ numEntries);
Review Comment:
`OzoneManagerVersion.LIGHTWEIGHT_LIST_STATUS` is an existing version, added in HDDS-11414. The method being called is new in this PR:
https://github.com/apache/ozone/blob/bd264b5d415466743cf25825f9d4d75d1af26143/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java#L1000-L1001
So this is not compatible with versions of Ozone Manager between these commits (8ca33c718b and the one that will precede this on `master` when merged).
My suggestion:
- remove the new method from `OzoneManagerProtocol` and its implementations,
- call the existing method here:
```suggestion
return ozoneManagerClient.listStatusLight(keyArgs, recursive, startKey,
numEntries, false);
```
(It's not worth adding yet another `OzoneManagerVersion` constant, since the new method is a subset of the existing.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1846154926
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -985,8 +985,23 @@ public OzoneDataStreamOutput createStreamFile(String keyName, long size,
*/
public List<OzoneFileStatus> listStatus(String keyName, boolean recursive,
String startKey, long numEntries) throws IOException {
- return proxy
- .listStatus(volumeName, name, keyName, recursive, startKey, numEntries);
+ return proxy.listStatus(volumeName, name, keyName, recursive, startKey, numEntries);
+ }
+
+ /**
+ * List the lightweight status for a file or a directory and its contents.
+ *
+ * @param keyName Absolute path of the entry to be listed
+ * @param recursive For a directory if true all the descendants of a
+ * particular directory are listed
+ * @param startKey Key from which listing needs to start. If startKey exists
+ * its status is included in the final list.
+ * @param numEntries Number of entries to list from the start key
+ * @return list of file status
+ */
+ public List<OzoneFileStatusLight> listStatusLight(String keyName, boolean recursive,
Review Comment:
This method returns a list of OzoneFileStatusLight while listSTatus method returns a List<OzoneFileStatus>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #7440:
URL: https://github.com/apache/ozone/pull/7440#issuecomment-2483454332
@kerneltime Let me know if I have missed anything on this patch, we can create a follow up for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1846078280
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneClientAdapterImpl.java:
##########
@@ -1039,6 +1053,31 @@ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatus status,
);
}
+ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatusLight status,
+ String owner, URI defaultUri, Path workingDir, String ofsPathPrefix) {
+ BasicOmKeyInfo keyInfo = status.getKeyInfo();
+ short replication = (short) keyInfo.getReplicationConfig()
+ .getRequiredNodes();
+ return new FileStatusAdapter(
+ keyInfo.getDataSize(),
+ keyInfo.getReplicatedSize(),
+ new Path(ofsPathPrefix + OZONE_URI_DELIMITER + keyInfo.getKeyName())
+ .makeQualified(defaultUri, workingDir),
+ status.isDirectory(),
+ replication,
+ status.getBlockSize(),
+ keyInfo.getModificationTime(),
+ keyInfo.getModificationTime(),
+ status.isDirectory() ? (short) 00777 : (short) 00666,
Review Comment:
done
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1007,10 +1011,12 @@ private final class OzoneFileStatusIterator<T extends FileStatus>
* @param p path to file/directory.
* @throws IOException
*/
- private OzoneFileStatusIterator(Path p) throws IOException {
+ private OzoneFileStatusIterator(Path p, Function<FileStatus, T> transformFunc, boolean lite) throws IOException {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1846152126
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -985,8 +985,23 @@ public OzoneDataStreamOutput createStreamFile(String keyName, long size,
*/
public List<OzoneFileStatus> listStatus(String keyName, boolean recursive,
String startKey, long numEntries) throws IOException {
- return proxy
- .listStatus(volumeName, name, keyName, recursive, startKey, numEntries);
+ return proxy.listStatus(volumeName, name, keyName, recursive, startKey, numEntries);
+ }
+
+ /**
+ * List the lightweight status for a file or a directory and its contents.
+ *
+ * @param keyName Absolute path of the entry to be listed
+ * @param recursive For a directory if true all the descendants of a
+ * particular directory are listed
+ * @param startKey Key from which listing needs to start. If startKey exists
+ * its status is included in the final list.
+ * @param numEntries Number of entries to list from the start key
+ * @return list of file status
+ */
+ public List<OzoneFileStatusLight> listStatusLight(String keyName, boolean recursive,
Review Comment:
do we need expose this interface using OzoneBucket? or can be in listStatus with flag "lite" as used for FS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi merged PR #7440:
URL: https://github.com/apache/ozone/pull/7440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1846058185
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneClientAdapterImpl.java:
##########
@@ -549,6 +558,31 @@ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatus status,
);
}
+ private FileStatusAdapter toFileStatusAdapter(OzoneFileStatusLight status,
+ String owner, URI defaultUri, Path workingDir) {
+ BasicOmKeyInfo keyInfo = status.getKeyInfo();
+ short replication = (short) keyInfo.getReplicationConfig()
+ .getRequiredNodes();
+ return new FileStatusAdapter(
+ keyInfo.getDataSize(),
+ keyInfo.getReplicatedSize(),
+ new Path(OZONE_URI_DELIMITER + keyInfo.getKeyName())
+ .makeQualified(defaultUri, workingDir),
+ status.isDirectory(),
+ replication,
+ status.getBlockSize(),
+ keyInfo.getModificationTime(),
+ keyInfo.getModificationTime(),
+ status.isDirectory() ? (short) 00777 : (short) 00666,
Review Comment:
yeah we should pursue this in a followup task for this. There is too much duplication b/w BasicOzoneFIleSystem & BasicRootedOzoneFileSystem, BasicOzoneClientAdapterImpl & BasicRootedOzoneClientAdapterImpl. This created a lot of confusion while implementing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1846080366
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -1088,15 +1094,15 @@ public T next() throws IOException {
* @return list of file status.
* @throws IOException
*/
- private List<FileStatus> listFileStatus(Path f, String startPath)
+ private List<FileStatus> listFileStatus(Path f, String startPath, boolean lite)
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #7440:
URL: https://github.com/apache/ozone/pull/7440#issuecomment-2483451561
Thanks for reviews on the patch @adoroszlai @sumitagrawl @tanvipenumudy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1845563156
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataReader.java:
##########
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1845563873
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataReader.java:
##########
Review Comment:
Not really:
https://github.com/apache/ozone/pull/7440/files#diff-2704fe9d507507852a26299d8dbcd929f4127a28c053e42d46f71bb39f7f9cc1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11697. Integrate Ozone Filesystem Implementation with Ozone ListStatusLight API [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #7440:
URL: https://github.com/apache/ozone/pull/7440#discussion_r1846095875
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -2311,6 +2311,22 @@ public List<OzoneFileStatus> listStatus(String volumeName, String bucketName,
allowPartialPrefixes);
}
+ @Override
+ public List<OzoneFileStatusLight> listStatusLight(String volumeName, String bucketName, String keyName,
+ boolean recursive, String startKey, long numEntries) throws IOException {
+ OmKeyArgs keyArgs = prepareOmKeyArgs(volumeName, bucketName, keyName);
+ if (omVersion.compareTo(OzoneManagerVersion.LIGHTWEIGHT_LIST_STATUS) >= 0) {
+ return ozoneManagerClient.listStatusLight(keyArgs, recursive, startKey,
+ numEntries);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org