You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2023/04/14 00:38:28 UTC
[GitHub] [incubator-opendal] Xuanwo opened a new issue, #1938: feat: Adopt WebHDFS LISTSTATUS_BATCH for better performance
Xuanwo opened a new issue, #1938:
URL: https://github.com/apache/incubator-opendal/issues/1938
WebHDFS only supports `LISTSTATUS` for now which will read dir content at once. It could be very slow for a large dir. By adopting `LISTSTATUS_BATCH`, we can list dir Iteratively.
But this feature only added in recent HDFS versions, we should provide a config for users to enable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo closed issue #1938: feat: Adopt WebHDFS LISTSTATUS_BATCH for better performance
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed issue #1938: feat: Adopt WebHDFS LISTSTATUS_BATCH for better performance
URL: https://github.com/apache/incubator-opendal/issues/1938
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] morristai commented on issue #1938: feat: Adopt WebHDFS LISTSTATUS_BATCH for better performance
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on issue #1938:
URL: https://github.com/apache/incubator-opendal/issues/1938#issuecomment-1591785679
Hi @Xuanwo, is this issue urgent? If not, I would like to take a look at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on issue #1938: feat: Adopt WebHDFS LISTSTATUS_BATCH for better performance
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1938:
URL: https://github.com/apache/incubator-opendal/issues/1938#issuecomment-1592187121
> Hi @Xuanwo, is this issue urgent? If not, I would like to take a look at it.
Great! Take your time and have fun!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on issue #1938: feat: Adopt WebHDFS LISTSTATUS_BATCH for better performance
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1938:
URL: https://github.com/apache/incubator-opendal/issues/1938#issuecomment-1518585154
`LISTSTATUS_BATCH` only exists in newer versions, we should provide a config to allow user to switch. For example: `enable_list_status_batch`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] morristai commented on issue #1938: feat: Adopt WebHDFS LISTSTATUS_BATCH for better performance
Posted by "morristai (via GitHub)" <gi...@apache.org>.
morristai commented on issue #1938:
URL: https://github.com/apache/incubator-opendal/issues/1938#issuecomment-1595622117
Hi @Xuanwo, just want to confirm couple things before implemention:
1. Do you think adding switch config into https://github.com/apache/incubator-opendal/blob/2a7e75d9b21fc9791b083f86f2b357277d2c8743/core/src/raw/ops.rs#L54
is a good idea? This would allow users to modify the state of calling the operator like this:
```rust
let mut lister = op.list_with("/path/to/hdfs/")
.enable_list_status_batch()
.await?;
```
2. The raw API of [Apache Hadoop 3.3.5 – WebHDFS REST API](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Iteratively_List_a_Directory) needs to use the previous response body's last `pathSuffix` to set `startAfter` argument in order to search for the next batch. If you agree with the first suggestion, then the scope of the change might involve modifying https://github.com/apache/incubator-opendal/blob/2a7e75d9b21fc9791b083f86f2b357277d2c8743/core/src/services/webhdfs/backend.rs#L488
By retrieving the preview response and extracting the pathSuffix value, then we can iterate and send requests until there are no more batches remaining. Finally, the combined response can be returned to the caller. (When the user receives the result of type `Lister`, it should be the whole list instead of batch result.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on issue #1938: feat: Adopt WebHDFS LISTSTATUS_BATCH for better performance
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #1938:
URL: https://github.com/apache/incubator-opendal/issues/1938#issuecomment-1595627086
> 1. This would allow users to modify the state of calling the operator like this:
No, this should not be exposed at `Operator` level.
We can use this feature by default and add a new config called `disable_list_batch` to fallback.
> If you agree with the first suggestion, then the scope of the change might involve modifying
This state should be preserved inside `Lister`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org