You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "morristai (via GitHub)" <gi...@apache.org> on 2023/06/20 04:28:19 UTC

[GitHub] [incubator-opendal] morristai commented on a diff in pull request #2499: feat(core): adopt WebHDFS LISTSTATUS_BATCH for better performance

morristai commented on code in PR #2499:
URL: https://github.com/apache/incubator-opendal/pull/2499#discussion_r1234719792


##########
core/src/services/webhdfs/backend.rs:
##########
@@ -276,6 +295,88 @@ impl WebhdfsBackend {
         Ok(req)
     }
 
+    pub(super) fn webhdfs_list_status_batch_request(
+        &self,
+        path: &str,
+        start_after: &Option<String>,
+    ) -> Result<Request<AsyncBody>> {
+        let p = build_abs_path(&self.root, path);
+
+        // if it's not the first time to call LISTSTATUS_BATCH, we will add &startAfter=<CHILD>
+        let start_after_param = match start_after {
+            Some(sa) if sa.is_empty() => String::new(),
+            Some(sa) => format!("&startAfter={}", sa),
+            None => String::new(),
+        };
+
+        let mut url = format!(
+            "{}/webhdfs/v1/{}?op=LISTSTATUS_BATCH{}",
+            self.endpoint,
+            percent_encode_path(&p),
+            start_after_param
+        );
+        if let Some(auth) = &self.auth {
+            url += format!("&{auth}").as_str();
+        }
+
+        let req = Request::get(&url)
+            .body(AsyncBody::Empty)
+            .map_err(new_request_build_error)?;
+        Ok(req)
+    }
+
+    /// parse the response of LISTSTATUS_BATCH
+    ///
+    /// # Note
+    ///
+    /// - if the response is 200, return the DirectoryListing
+    /// - if the response is 404, return an empty DirectoryListing
+    /// - otherwise, return an error
+    pub(super) async fn webhdfs_list_status_batch_parse(

Review Comment:
   > We don't need a seperate function for xxx_parse. They are simple and only be used once. We can write in related function body directly.
   
   I would like to point out that my intention to write an independent function is based on the fact that the `webhdfs_list_status_request` does not include parsing logic. The parser logic is instead located in the list action function. However, due to the different response bodies of `list_status` and `list_status_batch`, plus the `next` function in the webhdfs pager also requiring a call to the parser when querying for the next batch of list. I guess based on your suggestion, I should put it inside the request function like `webhdfs_list_status_batch_request`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org