You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2023/09/21 02:07:29 UTC

[GitHub] [incubator-opendal] Xuanwo opened a new issue, #3144: Integrate hdfs services with `hdfs-native`

Xuanwo opened a new issue, #3144:
URL: https://github.com/apache/incubator-opendal/issues/3144

   https://github.com/Kimahriman/hdfs-native is a pure Rust implementation of an HDFS client. It might be worth considering integrating it as a new service in opendal, allowing our users to give it a try.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3144:
URL: https://github.com/apache/opendal/issues/3144#issuecomment-1936267690

   > Is there some information about how to add the behavior test?
   
   Our behavior test exists at https://github.com/apache/opendal/tree/main/core/tests/behavior
   
   We can follow the same content from hdfs for our hdfs native tests:
   
   - fixtures: we use those fixtures to setup our services,  https://github.com/apache/opendal/tree/main/fixtures/hdfs
   - setup: we have a automate workflow for all services. We just need to add new setup for `hdfs_native` like https://github.com/apache/opendal/tree/main/.github/services/hdfs/hdfs_default
   
   The test should be run automanticly in our CI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [opendal]

Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3144:
URL: https://github.com/apache/opendal/issues/3144#issuecomment-2059937470

   @Xuanwo Can I take a swing at implementing the read and write?
   Thank you!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [incubator-opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3144:
URL: https://github.com/apache/incubator-opendal/issues/3144#issuecomment-1873859684

   hdfs-native 0.5 has been released, let's do this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [opendal]

Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
jihuayu commented on issue #3144:
URL: https://github.com/apache/opendal/issues/3144#issuecomment-2048854727

   @Xuanwo Thank you. I love the new trait. 
   I've been quite busy lately, and I'll be back in a few months.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [incubator-opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3144:
URL: https://github.com/apache/incubator-opendal/issues/3144#issuecomment-1867123185

   Hi, @shbhmrzd, I'm waiting for https://github.com/Kimahriman/hdfs-native/issues/13.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [opendal]

Posted by "suyanhanx (via GitHub)" <gi...@apache.org>.
suyanhanx commented on issue #3144:
URL: https://github.com/apache/opendal/issues/3144#issuecomment-5644042526

   closed by https://github.com/apache/opendal/pull/5617


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [opendal]

Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
jihuayu commented on issue #3144:
URL: https://github.com/apache/opendal/issues/3144#issuecomment-1962172879

   @Xuanwo Thank you!
   I did encounter some trouble:
   When I was preparing to write test cases, I found that the reader and writer for hdfs_native were not implemented, so I decided to implement them first.
   Since I'm not very familiar with asynchronous Rust, my code has not been functioning properly. Could you help me take a look at how I should write it?
   https://github.com/jihuayu/opendal/blob/f/hdfs-test/core/src/services/hdfs_native/writer.rs#L43
   After running it, I found that the write never stops.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [opendal]

Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
jihuayu commented on issue #3144:
URL: https://github.com/apache/opendal/issues/3144#issuecomment-1962254635

   @Xuanwo Ohhh! I know! Thank you. I will have a try!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [incubator-opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3144:
URL: https://github.com/apache/incubator-opendal/issues/3144#issuecomment-1867161756

   > Sure @Xuanwo I would like to pick up OpenDAL integration when it is planned :)
   
   Thanks! I will ping you here when it's ready.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3144:
URL: https://github.com/apache/opendal/issues/3144#issuecomment-1935377333

   Thanks to @shbhmrzd and @jihuayu's efforts. We have the basic feature set now. Maybe it's time for us to establish the behavior test before adding more features.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3144:
URL: https://github.com/apache/opendal/issues/3144#issuecomment-1960992554

   Hi, @jihuayu, do you need some help from me?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [opendal]

Posted by "suyanhanx (via GitHub)" <gi...@apache.org>.
suyanhanx closed issue #3144: Integrate hdfs services with `hdfs-native`
URL: https://github.com/apache/opendal/issues/3144


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3144:
URL: https://github.com/apache/opendal/issues/3144#issuecomment-2062979918

   > Can I take a swing at implementing the read and write?
   
   Of course, have fun!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [incubator-opendal]

Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3144:
URL: https://github.com/apache/incubator-opendal/issues/3144#issuecomment-1866879774

   I am looking for basic hdfs operations read, write and delete in my app.
   The current `hdfs-sys` based approach is costing in terms of image size due to the hadoop jars which need to be pulled in. If this is a pure rust implementation then it could be worth investing.
   I can take this up as we already use OpenDAL in our application, and having support for `hdfs-native` will reduce need for an additional layer in my app.
   Please let me know if this is up for grab and I can spend time on it.
   Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [incubator-opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3144:
URL: https://github.com/apache/incubator-opendal/issues/3144#issuecomment-1874713052

   > Awesome Can I take it up ?
   
   Welcome! Have fun.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [incubator-opendal]

Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3144:
URL: https://github.com/apache/incubator-opendal/issues/3144#issuecomment-1867153935

   Sure 
   I would like to pick up OpenDAL integration when it is planned


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [incubator-opendal]

Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3144:
URL: https://github.com/apache/incubator-opendal/issues/3144#issuecomment-1874457637

   Awesome 
   Can I take it up ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3144:
URL: https://github.com/apache/opendal/issues/3144#issuecomment-1962242415

   > After running it, I found that the write never stops.
   
   `FileWrtier::write` is an async function, every call to `f.write()` will create a new future. So you will need to store this future and poll it until `Ready`.
   
   For example:
   
   https://github.com/apache/opendal/blob/ab52b437fdb19b7fbaa4be850e00c84f360d06f1/core/src/raw/oio/read/range_read.rs#L51-L56


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3144:
URL: https://github.com/apache/opendal/issues/3144#issuecomment-2047658867

   Hi, @jihuayu, I did a refactor to the whole opendal's IO trait. Would you like to take another try?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Integrate hdfs services with `hdfs-native` [opendal]

Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
jihuayu commented on issue #3144:
URL: https://github.com/apache/opendal/issues/3144#issuecomment-1935602825

   @Xuanwo Hi, I want to add the behavior tests for it.
   Is there some information about how to add the behavior test?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org