You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "xiaguan (via GitHub)" <gi...@apache.org> on 2024/02/22 10:24:14 UTC

[I] Best Practices for Concurrent Usage Doubts [opendal]

xiaguan opened a new issue, #4248:
URL: https://github.com/apache/opendal/issues/4248

   We've got a bunch of write back tasks in our file system for pushing dirty data back to cloud storage. I'm trying to figure out if it's best to share a single `Arc<Operator>` among all these tasks or if we should be giving each task its own `Operator` instance. Is there even a difference between these two approaches when it comes to performance or how things work?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Best Practices for Concurrent Usage Doubts [opendal]

Posted by "xiaguan (via GitHub)" <gi...@apache.org>.
xiaguan commented on issue #4248:
URL: https://github.com/apache/opendal/issues/4248#issuecomment-1959149128

   And another thing.
   I've been feeling like the Memory Service is kinda slow, especially with lotfs of data? Though, I've only noticed this during testing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Best Practices for Concurrent Usage Doubts [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4248:
URL: https://github.com/apache/opendal/issues/4248#issuecomment-1960655969

   > I'm trying to figure out if it's best to share a single `Arc<Operator>` among all these tasks
   
   `Operator` itself is an `Arc`. We don't need to have `Arc<Operator>`.
   
   > if we should be giving each task its own `Operator` instance.
   
   It depends. For services like s3, they have seperate http clients; For services like memory, their data are not shared; For services like fs, they are the same.
   
   In general, we don't need to giving each task its own `Operator` instance if they are backed by the same storage.
   
   > I've been feeling like the Memory Service is kinda slow, especially with lotfs of data?
   
   Yes, the memory service is a basic `Arc<Mutex<HashMap<String, Vec<u8>>>>`, which isn't suitable for handling large amounts of data, especially when used concurrently. Maybe you could try `dashmap` or other storage services.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Best Practices for Concurrent Usage Doubts [opendal]

Posted by "xiaguan (via GitHub)" <gi...@apache.org>.
xiaguan commented on issue #4248:
URL: https://github.com/apache/opendal/issues/4248#issuecomment-1960679673

   I got it , thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Best Practices for Concurrent Usage Doubts [opendal]

Posted by "xiaguan (via GitHub)" <gi...@apache.org>.
xiaguan closed issue #4248: Best Practices for Concurrent Usage Doubts
URL: https://github.com/apache/opendal/issues/4248


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org