Re: [I] refactor: piplineutil async mode [geaflow]

["cbqiao (via GitHub)" <gi...@apache.org>]

cbqiao commented on issue #607:
URL: https://github.com/apache/geaflow/issues/607#issuecomment-3485106133

   @kitalkuyo-gita Thank you very much for your proposal. It appears you have given some deep thought to the resource allocation aspect.
   
   I need to clarify that the async mode design here is intended for OLAP scenarios, where multiple drivers are created to provide higher concurrency for query processing. Each driver involves considerations about how resources should be allocated, which graph shards should be cached on each worker, and how tasks submitted from different drivers can share read access to worker shards during runtime.
   
   The current proposal focuses more on how drivers allocate resources based on queries, but it doesn't address the distribution of graph shards on workers or the shared read access issues. Therefore, there are some conflicts between this proposal and the original async mode problem we aimed to solve in this issue.
   
   If we only consider the offline graph computing scenario—similar to Spark's approach of processing multiple computing tasks continuously or in parallel—the resource allocation based on workload discussed above does have some value. For this scenarios, we'd better open a new issue to further discuss and break down the plan.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@geaflow.apache.org
For additional commands, e-mail: issues-help@geaflow.apache.org