[jira] [Commented] (MESOS-8587) Introduce a parallel for each loop (and other parallel algorithms).

["Alexander Rukletsov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-8587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559671#comment-16559671 ] 

Alexander Rukletsov commented on MESOS-8587:
--------------------------------------------

An alternative suggestion built around parallel {{async}} idea:
{noformat}
Future<Nothing> async(const C& items, const F& f)
{
  std::vector<Future<Nothing>> futures;

  foreach (const T& item, items) {
    Future<Nothing> finished = process::async(f, item);
    futures.push_back(finished);
  }

  return process::collect(futures).then([] { return Nothing(); });
}
{noformat}
Can be used for example like this:
{noformat}
Future<Nothing> finished = process::async(
    requests,
    [this](const Request& request) {
  // process the request and send the response
});

// Block the actor until all workers have generated responses.
finished.await();
{noformat}

> Introduce a parallel for each loop (and other parallel algorithms).
> -------------------------------------------------------------------
>
>                 Key: MESOS-8587
>                 URL: https://issues.apache.org/jira/browse/MESOS-8587
>             Project: Mesos
>          Issue Type: Improvement
>          Components: libprocess
>            Reporter: Benjamin Mahler
>            Priority: Major
>              Labels: performance
>
> Consider the following code:
> {code}
> SomeProcess::func()
> {
>   foreach (const Item& item, items) {
>     // Perform some const work on item.
>   }
> }
> {code}
> When {{items}} becomes very large, this code would benefit from some parallelism. With a parallel loop construct, we could improve the performance of this type of code significantly:
> {code}
> SomeProcess::func()
> {
>   foreach_parallel (items, [=](const Item& item) {
>     // Perform some const work on item.
>   });
> }
> {code}
> Ideally, this could enforce const-access to the current Process for safety. An implementation of this would need to do something like:
> # Split the iteration of {{items}} into 1 <= N <= num_worker_threads segments.
> # Spawn N-1 additional temporary execution Processes (or re-use from a pool)
> # Dispatch to these N-1 additional processes for them to perform their segment of the iteration.
> # Perform the 1st segment on the current Process.
> # Have the current Process block to wait for the others to finish. (note need to avoid deadlocking the worker threads here! See MESOS-8256)
> An alternative implementation would be to pull work from a shared queue:
> # Split the iteration of {{items}} into 1 <= N <= num_worker_threads segments. Store these segments in a lock free queue.
> # Spawn N-1 additional temporary execution Processes (or re-use from a pool)
> # Perform the 1st segment on the current Process.
> # Each process pulls a segment from the queue and executes a segment.
> # If the current Process finds the queue empty, it then needs to block waiting for outstanding segments to finish. Note that this cannot deadlock: if an item was pulled from the queue, a worker is executing it.
> An example use case for this is the task reconciliation loops in the master:
> https://github.com/apache/mesos/blob/1.5.0/src/master/master.cpp#L8385-L8419
> This generalizes to many other algorithms rather than just iteration. It may be good to align this with the C++ Parallelism TS, which shows how many of the C++ algorithms have potential for parallel counterparts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)