You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Xiangrui Meng (Jira)" <ji...@apache.org> on 2020/01/28 19:31:00 UTC

[jira] [Created] (SPARK-30667) Support simple all gather in barrier task context

Xiangrui Meng created SPARK-30667:
-------------------------------------

             Summary: Support simple all gather in barrier task context
                 Key: SPARK-30667
                 URL: https://issues.apache.org/jira/browse/SPARK-30667
             Project: Spark
          Issue Type: New Feature
          Components: PySpark, Spark Core
    Affects Versions: 3.0.0
            Reporter: Xiangrui Meng


Currently we offer task.barrier() to coordinate tasks in barrier mode. Tasks can see all IP addresses from BarrierTaskContext. It would be simpler to integrate with distributed frameworks like TensorFlow DistributionStrategy if we provide all gather that can let tasks share additional information with others, e.g., an available port.

Note that with all gather, tasks are share their IP addresses as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org