You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2019/07/16 16:41:02 UTC
[jira] [Updated] (SPARK-26498) Integrate barrier execution with
MMLSpark's LightGBM
[ https://issues.apache.org/jira/browse/SPARK-26498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-26498:
----------------------------------
Affects Version/s: (was: 2.4.0)
3.0.0
> Integrate barrier execution with MMLSpark's LightGBM
> ----------------------------------------------------
>
> Key: SPARK-26498
> URL: https://issues.apache.org/jira/browse/SPARK-26498
> Project: Spark
> Issue Type: New Feature
> Components: ML, MLlib
> Affects Versions: 3.0.0
> Reporter: Ilya Matiach
> Priority: Major
>
> I would like to use the new barrier execution mode introduced in spark 2.4 with LightGBM in the spark package mmlspark but I ran into some issues.
> Currently, the LightGBM distributed learner tries to figure out the number of cores on the cluster and then does a coalesce and a mapPartitions, and inside the mapPartitions we do a NetworkInit (where the address:port of all workers needs to be passed in the constructor) and pass the data in-memory to the native layer of the distributed lightgbm learner.
> With barrier execution mode, I think the code would become much more robust. However, there are several issues that I am running into when trying to move my code over to the new barrier execution mode scheduler:
> Does not support dynamic allocation – however, I think it would be convenient if it restarted the job when the number of workers has decreased and allowed the dev to decide whether to restart the job if the number of workers increased
> Does not work with DataFrame or Dataset API, but I think it would be much more convenient if it did.
> How does barrier execution mode deal with #partitions > #tasks? If the number of partitions is larger than the number of “tasks” or workers, can barrier execution mode automatically coalesce the dataset to have # partitions == # tasks?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org