You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Daniel Galvez <dt...@gmail.com> on 2018/05/16 05:58:06 UTC
Interest in adding ability to request GPU's to the spark client?
Hi all,
Is anyone here interested in adding the ability to request GPUs to Spark's
client (i.e, spark-submit)? As of now, Yarn 3.0's resource manager server
has the ability to schedule GPUs as resources via cgroups, but the Spark
client lacks an ability to request these.
The ability to guarantee GPU resources would be practically useful for my
organization. Right now, the only way to do that is to request the entire
memory (or all CPU's) on a node, which is very kludgey and wastes
resources, especially if your node has more than 1 GPU and your code was
written such that an executor can use only one GPU at a time.
I'm just not sure of a good way to make use of libraries like Databricks' Deep
Learning pipelines <https://github.com/databricks/spark-deep-learning> for
GPU-heavy computation otherwise, unless you are luckily in an organization
which is able to virtualize computer nodes such that each node will have
only one GPU. Of course, I realize that many Databricks customers are using
Azure or AWS, which allow you to do this facilely. Is this what people
normally do in industry?
This is something I am interested in working on, unless others out there
have advice on why this is a bad idea.
Unfortunately, I am not familiar enough with Mesos and Kubernetes right now
to know how they schedule gpu resources and whether adding support for
requesting GPU's from them to the spark-submit client would be simple.
Daniel
--
Daniel Galvez
http://danielgalvez.me
https://github.com/galv
Re: Interest in adding ability to request GPU's to the spark client?
Posted by "Susan X. Huynh" <xh...@mesosphere.io>.
There's some discussion and proposal of supporting GPUs in this Spark JIRA:
https://jira.apache.org/jira/browse/SPARK-24615 "Accelerator-aware task
scheduling for Spark"
Susan
On Thu, Jul 12, 2018 at 11:17 AM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> wrote:
> I agree.
>
> Adding GPU capability to Spark in my opinion is a must for Advanced
> Analytics.
>
> HTH
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Thu, 12 Jul 2018 at 19:14, Maximiliano Felice <
> maximilianofelice@gmail.com> wrote:
>
>> Hi,
>>
>> I've been meaning to reply to this email for a while now, sorry for
>> taking so much time.
>>
>> I personally think that adding GPU resource management will allow us to
>> boost some ETL performance a lot. For the last year, I've worked in
>> transforming some Machine Learning pipelines from Python in Numpy/Pandas to
>> Spark. Adding GPU capabilities to Spark would:
>>
>>
>> - Accelerate many matrix and batch computations we currently have in
>> Tensorflow
>> - Allow us to use spark for the whole pipeline (combined with
>> possibly better online serving)
>> - Let us trigger better Hyperparameter selection directly from Spark
>>
>>
>> There will be many more aspects of this that we could explode. What do
>> the rest of the list think?
>>
>> See you
>>
>> El mié., 16 may. 2018 a las 2:58, Daniel Galvez (<dt...@gmail.com>)
>> escribió:
>>
>>> Hi all,
>>>
>>> Is anyone here interested in adding the ability to request GPUs to
>>> Spark's client (i.e, spark-submit)? As of now, Yarn 3.0's resource manager
>>> server has the ability to schedule GPUs as resources via cgroups, but the
>>> Spark client lacks an ability to request these.
>>>
>>> The ability to guarantee GPU resources would be practically useful for
>>> my organization. Right now, the only way to do that is to request the
>>> entire memory (or all CPU's) on a node, which is very kludgey and wastes
>>> resources, especially if your node has more than 1 GPU and your code was
>>> written such that an executor can use only one GPU at a time.
>>>
>>> I'm just not sure of a good way to make use of libraries like
>>> Databricks' Deep Learning pipelines
>>> <https://github.com/databricks/spark-deep-learning> for GPU-heavy
>>> computation otherwise, unless you are luckily in an organization which is
>>> able to virtualize computer nodes such that each node will have only one
>>> GPU. Of course, I realize that many Databricks customers are using Azure or
>>> AWS, which allow you to do this facilely. Is this what people normally do
>>> in industry?
>>>
>>> This is something I am interested in working on, unless others out there
>>> have advice on why this is a bad idea.
>>>
>>> Unfortunately, I am not familiar enough with Mesos and Kubernetes right
>>> now to know how they schedule gpu resources and whether adding support for
>>> requesting GPU's from them to the spark-submit client would be simple.
>>>
>>> Daniel
>>>
>>> --
>>> Daniel Galvez
>>> http://danielgalvez.me
>>> https://github.com/galv
>>>
>>
--
Susan X. Huynh
Software engineer, Data Agility
xhuynh@mesosphere.com
Re: Interest in adding ability to request GPU's to the spark client?
Posted by Mich Talebzadeh <mi...@gmail.com>.
I agree.
Adding GPU capability to Spark in my opinion is a must for Advanced
Analytics.
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Thu, 12 Jul 2018 at 19:14, Maximiliano Felice <
maximilianofelice@gmail.com> wrote:
> Hi,
>
> I've been meaning to reply to this email for a while now, sorry for taking
> so much time.
>
> I personally think that adding GPU resource management will allow us to
> boost some ETL performance a lot. For the last year, I've worked in
> transforming some Machine Learning pipelines from Python in Numpy/Pandas to
> Spark. Adding GPU capabilities to Spark would:
>
>
> - Accelerate many matrix and batch computations we currently have in
> Tensorflow
> - Allow us to use spark for the whole pipeline (combined with possibly
> better online serving)
> - Let us trigger better Hyperparameter selection directly from Spark
>
>
> There will be many more aspects of this that we could explode. What do the
> rest of the list think?
>
> See you
>
> El mié., 16 may. 2018 a las 2:58, Daniel Galvez (<dt...@gmail.com>)
> escribió:
>
>> Hi all,
>>
>> Is anyone here interested in adding the ability to request GPUs to
>> Spark's client (i.e, spark-submit)? As of now, Yarn 3.0's resource manager
>> server has the ability to schedule GPUs as resources via cgroups, but the
>> Spark client lacks an ability to request these.
>>
>> The ability to guarantee GPU resources would be practically useful for my
>> organization. Right now, the only way to do that is to request the entire
>> memory (or all CPU's) on a node, which is very kludgey and wastes
>> resources, especially if your node has more than 1 GPU and your code was
>> written such that an executor can use only one GPU at a time.
>>
>> I'm just not sure of a good way to make use of libraries like Databricks' Deep
>> Learning pipelines <https://github.com/databricks/spark-deep-learning>
>> for GPU-heavy computation otherwise, unless you are luckily in an
>> organization which is able to virtualize computer nodes such that each node
>> will have only one GPU. Of course, I realize that many Databricks customers
>> are using Azure or AWS, which allow you to do this facilely. Is this what
>> people normally do in industry?
>>
>> This is something I am interested in working on, unless others out there
>> have advice on why this is a bad idea.
>>
>> Unfortunately, I am not familiar enough with Mesos and Kubernetes right
>> now to know how they schedule gpu resources and whether adding support for
>> requesting GPU's from them to the spark-submit client would be simple.
>>
>> Daniel
>>
>> --
>> Daniel Galvez
>> http://danielgalvez.me
>> https://github.com/galv
>>
>
Re: Interest in adding ability to request GPU's to the spark client?
Posted by Maximiliano Felice <ma...@gmail.com>.
Hi,
I've been meaning to reply to this email for a while now, sorry for taking
so much time.
I personally think that adding GPU resource management will allow us to
boost some ETL performance a lot. For the last year, I've worked in
transforming some Machine Learning pipelines from Python in Numpy/Pandas to
Spark. Adding GPU capabilities to Spark would:
- Accelerate many matrix and batch computations we currently have in
Tensorflow
- Allow us to use spark for the whole pipeline (combined with possibly
better online serving)
- Let us trigger better Hyperparameter selection directly from Spark
There will be many more aspects of this that we could explode. What do the
rest of the list think?
See you
El mié., 16 may. 2018 a las 2:58, Daniel Galvez (<dt...@gmail.com>)
escribió:
> Hi all,
>
> Is anyone here interested in adding the ability to request GPUs to Spark's
> client (i.e, spark-submit)? As of now, Yarn 3.0's resource manager server
> has the ability to schedule GPUs as resources via cgroups, but the Spark
> client lacks an ability to request these.
>
> The ability to guarantee GPU resources would be practically useful for my
> organization. Right now, the only way to do that is to request the entire
> memory (or all CPU's) on a node, which is very kludgey and wastes
> resources, especially if your node has more than 1 GPU and your code was
> written such that an executor can use only one GPU at a time.
>
> I'm just not sure of a good way to make use of libraries like Databricks' Deep
> Learning pipelines <https://github.com/databricks/spark-deep-learning>
> for GPU-heavy computation otherwise, unless you are luckily in an
> organization which is able to virtualize computer nodes such that each node
> will have only one GPU. Of course, I realize that many Databricks customers
> are using Azure or AWS, which allow you to do this facilely. Is this what
> people normally do in industry?
>
> This is something I am interested in working on, unless others out there
> have advice on why this is a bad idea.
>
> Unfortunately, I am not familiar enough with Mesos and Kubernetes right
> now to know how they schedule gpu resources and whether adding support for
> requesting GPU's from them to the spark-submit client would be simple.
>
> Daniel
>
> --
> Daniel Galvez
> http://danielgalvez.me
> https://github.com/galv
>