You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Gaurav1809 <ga...@gmail.com> on 2017/07/06 04:03:21 UTC
Re: Do we anything for Deep Learning in Spark?
Thanks Roope for the inputs.
On Wed, Jul 5, 2017 at 11:41 PM, Roope [via Apache Spark User List] <
ml+s1001560n28824h4@n3.nabble.com> wrote:
> Microsoft Machine Learning Library for Apache Spark lets you run CNTK deep
> learning models on Spark.
>
> https://github.com/Azure/mmlspark
>
> The library APIs are focused on image processing scenarios, and are
> compatible with SparkML Pipelines.
>
> Cheers,
> Roope - Microsoft Cloud AI Team
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-
> anything-for-Deep-Learning-in-Spark-tp28772p28824.html
> To start a new topic under Apache Spark User List, email
> ml+s1001560n1h91@n3.nabble.com
> To unsubscribe from Do we anything for Deep Learning in Spark?, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=28772&code=Z2F1cmF2aHBhbmR5YUBnbWFpbC5jb218Mjg3NzJ8NTQ0MzUyNjA4>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-anything-for-Deep-Learning-in-Spark-tp28772p28826.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
RE: Do we anything for Deep Learning in Spark?
Posted by hosur narahari <hn...@gmail.com>.
Thank you.
Best Regards,
Hari
On 7 Jul 2017 3:59 a.m., "Roope Astala" <ro...@microsoft.com> wrote:
> You can use an attached GPU VM for DNN training, and do other processing
> on regular CPU nodes. You can even deallocate the GPU VM to save costs when
> not using it. The GPU branch has instructions how to set up such compute
> environment:
>
>
>
> https://github.com/Azure/mmlspark/tree/gpu#gpu-vm-setup
>
>
>
> Cheers,
>
> Roope – Microsoft Cloud AI Team
>
>
>
> *From:* hosur narahari [mailto:hnr1992@gmail.com]
> *Sent:* Thursday, July 6, 2017 1:54 AM
> *To:* Gaurav1809 <ga...@gmail.com>
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: Do we anything for Deep Learning in Spark?
>
>
>
> Hi Roope,
>
>
>
> Does this mmlspark project uses GPGPU for processing and just CPU cores
> since DL models are computationally very intensive.
>
>
>
> Best Regards,
>
> Hari
>
>
>
> On 6 Jul 2017 9:33 a.m., "Gaurav1809" <ga...@gmail.com> wrote:
>
> Thanks Roope for the inputs.
>
>
>
> On Wed, Jul 5, 2017 at 11:41 PM, Roope [via Apache Spark User List] <[hidden
> email] <http://user/SendEmail.jtp?type=node&node=28826&i=0>> wrote:
>
> Microsoft Machine Learning Library for Apache Spark lets you run CNTK deep
> learning models on Spark.
>
> https://github.com/Azure/mmlspark
> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fmmlspark&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=1FqnrRsLY6JrPb%2BdkG4AlLBN%2F8f2O6w3Ehi9DYvu6lc%3D&reserved=0>
>
> The library APIs are focused on image processing scenarios, and are
> compatible with SparkML Pipelines.
>
> Cheers,
> Roope - Microsoft Cloud AI Team
> ------------------------------
>
> *If you reply to this email, your message will be added to the discussion
> below:*
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-
> anything-for-Deep-Learning-in-Spark-tp28772p28824.html
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2FDo-we-anything-for-Deep-Learning-in-Spark-tp28772p28824.html&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=zFajYYH16xwzDRwgP1se9m%2BAA5fIufUS7FFxHjzTOl0%3D&reserved=0>
>
> To start a new topic under Apache Spark User List, email [hidden email]
> <http://user/SendEmail.jtp?type=node&node=28826&i=1>
> To unsubscribe from Do we anything for Deep Learning in Spark?, click here.
> NAML
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2Ftemplate%2FNamlServlet.jtp%3Fmacro%3Dmacro_viewer%26id%3Dinstant_html%2521nabble%253Aemail.naml%26base%3Dnabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace%26breadcrumbs%3Dnotify_subscribers%2521nabble%253Aemail.naml-instant_emails%2521nabble%253Aemail.naml-send_instant_email%2521nabble%253Aemail.naml&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=wzDT2AoJRfwqxlumkdoRlGq6b0PaeoypZtvvLgFlR1M%3D&reserved=0>
>
>
>
>
> ------------------------------
>
> View this message in context: Re: Do we anything for Deep Learning in
> Spark?
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2FDo-we-anything-for-Deep-Learning-in-Spark-tp28772p28826.html&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=dDprNB1TSm9WkAY1iuN1NjkG7vzx6JaiXB42DcXqUt4%3D&reserved=0>
> Sent from the Apache Spark User List mailing list archive
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2F&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=bI5METH7hpg6xCjUQF0%2BiIKY7qFy709YGC4g0n9MjW0%3D&reserved=0>
> at Nabble.com.
>
>
RE: Do we anything for Deep Learning in Spark?
Posted by Roope Astala <ro...@microsoft.com.INVALID>.
You can use an attached GPU VM for DNN training, and do other processing on regular CPU nodes. You can even deallocate the GPU VM to save costs when not using it. The GPU branch has instructions how to set up such compute environment:
https://github.com/Azure/mmlspark/tree/gpu#gpu-vm-setup
Cheers,
Roope – Microsoft Cloud AI Team
From: hosur narahari [mailto:hnr1992@gmail.com]
Sent: Thursday, July 6, 2017 1:54 AM
To: Gaurav1809 <ga...@gmail.com>
Cc: user <us...@spark.apache.org>
Subject: Re: Do we anything for Deep Learning in Spark?
Hi Roope,
Does this mmlspark project uses GPGPU for processing and just CPU cores since DL models are computationally very intensive.
Best Regards,
Hari
On 6 Jul 2017 9:33 a.m., "Gaurav1809" <ga...@gmail.com>> wrote:
Thanks Roope for the inputs.
On Wed, Jul 5, 2017 at 11:41 PM, Roope [via Apache Spark User List] <[hidden email]<http://user/SendEmail.jtp?type=node&node=28826&i=0>> wrote:
Microsoft Machine Learning Library for Apache Spark lets you run CNTK deep learning models on Spark.
https://github.com/Azure/mmlspark<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fmmlspark&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=1FqnrRsLY6JrPb%2BdkG4AlLBN%2F8f2O6w3Ehi9DYvu6lc%3D&reserved=0>
The library APIs are focused on image processing scenarios, and are compatible with SparkML Pipelines.
Cheers,
Roope - Microsoft Cloud AI Team
________________________________
If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-anything-for-Deep-Learning-in-Spark-tp28772p28824.html<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2FDo-we-anything-for-Deep-Learning-in-Spark-tp28772p28824.html&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=zFajYYH16xwzDRwgP1se9m%2BAA5fIufUS7FFxHjzTOl0%3D&reserved=0>
To start a new topic under Apache Spark User List, email [hidden email]<http://user/SendEmail.jtp?type=node&node=28826&i=1>
To unsubscribe from Do we anything for Deep Learning in Spark?, click here.
NAML<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2Ftemplate%2FNamlServlet.jtp%3Fmacro%3Dmacro_viewer%26id%3Dinstant_html%2521nabble%253Aemail.naml%26base%3Dnabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace%26breadcrumbs%3Dnotify_subscribers%2521nabble%253Aemail.naml-instant_emails%2521nabble%253Aemail.naml-send_instant_email%2521nabble%253Aemail.naml&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=wzDT2AoJRfwqxlumkdoRlGq6b0PaeoypZtvvLgFlR1M%3D&reserved=0>
________________________________
View this message in context: Re: Do we anything for Deep Learning in Spark?<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2FDo-we-anything-for-Deep-Learning-in-Spark-tp28772p28826.html&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=dDprNB1TSm9WkAY1iuN1NjkG7vzx6JaiXB42DcXqUt4%3D&reserved=0>
Sent from the Apache Spark User List mailing list archive<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-spark-user-list.1001560.n3.nabble.com%2F&data=02%7C01%7Croastala%40microsoft.com%7Cc8e41a789e9d4837c81408d4c4337609%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636349172685703933&sdata=bI5METH7hpg6xCjUQF0%2BiIKY7qFy709YGC4g0n9MjW0%3D&reserved=0> at Nabble.com.
Re: Do we anything for Deep Learning in Spark?
Posted by hosur narahari <hn...@gmail.com>.
Hi Roope,
Does this mmlspark project uses GPGPU for processing and just CPU cores
since DL models are computationally very intensive.
Best Regards,
Hari
On 6 Jul 2017 9:33 a.m., "Gaurav1809" <ga...@gmail.com> wrote:
> Thanks Roope for the inputs.
>
> On Wed, Jul 5, 2017 at 11:41 PM, Roope [via Apache Spark User List] <[hidden
> email] <http:///user/SendEmail.jtp?type=node&node=28826&i=0>> wrote:
>
>> Microsoft Machine Learning Library for Apache Spark lets you run CNTK
>> deep learning models on Spark.
>>
>> https://github.com/Azure/mmlspark
>>
>> The library APIs are focused on image processing scenarios, and are
>> compatible with SparkML Pipelines.
>>
>> Cheers,
>> Roope - Microsoft Cloud AI Team
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-an
>> ything-for-Deep-Learning-in-Spark-tp28772p28824.html
>> To start a new topic under Apache Spark User List, email [hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=28826&i=1>
>> To unsubscribe from Do we anything for Deep Learning in Spark?, click
>> here.
>> NAML
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Do we anything for Deep Learning in
> Spark?
> <http://apache-spark-user-list.1001560.n3.nabble.com/Do-we-anything-for-Deep-Learning-in-Spark-tp28772p28826.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>