You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Bobby <wb...@gmail.com> on 2025/01/21 07:13:51 UTC
Re: [DISCUSS] Support spark.ml on Spark Connect
Hi all,
Thank you for reviewing and merging PR #48791
<https://github.com/apache/spark/pull/48791>, which enables running spark.ml
on Connect starting from Spark 4.0. With this PR, it becomes extremely easy
to support third-party Spark ML libraries on Connect.
I also want to highlight PR #49503
<https://github.com/apache/spark/pull/49503>, which adds support for the
Connect ML backend. This PR allows users to leverage GPU-accelerated
spark-ml libraries to speed up the built-in spark.ml algorithms without
requiring any changes to user code.
You can find the full story at SPARK-50812
<https://issues.apache.org/jira/browse/SPARK-50812>.
Thank you again for your feedback and for merging these PRs!
Best Regards,
Bobby Wang
Mich Talebzadeh <mi...@gmail.com> 于2024年11月13日周三 21:17写道：
> OK I added a comment to PR
>
> HTH,
>
> Mich Talebzadeh,
>
> Architect | Data Engineer | Data Science | Financial Crime
> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College
> London <https://en.wikipedia.org/wiki/Imperial_College_London>
> London, United Kingdom
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* The information provided is correct to the best of my
> knowledge but of course cannot be guaranteed . It is essential to note
> that, as with any advice, quote "one test result is worth one-thousand
> expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von
> Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>
>
> On Wed, 13 Nov 2024 at 09:17, Bobby <wb...@gmail.com> wrote:
>
>> Hi all,
>>
>> Finally, I got the draft PR up https://github.com/apache/spark/pull/48791
>> last week, but I would like to send it out here and hope to get
>> your feedback and reviews.
>>
>> Thx very much.
>>
>> Mich Talebzadeh <mi...@gmail.com> 于2024年10月15日周二 15:47写道：
>>
>>> +1
>>>
>>> It will be a desirable feature
>>>
>>> Mich Talebzadeh,
>>>
>>> Architect | Data Engineer | Data Science | Financial Crime
>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial
>>> College London <https://en.wikipedia.org/wiki/Imperial_College_London>
>>> London, United Kingdom
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* The information provided is correct to the best of my
>>> knowledge but of course cannot be guaranteed . It is essential to note
>>> that, as with any advice, quote "one test result is worth one-thousand
>>> expert opinions (Werner
>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun
>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>>>
>>>
>>> On Tue, 15 Oct 2024 at 08:24, Bobby <wb...@gmail.com> wrote:
>>>
>>>> Thank you for your kind response. I will prepare a formal PR for Spark.
>>>>
>>>>
>>>>
>>>> Niranjan Jayakar <ni...@databricks.com> 于2024年10月11日周五 22:45写道：
>>>>
>>>>> +1
>>>>>
>>>>> On Thu, Oct 10, 2024 at 5:28 PM Xiao Li <ga...@gmail.com> wrote:
>>>>>
>>>>>> Thank you for working on this!
>>>>>>
>>>>>> Xiao
>>>>>>
>>>>>> Martin Grund <ma...@databricks.com.invalid> 于2024年10月10日周四 03:01写道：
>>>>>>
>>>>>>>
>>>>>>> Hi Bobby,
>>>>>>>
>>>>>>> Awesome to see the proposal! I'm very much looking forward to the
>>>>>>> contributions!
>>>>>>>
>>>>>>> Martin
>>>>>>>
>>>>>>> On Thu, Oct 10, 2024 at 4:16 AM Ángel <
>>>>>>> angel.alvarez.pascua@gmail.com> wrote:
>>>>>>>
>>>>>>>> You have my vote (btw, great idea, ML is so sexy nowadays 😉)
>>>>>>>>
>>>>>>>> El jue, 10 oct 2024 a las 3:19, Bobby (<wb...@gmail.com>)
>>>>>>>> escribió:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'd like to start a discussion about support spark.ml on Connect.
>>>>>>>>> With this feature, Users don't need to change their code to run Spark ML
>>>>>>>>> cases on Connect.
>>>>>>>>>
>>>>>>>>> Please refer to the JIRA:
>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-49907
>>>>>>>>>
>>>>>>>>> Thx,
>>>>>>>>> Bobby Wang
>>>>>>>>>
>>>>>>>>
Re: [DISCUSS] Support spark.ml on Spark Connect
Posted by Mich Talebzadeh <mi...@gmail.com>.
Given our recent discussion on using spark connect as a stable API, this
will be another positive step.
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Tue, 21 Jan 2025 at 07:44, Martin Grund <ma...@databricks.com> wrote:
> Awesome progress!
>
> On Tue, Jan 21, 2025 at 8:15 AM Bobby <wb...@gmail.com> wrote:
>
>> Hi all,
>>
>> Thank you for reviewing and merging PR #48791
>> <https://github.com/apache/spark/pull/48791>, which enables running
>> spark.ml on Connect starting from Spark 4.0. With this PR, it becomes
>> extremely easy to support third-party Spark ML libraries on Connect.
>>
>> I also want to highlight PR #49503
>> <https://github.com/apache/spark/pull/49503>, which adds support for the
>> Connect ML backend. This PR allows users to leverage GPU-accelerated
>> spark-ml libraries to speed up the built-in spark.ml algorithms without
>> requiring any changes to user code.
>>
>> You can find the full story at SPARK-50812
>> <https://issues.apache.org/jira/browse/SPARK-50812>.
>>
>> Thank you again for your feedback and for merging these PRs!
>>
>> Best Regards,
>> Bobby Wang
>>
>> Mich Talebzadeh <mi...@gmail.com> 于2024年11月13日周三 21:17写道：
>>
>>> OK I added a comment to PR
>>>
>>> HTH,
>>>
>>> Mich Talebzadeh,
>>>
>>> Architect | Data Engineer | Data Science | Financial Crime
>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial
>>> College London <https://en.wikipedia.org/wiki/Imperial_College_London>
>>> London, United Kingdom
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* The information provided is correct to the best of my
>>> knowledge but of course cannot be guaranteed . It is essential to note
>>> that, as with any advice, quote "one test result is worth one-thousand
>>> expert opinions (Werner
>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun
>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>>>
>>>
>>> On Wed, 13 Nov 2024 at 09:17, Bobby <wb...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Finally, I got the draft PR up
>>>> https://github.com/apache/spark/pull/48791 last week, but I would like
>>>> to send it out here and hope to get your feedback and reviews.
>>>>
>>>> Thx very much.
>>>>
>>>> Mich Talebzadeh <mi...@gmail.com> 于2024年10月15日周二 15:47写道：
>>>>
>>>>> +1
>>>>>
>>>>> It will be a desirable feature
>>>>>
>>>>> Mich Talebzadeh,
>>>>>
>>>>> Architect | Data Engineer | Data Science | Financial Crime
>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial
>>>>> College London <https://en.wikipedia.org/wiki/Imperial_College_London>
>>>>>
>>>>> London, United Kingdom
>>>>>
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> *Disclaimer:* The information provided is correct to the best of my
>>>>> knowledge but of course cannot be guaranteed . It is essential to note
>>>>> that, as with any advice, quote "one test result is worth one-thousand
>>>>> expert opinions (Werner
>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun
>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>>>>>
>>>>>
>>>>> On Tue, 15 Oct 2024 at 08:24, Bobby <wb...@gmail.com> wrote:
>>>>>
>>>>>> Thank you for your kind response. I will prepare a formal PR for
>>>>>> Spark.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Niranjan Jayakar <ni...@databricks.com> 于2024年10月11日周五 22:45写道：
>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> On Thu, Oct 10, 2024 at 5:28 PM Xiao Li <ga...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thank you for working on this!
>>>>>>>>
>>>>>>>> Xiao
>>>>>>>>
>>>>>>>> Martin Grund <ma...@databricks.com.invalid> 于2024年10月10日周四
>>>>>>>> 03:01写道：
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Bobby,
>>>>>>>>>
>>>>>>>>> Awesome to see the proposal! I'm very much looking forward to the
>>>>>>>>> contributions!
>>>>>>>>>
>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>> On Thu, Oct 10, 2024 at 4:16 AM Ángel <
>>>>>>>>> angel.alvarez.pascua@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> You have my vote (btw, great idea, ML is so sexy nowadays 😉)
>>>>>>>>>>
>>>>>>>>>> El jue, 10 oct 2024 a las 3:19, Bobby (<wb...@gmail.com>)
>>>>>>>>>> escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I'd like to start a discussion about support spark.ml on
>>>>>>>>>>> Connect. With this feature, Users don't need to change their code to run
>>>>>>>>>>> Spark ML cases on Connect.
>>>>>>>>>>>
>>>>>>>>>>> Please refer to the JIRA:
>>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-49907
>>>>>>>>>>>
>>>>>>>>>>> Thx,
>>>>>>>>>>> Bobby Wang
>>>>>>>>>>>
>>>>>>>>>>
Re: [DISCUSS] Support spark.ml on Spark Connect
Posted by Martin Grund <ma...@databricks.com.INVALID>.
Awesome progress!
On Tue, Jan 21, 2025 at 8:15 AM Bobby <wb...@gmail.com> wrote:
> Hi all,
>
> Thank you for reviewing and merging PR #48791
> <https://github.com/apache/spark/pull/48791>, which enables running
> spark.ml on Connect starting from Spark 4.0. With this PR, it becomes
> extremely easy to support third-party Spark ML libraries on Connect.
>
> I also want to highlight PR #49503
> <https://github.com/apache/spark/pull/49503>, which adds support for the
> Connect ML backend. This PR allows users to leverage GPU-accelerated
> spark-ml libraries to speed up the built-in spark.ml algorithms without
> requiring any changes to user code.
>
> You can find the full story at SPARK-50812
> <https://issues.apache.org/jira/browse/SPARK-50812>.
>
> Thank you again for your feedback and for merging these PRs!
>
> Best Regards,
> Bobby Wang
>
> Mich Talebzadeh <mi...@gmail.com> 于2024年11月13日周三 21:17写道：
>
>> OK I added a comment to PR
>>
>> HTH,
>>
>> Mich Talebzadeh,
>>
>> Architect | Data Engineer | Data Science | Financial Crime
>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial
>> College London <https://en.wikipedia.org/wiki/Imperial_College_London>
>> London, United Kingdom
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* The information provided is correct to the best of my
>> knowledge but of course cannot be guaranteed . It is essential to note
>> that, as with any advice, quote "one test result is worth one-thousand
>> expert opinions (Werner
>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun
>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>>
>>
>> On Wed, 13 Nov 2024 at 09:17, Bobby <wb...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> Finally, I got the draft PR up
>>> https://github.com/apache/spark/pull/48791 last week, but I would like
>>> to send it out here and hope to get your feedback and reviews.
>>>
>>> Thx very much.
>>>
>>> Mich Talebzadeh <mi...@gmail.com> 于2024年10月15日周二 15:47写道：
>>>
>>>> +1
>>>>
>>>> It will be a desirable feature
>>>>
>>>> Mich Talebzadeh,
>>>>
>>>> Architect | Data Engineer | Data Science | Financial Crime
>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial
>>>> College London <https://en.wikipedia.org/wiki/Imperial_College_London>
>>>> London, United Kingdom
>>>>
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>
>>>>
>>>>
>>>> *Disclaimer:* The information provided is correct to the best of my
>>>> knowledge but of course cannot be guaranteed . It is essential to note
>>>> that, as with any advice, quote "one test result is worth one-thousand
>>>> expert opinions (Werner
>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun
>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)".
>>>>
>>>>
>>>> On Tue, 15 Oct 2024 at 08:24, Bobby <wb...@gmail.com> wrote:
>>>>
>>>>> Thank you for your kind response. I will prepare a formal PR for Spark.
>>>>>
>>>>>
>>>>>
>>>>> Niranjan Jayakar <ni...@databricks.com> 于2024年10月11日周五 22:45写道：
>>>>>
>>>>>> +1
>>>>>>
>>>>>> On Thu, Oct 10, 2024 at 5:28 PM Xiao Li <ga...@gmail.com> wrote:
>>>>>>
>>>>>>> Thank you for working on this!
>>>>>>>
>>>>>>> Xiao
>>>>>>>
>>>>>>> Martin Grund <ma...@databricks.com.invalid> 于2024年10月10日周四 03:01写道：
>>>>>>>
>>>>>>>>
>>>>>>>> Hi Bobby,
>>>>>>>>
>>>>>>>> Awesome to see the proposal! I'm very much looking forward to the
>>>>>>>> contributions!
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>> On Thu, Oct 10, 2024 at 4:16 AM Ángel <
>>>>>>>> angel.alvarez.pascua@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> You have my vote (btw, great idea, ML is so sexy nowadays 😉)
>>>>>>>>>
>>>>>>>>> El jue, 10 oct 2024 a las 3:19, Bobby (<wb...@gmail.com>)
>>>>>>>>> escribió:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I'd like to start a discussion about support spark.ml on
>>>>>>>>>> Connect. With this feature, Users don't need to change their code to run
>>>>>>>>>> Spark ML cases on Connect.
>>>>>>>>>>
>>>>>>>>>> Please refer to the JIRA:
>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-49907
>>>>>>>>>>
>>>>>>>>>> Thx,
>>>>>>>>>> Bobby Wang
>>>>>>>>>>
>>>>>>>>>