Ammonite as REPL for Spark Connect

[Herman van Hovell <he...@databricks.com.INVALID>]

Hi All,

For Spark Connect Scala Client we are working on making the REPL experience
a bit nicer <https://github.com/apache/spark/pull/40515>. In a nutshell we
want to give users a turn key scala REPL, that works even if you don't have
a Spark distribution on your machine (through coursier
<https://get-coursier.io/>). We are using Ammonite <https://ammonite.io/>
instead of the standard scala REPL for this, the main reason for going with
Ammonite is that it is easier to customize, and IMO has a superior user
experience.

Does anyone object to doing this?

Kind regards,
Herman

Re: Ammonite as REPL for Spark Connect

[John Zhuge <jz...@apache.org>]

+1 on better notebook and other REPL experience

On Thu, Mar 23, 2023 at 9:17 AM Dongjoon Hyun <do...@gmail.com>
wrote:

> I also support Herman's `SPARK-42884 Add Ammonite REPL integration` PR.
>
> Thanks,
> Dongjoon.
>
>
> On Thu, Mar 23, 2023 at 7:51 AM Mridul Muralidharan <mr...@gmail.com>
> wrote:
>
>>
>> Sounds good, thanks for clarifying !
>>
>> Regards,
>> Mridul
>>
>> On Thu, Mar 23, 2023 at 9:09 AM Herman van Hovell <he...@databricks.com>
>> wrote:
>>
>>> The goal of adding this, is to make it easy for a user to connect a
>>> scala REPL to a Spark Connect server. Just like Spark shell makes it easy
>>> to work with a regular Spark environment.
>>>
>>> It is not meant as a Spark shell replacement. They represent two
>>> different modes of working with Spark, and they have very different API
>>> surfaces (Connect being a subset of what regular Spark has to offer). I do
>>> think we should consider using ammonite for Spark shell at some point,
>>> since this has better UX and does not require us to fork a REPL. That
>>> discussion is for another day though.
>>>
>>> I guess you can use it as an example of building an integration. In
>>> itself I wouldn't call it that because I think this a key part of getting
>>> started with connect, and/or doing debugging.
>>>
>>> On Thu, Mar 23, 2023 at 4:00 AM Mridul Muralidharan <mr...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> What is unclear to me is why we are introducing this integration, how
>>>> users will leverage it.
>>>>
>>>> * Are we replacing spark-shell with it ?
>>>> Given the existing gaps, this is not the case.
>>>>
>>>> * Is it an example to showcase how to build an integration ?
>>>> That could be interesting, and we can add it to external/
>>>>
>>>> Anything else I am missing ?
>>>>
>>>> Regards,
>>>> Mridul
>>>>
>>>>
>>>>
>>>> On Wed, Mar 22, 2023 at 6:58 PM Herman van Hovell <
>>>> herman@databricks.com> wrote:
>>>>
>>>>> Ammonite is maintained externally by Li Haoyi et al. We are including
>>>>> it as a 'provided' dependency. The integration bits and pieces (1 file) are
>>>>> included in Apache Spark.
>>>>>
>>>>> On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan <mr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Will this be maintained externally or included into Apache Spark ?
>>>>>>
>>>>>> Regards ,
>>>>>> Mridul
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>>>>>> <he...@databricks.com.invalid> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> For Spark Connect Scala Client we are working on making the REPL
>>>>>>> experience a bit nicer <https://github.com/apache/spark/pull/40515>.
>>>>>>> In a nutshell we want to give users a turn key scala REPL, that works even
>>>>>>> if you don't have a Spark distribution on your machine (through
>>>>>>> coursier <https://get-coursier.io/>). We are using Ammonite
>>>>>>> <https://ammonite.io/> instead of the standard scala REPL for this,
>>>>>>> the main reason for going with Ammonite is that it is easier to customize,
>>>>>>> and IMO has a superior user experience.
>>>>>>>
>>>>>>> Does anyone object to doing this?
>>>>>>>
>>>>>>> Kind regards,
>>>>>>> Herman
>>>>>>>
>>>>>>>

-- 
John Zhuge

Re: Ammonite as REPL for Spark Connect

[Dongjoon Hyun <do...@gmail.com>]

I also support Herman's `SPARK-42884 Add Ammonite REPL integration` PR.

Thanks,
Dongjoon.


On Thu, Mar 23, 2023 at 7:51 AM Mridul Muralidharan <mr...@gmail.com>
wrote:

>
> Sounds good, thanks for clarifying !
>
> Regards,
> Mridul
>
> On Thu, Mar 23, 2023 at 9:09 AM Herman van Hovell <he...@databricks.com>
> wrote:
>
>> The goal of adding this, is to make it easy for a user to connect a scala
>> REPL to a Spark Connect server. Just like Spark shell makes it easy to work
>> with a regular Spark environment.
>>
>> It is not meant as a Spark shell replacement. They represent two
>> different modes of working with Spark, and they have very different API
>> surfaces (Connect being a subset of what regular Spark has to offer). I do
>> think we should consider using ammonite for Spark shell at some point,
>> since this has better UX and does not require us to fork a REPL. That
>> discussion is for another day though.
>>
>> I guess you can use it as an example of building an integration. In
>> itself I wouldn't call it that because I think this a key part of getting
>> started with connect, and/or doing debugging.
>>
>> On Thu, Mar 23, 2023 at 4:00 AM Mridul Muralidharan <mr...@gmail.com>
>> wrote:
>>
>>>
>>> What is unclear to me is why we are introducing this integration, how
>>> users will leverage it.
>>>
>>> * Are we replacing spark-shell with it ?
>>> Given the existing gaps, this is not the case.
>>>
>>> * Is it an example to showcase how to build an integration ?
>>> That could be interesting, and we can add it to external/
>>>
>>> Anything else I am missing ?
>>>
>>> Regards,
>>> Mridul
>>>
>>>
>>>
>>> On Wed, Mar 22, 2023 at 6:58 PM Herman van Hovell <he...@databricks.com>
>>> wrote:
>>>
>>>> Ammonite is maintained externally by Li Haoyi et al. We are including
>>>> it as a 'provided' dependency. The integration bits and pieces (1 file) are
>>>> included in Apache Spark.
>>>>
>>>> On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan <mr...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> Will this be maintained externally or included into Apache Spark ?
>>>>>
>>>>> Regards ,
>>>>> Mridul
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>>>>> <he...@databricks.com.invalid> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> For Spark Connect Scala Client we are working on making the REPL
>>>>>> experience a bit nicer <https://github.com/apache/spark/pull/40515>.
>>>>>> In a nutshell we want to give users a turn key scala REPL, that works even
>>>>>> if you don't have a Spark distribution on your machine (through
>>>>>> coursier <https://get-coursier.io/>). We are using Ammonite
>>>>>> <https://ammonite.io/> instead of the standard scala REPL for this,
>>>>>> the main reason for going with Ammonite is that it is easier to customize,
>>>>>> and IMO has a superior user experience.
>>>>>>
>>>>>> Does anyone object to doing this?
>>>>>>
>>>>>> Kind regards,
>>>>>> Herman
>>>>>>
>>>>>>

Re: Ammonite as REPL for Spark Connect

[Mridul Muralidharan <mr...@gmail.com>]

Sounds good, thanks for clarifying !

Regards,
Mridul

On Thu, Mar 23, 2023 at 9:09 AM Herman van Hovell <he...@databricks.com>
wrote:

> The goal of adding this, is to make it easy for a user to connect a scala
> REPL to a Spark Connect server. Just like Spark shell makes it easy to work
> with a regular Spark environment.
>
> It is not meant as a Spark shell replacement. They represent two different
> modes of working with Spark, and they have very different API surfaces
> (Connect being a subset of what regular Spark has to offer). I do think we
> should consider using ammonite for Spark shell at some point, since this
> has better UX and does not require us to fork a REPL. That discussion is
> for another day though.
>
> I guess you can use it as an example of building an integration. In itself
> I wouldn't call it that because I think this a key part of getting started
> with connect, and/or doing debugging.
>
> On Thu, Mar 23, 2023 at 4:00 AM Mridul Muralidharan <mr...@gmail.com>
> wrote:
>
>>
>> What is unclear to me is why we are introducing this integration, how
>> users will leverage it.
>>
>> * Are we replacing spark-shell with it ?
>> Given the existing gaps, this is not the case.
>>
>> * Is it an example to showcase how to build an integration ?
>> That could be interesting, and we can add it to external/
>>
>> Anything else I am missing ?
>>
>> Regards,
>> Mridul
>>
>>
>>
>> On Wed, Mar 22, 2023 at 6:58 PM Herman van Hovell <he...@databricks.com>
>> wrote:
>>
>>> Ammonite is maintained externally by Li Haoyi et al. We are including it
>>> as a 'provided' dependency. The integration bits and pieces (1 file) are
>>> included in Apache Spark.
>>>
>>> On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan <mr...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> Will this be maintained externally or included into Apache Spark ?
>>>>
>>>> Regards ,
>>>> Mridul
>>>>
>>>>
>>>>
>>>> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>>>> <he...@databricks.com.invalid> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> For Spark Connect Scala Client we are working on making the REPL
>>>>> experience a bit nicer <https://github.com/apache/spark/pull/40515>.
>>>>> In a nutshell we want to give users a turn key scala REPL, that works even
>>>>> if you don't have a Spark distribution on your machine (through
>>>>> coursier <https://get-coursier.io/>). We are using Ammonite
>>>>> <https://ammonite.io/> instead of the standard scala REPL for this,
>>>>> the main reason for going with Ammonite is that it is easier to customize,
>>>>> and IMO has a superior user experience.
>>>>>
>>>>> Does anyone object to doing this?
>>>>>
>>>>> Kind regards,
>>>>> Herman
>>>>>
>>>>>

Re: Ammonite as REPL for Spark Connect

[Herman van Hovell <he...@databricks.com.INVALID>]

The goal of adding this, is to make it easy for a user to connect a scala
REPL to a Spark Connect server. Just like Spark shell makes it easy to work
with a regular Spark environment.

It is not meant as a Spark shell replacement. They represent two different
modes of working with Spark, and they have very different API surfaces
(Connect being a subset of what regular Spark has to offer). I do think we
should consider using ammonite for Spark shell at some point, since this
has better UX and does not require us to fork a REPL. That discussion is
for another day though.

I guess you can use it as an example of building an integration. In itself
I wouldn't call it that because I think this a key part of getting started
with connect, and/or doing debugging.

On Thu, Mar 23, 2023 at 4:00 AM Mridul Muralidharan <mr...@gmail.com>
wrote:

>
> What is unclear to me is why we are introducing this integration, how
> users will leverage it.
>
> * Are we replacing spark-shell with it ?
> Given the existing gaps, this is not the case.
>
> * Is it an example to showcase how to build an integration ?
> That could be interesting, and we can add it to external/
>
> Anything else I am missing ?
>
> Regards,
> Mridul
>
>
>
> On Wed, Mar 22, 2023 at 6:58 PM Herman van Hovell <he...@databricks.com>
> wrote:
>
>> Ammonite is maintained externally by Li Haoyi et al. We are including it
>> as a 'provided' dependency. The integration bits and pieces (1 file) are
>> included in Apache Spark.
>>
>> On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan <mr...@gmail.com>
>> wrote:
>>
>>>
>>> Will this be maintained externally or included into Apache Spark ?
>>>
>>> Regards ,
>>> Mridul
>>>
>>>
>>>
>>> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>>> <he...@databricks.com.invalid> wrote:
>>>
>>>> Hi All,
>>>>
>>>> For Spark Connect Scala Client we are working on making the REPL
>>>> experience a bit nicer <https://github.com/apache/spark/pull/40515>.
>>>> In a nutshell we want to give users a turn key scala REPL, that works even
>>>> if you don't have a Spark distribution on your machine (through
>>>> coursier <https://get-coursier.io/>). We are using Ammonite
>>>> <https://ammonite.io/> instead of the standard scala REPL for this,
>>>> the main reason for going with Ammonite is that it is easier to customize,
>>>> and IMO has a superior user experience.
>>>>
>>>> Does anyone object to doing this?
>>>>
>>>> Kind regards,
>>>> Herman
>>>>
>>>>

Re: Ammonite as REPL for Spark Connect

[Mridul Muralidharan <mr...@gmail.com>]

What is unclear to me is why we are introducing this integration, how users
will leverage it.

* Are we replacing spark-shell with it ?
Given the existing gaps, this is not the case.

* Is it an example to showcase how to build an integration ?
That could be interesting, and we can add it to external/

Anything else I am missing ?

Regards,
Mridul



On Wed, Mar 22, 2023 at 6:58 PM Herman van Hovell <he...@databricks.com>
wrote:

> Ammonite is maintained externally by Li Haoyi et al. We are including it
> as a 'provided' dependency. The integration bits and pieces (1 file) are
> included in Apache Spark.
>
> On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan <mr...@gmail.com>
> wrote:
>
>>
>> Will this be maintained externally or included into Apache Spark ?
>>
>> Regards ,
>> Mridul
>>
>>
>>
>> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>> <he...@databricks.com.invalid> wrote:
>>
>>> Hi All,
>>>
>>> For Spark Connect Scala Client we are working on making the REPL
>>> experience a bit nicer <https://github.com/apache/spark/pull/40515>. In
>>> a nutshell we want to give users a turn key scala REPL, that works even if
>>> you don't have a Spark distribution on your machine (through coursier
>>> <https://get-coursier.io/>). We are using Ammonite
>>> <https://ammonite.io/> instead of the standard scala REPL for this, the
>>> main reason for going with Ammonite is that it is easier to customize, and
>>> IMO has a superior user experience.
>>>
>>> Does anyone object to doing this?
>>>
>>> Kind regards,
>>> Herman
>>>
>>>

Re: Ammonite as REPL for Spark Connect

[Holden Karau <ho...@pigscanfly.ca>]

I am +1 to the general concept of including Ammonite magic 🪄.

On Wed, Mar 22, 2023 at 4:58 PM Herman van Hovell
<he...@databricks.com.invalid> wrote:

> Ammonite is maintained externally by Li Haoyi et al. We are including it
> as a 'provided' dependency. The integration bits and pieces (1 file) are
> included in Apache Spark.
>
> On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan <mr...@gmail.com>
> wrote:
>
>>
>> Will this be maintained externally or included into Apache Spark ?
>>
>> Regards ,
>> Mridul
>>
>>
>>
>> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
>> <he...@databricks.com.invalid> wrote:
>>
>>> Hi All,
>>>
>>> For Spark Connect Scala Client we are working on making the REPL
>>> experience a bit nicer <https://github.com/apache/spark/pull/40515>. In
>>> a nutshell we want to give users a turn key scala REPL, that works even if
>>> you don't have a Spark distribution on your machine (through coursier
>>> <https://get-coursier.io/>). We are using Ammonite
>>> <https://ammonite.io/> instead of the standard scala REPL for this, the
>>> main reason for going with Ammonite is that it is easier to customize, and
>>> IMO has a superior user experience.
>>>
>>> Does anyone object to doing this?
>>>
>>> Kind regards,
>>> Herman
>>>
>>> --
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: Ammonite as REPL for Spark Connect

[Herman van Hovell <he...@databricks.com.INVALID>]

Ammonite is maintained externally by Li Haoyi et al. We are including it as
a 'provided' dependency. The integration bits and pieces (1 file) are
included in Apache Spark.

On Wed, Mar 22, 2023 at 7:53 PM Mridul Muralidharan <mr...@gmail.com>
wrote:

>
> Will this be maintained externally or included into Apache Spark ?
>
> Regards ,
> Mridul
>
>
>
> On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
> <he...@databricks.com.invalid> wrote:
>
>> Hi All,
>>
>> For Spark Connect Scala Client we are working on making the REPL
>> experience a bit nicer <https://github.com/apache/spark/pull/40515>. In
>> a nutshell we want to give users a turn key scala REPL, that works even if
>> you don't have a Spark distribution on your machine (through coursier
>> <https://get-coursier.io/>). We are using Ammonite <https://ammonite.io/>
>> instead of the standard scala REPL for this, the main reason for going with
>> Ammonite is that it is easier to customize, and IMO has a superior user
>> experience.
>>
>> Does anyone object to doing this?
>>
>> Kind regards,
>> Herman
>>
>>

Re: Ammonite as REPL for Spark Connect

[Mridul Muralidharan <mr...@gmail.com>]

Will this be maintained externally or included into Apache Spark ?

Regards ,
Mridul



On Wed, Mar 22, 2023 at 6:50 PM Herman van Hovell
<he...@databricks.com.invalid> wrote:

> Hi All,
>
> For Spark Connect Scala Client we are working on making the REPL
> experience a bit nicer <https://github.com/apache/spark/pull/40515>. In a
> nutshell we want to give users a turn key scala REPL, that works even if
> you don't have a Spark distribution on your machine (through coursier
> <https://get-coursier.io/>). We are using Ammonite <https://ammonite.io/>
> instead of the standard scala REPL for this, the main reason for going with
> Ammonite is that it is easier to customize, and IMO has a superior user
> experience.
>
> Does anyone object to doing this?
>
> Kind regards,
> Herman
>
>