You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Nicholas Chammas <ni...@gmail.com> on 2026/06/21 14:57:58 UTC
Eliminating startup time in local dev/test loop
A recent SPIP <https://docs.google.com/document/d/1Nphejrf_vh4YRECn0JPgKClqxDS_lB6wufZFJQxyY98/> proposed to improve Spark’s performance on small and local datasets. On that SPIP I raised a related issue <https://docs.google.com/document/d/1Nphejrf_vh4YRECn0JPgKClqxDS_lB6wufZFJQxyY98/edit?disco=AAAB5rOuVBw> that I would like to surface here, and that is the time it takes to create a Spark session locally.
import time
from pyspark.sql import SparkSession
start = time.perf_counter()
session = (
SparkSession.builder.remote("local[*]")
.getOrCreate()
)
elapsed = time.perf_counter() - start
print(f"SparkSession startup: {elapsed:.3f}s")
On my M2 MacBook this consistently takes ~3 seconds.
If you’re working on an application that uses Spark and have a local dev/test loop setup, every loop will incur this startup cost. This makes the entire experience feel incredibly sluggish.
A straightforward solution is to start a persistent Connect server using sbin/start-connect-server.sh and set your remote to sc://localhost:{port}. In my testing, this cuts the startup time from ~3 seconds to <1 second.
This is good, but as a solution it has some problems:
It’s not discoverable. Users are unlikely to figure this out by themselves.
It’s not the default behavior. Users ideally should not have anything to figure out at all. It should just work like this in the background.
A background process needs some tooling to help manage.
I think we can address these problems by doing something like this:
Make .remote("local") create a persistent Connect server in the background by default, and restart/reconnect to it as needed.
Add a basic CLI to manage the Connect server, like spark connect {start | stop | show}. This CLI can perhaps just be a wrapper around the scripts in sbin/.
There are some details to figure out related to server idle timeouts, server discoverability, etc. But before exploring this further with a prototype, I wanted to get a reaction from the list.
What do you think?
Nick
Re: Eliminating startup time in local dev/test loop
Posted by Nicholas Chammas <ni...@gmail.com>.
> On Jun 22, 2026, at 3:01 PM, Nicholas Chammas <ni...@gmail.com> wrote:
>
> OK, so are you in support of the weaker solution I outlined then? That is, we add a `spark connect` CLI with some accompanying user guide documentation, and make any existing Connect server discoverable by `.remote(“local”)`. The user still has to explicitly start up a persistent server.
To close the loop here, I have expanded this idea of a `spark connect` CLI into a more general `spark` CLI, which I proposed in this dev list thread <https://lists.apache.org/thread/hby7fw6kxbzk6szzmgyocw662415ls6r>.
I am ready to move forward with the idea but need committer support. If you are interested, please chime in on that thread.
Nick
Re: Eliminating startup time in local dev/test loop
Posted by Hyukjin Kwon <gu...@apache.org>.
Ignore my previous emails. I rushed to read them :-).
On Tue, 23 Jun 2026 at 07:59, Hyukjin Kwon <gu...@apache.org> wrote:
> > but I think it doesn't work with the interpreter way Nicholas mentioned.
>
> Oh oops sorry I forgot that I added that support as well.
> Anyway it's still related to
> https://lists.apache.org/thread/j382to15zgy8mr2pvrtcod9c02zj1org
>
> On Tue, 23 Jun 2026 at 07:57, Hyukjin Kwon <gu...@apache.org> wrote:
>
>> Spark already supports `bin/pyspark --remote local` now but I think it
>> doesn't work with the interpreter way Nicholas mentioned.
>> This is probably related to
>> https://lists.apache.org/thread/j382to15zgy8mr2pvrtcod9c02zj1org as well.
>>
>>
>> On Tue, 23 Jun 2026 at 05:05, Tian Gao via dev <de...@spark.apache.org>
>> wrote:
>>
>>> Yes I think that's totally fine.
>>>
>>> I even support `.remote("local").getOrCreate()` as long as `"local"` has
>>> no other ambiguity (is it used now? if not I'm fine to have this as a
>>> special value). What I don't support is that we change the behavior for
>>> something we are supporting now. For example,
>>> .remote("sc://localhost:15002").getOrCreate() should not magically create
>>> the server if there's no existing server.
>>>
>>> Tian
>>>
>>> On Mon, Jun 22, 2026 at 12:01 PM Nicholas Chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>>>
>>>> > On Jun 22, 2026, at 2:45 PM, Tian Gao <ti...@databricks.com>
>>>> wrote:
>>>> >
>>>> > All I'm saying is that this is a tradeoff, it has benefits in some
>>>> cases, but it's not a free lunch. I support having this as a feature when
>>>> users know what they are doing, aka explicit consent. I'm against making
>>>> this an implicit default to replace the existing behavior.
>>>>
>>>> OK, so are you in support of the weaker solution I outlined then? That
>>>> is, we add a `spark connect` CLI with some accompanying user guide
>>>> documentation, and make any existing Connect server discoverable by
>>>> `.remote(“local”)`. The user still has to explicitly start up a persistent
>>>> server.
>>>>
>>>> I think down the line we want to consider making this part of the
>>>> default local workflow, but we can revisit that after this first step.
>>>
>>>
Re: Eliminating startup time in local dev/test loop
Posted by Hyukjin Kwon <gu...@apache.org>.
> but I think it doesn't work with the interpreter way Nicholas mentioned.
Oh oops sorry I forgot that I added that support as well.
Anyway it's still related to
https://lists.apache.org/thread/j382to15zgy8mr2pvrtcod9c02zj1org
On Tue, 23 Jun 2026 at 07:57, Hyukjin Kwon <gu...@apache.org> wrote:
> Spark already supports `bin/pyspark --remote local` now but I think it
> doesn't work with the interpreter way Nicholas mentioned.
> This is probably related to
> https://lists.apache.org/thread/j382to15zgy8mr2pvrtcod9c02zj1org as well.
>
>
> On Tue, 23 Jun 2026 at 05:05, Tian Gao via dev <de...@spark.apache.org>
> wrote:
>
>> Yes I think that's totally fine.
>>
>> I even support `.remote("local").getOrCreate()` as long as `"local"` has
>> no other ambiguity (is it used now? if not I'm fine to have this as a
>> special value). What I don't support is that we change the behavior for
>> something we are supporting now. For example,
>> .remote("sc://localhost:15002").getOrCreate() should not magically create
>> the server if there's no existing server.
>>
>> Tian
>>
>> On Mon, Jun 22, 2026 at 12:01 PM Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>>
>>> > On Jun 22, 2026, at 2:45 PM, Tian Gao <ti...@databricks.com> wrote:
>>> >
>>> > All I'm saying is that this is a tradeoff, it has benefits in some
>>> cases, but it's not a free lunch. I support having this as a feature when
>>> users know what they are doing, aka explicit consent. I'm against making
>>> this an implicit default to replace the existing behavior.
>>>
>>> OK, so are you in support of the weaker solution I outlined then? That
>>> is, we add a `spark connect` CLI with some accompanying user guide
>>> documentation, and make any existing Connect server discoverable by
>>> `.remote(“local”)`. The user still has to explicitly start up a persistent
>>> server.
>>>
>>> I think down the line we want to consider making this part of the
>>> default local workflow, but we can revisit that after this first step.
>>
>>
Re: Eliminating startup time in local dev/test loop
Posted by Hyukjin Kwon <gu...@apache.org>.
Spark already supports `bin/pyspark --remote local` now but I think it
doesn't work with the interpreter way Nicholas mentioned.
This is probably related to
https://lists.apache.org/thread/j382to15zgy8mr2pvrtcod9c02zj1org as well.
On Tue, 23 Jun 2026 at 05:05, Tian Gao via dev <de...@spark.apache.org> wrote:
> Yes I think that's totally fine.
>
> I even support `.remote("local").getOrCreate()` as long as `"local"` has
> no other ambiguity (is it used now? if not I'm fine to have this as a
> special value). What I don't support is that we change the behavior for
> something we are supporting now. For example,
> .remote("sc://localhost:15002").getOrCreate() should not magically create
> the server if there's no existing server.
>
> Tian
>
> On Mon, Jun 22, 2026 at 12:01 PM Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>>
>> > On Jun 22, 2026, at 2:45 PM, Tian Gao <ti...@databricks.com> wrote:
>> >
>> > All I'm saying is that this is a tradeoff, it has benefits in some
>> cases, but it's not a free lunch. I support having this as a feature when
>> users know what they are doing, aka explicit consent. I'm against making
>> this an implicit default to replace the existing behavior.
>>
>> OK, so are you in support of the weaker solution I outlined then? That
>> is, we add a `spark connect` CLI with some accompanying user guide
>> documentation, and make any existing Connect server discoverable by
>> `.remote(“local”)`. The user still has to explicitly start up a persistent
>> server.
>>
>> I think down the line we want to consider making this part of the default
>> local workflow, but we can revisit that after this first step.
>
>
Re: Eliminating startup time in local dev/test loop
Posted by Tian Gao via dev <de...@spark.apache.org>.
Yes I think that's totally fine.
I even support `.remote("local").getOrCreate()` as long as `"local"` has no
other ambiguity (is it used now? if not I'm fine to have this as a special
value). What I don't support is that we change the behavior for something
we are supporting now. For example,
.remote("sc://localhost:15002").getOrCreate() should not magically create
the server if there's no existing server.
Tian
On Mon, Jun 22, 2026 at 12:01 PM Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
>
> > On Jun 22, 2026, at 2:45 PM, Tian Gao <ti...@databricks.com> wrote:
> >
> > All I'm saying is that this is a tradeoff, it has benefits in some
> cases, but it's not a free lunch. I support having this as a feature when
> users know what they are doing, aka explicit consent. I'm against making
> this an implicit default to replace the existing behavior.
>
> OK, so are you in support of the weaker solution I outlined then? That is,
> we add a `spark connect` CLI with some accompanying user guide
> documentation, and make any existing Connect server discoverable by
> `.remote(“local”)`. The user still has to explicitly start up a persistent
> server.
>
> I think down the line we want to consider making this part of the default
> local workflow, but we can revisit that after this first step.
Re: Eliminating startup time in local dev/test loop
Posted by Nicholas Chammas <ni...@gmail.com>.
> On Jun 22, 2026, at 2:45 PM, Tian Gao <ti...@databricks.com> wrote:
>
> All I'm saying is that this is a tradeoff, it has benefits in some cases, but it's not a free lunch. I support having this as a feature when users know what they are doing, aka explicit consent. I'm against making this an implicit default to replace the existing behavior.
OK, so are you in support of the weaker solution I outlined then? That is, we add a `spark connect` CLI with some accompanying user guide documentation, and make any existing Connect server discoverable by `.remote(“local”)`. The user still has to explicitly start up a persistent server.
I think down the line we want to consider making this part of the default local workflow, but we can revisit that after this first step.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Eliminating startup time in local dev/test loop
Posted by Tian Gao via dev <de...@spark.apache.org>.
Imagine a behavior that depends on the cache. Running the test from scratch
will differ from running it when the backend server has been established
for some time. The users could have issues when the backend server is
shutting down - race conditions that are hard to reproduce.
Without explicitly knowing that a persistent backend server is established,
the user might believe this is a spark error, but none of the
developers can reproduce the issue with a new state.
We should not assume the server will "just work" - it won't. Having a brand
new server for every run has its cost, but it's also more deterministic. We
won't able to catch every issue with the server with a health checker - any
health checker. We will miss some issues, and the server will sometimes be
left in a bad state. It's horrible if the user is running some valid code
but hit errors because the backend persistent server is in a bad state that
we can't recover.
All I'm saying is that this is a tradeoff, it has benefits in some cases,
but it's not a free lunch. I support having this as a feature when users
know what they are doing, aka explicit consent. I'm against making this an
implicit default to replace the existing behavior.
Tian
On Mon, Jun 22, 2026 at 11:09 AM Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
>
> > On Jun 22, 2026, at 1:14 PM, Tian Gao via dev <de...@spark.apache.org>
> wrote:
> >
> > I just want to add that an implicitly persistent server might be a
> hotbed for nondeterministic behaviors, especially when the background
> server has an arbitrary shutdown timeout. I think it's nice to have
> something like this when users *explicitly* ask for it.
>
> Could you elaborate with some examples of what you mean?
Re: Eliminating startup time in local dev/test loop
Posted by Nicholas Chammas <ni...@gmail.com>.
> On Jun 22, 2026, at 1:14 PM, Tian Gao via dev <de...@spark.apache.org> wrote:
>
> I just want to add that an implicitly persistent server might be a hotbed for nondeterministic behaviors, especially when the background server has an arbitrary shutdown timeout. I think it's nice to have something like this when users *explicitly* ask for it.
Could you elaborate with some examples of what you mean?
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Eliminating startup time in local dev/test loop
Posted by Tian Gao via dev <de...@spark.apache.org>.
I just want to add that an implicitly persistent server might be a hotbed
for nondeterministic behaviors, especially when the background server has
an arbitrary shutdown timeout. I think it's nice to have something like
this when users *explicitly* ask for it.
Tian
On Mon, Jun 22, 2026 at 8:08 AM Nicholas Chammas <ni...@gmail.com>
wrote:
>
> > On Jun 21, 2026, at 5:45 PM, Holden Karau <ho...@gmail.com>
> wrote:
> >
> > I have mixed feelings about starting a persistent server perhaps
> unexpectedly especially since if it gets in a bad state (say driver JVM
> thrashjng on GC) a reasonable user might restart their Python process and
> expect it to also kill the server (as it has done until now).
> >
> > Not saying I’m against this yet just there’s some downsides with
> changing a default like this and we’d probably want to be careful with
> messaging to users so they don’t get stuck / surpised (and give clear
> instructions when connecting to an existing server etc).
>
> A weaker but still useful solution is to add the Connect server management
> CLI along with documentation in the user guide, and have `.remote(“local”)`
> have a way to automatically discover and connect to an existing server
> without doing anything to the running server itself.
>
> A stronger solution would extend this approach by having the call to
> `.remote(…).getOrCreate()` start a server if one isn’t already running, or
> restart it if it’s running but fails some kind of health check.
>
> In both cases we would want to perhaps print some INFO-level log messages
> about whether a server was found, started, or restarted.
>
> Nick
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: Eliminating startup time in local dev/test loop
Posted by Nicholas Chammas <ni...@gmail.com>.
> On Jun 21, 2026, at 5:45 PM, Holden Karau <ho...@gmail.com> wrote:
>
> I have mixed feelings about starting a persistent server perhaps unexpectedly especially since if it gets in a bad state (say driver JVM thrashjng on GC) a reasonable user might restart their Python process and expect it to also kill the server (as it has done until now).
>
> Not saying I’m against this yet just there’s some downsides with changing a default like this and we’d probably want to be careful with messaging to users so they don’t get stuck / surpised (and give clear instructions when connecting to an existing server etc).
A weaker but still useful solution is to add the Connect server management CLI along with documentation in the user guide, and have `.remote(“local”)` have a way to automatically discover and connect to an existing server without doing anything to the running server itself.
A stronger solution would extend this approach by having the call to `.remote(…).getOrCreate()` start a server if one isn’t already running, or restart it if it’s running but fails some kind of health check.
In both cases we would want to perhaps print some INFO-level log messages about whether a server was found, started, or restarted.
Nick
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Eliminating startup time in local dev/test loop
Posted by Holden Karau <ho...@gmail.com>.
I have mixed feelings about starting a persistent server perhaps
unexpectedly especially since if it gets in a bad state (say driver JVM
thrashjng on GC) a reasonable user might restart their Python process and
expect it to also kill the server (as it has done until now).
Not saying I’m against this yet just there’s some downsides with changing a
default like this and we’d probably want to be careful with messaging to
users so they don’t get stuck / surpised (and give clear instructions when
connecting to an existing server etc).
Twitter: https://twitter.com/holdenkarau
Fight Health Insurance: https://www.fighthealthinsurance.com/
<https://www.fighthealthinsurance.com/?q=hk_email>
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her
On Sun, Jun 21, 2026 at 2:36 PM serge rielau.com <se...@rielau.com> wrote:
> Oh, yes please!
>
>
> On Jun 21, 2026, at 7:57 AM, Nicholas Chammas <ni...@gmail.com>
> wrote:
>
> A recent SPIP
> <https://docs.google.com/document/d/1Nphejrf_vh4YRECn0JPgKClqxDS_lB6wufZFJQxyY98/> proposed
> to improve Spark’s performance on small and local datasets. On that SPIP I
> raised a related issue
> <https://docs.google.com/document/d/1Nphejrf_vh4YRECn0JPgKClqxDS_lB6wufZFJQxyY98/edit?disco=AAAB5rOuVBw> that
> I would like to surface here, and that is the time it takes to create a
> Spark session locally.
>
> import timefrom pyspark.sql import SparkSession
>
> start = time.perf_counter()
> session = (
> SparkSession.builder.remote("local[*]")
> .getOrCreate()
> )
> elapsed = time.perf_counter() - startprint(f"SparkSession startup: {elapsed:.3f}s")
>
> On my M2 MacBook this consistently takes ~3 seconds.
>
> If you’re working on an application that uses Spark and have a local
> dev/test loop setup, every loop will incur this startup cost. This makes
> the entire experience feel incredibly sluggish.
>
> A straightforward solution is to start a persistent Connect server using
> sbin/start-connect-server.sh and set your remote to sc://localhost:{port}.
> In my testing, this cuts the startup time from ~3 seconds to <1 second.
>
> This is good, but as a solution it has some problems:
>
> 1. It’s not discoverable. Users are unlikely to figure this out by
> themselves.
> 2. It’s not the default behavior. Users ideally should not have
> anything to figure out at all. It should just work like this in the
> background.
> 3. A background process needs some tooling to help manage.
>
> I think we can address these problems by doing something like this:
>
> 1. Make .remote("local") create a persistent Connect server in the
> background by default, and restart/reconnect to it as needed.
> 2. Add a basic CLI to manage the Connect server, like spark connect
> {start | stop | show}. This CLI can perhaps just be a wrapper around
> the scripts in sbin/.
>
> There are some details to figure out related to server idle timeouts,
> server discoverability, etc. But before exploring this further with a
> prototype, I wanted to get a reaction from the list.
>
> What do you think?
>
> Nick
>
>
>
Re: Eliminating startup time in local dev/test loop
Posted by "serge rielau.com" <se...@rielau.com>.
Oh, yes please!
On Jun 21, 2026, at 7:57 AM, Nicholas Chammas <ni...@gmail.com> wrote:
A recent SPIP<https://docs.google.com/document/d/1Nphejrf_vh4YRECn0JPgKClqxDS_lB6wufZFJQxyY98/> proposed to improve Spark’s performance on small and local datasets. On that SPIP I raised a related issue<https://docs.google.com/document/d/1Nphejrf_vh4YRECn0JPgKClqxDS_lB6wufZFJQxyY98/edit?disco=AAAB5rOuVBw> that I would like to surface here, and that is the time it takes to create a Spark session locally.
import time
from pyspark.sql import SparkSession
start = time.perf_counter()
session = (
SparkSession.builder.remote("local[*]")
.getOrCreate()
)
elapsed = time.perf_counter() - start
print(f"SparkSession startup: {elapsed:.3f}s")
On my M2 MacBook this consistently takes ~3 seconds.
If you’re working on an application that uses Spark and have a local dev/test loop setup, every loop will incur this startup cost. This makes the entire experience feel incredibly sluggish.
A straightforward solution is to start a persistent Connect server using sbin/start-connect-server.sh and set your remote to sc://localhost:{port}. In my testing, this cuts the startup time from ~3 seconds to <1 second.
This is good, but as a solution it has some problems:
1. It’s not discoverable. Users are unlikely to figure this out by themselves.
2. It’s not the default behavior. Users ideally should not have anything to figure out at all. It should just work like this in the background.
3. A background process needs some tooling to help manage.
I think we can address these problems by doing something like this:
1. Make .remote("local") create a persistent Connect server in the background by default, and restart/reconnect to it as needed.
2. Add a basic CLI to manage the Connect server, like spark connect {start | stop | show}. This CLI can perhaps just be a wrapper around the scripts in sbin/.
There are some details to figure out related to server idle timeouts, server discoverability, etc. But before exploring this further with a prototype, I wanted to get a reaction from the list.
What do you think?
Nick