You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Hyukjin Kwon <gu...@apache.org> on 2026/06/04 06:12:16 UTC
[DISCUSS] Replacing Py4J to Spark Connect in PySpark (possibly with RDD API and UI)
Hi all,
Firstly, I wanted to check other opinions (rather saying this to push my
opinion hard in this way).
Lately, I have been looking through the feasibility of replacing Py4J to
Spark Connect in PySpark.
More specifically I mean that use Spark Connect server, and leave Py4J
server as a fallback option to avoid breakage.
As we know, Py4J gateway server itself already works similarly with Spark
Connect server.
It is a bit of a difficult story in Scala because there wasn't an
intermediate server in Classic but in PySpark there is already a Py4J
server running.
There are few downsides of using Py4J:
- Py4J server itself exposes arbitrary access to the JVM machine which
actually is risky.
- Performance is quite slow when sending large data. Note that we are
switching to the raw sockets when we send large binaries
- Difficult to debug the errors
With Spark Connect
- We could limit those accesses.
- We use Arrow batches to send the data should be more efficient
- Error handling are quite implemented well in Python Spark Connect
Lastly, structurally they are quite the same. It is not like introducing a
new layer.
Last time when we chatted about enabling Spark Connect on (for both Scala,
Python etc.), the biggest pushback was RDD API missing. I recently
prototyped, and concluded that, for Python specifically, we can add the
full support of RDD API, and most of SparkContext API (
https://github.com/apache/spark/pull/55888).
I also prototyped Python client side UI (
https://github.com/apache/spark/pull/56053) here too, so for Python
specifically, I believe there is not quite much gap between classic and
connect. API-wise, structure-wise, etc.
I have been thinking about this idea for quite a long time, and I concluded
that it is feasible. So I would like to know what others think about this.
Thanks!
Re: [DISCUSS] Replacing Py4J to Spark Connect in PySpark (possibly with RDD API and UI)
Posted by Hyukjin Kwon <gu...@apache.org>.
Yes they will be supported all the same from my understanding.
On Fri, 5 Jun 2026 at 06:44, Sem <ss...@apache.org> wrote:
> > To Sem, I fully agree that there are existent use cases of Py4J through
> PySpark. Note that the original intention is not to support JVM accesses
> through Py4J, and they are not officially API in PySpark. Nevertheless, I
> do understand those existent use cases, and I am thinking about reducing
> those breakage by having Py4J as an option for the time being.
>
> Thanks for the explanation! Is there any chance that Spark Connect will
> support dynamic resolution of the plugin classes
> ("o.a.s.sql.connect.plugin.RelationPlugin" and friends) for running
> clusters? From my point of view it would be a parity with plugging via
> py4j: users can specify `--package` or `spark.jars` and get the same
> experience as with classic/py4j -- plugin's JAR is attached to the running
> Spark Cluster and resolved in runtime, no code change is required from user.
>
> Best regards,
> Sem
>
> On Fri, 2026-06-05 at 06:31 +0900, Hyukjin Kwon wrote:
>
> > A question: does that mean that in the future, all Connect dependencies
> will be required for PySpark Classic?
>
> For now, I am still thinking about the Connect dependencies as an option
> to reduce the breakage. Maybe yes in Spark 5 but nothing is decided. Was
> more to hear what you guys think. I don't intend to open the vote for this
> very soon.
>
> To Sem, I fully agree that there are existent use cases of Py4J through
> PySpark. Note that the original intention is not to support JVM accesses
> through Py4J, and they are not officially API in PySpark. Nevertheless, I
> do understand those existent use cases, and I am thinking about reducing
> those breakage by having Py4J as an option for the time being.
>
>
> On Fri, 5 Jun 2026 at 04:11, Sem <ss...@apache.org> wrote:
>
> Hello!
>
> Correct me if I'm wrong, but for me it looks like py4j is currently the
> main escape hatch for Python users to access JVM-side APIs from an already
> running PySpark Classic session..
>
> Let me explain it on the example of GraphFrames project where I'm a
> maintainer. At the moment the project provide ~99% parity for classic (via
> py4j) and connect (via "o.a.s.sql.connect.plugin.RelationPlugin") from the
> API point of view.
> But for users it is very different. If we imagine a running cluster like
> YARN, users can just drop JAR to the CP and access it via GF py4j API from
> their session. With connect it is different because the conf
> "spark.connect.extensions.relation.classes" cannot be changed for the
> running cluster: it is a "static conf" isn't it? So, all the
> implementations of the "RelationPlugin" should be known before the cluster
> started. So, from the user's perspective it means they need not just add a
> dependency via "SparkSession.builder.conf("spark.jars", ...)" but to start
> the dedicated spark cluster with a correct list of all the Conect's
> plugins. At least that is what I know based on my experiments with Spark
> Connect at the version ~3.5.x and early 4.x RC-builds.
>
> On top of this I would like to highlight that at the moment none of the
> major Spark's vendors supports changing of the
> "spark.connect.extensions.relation.classes" at least I did not find
> anything about it in their docs. And py4j API is the only way for users to
> access 3d party packages.
>
> I'm not a contributor of Spark, so feel free to ignore. I just wanted to
> highlight this, because for me Apache Spark was always not only the
> framework but a broad ecosystem of 3d party packages and extensions.
>
> Best regards,
> Sem
>
> On Thu, 2026-06-04 at 11:04 -0700, Tian Gao via dev wrote:
>
> I think deprecating py4j is what we need to do eventually. A question:
> does that mean that in the future, all Connect dependencies will be
> required for PySpark Classic?
>
> Tian
>
> On Thu, Jun 4, 2026 at 9:59 AM Holden Karau <ho...@gmail.com>
> wrote:
>
> Spark 5 sounds like a good time to target these changes, I think we're
> going to also want to audit and look for OSS code which uses Py4J to access
> direct JVM objects and see which additional APIs we might want to / need to
> expose for power users (the answer might just be write some scala code and
> add it as a plugin).
>
> On Thu, Jun 4, 2026 at 9:50 AM Dongjoon Hyun <do...@apache.org> wrote:
>
> Thank you so much for sharing the progress, Hyukjin.
>
> This sounds reasonable to me. This will reduce the gap between classic and
> connect mode greatly.
>
> Given that Apache Spark Release Cadence already defines the path to
> introduce a new feature with its feature flag, I hope we are able to start
> to test this in Apache Spark 4.3.0.
>
> After stabilizing this new architecture via Spark 4.3.0, we may choose
> more migration steps in Spark 5 in 2027 or Spark 6 in 2028.
>
> Thanks,
> Dongjoon.
>
> On 2026/06/04 06:12:16 Hyukjin Kwon wrote:
> > Hi all,
> >
> > Firstly, I wanted to check other opinions (rather saying this to push my
> > opinion hard in this way).
> >
> > Lately, I have been looking through the feasibility of replacing Py4J to
> > Spark Connect in PySpark.
> > More specifically I mean that use Spark Connect server, and leave Py4J
> > server as a fallback option to avoid breakage.
> >
> > As we know, Py4J gateway server itself already works similarly with Spark
> > Connect server.
> > It is a bit of a difficult story in Scala because there wasn't an
> > intermediate server in Classic but in PySpark there is already a Py4J
> > server running.
> >
> > There are few downsides of using Py4J:
> > - Py4J server itself exposes arbitrary access to the JVM machine which
> > actually is risky.
> > - Performance is quite slow when sending large data. Note that we are
> > switching to the raw sockets when we send large binaries
> > - Difficult to debug the errors
> >
> > With Spark Connect
> > - We could limit those accesses.
> > - We use Arrow batches to send the data should be more efficient
> > - Error handling are quite implemented well in Python Spark Connect
> >
> > Lastly, structurally they are quite the same. It is not like introducing
> a
> > new layer.
> >
> > Last time when we chatted about enabling Spark Connect on (for both
> Scala,
> > Python etc.), the biggest pushback was RDD API missing. I recently
> > prototyped, and concluded that, for Python specifically, we can add the
> > full support of RDD API, and most of SparkContext API (
> > https://github.com/apache/spark/pull/55888).
> >
> > I also prototyped Python client side UI (
> > https://github.com/apache/spark/pull/56053) here too, so for Python
> > specifically, I believe there is not quite much gap between classic and
> > connect. API-wise, structure-wise, etc.
> >
> > I have been thinking about this idea for quite a long time, and I
> concluded
> > that it is feasible. So I would like to know what others think about
> this.
> >
> > Thanks!
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
>
>
>
Re: [DISCUSS] Replacing Py4J to Spark Connect in PySpark (possibly with RDD API and UI)
Posted by Sem <ss...@apache.org>.
> To Sem, I fully agree that there are existent use cases of Py4J
through PySpark. Note that the original intention is not to support JVM
accesses through Py4J, and they are not officially API in PySpark.
Nevertheless, I do understand those existent use cases, and I am
thinking about reducing those breakage by having Py4J as an option for
the time being.
Thanks for the explanation! Is there any chance that Spark Connect will
support dynamic resolution of the plugin classes
("o.a.s.sql.connect.plugin.RelationPlugin" and friends) for running
clusters? From my point of view it would be a parity with plugging via
py4j: users can specify `--package` or `spark.jars` and get the same
experience as with classic/py4j -- plugin's JAR is attached to the
running Spark Cluster and resolved in runtime, no code change is
required from user.
Best regards,
Sem
On Fri, 2026-06-05 at 06:31 +0900, Hyukjin Kwon wrote:
> > A question: does that mean that in the future, all Connect
> dependencies will be required for PySpark Classic?
>
> For now, I am still thinking about the Connect dependencies as an
> option to reduce the breakage. Maybe yes in Spark 5 but nothing is
> decided. Was more to hear what you guys think. I don't intend to open
> the vote for this very soon.
>
> To Sem, I fully agree that there are existent use cases of Py4J
> through PySpark. Note that the original intention is not to support
> JVM accesses through Py4J, and they are not officially API in
> PySpark. Nevertheless, I do understand those existent use cases, and
> I am thinking about reducing those breakage by having Py4J as an
> option for the time being.
>
>
> On Fri, 5 Jun 2026 at 04:11, Sem <ss...@apache.org> wrote:
> > Hello!
> >
> > Correct me if I'm wrong, but for me it looks like py4j is currently
> > the main escape hatch for Python users to access JVM-side APIs from
> > an already running PySpark Classic session..
> >
> > Let me explain it on the example of GraphFrames project where I'm a
> > maintainer. At the moment the project provide ~99% parity for
> > classic (via py4j) and connect (via
> > "o.a.s.sql.connect.plugin.RelationPlugin") from the API point of
> > view.
> > But for users it is very different. If we imagine a running cluster
> > like YARN, users can just drop JAR to the CP and access it via GF
> > py4j API from their session. With connect it is different because
> > the conf "spark.connect.extensions.relation.classes" cannot be
> > changed for the running cluster: it is a "static conf" isn't it?
> > So, all the implementations of the "RelationPlugin" should be known
> > before the cluster started. So, from the user's perspective it
> > means they need not just add a dependency via
> > "SparkSession.builder.conf("spark.jars", ...)" but to start the
> > dedicated spark cluster with a correct list of all the Conect's
> > plugins. At least that is what I know based on my experiments with
> > Spark Connect at the version ~3.5.x and early 4.x RC-builds.
> >
> > On top of this I would like to highlight that at the moment none of
> > the major Spark's vendors supports changing of the
> > "spark.connect.extensions.relation.classes" at least I did not find
> > anything about it in their docs. And py4j API is the only way for
> > users to access 3d party packages.
> >
> > I'm not a contributor of Spark, so feel free to ignore. I just
> > wanted to highlight this, because for me Apache Spark was always
> > not only the framework but a broad ecosystem of 3d party packages
> > and extensions.
> >
> > Best regards,
> > Sem
> >
> > On Thu, 2026-06-04 at 11:04 -0700, Tian Gao via dev wrote:
> > > I think deprecating py4j is what we need to do eventually. A
> > > question: does that mean that in the future, all Connect
> > > dependencies will be required for PySpark Classic?
> > >
> > > Tian
> > >
> > > On Thu, Jun 4, 2026 at 9:59 AM Holden Karau
> > > <ho...@gmail.com> wrote:
> > > > Spark 5 sounds like a good time to target these changes, I
> > > > think we're going to also want to audit and look for OSS code
> > > > which uses Py4J to access direct JVM objects and see which
> > > > additional APIs we might want to / need to expose for power
> > > > users (the answer might just be write some scala code and add
> > > > it as a plugin).
> > > >
> > > > On Thu, Jun 4, 2026 at 9:50 AM Dongjoon Hyun
> > > > <do...@apache.org> wrote:
> > > > > Thank you so much for sharing the progress, Hyukjin.
> > > > >
> > > > > This sounds reasonable to me. This will reduce the gap
> > > > > between classic and connect mode greatly.
> > > > >
> > > > > Given that Apache Spark Release Cadence already defines the
> > > > > path to introduce a new feature with its feature flag, I hope
> > > > > we are able to start to test this in Apache Spark 4.3.0.
> > > > >
> > > > > After stabilizing this new architecture via Spark 4.3.0, we
> > > > > may choose more migration steps in Spark 5 in 2027 or Spark 6
> > > > > in 2028.
> > > > >
> > > > > Thanks,
> > > > > Dongjoon.
> > > > >
> > > > > On 2026/06/04 06:12:16 Hyukjin Kwon wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > Firstly, I wanted to check other opinions (rather saying
> > > > > this to push my
> > > > > > opinion hard in this way).
> > > > > >
> > > > > > Lately, I have been looking through the feasibility of
> > > > > replacing Py4J to
> > > > > > Spark Connect in PySpark.
> > > > > > More specifically I mean that use Spark Connect server, and
> > > > > leave Py4J
> > > > > > server as a fallback option to avoid breakage.
> > > > > >
> > > > > > As we know, Py4J gateway server itself already works
> > > > > similarly with Spark
> > > > > > Connect server.
> > > > > > It is a bit of a difficult story in Scala because there
> > > > > wasn't an
> > > > > > intermediate server in Classic but in PySpark there is
> > > > > already a Py4J
> > > > > > server running.
> > > > > >
> > > > > > There are few downsides of using Py4J:
> > > > > > - Py4J server itself exposes arbitrary access to the JVM
> > > > > machine which
> > > > > > actually is risky.
> > > > > > - Performance is quite slow when sending large data. Note
> > > > > that we are
> > > > > > switching to the raw sockets when we send large binaries
> > > > > > - Difficult to debug the errors
> > > > > >
> > > > > > With Spark Connect
> > > > > > - We could limit those accesses.
> > > > > > - We use Arrow batches to send the data should be more
> > > > > efficient
> > > > > > - Error handling are quite implemented well in Python Spark
> > > > > Connect
> > > > > >
> > > > > > Lastly, structurally they are quite the same. It is not
> > > > > like introducing a
> > > > > > new layer.
> > > > > >
> > > > > > Last time when we chatted about enabling Spark Connect on
> > > > > (for both Scala,
> > > > > > Python etc.), the biggest pushback was RDD API missing. I
> > > > > recently
> > > > > > prototyped, and concluded that, for Python specifically, we
> > > > > can add the
> > > > > > full support of RDD API, and most of SparkContext API (
> > > > > > https://github.com/apache/spark/pull/55888).
> > > > > >
> > > > > > I also prototyped Python client side UI (
> > > > > > https://github.com/apache/spark/pull/56053) here too, so
> > > > > for Python
> > > > > > specifically, I believe there is not quite much gap between
> > > > > classic and
> > > > > > connect. API-wise, structure-wise, etc.
> > > > > >
> > > > > > I have been thinking about this idea for quite a long time,
> > > > > and I concluded
> > > > > > that it is feasible. So I would like to know what others
> > > > > think about this.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > >
> > > > > -------------------------------------------------------------
> > > > > --------
> > > > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > > > >
> > > >
> > > >
> > > > --
> > > > Twitter: https://twitter.com/holdenkarau
> > > > Fight Health Insurance: https://www.fighthealthinsurance.com/
> > > > [1]
> > > > Books (Learning Spark, High Performance Spark,
> > > > etc.): https://amzn.to/2MaRAG9 [2]
> > > > YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> > > > Pronouns: she/her
> >
> >
[1] https://www.fighthealthinsurance.com/
https://www.fighthealthinsurance.com/?q=hk_email
[2] https://amzn.to/2MaRAG9 https://amzn.to/2MaRAG9
Re: [DISCUSS] Replacing Py4J to Spark Connect in PySpark (possibly with RDD API and UI)
Posted by Hyukjin Kwon <gu...@apache.org>.
> A question: does that mean that in the future, all Connect dependencies
will be required for PySpark Classic?
For now, I am still thinking about the Connect dependencies as an option to
reduce the breakage. Maybe yes in Spark 5 but nothing is decided. Was more
to hear what you guys think. I don't intend to open the vote for this very
soon.
To Sem, I fully agree that there are existent use cases of Py4J through
PySpark. Note that the original intention is not to support JVM accesses
through Py4J, and they are not officially API in PySpark. Nevertheless, I
do understand those existent use cases, and I am thinking about reducing
those breakage by having Py4J as an option for the time being.
On Fri, 5 Jun 2026 at 04:11, Sem <ss...@apache.org> wrote:
> Hello!
>
> Correct me if I'm wrong, but for me it looks like py4j is currently the
> main escape hatch for Python users to access JVM-side APIs from an already
> running PySpark Classic session..
>
> Let me explain it on the example of GraphFrames project where I'm a
> maintainer. At the moment the project provide ~99% parity for classic (via
> py4j) and connect (via "o.a.s.sql.connect.plugin.RelationPlugin") from the
> API point of view.
> But for users it is very different. If we imagine a running cluster like
> YARN, users can just drop JAR to the CP and access it via GF py4j API from
> their session. With connect it is different because the conf
> "spark.connect.extensions.relation.classes" cannot be changed for the
> running cluster: it is a "static conf" isn't it? So, all the
> implementations of the "RelationPlugin" should be known before the cluster
> started. So, from the user's perspective it means they need not just add a
> dependency via "SparkSession.builder.conf("spark.jars", ...)" but to start
> the dedicated spark cluster with a correct list of all the Conect's
> plugins. At least that is what I know based on my experiments with Spark
> Connect at the version ~3.5.x and early 4.x RC-builds.
>
> On top of this I would like to highlight that at the moment none of the
> major Spark's vendors supports changing of the
> "spark.connect.extensions.relation.classes" at least I did not find
> anything about it in their docs. And py4j API is the only way for users to
> access 3d party packages.
>
> I'm not a contributor of Spark, so feel free to ignore. I just wanted to
> highlight this, because for me Apache Spark was always not only the
> framework but a broad ecosystem of 3d party packages and extensions.
>
> Best regards,
> Sem
>
> On Thu, 2026-06-04 at 11:04 -0700, Tian Gao via dev wrote:
>
> I think deprecating py4j is what we need to do eventually. A question:
> does that mean that in the future, all Connect dependencies will be
> required for PySpark Classic?
>
> Tian
>
> On Thu, Jun 4, 2026 at 9:59 AM Holden Karau <ho...@gmail.com>
> wrote:
>
> Spark 5 sounds like a good time to target these changes, I think we're
> going to also want to audit and look for OSS code which uses Py4J to access
> direct JVM objects and see which additional APIs we might want to / need to
> expose for power users (the answer might just be write some scala code and
> add it as a plugin).
>
> On Thu, Jun 4, 2026 at 9:50 AM Dongjoon Hyun <do...@apache.org> wrote:
>
> Thank you so much for sharing the progress, Hyukjin.
>
> This sounds reasonable to me. This will reduce the gap between classic and
> connect mode greatly.
>
> Given that Apache Spark Release Cadence already defines the path to
> introduce a new feature with its feature flag, I hope we are able to start
> to test this in Apache Spark 4.3.0.
>
> After stabilizing this new architecture via Spark 4.3.0, we may choose
> more migration steps in Spark 5 in 2027 or Spark 6 in 2028.
>
> Thanks,
> Dongjoon.
>
> On 2026/06/04 06:12:16 Hyukjin Kwon wrote:
> > Hi all,
> >
> > Firstly, I wanted to check other opinions (rather saying this to push my
> > opinion hard in this way).
> >
> > Lately, I have been looking through the feasibility of replacing Py4J to
> > Spark Connect in PySpark.
> > More specifically I mean that use Spark Connect server, and leave Py4J
> > server as a fallback option to avoid breakage.
> >
> > As we know, Py4J gateway server itself already works similarly with Spark
> > Connect server.
> > It is a bit of a difficult story in Scala because there wasn't an
> > intermediate server in Classic but in PySpark there is already a Py4J
> > server running.
> >
> > There are few downsides of using Py4J:
> > - Py4J server itself exposes arbitrary access to the JVM machine which
> > actually is risky.
> > - Performance is quite slow when sending large data. Note that we are
> > switching to the raw sockets when we send large binaries
> > - Difficult to debug the errors
> >
> > With Spark Connect
> > - We could limit those accesses.
> > - We use Arrow batches to send the data should be more efficient
> > - Error handling are quite implemented well in Python Spark Connect
> >
> > Lastly, structurally they are quite the same. It is not like introducing
> a
> > new layer.
> >
> > Last time when we chatted about enabling Spark Connect on (for both
> Scala,
> > Python etc.), the biggest pushback was RDD API missing. I recently
> > prototyped, and concluded that, for Python specifically, we can add the
> > full support of RDD API, and most of SparkContext API (
> > https://github.com/apache/spark/pull/55888).
> >
> > I also prototyped Python client side UI (
> > https://github.com/apache/spark/pull/56053) here too, so for Python
> > specifically, I believe there is not quite much gap between classic and
> > connect. API-wise, structure-wise, etc.
> >
> > I have been thinking about this idea for quite a long time, and I
> concluded
> > that it is feasible. So I would like to know what others think about
> this.
> >
> > Thanks!
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
>
>
Re: [DISCUSS] Replacing Py4J to Spark Connect in PySpark (possibly with RDD API and UI)
Posted by Sem <ss...@apache.org>.
Hello!
Correct me if I'm wrong, but for me it looks like py4j is currently the
main escape hatch for Python users to access JVM-side APIs from an
already running PySpark Classic session..
Let me explain it on the example of GraphFrames project where I'm a
maintainer. At the moment the project provide ~99% parity for classic
(via py4j) and connect (via "o.a.s.sql.connect.plugin.RelationPlugin")
from the API point of view.
But for users it is very different. If we imagine a running cluster
like YARN, users can just drop JAR to the CP and access it via GF py4j
API from their session. With connect it is different because the conf
"spark.connect.extensions.relation.classes" cannot be changed for the
running cluster: it is a "static conf" isn't it? So, all the
implementations of the "RelationPlugin" should be known before the
cluster started. So, from the user's perspective it means they need not
just add a dependency via "SparkSession.builder.conf("spark.jars",
...)" but to start the dedicated spark cluster with a correct list of
all the Conect's plugins. At least that is what I know based on my
experiments with Spark Connect at the version ~3.5.x and early 4.x RC-
builds.
On top of this I would like to highlight that at the moment none of the
major Spark's vendors supports changing of the
"spark.connect.extensions.relation.classes" at least I did not find
anything about it in their docs. And py4j API is the only way for users
to access 3d party packages.
I'm not a contributor of Spark, so feel free to ignore. I just wanted
to highlight this, because for me Apache Spark was always not only the
framework but a broad ecosystem of 3d party packages and extensions.
Best regards,
Sem
On Thu, 2026-06-04 at 11:04 -0700, Tian Gao via dev wrote:
> I think deprecating py4j is what we need to do eventually. A
> question: does that mean that in the future, all Connect dependencies
> will be required for PySpark Classic?
>
> Tian
>
> On Thu, Jun 4, 2026 at 9:59 AM Holden Karau <ho...@gmail.com>
> wrote:
> > Spark 5 sounds like a good time to target these changes, I think
> > we're going to also want to audit and look for OSS code which uses
> > Py4J to access direct JVM objects and see which additional APIs we
> > might want to / need to expose for power users (the answer might
> > just be write some scala code and add it as a plugin).
> >
> > On Thu, Jun 4, 2026 at 9:50 AM Dongjoon Hyun <do...@apache.org>
> > wrote:
> > > Thank you so much for sharing the progress, Hyukjin.
> > >
> > > This sounds reasonable to me. This will reduce the gap between
> > > classic and connect mode greatly.
> > >
> > > Given that Apache Spark Release Cadence already defines the path
> > > to introduce a new feature with its feature flag, I hope we are
> > > able to start to test this in Apache Spark 4.3.0.
> > >
> > > After stabilizing this new architecture via Spark 4.3.0, we may
> > > choose more migration steps in Spark 5 in 2027 or Spark 6 in
> > > 2028.
> > >
> > > Thanks,
> > > Dongjoon.
> > >
> > > On 2026/06/04 06:12:16 Hyukjin Kwon wrote:
> > > > Hi all,
> > > >
> > > > Firstly, I wanted to check other opinions (rather saying this
> > > to push my
> > > > opinion hard in this way).
> > > >
> > > > Lately, I have been looking through the feasibility of
> > > replacing Py4J to
> > > > Spark Connect in PySpark.
> > > > More specifically I mean that use Spark Connect server, and
> > > leave Py4J
> > > > server as a fallback option to avoid breakage.
> > > >
> > > > As we know, Py4J gateway server itself already works similarly
> > > with Spark
> > > > Connect server.
> > > > It is a bit of a difficult story in Scala because there wasn't
> > > an
> > > > intermediate server in Classic but in PySpark there is already
> > > a Py4J
> > > > server running.
> > > >
> > > > There are few downsides of using Py4J:
> > > > - Py4J server itself exposes arbitrary access to the JVM
> > > machine which
> > > > actually is risky.
> > > > - Performance is quite slow when sending large data. Note that
> > > we are
> > > > switching to the raw sockets when we send large binaries
> > > > - Difficult to debug the errors
> > > >
> > > > With Spark Connect
> > > > - We could limit those accesses.
> > > > - We use Arrow batches to send the data should be more
> > > efficient
> > > > - Error handling are quite implemented well in Python Spark
> > > Connect
> > > >
> > > > Lastly, structurally they are quite the same. It is not like
> > > introducing a
> > > > new layer.
> > > >
> > > > Last time when we chatted about enabling Spark Connect on (for
> > > both Scala,
> > > > Python etc.), the biggest pushback was RDD API missing. I
> > > recently
> > > > prototyped, and concluded that, for Python specifically, we can
> > > add the
> > > > full support of RDD API, and most of SparkContext API (
> > > > https://github.com/apache/spark/pull/55888).
> > > >
> > > > I also prototyped Python client side UI (
> > > > https://github.com/apache/spark/pull/56053) here too, so for
> > > Python
> > > > specifically, I believe there is not quite much gap between
> > > classic and
> > > > connect. API-wise, structure-wise, etc.
> > > >
> > > > I have been thinking about this idea for quite a long time, and
> > > I concluded
> > > > that it is feasible. So I would like to know what others think
> > > about this.
> > > >
> > > > Thanks!
> > > >
> > >
> > > -----------------------------------------------------------------
> > > ----
> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > >
> >
> >
> > --
> > Twitter: https://twitter.com/holdenkarau
> > Fight Health Insurance: https://www.fighthealthinsurance.com/ [1]
> > Books (Learning Spark, High Performance Spark,
> > etc.): https://amzn.to/2MaRAG9 [2]
> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> > Pronouns: she/her
[1] https://www.fighthealthinsurance.com/
https://www.fighthealthinsurance.com/?q=hk_email
[2] https://amzn.to/2MaRAG9 https://amzn.to/2MaRAG9
Re: [DISCUSS] Replacing Py4J to Spark Connect in PySpark (possibly with RDD API and UI)
Posted by Tian Gao via dev <de...@spark.apache.org>.
I think deprecating py4j is what we need to do eventually. A question: does
that mean that in the future, all Connect dependencies will be required for
PySpark Classic?
Tian
On Thu, Jun 4, 2026 at 9:59 AM Holden Karau <ho...@gmail.com> wrote:
> Spark 5 sounds like a good time to target these changes, I think we're
> going to also want to audit and look for OSS code which uses Py4J to access
> direct JVM objects and see which additional APIs we might want to / need to
> expose for power users (the answer might just be write some scala code and
> add it as a plugin).
>
> On Thu, Jun 4, 2026 at 9:50 AM Dongjoon Hyun <do...@apache.org> wrote:
>
>> Thank you so much for sharing the progress, Hyukjin.
>>
>> This sounds reasonable to me. This will reduce the gap between classic
>> and connect mode greatly.
>>
>> Given that Apache Spark Release Cadence already defines the path to
>> introduce a new feature with its feature flag, I hope we are able to start
>> to test this in Apache Spark 4.3.0.
>>
>> After stabilizing this new architecture via Spark 4.3.0, we may choose
>> more migration steps in Spark 5 in 2027 or Spark 6 in 2028.
>>
>> Thanks,
>> Dongjoon.
>>
>> On 2026/06/04 06:12:16 Hyukjin Kwon wrote:
>> > Hi all,
>> >
>> > Firstly, I wanted to check other opinions (rather saying this to push my
>> > opinion hard in this way).
>> >
>> > Lately, I have been looking through the feasibility of replacing Py4J to
>> > Spark Connect in PySpark.
>> > More specifically I mean that use Spark Connect server, and leave Py4J
>> > server as a fallback option to avoid breakage.
>> >
>> > As we know, Py4J gateway server itself already works similarly with
>> Spark
>> > Connect server.
>> > It is a bit of a difficult story in Scala because there wasn't an
>> > intermediate server in Classic but in PySpark there is already a Py4J
>> > server running.
>> >
>> > There are few downsides of using Py4J:
>> > - Py4J server itself exposes arbitrary access to the JVM machine which
>> > actually is risky.
>> > - Performance is quite slow when sending large data. Note that we are
>> > switching to the raw sockets when we send large binaries
>> > - Difficult to debug the errors
>> >
>> > With Spark Connect
>> > - We could limit those accesses.
>> > - We use Arrow batches to send the data should be more efficient
>> > - Error handling are quite implemented well in Python Spark Connect
>> >
>> > Lastly, structurally they are quite the same. It is not like
>> introducing a
>> > new layer.
>> >
>> > Last time when we chatted about enabling Spark Connect on (for both
>> Scala,
>> > Python etc.), the biggest pushback was RDD API missing. I recently
>> > prototyped, and concluded that, for Python specifically, we can add the
>> > full support of RDD API, and most of SparkContext API (
>> > https://github.com/apache/spark/pull/55888).
>> >
>> > I also prototyped Python client side UI (
>> > https://github.com/apache/spark/pull/56053) here too, so for Python
>> > specifically, I believe there is not quite much gap between classic and
>> > connect. API-wise, structure-wise, etc.
>> >
>> > I have been thinking about this idea for quite a long time, and I
>> concluded
>> > that it is feasible. So I would like to know what others think about
>> this.
>> >
>> > Thanks!
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
Re: [DISCUSS] Replacing Py4J to Spark Connect in PySpark (possibly with RDD API and UI)
Posted by Holden Karau <ho...@gmail.com>.
Spark 5 sounds like a good time to target these changes, I think we're
going to also want to audit and look for OSS code which uses Py4J to access
direct JVM objects and see which additional APIs we might want to / need to
expose for power users (the answer might just be write some scala code and
add it as a plugin).
On Thu, Jun 4, 2026 at 9:50 AM Dongjoon Hyun <do...@apache.org> wrote:
> Thank you so much for sharing the progress, Hyukjin.
>
> This sounds reasonable to me. This will reduce the gap between classic and
> connect mode greatly.
>
> Given that Apache Spark Release Cadence already defines the path to
> introduce a new feature with its feature flag, I hope we are able to start
> to test this in Apache Spark 4.3.0.
>
> After stabilizing this new architecture via Spark 4.3.0, we may choose
> more migration steps in Spark 5 in 2027 or Spark 6 in 2028.
>
> Thanks,
> Dongjoon.
>
> On 2026/06/04 06:12:16 Hyukjin Kwon wrote:
> > Hi all,
> >
> > Firstly, I wanted to check other opinions (rather saying this to push my
> > opinion hard in this way).
> >
> > Lately, I have been looking through the feasibility of replacing Py4J to
> > Spark Connect in PySpark.
> > More specifically I mean that use Spark Connect server, and leave Py4J
> > server as a fallback option to avoid breakage.
> >
> > As we know, Py4J gateway server itself already works similarly with Spark
> > Connect server.
> > It is a bit of a difficult story in Scala because there wasn't an
> > intermediate server in Classic but in PySpark there is already a Py4J
> > server running.
> >
> > There are few downsides of using Py4J:
> > - Py4J server itself exposes arbitrary access to the JVM machine which
> > actually is risky.
> > - Performance is quite slow when sending large data. Note that we are
> > switching to the raw sockets when we send large binaries
> > - Difficult to debug the errors
> >
> > With Spark Connect
> > - We could limit those accesses.
> > - We use Arrow batches to send the data should be more efficient
> > - Error handling are quite implemented well in Python Spark Connect
> >
> > Lastly, structurally they are quite the same. It is not like introducing
> a
> > new layer.
> >
> > Last time when we chatted about enabling Spark Connect on (for both
> Scala,
> > Python etc.), the biggest pushback was RDD API missing. I recently
> > prototyped, and concluded that, for Python specifically, we can add the
> > full support of RDD API, and most of SparkContext API (
> > https://github.com/apache/spark/pull/55888).
> >
> > I also prototyped Python client side UI (
> > https://github.com/apache/spark/pull/56053) here too, so for Python
> > specifically, I believe there is not quite much gap between classic and
> > connect. API-wise, structure-wise, etc.
> >
> > I have been thinking about this idea for quite a long time, and I
> concluded
> > that it is feasible. So I would like to know what others think about
> this.
> >
> > Thanks!
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
--
Twitter: https://twitter.com/holdenkarau
Fight Health Insurance: https://www.fighthealthinsurance.com/
<https://www.fighthealthinsurance.com/?q=hk_email>
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her
Re: [DISCUSS] Replacing Py4J to Spark Connect in PySpark (possibly with RDD API and UI)
Posted by Dongjoon Hyun <do...@apache.org>.
Thank you so much for sharing the progress, Hyukjin.
This sounds reasonable to me. This will reduce the gap between classic and connect mode greatly.
Given that Apache Spark Release Cadence already defines the path to introduce a new feature with its feature flag, I hope we are able to start to test this in Apache Spark 4.3.0.
After stabilizing this new architecture via Spark 4.3.0, we may choose more migration steps in Spark 5 in 2027 or Spark 6 in 2028.
Thanks,
Dongjoon.
On 2026/06/04 06:12:16 Hyukjin Kwon wrote:
> Hi all,
>
> Firstly, I wanted to check other opinions (rather saying this to push my
> opinion hard in this way).
>
> Lately, I have been looking through the feasibility of replacing Py4J to
> Spark Connect in PySpark.
> More specifically I mean that use Spark Connect server, and leave Py4J
> server as a fallback option to avoid breakage.
>
> As we know, Py4J gateway server itself already works similarly with Spark
> Connect server.
> It is a bit of a difficult story in Scala because there wasn't an
> intermediate server in Classic but in PySpark there is already a Py4J
> server running.
>
> There are few downsides of using Py4J:
> - Py4J server itself exposes arbitrary access to the JVM machine which
> actually is risky.
> - Performance is quite slow when sending large data. Note that we are
> switching to the raw sockets when we send large binaries
> - Difficult to debug the errors
>
> With Spark Connect
> - We could limit those accesses.
> - We use Arrow batches to send the data should be more efficient
> - Error handling are quite implemented well in Python Spark Connect
>
> Lastly, structurally they are quite the same. It is not like introducing a
> new layer.
>
> Last time when we chatted about enabling Spark Connect on (for both Scala,
> Python etc.), the biggest pushback was RDD API missing. I recently
> prototyped, and concluded that, for Python specifically, we can add the
> full support of RDD API, and most of SparkContext API (
> https://github.com/apache/spark/pull/55888).
>
> I also prototyped Python client side UI (
> https://github.com/apache/spark/pull/56053) here too, so for Python
> specifically, I believe there is not quite much gap between classic and
> connect. API-wise, structure-wise, etc.
>
> I have been thinking about this idea for quite a long time, and I concluded
> that it is feasible. So I would like to know what others think about this.
>
> Thanks!
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] Replacing Py4J to Spark Connect in PySpark (possibly with RDD API and UI)
Posted by Ángel Álvarez Pascua <an...@gmail.com>.
I’m +0.5 / cautiously supportive.
For PySpark specifically, I agree this is feasible and probably
directionally good: Py4J is leaky, unsafe, hard to debug, and not a great
long-term protocol boundary. Spark Connect gives us a cleaner, constrained,
language-neutral boundary.
But I would be careful not to describe this as “structurally the same” or
just replacing one intermediate server with another. Spark Connect changes
the contract from “Python can reach into the Spark driver JVM” to “Python
describes operations through a protocol.” That is a major semantic change,
even if we can paper over much of the public API.
The hard part is not only API parity. It is preserving, rejecting, or
replacing the ecosystem behaviors that grew around Py4J: _jvm, _jsc, _jdf,
custom Java integrations, RDD closure semantics, SparkContext-side effects,
debugging habits, and local-mode expectations.
I’d support an experimental/default-off path with Py4J fallback, a public
compatibility matrix, benchmarks, and explicit migration guidance. If we
can show that RDD/SparkContext support is real rather than a compatibility
façade, and that common JVM-dependent extension patterns have a story, then
this could be a very good direction.
El jue, 4 jun 2026 a las 8:13, Hyukjin Kwon (<gu...@apache.org>)
escribió:
> Hi all,
>
> Firstly, I wanted to check other opinions (rather saying this to push my
> opinion hard in this way).
>
> Lately, I have been looking through the feasibility of replacing Py4J to
> Spark Connect in PySpark.
> More specifically I mean that use Spark Connect server, and leave Py4J
> server as a fallback option to avoid breakage.
>
> As we know, Py4J gateway server itself already works similarly with Spark
> Connect server.
> It is a bit of a difficult story in Scala because there wasn't an
> intermediate server in Classic but in PySpark there is already a Py4J
> server running.
>
> There are few downsides of using Py4J:
> - Py4J server itself exposes arbitrary access to the JVM machine which
> actually is risky.
> - Performance is quite slow when sending large data. Note that we are
> switching to the raw sockets when we send large binaries
> - Difficult to debug the errors
>
> With Spark Connect
> - We could limit those accesses.
> - We use Arrow batches to send the data should be more efficient
> - Error handling are quite implemented well in Python Spark Connect
>
> Lastly, structurally they are quite the same. It is not like introducing a
> new layer.
>
> Last time when we chatted about enabling Spark Connect on (for both Scala,
> Python etc.), the biggest pushback was RDD API missing. I recently
> prototyped, and concluded that, for Python specifically, we can add the
> full support of RDD API, and most of SparkContext API (
> https://github.com/apache/spark/pull/55888).
>
> I also prototyped Python client side UI (
> https://github.com/apache/spark/pull/56053) here too, so for Python
> specifically, I believe there is not quite much gap between classic and
> connect. API-wise, structure-wise, etc.
>
> I have been thinking about this idea for quite a long time, and I
> concluded that it is feasible. So I would like to know what others think
> about this.
>
> Thanks!
>