You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Dongjoon Hyun <do...@apache.org> on 2025/01/21 22:31:26 UTC
FYI: A Hallucination about Spark Connect Stability in Spark 4
It seems that there is misinformation about the stability of Spark Connect
in Spark 4. I would like to reduce the gap in our dev mailing list.
Frequently, some people claim `Spark Connect` is stable because it uses
Protobuf. Yes, we standardize the interface layer. However, may I ask if it
implies its implementation's stability?
Since Apache Spark is an open source community, you can see the stability
of implementation in our public CI. In our CI, the PySpark Connect client
has been technically broken most of the time.
1.
https://github.com/apache/spark/actions/workflows/build_python_connect.yml
(Spark Connect Python-only in master)
In addition, the Spark 3.5 client seems to face another difficulty talking
with Spark 4 server.
2.
https://github.com/apache/spark/actions/workflows/build_python_connect35.yml
(Spark Connect Python-only:master-server, 35-client)
3. What about the stability and the feature parities in different
languages? Do they work well with Apache Spark 4? I'm wondering if there is
any clue for the Apache Spark community to do assessment?
Given (1), (2), and (3), how can we make sure that `Spark Connect` is
stable or ready in Spark 4? From my perspective, this is still actively
under development with an open end.
The bottom line is `Spark Connect` needs more community love in order to be
claimed as Stable in Apache Spark 4. I'm looking forward to seeing the
healthy Spark Connect CI in Spark 4. Until then, let's clarify what is
stable in `Spark Connect` and what is not yet.
Best Regards,
Dongjoon.
PS.
This is a seperate thread from the previous flakiness issues.
https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq
([FYI] Known `Spark Connect` Test Suite Flakiness)
Re: FYI: A Hallucination about Spark Connect Stability in Spark 4
Posted by Dongjoon Hyun <do...@apache.org>.
To be clear, (1) is `PySpark 4.0 Client` + `Spark 4.0 Server`, which is more severe.
And, your point matches with (2) exactly. Thank you for your reply, Holden.
Dongjoon.
On 2025/01/21 22:38:20 Holden Karau wrote:
> Interesting. So given one of the features of Spark connect should be
> simpler migrations we should (in my mind) only declare it stable once we’ve
> gone through two releases where the previous client + its code can talk to
> the new server.
>
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
>
> On Tue, Jan 21, 2025 at 12:31 PM Dongjoon Hyun <do...@apache.org> wrote:
>
> > It seems that there is misinformation about the stability of Spark Connect
> > in Spark 4. I would like to reduce the gap in our dev mailing list.
> >
> > Frequently, some people claim `Spark Connect` is stable because it uses
> > Protobuf. Yes, we standardize the interface layer. However, may I ask if it
> > implies its implementation's stability?
> >
> > Since Apache Spark is an open source community, you can see the stability
> > of implementation in our public CI. In our CI, the PySpark Connect client
> > has been technically broken most of the time.
> >
> > 1.
> > https://github.com/apache/spark/actions/workflows/build_python_connect.yml
> > (Spark Connect Python-only in master)
> >
> > In addition, the Spark 3.5 client seems to face another difficulty talking
> > with Spark 4 server.
> >
> > 2.
> > https://github.com/apache/spark/actions/workflows/build_python_connect35.yml
> > (Spark Connect Python-only:master-server, 35-client)
> >
> > 3. What about the stability and the feature parities in different
> > languages? Do they work well with Apache Spark 4? I'm wondering if there is
> > any clue for the Apache Spark community to do assessment?
> >
> > Given (1), (2), and (3), how can we make sure that `Spark Connect` is
> > stable or ready in Spark 4? From my perspective, this is still actively
> > under development with an open end.
> >
> > The bottom line is `Spark Connect` needs more community love in order to
> > be claimed as Stable in Apache Spark 4. I'm looking forward to seeing the
> > healthy Spark Connect CI in Spark 4. Until then, let's clarify what is
> > stable in `Spark Connect` and what is not yet.
> >
> > Best Regards,
> > Dongjoon.
> >
> > PS.
> > This is a seperate thread from the previous flakiness issues.
> > https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq
> > ([FYI] Known `Spark Connect` Test Suite Flakiness)
> >
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
RE: FYI: A Hallucination about Spark Connect Stability in Spark 4
Posted by Balaji Sudharsanam V <ba...@ibm.com.INVALID>.
Do we have a Java Client for Spark Connect which is something like PySpark?
From: Mich Talebzadeh <mi...@gmail.com>
Sent: 22 January 2025 15:05
To: Hyukjin Kwon <gu...@apache.org>
Cc: Martin Grund <ma...@databricks.com.invalid>; Holden Karau <ho...@gmail.com>; Dongjoon Hyun <do...@apache.org>; dev <de...@spark.apache.org>
Subject: [EXTERNAL] Re: FYI: A Hallucination about Spark Connect Stability in Spark 4
CI broken is really an operational aspect albeit in this case was quote temporary. We should put that aside and move on as 1) product is sound and 2) spark connect is strategic for the future of Spark. HTH Mich Talebzadeh, Architect | Data Science
CI broken is really an operational aspect albeit in this case was quote temporary. We should put that aside and move on as 1) product is sound and 2) spark connect is strategic for the future of Spark.
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
[https://ci3.googleusercontent.com/mail-sig/AIorK4zholKucR2Q9yMrKbHNn-o1TuS4mYXyi2KO6Xmx6ikHPySa9MLaLZ8t2hrA6AUcxSxDgHIwmKE] view my Linkedin profile<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/ >
On Wed, 22 Jan 2025 at 09:26, Hyukjin Kwon <gu...@apache.org>> wrote:
While it might be a bit too much to talk about its stability, it is true that the CI dedicated for Spark Connect compat was broken there for a couple of weeks, and the errors from the tests look confusing.
I agree that tests and builds could be one of the easiest measurements to tell the state of a project, and we should probably make sure those builds pass properly.
I don't particularly agree with either side (either it is unstable as CI is broken or, it is perfectly stable with CI broken).
The truth is that the scheduled build for Spark Connect is broken for a couple of weeks, which is bad. We should fix it to keep the project running healthily.
On Wed, 22 Jan 2025 at 18:13, Martin Grund <ma...@databricks.com.invalid>> wrote:
I'm very confused about how we use stability in CI as a measure to discuss the strategy of a particular feature, particularly because we call these "hallucinations."
From real-world experience, I can say that we have thousands of clients using Spark Connect across many different versions in our infrastructure without any issues and this since Spark 3.4 (e.g. Spark 3.4 clients talking to Spark 3.5 and above).
Second, from maintaining the Golang client the Spark 3.5 based Golang client works nicely against a Spark 4 preview build without any issue.
If you look at this then we have multiple releases already where we successfully retained parity without any issues.
From this perspective, I think it's perfectly fine to declare Spark Connect as stable.
On Tue, Jan 21, 2025 at 11:40 PM Holden Karau <ho...@gmail.com>> wrote:
Interesting. So given one of the features of Spark connect should be simpler migrations we should (in my mind) only declare it stable once we’ve gone through two releases where the previous client + its code can talk to the new server.
Twitter: https://twitter.com/holdenkarau<https://twitter.com/holdenkarau >
Fight Health Insurance: https://www.fighthealthinsurance.com/<https://www.fighthealthinsurance.com/?q=hk_email >
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9 >
YouTube Live Streams: https://www.youtube.com/user/holdenkarau<https://www.youtube.com/user/holdenkarau >
Pronouns: she/her
On Tue, Jan 21, 2025 at 12:31 PM Dongjoon Hyun <do...@apache.org>> wrote:
It seems that there is misinformation about the stability of Spark Connect in Spark 4. I would like to reduce the gap in our dev mailing list.
Frequently, some people claim `Spark Connect` is stable because it uses Protobuf. Yes, we standardize the interface layer. However, may I ask if it implies its implementation's stability?
Since Apache Spark is an open source community, you can see the stability of implementation in our public CI. In our CI, the PySpark Connect client has been technically broken most of the time.
1. https://github.com/apache/spark/actions/workflows/build_python_connect.yml<https://github.com/apache/spark/actions/workflows/build_python_connect.yml >
(Spark Connect Python-only in master)
In addition, the Spark 3.5 client seems to face another difficulty talking with Spark 4 server.
2. https://github.com/apache/spark/actions/workflows/build_python_connect35.yml<https://github.com/apache/spark/actions/workflows/build_python_connect35.yml >
(Spark Connect Python-only:master-server, 35-client)
3. What about the stability and the feature parities in different languages? Do they work well with Apache Spark 4? I'm wondering if there is any clue for the Apache Spark community to do assessment?
Given (1), (2), and (3), how can we make sure that `Spark Connect` is stable or ready in Spark 4? From my perspective, this is still actively under development with an open end.
The bottom line is `Spark Connect` needs more community love in order to be claimed as Stable in Apache Spark 4. I'm looking forward to seeing the healthy Spark Connect CI in Spark 4. Until then, let's clarify what is stable in `Spark Connect` and what is not yet.
Best Regards,
Dongjoon.
PS.
This is a seperate thread from the previous flakiness issues.
https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq<https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq >
([FYI] Known `Spark Connect` Test Suite Flakiness)
Re: FYI: A Hallucination about Spark Connect Stability in Spark 4
Posted by Mich Talebzadeh <mi...@gmail.com>.
CI broken is really an operational aspect albeit in this case was quote
temporary. We should put that aside and move on as 1) product is sound and
2) spark connect is strategic for the future of Spark.
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Wed, 22 Jan 2025 at 09:26, Hyukjin Kwon <gu...@apache.org> wrote:
> While it might be a bit too much to talk about its stability, it is true
> that the CI dedicated for Spark Connect compat was broken there for a
> couple of weeks, and the errors from the tests look confusing.
> I agree that tests and builds could be one of the easiest measurements to
> tell the state of a project, and we should probably make sure those builds
> pass properly.
>
> I don't particularly agree with either side (either it is unstable as CI
> is broken or, it is perfectly stable with CI broken).
> The truth is that the scheduled build for Spark Connect is broken for a
> couple of weeks, which is bad. We should fix it to keep the project running
> healthily.
>
>
> On Wed, 22 Jan 2025 at 18:13, Martin Grund <ma...@databricks.com.invalid>
> wrote:
>
>> I'm very confused about how we use stability in CI as a measure to
>> discuss the strategy of a particular feature, particularly because we call
>> these "hallucinations."
>>
>> From real-world experience, I can say that we have thousands of clients
>> using Spark Connect across many different versions in our infrastructure
>> without any issues and this since Spark 3.4 (e.g. Spark 3.4 clients talking
>> to Spark 3.5 and above).
>>
>> Second, from maintaining the Golang client the Spark 3.5 based Golang
>> client works nicely against a Spark 4 preview build without any issue.
>>
>> If you look at this then we have multiple releases already where we
>> successfully retained parity without any issues.
>>
>> From this perspective, I think it's perfectly fine to declare Spark
>> Connect as stable.
>>
>> On Tue, Jan 21, 2025 at 11:40 PM Holden Karau <ho...@gmail.com>
>> wrote:
>>
>>> Interesting. So given one of the features of Spark connect should be
>>> simpler migrations we should (in my mind) only declare it stable once we’ve
>>> gone through two releases where the previous client + its code can talk to
>>> the new server.
>>>
>>> Twitter: https://twitter.com/holdenkarau
>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>> Pronouns: she/her
>>>
>>>
>>> On Tue, Jan 21, 2025 at 12:31 PM Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>>
>>>> It seems that there is misinformation about the stability of Spark
>>>> Connect in Spark 4. I would like to reduce the gap in our dev mailing list.
>>>>
>>>> Frequently, some people claim `Spark Connect` is stable because it uses
>>>> Protobuf. Yes, we standardize the interface layer. However, may I ask if it
>>>> implies its implementation's stability?
>>>>
>>>> Since Apache Spark is an open source community, you can see the
>>>> stability of implementation in our public CI. In our CI, the PySpark
>>>> Connect client has been technically broken most of the time.
>>>>
>>>> 1.
>>>> https://github.com/apache/spark/actions/workflows/build_python_connect.yml
>>>> (Spark Connect Python-only in master)
>>>>
>>>> In addition, the Spark 3.5 client seems to face another difficulty
>>>> talking with Spark 4 server.
>>>>
>>>> 2.
>>>> https://github.com/apache/spark/actions/workflows/build_python_connect35.yml
>>>> (Spark Connect Python-only:master-server, 35-client)
>>>>
>>>> 3. What about the stability and the feature parities in different
>>>> languages? Do they work well with Apache Spark 4? I'm wondering if there is
>>>> any clue for the Apache Spark community to do assessment?
>>>>
>>>> Given (1), (2), and (3), how can we make sure that `Spark Connect` is
>>>> stable or ready in Spark 4? From my perspective, this is still actively
>>>> under development with an open end.
>>>>
>>>> The bottom line is `Spark Connect` needs more community love in order
>>>> to be claimed as Stable in Apache Spark 4. I'm looking forward to seeing
>>>> the healthy Spark Connect CI in Spark 4. Until then, let's clarify what is
>>>> stable in `Spark Connect` and what is not yet.
>>>>
>>>> Best Regards,
>>>> Dongjoon.
>>>>
>>>> PS.
>>>> This is a seperate thread from the previous flakiness issues.
>>>> https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq
>>>> ([FYI] Known `Spark Connect` Test Suite Flakiness)
>>>>
>>>
Re: FYI: A Hallucination about Spark Connect Stability in Spark 4
Posted by Hyukjin Kwon <gu...@apache.org>.
While it might be a bit too much to talk about its stability, it is true
that the CI dedicated for Spark Connect compat was broken there for a
couple of weeks, and the errors from the tests look confusing.
I agree that tests and builds could be one of the easiest measurements to
tell the state of a project, and we should probably make sure those builds
pass properly.
I don't particularly agree with either side (either it is unstable as CI is
broken or, it is perfectly stable with CI broken).
The truth is that the scheduled build for Spark Connect is broken for a
couple of weeks, which is bad. We should fix it to keep the project running
healthily.
On Wed, 22 Jan 2025 at 18:13, Martin Grund <ma...@databricks.com.invalid>
wrote:
> I'm very confused about how we use stability in CI as a measure to discuss
> the strategy of a particular feature, particularly because we call these
> "hallucinations."
>
> From real-world experience, I can say that we have thousands of clients
> using Spark Connect across many different versions in our infrastructure
> without any issues and this since Spark 3.4 (e.g. Spark 3.4 clients talking
> to Spark 3.5 and above).
>
> Second, from maintaining the Golang client the Spark 3.5 based Golang
> client works nicely against a Spark 4 preview build without any issue.
>
> If you look at this then we have multiple releases already where we
> successfully retained parity without any issues.
>
> From this perspective, I think it's perfectly fine to declare Spark
> Connect as stable.
>
> On Tue, Jan 21, 2025 at 11:40 PM Holden Karau <ho...@gmail.com>
> wrote:
>
>> Interesting. So given one of the features of Spark connect should be
>> simpler migrations we should (in my mind) only declare it stable once we’ve
>> gone through two releases where the previous client + its code can talk to
>> the new server.
>>
>> Twitter: https://twitter.com/holdenkarau
>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>> <https://www.fighthealthinsurance.com/?q=hk_email>
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>> Pronouns: she/her
>>
>>
>> On Tue, Jan 21, 2025 at 12:31 PM Dongjoon Hyun <do...@apache.org>
>> wrote:
>>
>>> It seems that there is misinformation about the stability of Spark
>>> Connect in Spark 4. I would like to reduce the gap in our dev mailing list.
>>>
>>> Frequently, some people claim `Spark Connect` is stable because it uses
>>> Protobuf. Yes, we standardize the interface layer. However, may I ask if it
>>> implies its implementation's stability?
>>>
>>> Since Apache Spark is an open source community, you can see the
>>> stability of implementation in our public CI. In our CI, the PySpark
>>> Connect client has been technically broken most of the time.
>>>
>>> 1.
>>> https://github.com/apache/spark/actions/workflows/build_python_connect.yml
>>> (Spark Connect Python-only in master)
>>>
>>> In addition, the Spark 3.5 client seems to face another difficulty
>>> talking with Spark 4 server.
>>>
>>> 2.
>>> https://github.com/apache/spark/actions/workflows/build_python_connect35.yml
>>> (Spark Connect Python-only:master-server, 35-client)
>>>
>>> 3. What about the stability and the feature parities in different
>>> languages? Do they work well with Apache Spark 4? I'm wondering if there is
>>> any clue for the Apache Spark community to do assessment?
>>>
>>> Given (1), (2), and (3), how can we make sure that `Spark Connect` is
>>> stable or ready in Spark 4? From my perspective, this is still actively
>>> under development with an open end.
>>>
>>> The bottom line is `Spark Connect` needs more community love in order to
>>> be claimed as Stable in Apache Spark 4. I'm looking forward to seeing the
>>> healthy Spark Connect CI in Spark 4. Until then, let's clarify what is
>>> stable in `Spark Connect` and what is not yet.
>>>
>>> Best Regards,
>>> Dongjoon.
>>>
>>> PS.
>>> This is a seperate thread from the previous flakiness issues.
>>> https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq
>>> ([FYI] Known `Spark Connect` Test Suite Flakiness)
>>>
>>
Re: FYI: A Hallucination about Spark Connect Stability in Spark 4
Posted by Martin Grund <ma...@databricks.com.INVALID>.
I'm very confused about how we use stability in CI as a measure to discuss
the strategy of a particular feature, particularly because we call these
"hallucinations."
From real-world experience, I can say that we have thousands of clients
using Spark Connect across many different versions in our infrastructure
without any issues and this since Spark 3.4 (e.g. Spark 3.4 clients talking
to Spark 3.5 and above).
Second, from maintaining the Golang client the Spark 3.5 based Golang
client works nicely against a Spark 4 preview build without any issue.
If you look at this then we have multiple releases already where we
successfully retained parity without any issues.
From this perspective, I think it's perfectly fine to declare Spark Connect
as stable.
On Tue, Jan 21, 2025 at 11:40 PM Holden Karau <ho...@gmail.com>
wrote:
> Interesting. So given one of the features of Spark connect should be
> simpler migrations we should (in my mind) only declare it stable once we’ve
> gone through two releases where the previous client + its code can talk to
> the new server.
>
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
>
> On Tue, Jan 21, 2025 at 12:31 PM Dongjoon Hyun <do...@apache.org>
> wrote:
>
>> It seems that there is misinformation about the stability of Spark
>> Connect in Spark 4. I would like to reduce the gap in our dev mailing list.
>>
>> Frequently, some people claim `Spark Connect` is stable because it uses
>> Protobuf. Yes, we standardize the interface layer. However, may I ask if it
>> implies its implementation's stability?
>>
>> Since Apache Spark is an open source community, you can see the stability
>> of implementation in our public CI. In our CI, the PySpark Connect client
>> has been technically broken most of the time.
>>
>> 1.
>> https://github.com/apache/spark/actions/workflows/build_python_connect.yml
>> (Spark Connect Python-only in master)
>>
>> In addition, the Spark 3.5 client seems to face another difficulty
>> talking with Spark 4 server.
>>
>> 2.
>> https://github.com/apache/spark/actions/workflows/build_python_connect35.yml
>> (Spark Connect Python-only:master-server, 35-client)
>>
>> 3. What about the stability and the feature parities in different
>> languages? Do they work well with Apache Spark 4? I'm wondering if there is
>> any clue for the Apache Spark community to do assessment?
>>
>> Given (1), (2), and (3), how can we make sure that `Spark Connect` is
>> stable or ready in Spark 4? From my perspective, this is still actively
>> under development with an open end.
>>
>> The bottom line is `Spark Connect` needs more community love in order to
>> be claimed as Stable in Apache Spark 4. I'm looking forward to seeing the
>> healthy Spark Connect CI in Spark 4. Until then, let's clarify what is
>> stable in `Spark Connect` and what is not yet.
>>
>> Best Regards,
>> Dongjoon.
>>
>> PS.
>> This is a seperate thread from the previous flakiness issues.
>> https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq
>> ([FYI] Known `Spark Connect` Test Suite Flakiness)
>>
>
Re: FYI: A Hallucination about Spark Connect Stability in Spark 4
Posted by Dongjoon Hyun <do...@apache.org>.
Thank you, Hyukjin!
Dongjoon
On Tue, Jan 21, 2025 at 16:10 Hyukjin Kwon <gu...@apache.org> wrote:
> Just a quick note on that: the major reason is 1. OOM we should figure out
> and fix the CI environment. 2. structured streaming test failure that is
> still in development.
> I made an umbrella JIRA (https://issues.apache.org/jira/browse/SPARK-50907),
> and I will work there. Should be easier to look at what was the actual
> issue there.
>
> On Wed, 22 Jan 2025 at 09:04, Hyukjin Kwon <gu...@apache.org> wrote:
>
>> Let me take a look. shouldn't be a major issue.
>>
>> On Wed, 22 Jan 2025 at 08:31, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> As discussed on a thread over the weekend, we agreed among us including
>>> Matei on a shift towards a more stable and version-independent APIs.
>>> Spark Connect IMO is a key enabler of this shift, allowing users and
>>> developers to build applications and libraries that are more resilient to
>>> changes in Spark's internals as opposed to RDDs. *Moreover, **maintaining
>>> backward compatibility fo*r the existing *RDD-based applications and
>>> libraries* is crucial during this transition window so the timeframe is
>>> another factor for consideration.
>>>
>>> HTH
>>>
>>> Mich Talebzadeh,
>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, 21 Jan 2025 at 22:40, Holden Karau <ho...@gmail.com>
>>> wrote:
>>>
>>>> Interesting. So given one of the features of Spark connect should be
>>>> simpler migrations we should (in my mind) only declare it stable once we’ve
>>>> gone through two releases where the previous client + its code can talk to
>>>> the new server.
>>>>
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>> Pronouns: she/her
>>>>
>>>>
>>>> On Tue, Jan 21, 2025 at 12:31 PM Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>>
>>>>> It seems that there is misinformation about the stability of Spark
>>>>> Connect in Spark 4. I would like to reduce the gap in our dev mailing list.
>>>>>
>>>>> Frequently, some people claim `Spark Connect` is stable because it
>>>>> uses Protobuf. Yes, we standardize the interface layer. However, may I ask
>>>>> if it implies its implementation's stability?
>>>>>
>>>>> Since Apache Spark is an open source community, you can see the
>>>>> stability of implementation in our public CI. In our CI, the PySpark
>>>>> Connect client has been technically broken most of the time.
>>>>>
>>>>> 1.
>>>>> https://github.com/apache/spark/actions/workflows/build_python_connect.yml
>>>>> (Spark Connect Python-only in master)
>>>>>
>>>>> In addition, the Spark 3.5 client seems to face another difficulty
>>>>> talking with Spark 4 server.
>>>>>
>>>>> 2.
>>>>> https://github.com/apache/spark/actions/workflows/build_python_connect35.yml
>>>>> (Spark Connect Python-only:master-server, 35-client)
>>>>>
>>>>> 3. What about the stability and the feature parities in different
>>>>> languages? Do they work well with Apache Spark 4? I'm wondering if there is
>>>>> any clue for the Apache Spark community to do assessment?
>>>>>
>>>>> Given (1), (2), and (3), how can we make sure that `Spark Connect` is
>>>>> stable or ready in Spark 4? From my perspective, this is still actively
>>>>> under development with an open end.
>>>>>
>>>>> The bottom line is `Spark Connect` needs more community love in order
>>>>> to be claimed as Stable in Apache Spark 4. I'm looking forward to seeing
>>>>> the healthy Spark Connect CI in Spark 4. Until then, let's clarify what is
>>>>> stable in `Spark Connect` and what is not yet.
>>>>>
>>>>> Best Regards,
>>>>> Dongjoon.
>>>>>
>>>>> PS.
>>>>> This is a seperate thread from the previous flakiness issues.
>>>>> https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq
>>>>> ([FYI] Known `Spark Connect` Test Suite Flakiness)
>>>>>
>>>>
Re: FYI: A Hallucination about Spark Connect Stability in Spark 4
Posted by Ángel <an...@gmail.com>.
I'm passionate about and have lots of experience fixing OOMs. Contact me if
you need some help.
El mié, 22 ene 2025, 1:10, Hyukjin Kwon <gu...@apache.org> escribió:
> Just a quick note on that: the major reason is 1. OOM we should figure out
> and fix the CI environment. 2. structured streaming test failure that is
> still in development.
> I made an umbrella JIRA (https://issues.apache.org/jira/browse/SPARK-50907),
> and I will work there. Should be easier to look at what was the actual
> issue there.
>
> On Wed, 22 Jan 2025 at 09:04, Hyukjin Kwon <gu...@apache.org> wrote:
>
>> Let me take a look. shouldn't be a major issue.
>>
>> On Wed, 22 Jan 2025 at 08:31, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> As discussed on a thread over the weekend, we agreed among us including
>>> Matei on a shift towards a more stable and version-independent APIs.
>>> Spark Connect IMO is a key enabler of this shift, allowing users and
>>> developers to build applications and libraries that are more resilient to
>>> changes in Spark's internals as opposed to RDDs. *Moreover, **maintaining
>>> backward compatibility fo*r the existing *RDD-based applications and
>>> libraries* is crucial during this transition window so the timeframe is
>>> another factor for consideration.
>>>
>>> HTH
>>>
>>> Mich Talebzadeh,
>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, 21 Jan 2025 at 22:40, Holden Karau <ho...@gmail.com>
>>> wrote:
>>>
>>>> Interesting. So given one of the features of Spark connect should be
>>>> simpler migrations we should (in my mind) only declare it stable once we’ve
>>>> gone through two releases where the previous client + its code can talk to
>>>> the new server.
>>>>
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>> Pronouns: she/her
>>>>
>>>>
>>>> On Tue, Jan 21, 2025 at 12:31 PM Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>>
>>>>> It seems that there is misinformation about the stability of Spark
>>>>> Connect in Spark 4. I would like to reduce the gap in our dev mailing list.
>>>>>
>>>>> Frequently, some people claim `Spark Connect` is stable because it
>>>>> uses Protobuf. Yes, we standardize the interface layer. However, may I ask
>>>>> if it implies its implementation's stability?
>>>>>
>>>>> Since Apache Spark is an open source community, you can see the
>>>>> stability of implementation in our public CI. In our CI, the PySpark
>>>>> Connect client has been technically broken most of the time.
>>>>>
>>>>> 1.
>>>>> https://github.com/apache/spark/actions/workflows/build_python_connect.yml
>>>>> (Spark Connect Python-only in master)
>>>>>
>>>>> In addition, the Spark 3.5 client seems to face another difficulty
>>>>> talking with Spark 4 server.
>>>>>
>>>>> 2.
>>>>> https://github.com/apache/spark/actions/workflows/build_python_connect35.yml
>>>>> (Spark Connect Python-only:master-server, 35-client)
>>>>>
>>>>> 3. What about the stability and the feature parities in different
>>>>> languages? Do they work well with Apache Spark 4? I'm wondering if there is
>>>>> any clue for the Apache Spark community to do assessment?
>>>>>
>>>>> Given (1), (2), and (3), how can we make sure that `Spark Connect` is
>>>>> stable or ready in Spark 4? From my perspective, this is still actively
>>>>> under development with an open end.
>>>>>
>>>>> The bottom line is `Spark Connect` needs more community love in order
>>>>> to be claimed as Stable in Apache Spark 4. I'm looking forward to seeing
>>>>> the healthy Spark Connect CI in Spark 4. Until then, let's clarify what is
>>>>> stable in `Spark Connect` and what is not yet.
>>>>>
>>>>> Best Regards,
>>>>> Dongjoon.
>>>>>
>>>>> PS.
>>>>> This is a seperate thread from the previous flakiness issues.
>>>>> https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq
>>>>> ([FYI] Known `Spark Connect` Test Suite Flakiness)
>>>>>
>>>>
Re: FYI: A Hallucination about Spark Connect Stability in Spark 4
Posted by Jules Damji <ju...@gmail.com>.
Thanks for update and looking into it.
Excuse the thumb typos
On Tue, 21 Jan 2025 at 4:09 PM, Hyukjin Kwon <gu...@apache.org> wrote:
> Just a quick note on that: the major reason is 1. OOM we should figure out
> and fix the CI environment. 2. structured streaming test failure that is
> still in development.
> I made an umbrella JIRA (https://issues.apache.org/jira/browse/SPARK-50907),
> and I will work there. Should be easier to look at what was the actual
> issue there.
>
> On Wed, 22 Jan 2025 at 09:04, Hyukjin Kwon <gu...@apache.org> wrote:
>
>> Let me take a look. shouldn't be a major issue.
>>
>> On Wed, 22 Jan 2025 at 08:31, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> As discussed on a thread over the weekend, we agreed among us including
>>> Matei on a shift towards a more stable and version-independent APIs.
>>> Spark Connect IMO is a key enabler of this shift, allowing users and
>>> developers to build applications and libraries that are more resilient to
>>> changes in Spark's internals as opposed to RDDs. *Moreover, **maintaining
>>> backward compatibility fo*r the existing *RDD-based applications and
>>> libraries* is crucial during this transition window so the timeframe is
>>> another factor for consideration.
>>>
>>> HTH
>>>
>>> Mich Talebzadeh,
>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, 21 Jan 2025 at 22:40, Holden Karau <ho...@gmail.com>
>>> wrote:
>>>
>>>> Interesting. So given one of the features of Spark connect should be
>>>> simpler migrations we should (in my mind) only declare it stable once we’ve
>>>> gone through two releases where the previous client + its code can talk to
>>>> the new server.
>>>>
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>> Pronouns: she/her
>>>>
>>>>
>>>> On Tue, Jan 21, 2025 at 12:31 PM Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>>
>>>>> It seems that there is misinformation about the stability of Spark
>>>>> Connect in Spark 4. I would like to reduce the gap in our dev mailing list.
>>>>>
>>>>> Frequently, some people claim `Spark Connect` is stable because it
>>>>> uses Protobuf. Yes, we standardize the interface layer. However, may I ask
>>>>> if it implies its implementation's stability?
>>>>>
>>>>> Since Apache Spark is an open source community, you can see the
>>>>> stability of implementation in our public CI. In our CI, the PySpark
>>>>> Connect client has been technically broken most of the time.
>>>>>
>>>>> 1.
>>>>> https://github.com/apache/spark/actions/workflows/build_python_connect.yml
>>>>> (Spark Connect Python-only in master)
>>>>>
>>>>> In addition, the Spark 3.5 client seems to face another difficulty
>>>>> talking with Spark 4 server.
>>>>>
>>>>> 2.
>>>>> https://github.com/apache/spark/actions/workflows/build_python_connect35.yml
>>>>> (Spark Connect Python-only:master-server, 35-client)
>>>>>
>>>>> 3. What about the stability and the feature parities in different
>>>>> languages? Do they work well with Apache Spark 4? I'm wondering if there is
>>>>> any clue for the Apache Spark community to do assessment?
>>>>>
>>>>> Given (1), (2), and (3), how can we make sure that `Spark Connect` is
>>>>> stable or ready in Spark 4? From my perspective, this is still actively
>>>>> under development with an open end.
>>>>>
>>>>> The bottom line is `Spark Connect` needs more community love in order
>>>>> to be claimed as Stable in Apache Spark 4. I'm looking forward to seeing
>>>>> the healthy Spark Connect CI in Spark 4. Until then, let's clarify what is
>>>>> stable in `Spark Connect` and what is not yet.
>>>>>
>>>>> Best Regards,
>>>>> Dongjoon.
>>>>>
>>>>> PS.
>>>>> This is a seperate thread from the previous flakiness issues.
>>>>> https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq
>>>>> ([FYI] Known `Spark Connect` Test Suite Flakiness)
>>>>>
>>>>
Re: FYI: A Hallucination about Spark Connect Stability in Spark 4
Posted by Hyukjin Kwon <gu...@apache.org>.
Just a quick note on that: the major reason is 1. OOM we should figure out
and fix the CI environment. 2. structured streaming test failure that is
still in development.
I made an umbrella JIRA (https://issues.apache.org/jira/browse/SPARK-50907),
and I will work there. Should be easier to look at what was the actual
issue there.
On Wed, 22 Jan 2025 at 09:04, Hyukjin Kwon <gu...@apache.org> wrote:
> Let me take a look. shouldn't be a major issue.
>
> On Wed, 22 Jan 2025 at 08:31, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> As discussed on a thread over the weekend, we agreed among us including
>> Matei on a shift towards a more stable and version-independent APIs.
>> Spark Connect IMO is a key enabler of this shift, allowing users and
>> developers to build applications and libraries that are more resilient to
>> changes in Spark's internals as opposed to RDDs. *Moreover, **maintaining
>> backward compatibility fo*r the existing *RDD-based applications and
>> libraries* is crucial during this transition window so the timeframe is
>> another factor for consideration.
>>
>> HTH
>>
>> Mich Talebzadeh,
>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>>
>>
>> On Tue, 21 Jan 2025 at 22:40, Holden Karau <ho...@gmail.com>
>> wrote:
>>
>>> Interesting. So given one of the features of Spark connect should be
>>> simpler migrations we should (in my mind) only declare it stable once we’ve
>>> gone through two releases where the previous client + its code can talk to
>>> the new server.
>>>
>>> Twitter: https://twitter.com/holdenkarau
>>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>>> <https://www.fighthealthinsurance.com/?q=hk_email>
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>> Pronouns: she/her
>>>
>>>
>>> On Tue, Jan 21, 2025 at 12:31 PM Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>>
>>>> It seems that there is misinformation about the stability of Spark
>>>> Connect in Spark 4. I would like to reduce the gap in our dev mailing list.
>>>>
>>>> Frequently, some people claim `Spark Connect` is stable because it uses
>>>> Protobuf. Yes, we standardize the interface layer. However, may I ask if it
>>>> implies its implementation's stability?
>>>>
>>>> Since Apache Spark is an open source community, you can see the
>>>> stability of implementation in our public CI. In our CI, the PySpark
>>>> Connect client has been technically broken most of the time.
>>>>
>>>> 1.
>>>> https://github.com/apache/spark/actions/workflows/build_python_connect.yml
>>>> (Spark Connect Python-only in master)
>>>>
>>>> In addition, the Spark 3.5 client seems to face another difficulty
>>>> talking with Spark 4 server.
>>>>
>>>> 2.
>>>> https://github.com/apache/spark/actions/workflows/build_python_connect35.yml
>>>> (Spark Connect Python-only:master-server, 35-client)
>>>>
>>>> 3. What about the stability and the feature parities in different
>>>> languages? Do they work well with Apache Spark 4? I'm wondering if there is
>>>> any clue for the Apache Spark community to do assessment?
>>>>
>>>> Given (1), (2), and (3), how can we make sure that `Spark Connect` is
>>>> stable or ready in Spark 4? From my perspective, this is still actively
>>>> under development with an open end.
>>>>
>>>> The bottom line is `Spark Connect` needs more community love in order
>>>> to be claimed as Stable in Apache Spark 4. I'm looking forward to seeing
>>>> the healthy Spark Connect CI in Spark 4. Until then, let's clarify what is
>>>> stable in `Spark Connect` and what is not yet.
>>>>
>>>> Best Regards,
>>>> Dongjoon.
>>>>
>>>> PS.
>>>> This is a seperate thread from the previous flakiness issues.
>>>> https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq
>>>> ([FYI] Known `Spark Connect` Test Suite Flakiness)
>>>>
>>>
Re: FYI: A Hallucination about Spark Connect Stability in Spark 4
Posted by Hyukjin Kwon <gu...@apache.org>.
Let me take a look. shouldn't be a major issue.
On Wed, 22 Jan 2025 at 08:31, Mich Talebzadeh <mi...@gmail.com>
wrote:
> As discussed on a thread over the weekend, we agreed among us including
> Matei on a shift towards a more stable and version-independent APIs.
> Spark Connect IMO is a key enabler of this shift, allowing users and
> developers to build applications and libraries that are more resilient to
> changes in Spark's internals as opposed to RDDs. *Moreover, **maintaining
> backward compatibility fo*r the existing *RDD-based applications and
> libraries* is crucial during this transition window so the timeframe is
> another factor for consideration.
>
> HTH
>
> Mich Talebzadeh,
> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
>
>
> On Tue, 21 Jan 2025 at 22:40, Holden Karau <ho...@gmail.com> wrote:
>
>> Interesting. So given one of the features of Spark connect should be
>> simpler migrations we should (in my mind) only declare it stable once we’ve
>> gone through two releases where the previous client + its code can talk to
>> the new server.
>>
>> Twitter: https://twitter.com/holdenkarau
>> Fight Health Insurance: https://www.fighthealthinsurance.com/
>> <https://www.fighthealthinsurance.com/?q=hk_email>
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>> Pronouns: she/her
>>
>>
>> On Tue, Jan 21, 2025 at 12:31 PM Dongjoon Hyun <do...@apache.org>
>> wrote:
>>
>>> It seems that there is misinformation about the stability of Spark
>>> Connect in Spark 4. I would like to reduce the gap in our dev mailing list.
>>>
>>> Frequently, some people claim `Spark Connect` is stable because it uses
>>> Protobuf. Yes, we standardize the interface layer. However, may I ask if it
>>> implies its implementation's stability?
>>>
>>> Since Apache Spark is an open source community, you can see the
>>> stability of implementation in our public CI. In our CI, the PySpark
>>> Connect client has been technically broken most of the time.
>>>
>>> 1.
>>> https://github.com/apache/spark/actions/workflows/build_python_connect.yml
>>> (Spark Connect Python-only in master)
>>>
>>> In addition, the Spark 3.5 client seems to face another difficulty
>>> talking with Spark 4 server.
>>>
>>> 2.
>>> https://github.com/apache/spark/actions/workflows/build_python_connect35.yml
>>> (Spark Connect Python-only:master-server, 35-client)
>>>
>>> 3. What about the stability and the feature parities in different
>>> languages? Do they work well with Apache Spark 4? I'm wondering if there is
>>> any clue for the Apache Spark community to do assessment?
>>>
>>> Given (1), (2), and (3), how can we make sure that `Spark Connect` is
>>> stable or ready in Spark 4? From my perspective, this is still actively
>>> under development with an open end.
>>>
>>> The bottom line is `Spark Connect` needs more community love in order to
>>> be claimed as Stable in Apache Spark 4. I'm looking forward to seeing the
>>> healthy Spark Connect CI in Spark 4. Until then, let's clarify what is
>>> stable in `Spark Connect` and what is not yet.
>>>
>>> Best Regards,
>>> Dongjoon.
>>>
>>> PS.
>>> This is a seperate thread from the previous flakiness issues.
>>> https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq
>>> ([FYI] Known `Spark Connect` Test Suite Flakiness)
>>>
>>
Re: FYI: A Hallucination about Spark Connect Stability in Spark 4
Posted by Mich Talebzadeh <mi...@gmail.com>.
As discussed on a thread over the weekend, we agreed among us including
Matei on a shift towards a more stable and version-independent APIs.
Spark Connect IMO is a key enabler of this shift, allowing users and
developers to build applications and libraries that are more resilient to
changes in Spark's internals as opposed to RDDs. *Moreover, **maintaining
backward compatibility fo*r the existing *RDD-based applications and
libraries* is crucial during this transition window so the timeframe is
another factor for consideration.
HTH
Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Tue, 21 Jan 2025 at 22:40, Holden Karau <ho...@gmail.com> wrote:
> Interesting. So given one of the features of Spark connect should be
> simpler migrations we should (in my mind) only declare it stable once we’ve
> gone through two releases where the previous client + its code can talk to
> the new server.
>
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
>
> On Tue, Jan 21, 2025 at 12:31 PM Dongjoon Hyun <do...@apache.org>
> wrote:
>
>> It seems that there is misinformation about the stability of Spark
>> Connect in Spark 4. I would like to reduce the gap in our dev mailing list.
>>
>> Frequently, some people claim `Spark Connect` is stable because it uses
>> Protobuf. Yes, we standardize the interface layer. However, may I ask if it
>> implies its implementation's stability?
>>
>> Since Apache Spark is an open source community, you can see the stability
>> of implementation in our public CI. In our CI, the PySpark Connect client
>> has been technically broken most of the time.
>>
>> 1.
>> https://github.com/apache/spark/actions/workflows/build_python_connect.yml
>> (Spark Connect Python-only in master)
>>
>> In addition, the Spark 3.5 client seems to face another difficulty
>> talking with Spark 4 server.
>>
>> 2.
>> https://github.com/apache/spark/actions/workflows/build_python_connect35.yml
>> (Spark Connect Python-only:master-server, 35-client)
>>
>> 3. What about the stability and the feature parities in different
>> languages? Do they work well with Apache Spark 4? I'm wondering if there is
>> any clue for the Apache Spark community to do assessment?
>>
>> Given (1), (2), and (3), how can we make sure that `Spark Connect` is
>> stable or ready in Spark 4? From my perspective, this is still actively
>> under development with an open end.
>>
>> The bottom line is `Spark Connect` needs more community love in order to
>> be claimed as Stable in Apache Spark 4. I'm looking forward to seeing the
>> healthy Spark Connect CI in Spark 4. Until then, let's clarify what is
>> stable in `Spark Connect` and what is not yet.
>>
>> Best Regards,
>> Dongjoon.
>>
>> PS.
>> This is a seperate thread from the previous flakiness issues.
>> https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq
>> ([FYI] Known `Spark Connect` Test Suite Flakiness)
>>
>
Re: FYI: A Hallucination about Spark Connect Stability in Spark 4
Posted by Holden Karau <ho...@gmail.com>.
Interesting. So given one of the features of Spark connect should be
simpler migrations we should (in my mind) only declare it stable once we’ve
gone through two releases where the previous client + its code can talk to
the new server.
Twitter: https://twitter.com/holdenkarau
Fight Health Insurance: https://www.fighthealthinsurance.com/
<https://www.fighthealthinsurance.com/?q=hk_email>
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her
On Tue, Jan 21, 2025 at 12:31 PM Dongjoon Hyun <do...@apache.org> wrote:
> It seems that there is misinformation about the stability of Spark Connect
> in Spark 4. I would like to reduce the gap in our dev mailing list.
>
> Frequently, some people claim `Spark Connect` is stable because it uses
> Protobuf. Yes, we standardize the interface layer. However, may I ask if it
> implies its implementation's stability?
>
> Since Apache Spark is an open source community, you can see the stability
> of implementation in our public CI. In our CI, the PySpark Connect client
> has been technically broken most of the time.
>
> 1.
> https://github.com/apache/spark/actions/workflows/build_python_connect.yml
> (Spark Connect Python-only in master)
>
> In addition, the Spark 3.5 client seems to face another difficulty talking
> with Spark 4 server.
>
> 2.
> https://github.com/apache/spark/actions/workflows/build_python_connect35.yml
> (Spark Connect Python-only:master-server, 35-client)
>
> 3. What about the stability and the feature parities in different
> languages? Do they work well with Apache Spark 4? I'm wondering if there is
> any clue for the Apache Spark community to do assessment?
>
> Given (1), (2), and (3), how can we make sure that `Spark Connect` is
> stable or ready in Spark 4? From my perspective, this is still actively
> under development with an open end.
>
> The bottom line is `Spark Connect` needs more community love in order to
> be claimed as Stable in Apache Spark 4. I'm looking forward to seeing the
> healthy Spark Connect CI in Spark 4. Until then, let's clarify what is
> stable in `Spark Connect` and what is not yet.
>
> Best Regards,
> Dongjoon.
>
> PS.
> This is a seperate thread from the previous flakiness issues.
> https://lists.apache.org/thread/r5dzdr3w4ly0dr99k24mqvld06r4mzmq
> ([FYI] Known `Spark Connect` Test Suite Flakiness)
>