You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Hyukjin Kwon <gu...@apache.org> on 2024/07/02 00:19:01 UTC
[DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
Hi all,
I would like to discuss moving Spark Connect server to builtin package.
Right now, users have to specify —packages when they run Spark Connect
server script, for example:
./sbin/start-connect-server.sh --jars `ls
connector/connect/server/target/**/spark-connect*SNAPSHOT.jar`
or
./sbin/start-connect-server.sh --packages
org.apache.spark:spark-connect_2.12:3.5.1
which is a little bit odd that sbin scripts should provide jars to start.
Moving it to builtin package is pretty straightforward because most of jars
are shaded, and the impact would be minimal, I have a prototype here
apache/spark/#47157 <https://github.com/apache/spark/pull/47157>. This also
simplifies Python local running logic a lot.
User facing API layer, Spark Connect Client, stays external but I would
like the internal/admin server layer, Spark Connect Server, implementation
to be built in Spark.
Please let me know if you have thoughts on this!
Re: [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
Posted by Hyukjin Kwon <gu...@apache.org>.
Alrighty, let me start the vote to make sure everybody is happy :-).
On Wed, 3 Jul 2024 at 09:55, Hyukjin Kwon <gu...@apache.org> wrote:
> It will be fine for non-connect users. When we are actually moving client
> one, I think we should go with an SPIP cuz that might affect end users....
>
> On Tue, 2 Jul 2024 at 23:05, Holden Karau <ho...@gmail.com> wrote:
>
>> I guess my one concern here would be are we going to expand the
>> dependencies that are visible on the class path for non-connect users?
>>
>> One of the pain points that folks experienced with upgrading can be from
>> those changing.
>>
>> Otherwise this seems pretty reasonable.
>>
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>>
>> On Tue, Jul 2, 2024 at 5:36 AM Matthew Powers <
>> matthewkevinpowers@gmail.com> wrote:
>>
>>> This is a great idea and would be a great quality of life improvement.
>>>
>>> +1 (non-binding)
>>>
>>> On Tue, Jul 2, 2024 at 4:56 AM Hyukjin Kwon <gu...@apache.org>
>>> wrote:
>>>
>>>> > while leaving the connect jvm client in a separate folder looks weird
>>>>
>>>> I plan to actually put it at the top level together but I feel like
>>>> this has to be done with SPIP so I am moving internal server side first
>>>> orthogonally
>>>>
>>>> On Tue, 2 Jul 2024 at 17:54, Cheng Pan <pa...@gmail.com> wrote:
>>>>
>>>>> Thanks for raising this discussion, I think putting the connect folder
>>>>> on the top level is a good idea to promote Spark Connect, while leaving the
>>>>> connect jvm client in a separate folder looks weird. I suppose there is no
>>>>> contract to leave all optional modules under `connector`? e.g.
>>>>> `resource-managers/kubernetes/{docker,integration-tests}`, `hadoop-cloud`.
>>>>> What about moving the whole `connect` folder to the top level?
>>>>>
>>>>> Thanks,
>>>>> Cheng Pan
>>>>>
>>>>>
>>>>> On Jul 2, 2024, at 08:19, Hyukjin Kwon <gu...@apache.org> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I would like to discuss moving Spark Connect server to builtin
>>>>> package. Right now, users have to specify —packages when they run Spark
>>>>> Connect server script, for example:
>>>>>
>>>>> ./sbin/start-connect-server.sh --jars `ls connector/connect/server/target/**/spark-connect*SNAPSHOT.jar`
>>>>>
>>>>> or
>>>>>
>>>>> ./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.1
>>>>>
>>>>> which is a little bit odd that sbin scripts should provide jars to
>>>>> start.
>>>>>
>>>>> Moving it to builtin package is pretty straightforward because most of
>>>>> jars are shaded, and the impact would be minimal, I have a prototype here
>>>>> apache/spark/#47157 <https://github.com/apache/spark/pull/47157>.
>>>>> This also simplifies Python local running logic a lot.
>>>>>
>>>>> User facing API layer, Spark Connect Client, stays external but I
>>>>> would like the internal/admin server layer, Spark Connect Server,
>>>>> implementation to be built in Spark.
>>>>>
>>>>> Please let me know if you have thoughts on this!
>>>>>
>>>>>
>>>>>
Re: [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
Posted by Hyukjin Kwon <gu...@apache.org>.
It will be fine for non-connect users. When we are actually moving client
one, I think we should go with an SPIP cuz that might affect end users....
On Tue, 2 Jul 2024 at 23:05, Holden Karau <ho...@gmail.com> wrote:
> I guess my one concern here would be are we going to expand the
> dependencies that are visible on the class path for non-connect users?
>
> One of the pain points that folks experienced with upgrading can be from
> those changing.
>
> Otherwise this seems pretty reasonable.
>
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
>
> On Tue, Jul 2, 2024 at 5:36 AM Matthew Powers <
> matthewkevinpowers@gmail.com> wrote:
>
>> This is a great idea and would be a great quality of life improvement.
>>
>> +1 (non-binding)
>>
>> On Tue, Jul 2, 2024 at 4:56 AM Hyukjin Kwon <gu...@apache.org> wrote:
>>
>>> > while leaving the connect jvm client in a separate folder looks weird
>>>
>>> I plan to actually put it at the top level together but I feel like this
>>> has to be done with SPIP so I am moving internal server side first
>>> orthogonally
>>>
>>> On Tue, 2 Jul 2024 at 17:54, Cheng Pan <pa...@gmail.com> wrote:
>>>
>>>> Thanks for raising this discussion, I think putting the connect folder
>>>> on the top level is a good idea to promote Spark Connect, while leaving the
>>>> connect jvm client in a separate folder looks weird. I suppose there is no
>>>> contract to leave all optional modules under `connector`? e.g.
>>>> `resource-managers/kubernetes/{docker,integration-tests}`, `hadoop-cloud`.
>>>> What about moving the whole `connect` folder to the top level?
>>>>
>>>> Thanks,
>>>> Cheng Pan
>>>>
>>>>
>>>> On Jul 2, 2024, at 08:19, Hyukjin Kwon <gu...@apache.org> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I would like to discuss moving Spark Connect server to builtin package.
>>>> Right now, users have to specify —packages when they run Spark Connect
>>>> server script, for example:
>>>>
>>>> ./sbin/start-connect-server.sh --jars `ls connector/connect/server/target/**/spark-connect*SNAPSHOT.jar`
>>>>
>>>> or
>>>>
>>>> ./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.1
>>>>
>>>> which is a little bit odd that sbin scripts should provide jars to
>>>> start.
>>>>
>>>> Moving it to builtin package is pretty straightforward because most of
>>>> jars are shaded, and the impact would be minimal, I have a prototype here
>>>> apache/spark/#47157 <https://github.com/apache/spark/pull/47157>. This
>>>> also simplifies Python local running logic a lot.
>>>>
>>>> User facing API layer, Spark Connect Client, stays external but I would
>>>> like the internal/admin server layer, Spark Connect Server, implementation
>>>> to be built in Spark.
>>>>
>>>> Please let me know if you have thoughts on this!
>>>>
>>>>
>>>>
Re: [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
Posted by Holden Karau <ho...@gmail.com>.
I guess my one concern here would be are we going to expand the
dependencies that are visible on the class path for non-connect users?
One of the pain points that folks experienced with upgrading can be from
those changing.
Otherwise this seems pretty reasonable.
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
On Tue, Jul 2, 2024 at 5:36 AM Matthew Powers <ma...@gmail.com>
wrote:
> This is a great idea and would be a great quality of life improvement.
>
> +1 (non-binding)
>
> On Tue, Jul 2, 2024 at 4:56 AM Hyukjin Kwon <gu...@apache.org> wrote:
>
>> > while leaving the connect jvm client in a separate folder looks weird
>>
>> I plan to actually put it at the top level together but I feel like this
>> has to be done with SPIP so I am moving internal server side first
>> orthogonally
>>
>> On Tue, 2 Jul 2024 at 17:54, Cheng Pan <pa...@gmail.com> wrote:
>>
>>> Thanks for raising this discussion, I think putting the connect folder
>>> on the top level is a good idea to promote Spark Connect, while leaving the
>>> connect jvm client in a separate folder looks weird. I suppose there is no
>>> contract to leave all optional modules under `connector`? e.g.
>>> `resource-managers/kubernetes/{docker,integration-tests}`, `hadoop-cloud`.
>>> What about moving the whole `connect` folder to the top level?
>>>
>>> Thanks,
>>> Cheng Pan
>>>
>>>
>>> On Jul 2, 2024, at 08:19, Hyukjin Kwon <gu...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> I would like to discuss moving Spark Connect server to builtin package.
>>> Right now, users have to specify —packages when they run Spark Connect
>>> server script, for example:
>>>
>>> ./sbin/start-connect-server.sh --jars `ls connector/connect/server/target/**/spark-connect*SNAPSHOT.jar`
>>>
>>> or
>>>
>>> ./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.1
>>>
>>> which is a little bit odd that sbin scripts should provide jars to start.
>>>
>>> Moving it to builtin package is pretty straightforward because most of
>>> jars are shaded, and the impact would be minimal, I have a prototype here
>>> apache/spark/#47157 <https://github.com/apache/spark/pull/47157>. This
>>> also simplifies Python local running logic a lot.
>>>
>>> User facing API layer, Spark Connect Client, stays external but I would
>>> like the internal/admin server layer, Spark Connect Server, implementation
>>> to be built in Spark.
>>>
>>> Please let me know if you have thoughts on this!
>>>
>>>
>>>
Re: [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
Posted by Matthew Powers <ma...@gmail.com>.
This is a great idea and would be a great quality of life improvement.
+1 (non-binding)
On Tue, Jul 2, 2024 at 4:56 AM Hyukjin Kwon <gu...@apache.org> wrote:
> > while leaving the connect jvm client in a separate folder looks weird
>
> I plan to actually put it at the top level together but I feel like this
> has to be done with SPIP so I am moving internal server side first
> orthogonally
>
> On Tue, 2 Jul 2024 at 17:54, Cheng Pan <pa...@gmail.com> wrote:
>
>> Thanks for raising this discussion, I think putting the connect folder on
>> the top level is a good idea to promote Spark Connect, while leaving the
>> connect jvm client in a separate folder looks weird. I suppose there is no
>> contract to leave all optional modules under `connector`? e.g.
>> `resource-managers/kubernetes/{docker,integration-tests}`, `hadoop-cloud`.
>> What about moving the whole `connect` folder to the top level?
>>
>> Thanks,
>> Cheng Pan
>>
>>
>> On Jul 2, 2024, at 08:19, Hyukjin Kwon <gu...@apache.org> wrote:
>>
>> Hi all,
>>
>> I would like to discuss moving Spark Connect server to builtin package.
>> Right now, users have to specify —packages when they run Spark Connect
>> server script, for example:
>>
>> ./sbin/start-connect-server.sh --jars `ls connector/connect/server/target/**/spark-connect*SNAPSHOT.jar`
>>
>> or
>>
>> ./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.1
>>
>> which is a little bit odd that sbin scripts should provide jars to start.
>>
>> Moving it to builtin package is pretty straightforward because most of
>> jars are shaded, and the impact would be minimal, I have a prototype here
>> apache/spark/#47157 <https://github.com/apache/spark/pull/47157>. This
>> also simplifies Python local running logic a lot.
>>
>> User facing API layer, Spark Connect Client, stays external but I would
>> like the internal/admin server layer, Spark Connect Server, implementation
>> to be built in Spark.
>>
>> Please let me know if you have thoughts on this!
>>
>>
>>
Re: [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
Posted by Hyukjin Kwon <gu...@apache.org>.
> while leaving the connect jvm client in a separate folder looks weird
I plan to actually put it at the top level together but I feel like this
has to be done with SPIP so I am moving internal server side first
orthogonally
On Tue, 2 Jul 2024 at 17:54, Cheng Pan <pa...@gmail.com> wrote:
> Thanks for raising this discussion, I think putting the connect folder on
> the top level is a good idea to promote Spark Connect, while leaving the
> connect jvm client in a separate folder looks weird. I suppose there is no
> contract to leave all optional modules under `connector`? e.g.
> `resource-managers/kubernetes/{docker,integration-tests}`, `hadoop-cloud`.
> What about moving the whole `connect` folder to the top level?
>
> Thanks,
> Cheng Pan
>
>
> On Jul 2, 2024, at 08:19, Hyukjin Kwon <gu...@apache.org> wrote:
>
> Hi all,
>
> I would like to discuss moving Spark Connect server to builtin package.
> Right now, users have to specify —packages when they run Spark Connect
> server script, for example:
>
> ./sbin/start-connect-server.sh --jars `ls connector/connect/server/target/**/spark-connect*SNAPSHOT.jar`
>
> or
>
> ./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.1
>
> which is a little bit odd that sbin scripts should provide jars to start.
>
> Moving it to builtin package is pretty straightforward because most of
> jars are shaded, and the impact would be minimal, I have a prototype here
> apache/spark/#47157 <https://github.com/apache/spark/pull/47157>. This
> also simplifies Python local running logic a lot.
>
> User facing API layer, Spark Connect Client, stays external but I would
> like the internal/admin server layer, Spark Connect Server, implementation
> to be built in Spark.
>
> Please let me know if you have thoughts on this!
>
>
>
Re: [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
Posted by Cheng Pan <pa...@gmail.com>.
Thanks for raising this discussion, I think putting the connect folder on the top level is a good idea to promote Spark Connect, while leaving the connect jvm client in a separate folder looks weird. I suppose there is no contract to leave all optional modules under `connector`? e.g. `resource-managers/kubernetes/{docker,integration-tests}`, `hadoop-cloud`. What about moving the whole `connect` folder to the top level?
Thanks,
Cheng Pan
> On Jul 2, 2024, at 08:19, Hyukjin Kwon <gu...@apache.org> wrote:
>
> Hi all,
>
> I would like to discuss moving Spark Connect server to builtin package. Right now, users have to specify —packages when they run Spark Connect server script, for example:
>
> ./sbin/start-connect-server.sh --jars `ls connector/connect/server/target/**/spark-connect*SNAPSHOT.jar`
> or
>
> ./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.1
> which is a little bit odd that sbin scripts should provide jars to start.
>
> Moving it to builtin package is pretty straightforward because most of jars are shaded, and the impact would be minimal, I have a prototype here apache/spark/#47157 <https://github.com/apache/spark/pull/47157>. This also simplifies Python local running logic a lot.
>
> User facing API layer, Spark Connect Client, stays external but I would like the internal/admin server layer, Spark Connect Server, implementation to be built in Spark.
>
> Please let me know if you have thoughts on this!
>
>
Re: [外部邮件] [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
Posted by Martin Grund <ma...@databricks.com.INVALID>.
+1
On Tue, Jul 2, 2024 at 7:19 AM yangjie01 <ya...@baidu.com.invalid>
wrote:
> I have manually attempted to only modify the `assembly/pom.xml` and
> examined the results of executing `dev/make-distribution.sh --tgz`. The
> `spark-connect_2.13-4.0.0-SNAPSHOT.jar` is indeed included in the jars
> directory. However, if rearranging the directories would result in a
> clearer project structure, I believe that would also be a viable approach.
>
>
>
> *发件人**: *Hyukjin Kwon <gu...@apache.org>
> *日期**: *2024年7月2日 星期二 12:00
> *收件人**: *yangjie01 <ya...@baidu.com.invalid>
> *抄送**: *dev <de...@spark.apache.org>
> *主题**: *Re: [外部邮件] [DISCUSS] Move Spark Connect server to builtin package
> (Client API layer stays external)
>
>
>
> My concern is that the `connector` directory is really for
> external/optional packages (and they aren't included in assembly IIRC).. so
> I am hesitant to just change the assembly.
> The actual changes are not quite large but it moves the files around.
>
>
>
> On Tue, 2 Jul 2024 at 12:23, yangjie01 <ya...@baidu.com.invalid>
> wrote:
>
> I'm supportive of this initiative. However, if the purpose is just to
> avoid the additional `--packages` option, it seems that making some
> adjustments to the `assembly/pom.xml` could potentially meet our goal. Is
> it really necessary to restructure the code directory?
>
>
>
> Jie Yang
>
>
>
> *发件人**: *Hyukjin Kwon <gu...@apache.org>
> *日期**: *2024年7月2日 星期二 08:19
> *收件人**: *dev <de...@spark.apache.org>
> *主题**: *[外部邮件] [DISCUSS] Move Spark Connect server to builtin package
> (Client API layer stays external)
>
>
>
> Hi all,
>
> I would like to discuss moving Spark Connect server to builtin package.
> Right now, users have to specify —packages when they run Spark Connect
> server script, for example:
>
> ./sbin/start-connect-server.sh --jars `ls connector/connect/server/target/**/spark-connect*SNAPSHOT.jar`
>
> or
>
> ./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.1
>
> which is a little bit odd that sbin scripts should provide jars to start.
>
> Moving it to builtin package is pretty straightforward because most of
> jars are shaded, and the impact would be minimal, I have a prototype here
> apache/spark/#47157
> <https://mailshield.baidu.com/check?q=%2fcy8z7%2fYZNJHeBln9msQEVrqe7rk5HGMbpa7h9lbvgUO6rNqy1kPDW2sYiQgCGHS>.
> This also simplifies Python local running logic a lot.
>
> User facing API layer, Spark Connect Client, stays external but I would
> like the internal/admin server layer, Spark Connect Server, implementation
> to be built in Spark.
>
> Please let me know if you have thoughts on this!
>
>
>
>
Re: [外部邮件] [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
Posted by yangjie01 <ya...@baidu.com.INVALID>.
I have manually attempted to only modify the `assembly/pom.xml` and examined the results of executing `dev/make-distribution.sh --tgz`. The `spark-connect_2.13-4.0.0-SNAPSHOT.jar` is indeed included in the jars directory. However, if rearranging the directories would result in a clearer project structure, I believe that would also be a viable approach.
发件人: Hyukjin Kwon <gu...@apache.org>
日期: 2024年7月2日 星期二 12:00
收件人: yangjie01 <ya...@baidu.com.invalid>
抄送: dev <de...@spark.apache.org>
主题: Re: [外部邮件] [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
My concern is that the `connector` directory is really for external/optional packages (and they aren't included in assembly IIRC).. so I am hesitant to just change the assembly.
The actual changes are not quite large but it moves the files around.
On Tue, 2 Jul 2024 at 12:23, yangjie01 <ya...@baidu.com.invalid> wrote:
I'm supportive of this initiative. However, if the purpose is just to avoid the additional `--packages` option, it seems that making some adjustments to the `assembly/pom.xml` could potentially meet our goal. Is it really necessary to restructure the code directory?
Jie Yang
发件人: Hyukjin Kwon <gu...@apache.org>>
日期: 2024年7月2日 星期二 08:19
收件人: dev <de...@spark.apache.org>>
主题: [外部邮件] [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
Hi all,
I would like to discuss moving Spark Connect server to builtin package. Right now, users have to specify —packages when they run Spark Connect server script, for example:
./sbin/start-connect-server.sh --jars `ls connector/connect/server/target/**/spark-connect*SNAPSHOT.jar`
or
./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.1
which is a little bit odd that sbin scripts should provide jars to start.
Moving it to builtin package is pretty straightforward because most of jars are shaded, and the impact would be minimal, I have a prototype here apache/spark/#47157<https://mailshield.baidu.com/check?q=%2fcy8z7%2fYZNJHeBln9msQEVrqe7rk5HGMbpa7h9lbvgUO6rNqy1kPDW2sYiQgCGHS>. This also simplifies Python local running logic a lot.
User facing API layer, Spark Connect Client, stays external but I would like the internal/admin server layer, Spark Connect Server, implementation to be built in Spark.
Please let me know if you have thoughts on this!
Re: [外部邮件] [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
Posted by Hyukjin Kwon <gu...@apache.org>.
My concern is that the `connector` directory is really for
external/optional packages (and they aren't included in assembly IIRC).. so
I am hesitant to just change the assembly.
The actual changes are not quite large but it moves the files around.
On Tue, 2 Jul 2024 at 12:23, yangjie01 <ya...@baidu.com.invalid> wrote:
> I'm supportive of this initiative. However, if the purpose is just to
> avoid the additional `--packages` option, it seems that making some
> adjustments to the `assembly/pom.xml` could potentially meet our goal. Is
> it really necessary to restructure the code directory?
>
>
>
> Jie Yang
>
>
>
> *发件人**: *Hyukjin Kwon <gu...@apache.org>
> *日期**: *2024年7月2日 星期二 08:19
> *收件人**: *dev <de...@spark.apache.org>
> *主题**: *[外部邮件] [DISCUSS] Move Spark Connect server to builtin package
> (Client API layer stays external)
>
>
>
> Hi all,
>
> I would like to discuss moving Spark Connect server to builtin package.
> Right now, users have to specify —packages when they run Spark Connect
> server script, for example:
>
> ./sbin/start-connect-server.sh --jars `ls connector/connect/server/target/**/spark-connect*SNAPSHOT.jar`
>
> or
>
> ./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.1
>
> which is a little bit odd that sbin scripts should provide jars to start.
>
> Moving it to builtin package is pretty straightforward because most of
> jars are shaded, and the impact would be minimal, I have a prototype here
> apache/spark/#47157
> <https://mailshield.baidu.com/check?q=%2fcy8z7%2fYZNJHeBln9msQEVrqe7rk5HGMbpa7h9lbvgUO6rNqy1kPDW2sYiQgCGHS>.
> This also simplifies Python local running logic a lot.
>
> User facing API layer, Spark Connect Client, stays external but I would
> like the internal/admin server layer, Spark Connect Server, implementation
> to be built in Spark.
>
> Please let me know if you have thoughts on this!
>
>
>
Re: [外部邮件] [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
Posted by yangjie01 <ya...@baidu.com.INVALID>.
I'm supportive of this initiative. However, if the purpose is just to avoid the additional `--packages` option, it seems that making some adjustments to the `assembly/pom.xml` could potentially meet our goal. Is it really necessary to restructure the code directory?
Jie Yang
发件人: Hyukjin Kwon <gu...@apache.org>
日期: 2024年7月2日 星期二 08:19
收件人: dev <de...@spark.apache.org>
主题: [外部邮件] [DISCUSS] Move Spark Connect server to builtin package (Client API layer stays external)
Hi all,
I would like to discuss moving Spark Connect server to builtin package. Right now, users have to specify —packages when they run Spark Connect server script, for example:
./sbin/start-connect-server.sh --jars `ls connector/connect/server/target/**/spark-connect*SNAPSHOT.jar`
or
./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.1
which is a little bit odd that sbin scripts should provide jars to start.
Moving it to builtin package is pretty straightforward because most of jars are shaded, and the impact would be minimal, I have a prototype here apache/spark/#47157<https://mailshield.baidu.com/check?q=%2fcy8z7%2fYZNJHeBln9msQEVrqe7rk5HGMbpa7h9lbvgUO6rNqy1kPDW2sYiQgCGHS>. This also simplifies Python local running logic a lot.
User facing API layer, Spark Connect Client, stays external but I would like the internal/admin server layer, Spark Connect Server, implementation to be built in Spark.
Please let me know if you have thoughts on this!