You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Dhruv Pratap via dev <de...@spark.apache.org> on 2026/07/20 16:50:32 UTC
[DISCUSS] Spark Connect support for the spark-sql CLI (completing SPARK-49194?)
Hi all,
The Spark Connect entry-point umbrella (SPARK-49194) has landed Connect
support for spark-shell (SPARK-48936), spark-submit (SPARK-48960), and
pyspark. The one interactive entry point still missing it is the spark-sql
CLI: there is currently no way to run spark-sql against a Connect server
(spark-sql --remote sc://...).
Before writing any code, two questions for the Connect maintainers:
1. Tracking: should this be a sub-task under SPARK-49194, or a new ticket?
The umbrella lists pyspark/spark-shell/spark-submit and its sub-tasks
are all resolved, so I want to confirm the right home.
2. Process: does this warrant a SPIP, or is it an incremental improvement
completing the already-accepted Connect pattern? It is user-facing but adds
no new public API (details below).
Proposed approach (kept minimal):
- A new SparkConnectSQLCLIDriver in sql/connect/client/jvm, selected
on --remote / spark.api.mode=connect, mirroring how ConnectRepl
backs spark-shell --remote (SPARK-48936).
- Builds a remote session via SparkSession.builder().client(...),
runs statements via spark.sql(...), and formats results client-side.
- No change to the existing SparkSQLCLIDriver / sql/hive-thriftserver
path; the legacy driver stays the default. No existing public API changes.
- MVP first (-e/-f + interactive loop + basic output); follow-ups for
full Hive-style output parity.
I have a short design write-up I can share/attach to the JIRA.
This is complementary to the draft "Add Unified Spark CLI" PR (#57313, cc
@nchammas): that unifies the bin/ scripts and its sql subcommand currently
wraps the existing spark-sql shell, so a Connect-native spark-sql would
slot under it cleanly. Happy to coordinate to avoid duplicate CLI churn.
Thanks,
Dhruv Pratap (Netflix)
Apache JIRA: dhruvpratap
dpratap@netflix.com · dhruv.pratap@gmail.com
Re: [DISCUSS] Spark Connect support for the spark-sql CLI (completing SPARK-49194?)
Posted by Cheng Pan <pa...@gmail.com>.
I reply in the GitHub PR.
In short, the idea that extending spark-sql to support Connect makes sense, but I have some concerns about the current approach - just building from the scratch start with a while loop ...
Thanks,
Cheng Pan
> On Jul 21, 2026, at 10:10, Dhruv Pratap <dp...@netflix.com> wrote:
>
> Hi Cheng Pan,
>
> Yes, I'd seen SPARK-54002 / #52706; nice addition, and I see it as complementary rather than overlapping. It gives an SQL CLI over Connect via BeeLine (the Hive sqlline client) + a JDBC driver; the proposal here is native `spark-sql --remote sc://...` the actual spark-sql command on the Spark Connect client directly (as spark-shell/pyspark --remote already do),
> no JDBC layer.
>
> Why native still seems worth it alongside BeeLine:
> - UX continuity: spark-sql users/scripts just add --remote, no tool switch for an edge-node -> remote-compute migration.
> - Consistency: spark-shell/spark-submit/pyspark all have native --remote under SPARK-49194; spark-sql is the only interactive entry point that doesn't.
> - Minimal, Hive-free stack: no JDBC driver or Hive-derived client. Aligned with the enterprises, such as ours, is moving off of Hive-coupled tooling that Connect is about.
>
> Does that make sense? Also, I've moved the PR to ready-for-review. Keen on your view.
>
> - PR (ready for review): https://github.com/apache/spark/pull/57383
> - JIRA: SPARK-58227 <https://issues.apache.org/jira/browse/SPARK-58227>
>
> Thanks,
> Dhruv Pratap (Netflix)
> Apache JIRA: dhruvpratap
>
>
> On Mon, Jul 20, 2026 at 8:25 PM Cheng Pan <pan3793@gmail.com <ma...@gmail.com>> wrote:
>> Just wondering if you have noticed SPARK-54002 (https://github.com/apache/spark/pull/52706), which makes BeeLine support the Spark Connect JDBC driver and provides an SQL CLI to connect to the Connect Server, providing a similar experience to spark-sql.
>>
>> Thanks,
>> Cheng Pan
>>
>>
>>
>>> On Jul 21, 2026, at 00:50, Dhruv Pratap via dev <dev@spark.apache.org <ma...@spark.apache.org>> wrote:
>>>
>>> Hi all,
>>>
>>> The Spark Connect entry-point umbrella (SPARK-49194) has landed Connect support for spark-shell (SPARK-48936), spark-submit (SPARK-48960), and pyspark. The one interactive entry point still missing it is the spark-sql CLI: there is currently no way to run spark-sql against a Connect server (spark-sql --remote sc://...).
>>>
>>> Before writing any code, two questions for the Connect maintainers:
>>>
>>> 1. Tracking: should this be a sub-task under SPARK-49194, or a new ticket? The umbrella lists pyspark/spark-shell/spark-submit and its sub-tasks are all resolved, so I want to confirm the right home.
>>> 2. Process: does this warrant a SPIP, or is it an incremental improvement completing the already-accepted Connect pattern? It is user-facing but adds no new public API (details below).
>>>
>>> Proposed approach (kept minimal):
>>> - A new SparkConnectSQLCLIDriver in sql/connect/client/jvm, selected on --remote / spark.api.mode=connect, mirroring how ConnectRepl backs spark-shell --remote (SPARK-48936).
>>> - Builds a remote session via SparkSession.builder().client(...), runs statements via spark.sql(...), and formats results client-side.
>>> - No change to the existing SparkSQLCLIDriver / sql/hive-thriftserver path; the legacy driver stays the default. No existing public API changes.
>>> - MVP first (-e/-f + interactive loop + basic output); follow-ups for full Hive-style output parity.
>>>
>>> I have a short design write-up I can share/attach to the JIRA.
>>>
>>> This is complementary to the draft "Add Unified Spark CLI" PR (#57313, cc @nchammas): that unifies the bin/ scripts and its sql subcommand currently wraps the existing spark-sql shell, so a Connect-native spark-sql would slot under it cleanly. Happy to coordinate to avoid duplicate CLI churn.
>>>
>>> Thanks,
>>> Dhruv Pratap (Netflix)
>>> Apache JIRA: dhruvpratap
>>> dpratap@netflix.com <ma...@netflix.com> · dhruv.pratap@gmail.com <ma...@gmail.com>
Re: [DISCUSS] Spark Connect support for the spark-sql CLI (completing SPARK-49194?)
Posted by Dhruv Pratap via dev <de...@spark.apache.org>.
Hi Cheng Pan,
Yes, I'd seen SPARK-54002 / #52706; nice addition, and I see it as
complementary rather than overlapping. It gives an SQL CLI over Connect via
BeeLine (the Hive sqlline client) + a JDBC driver; the proposal here is
native `spark-sql --remote sc://...` the actual spark-sql command on the
Spark Connect client directly (as spark-shell/pyspark --remote already do),
no JDBC layer.
Why native still seems worth it alongside BeeLine:
- UX continuity: spark-sql users/scripts just add --remote, no tool switch
for an edge-node -> remote-compute migration.
- Consistency: spark-shell/spark-submit/pyspark all have native --remote
under SPARK-49194; spark-sql is the only interactive entry point that
doesn't.
- Minimal, Hive-free stack: no JDBC driver or Hive-derived client.
Aligned with the enterprises, such as ours, is moving off of Hive-coupled
tooling that Connect is about.
Does that make sense? Also, I've moved the PR to ready-for-review. Keen on
your view.
- PR (ready for review): https://github.com/apache/spark/pull/57383
- JIRA: SPARK-58227 <https://issues.apache.org/jira/browse/SPARK-58227>
Thanks,
Dhruv Pratap (Netflix)
Apache JIRA: dhruvpratap
On Mon, Jul 20, 2026 at 8:25 PM Cheng Pan <pa...@gmail.com> wrote:
> Just wondering if you have noticed SPARK-54002 (
> https://github.com/apache/spark/pull/52706), which makes BeeLine support
> the Spark Connect JDBC driver and provides an SQL CLI to connect to the
> Connect Server, providing a similar experience to spark-sql.
>
> Thanks,
> Cheng Pan
>
>
>
> On Jul 21, 2026, at 00:50, Dhruv Pratap via dev <de...@spark.apache.org>
> wrote:
>
> Hi all,
>
> The Spark Connect entry-point umbrella (SPARK-49194) has landed Connect
> support for spark-shell (SPARK-48936), spark-submit (SPARK-48960), and
> pyspark. The one interactive entry point still missing it is the spark-sql
> CLI: there is currently no way to run spark-sql against a Connect server
> (spark-sql --remote sc://...).
>
> Before writing any code, two questions for the Connect maintainers:
>
> 1. Tracking: should this be a sub-task under SPARK-49194, or a new ticket?
> The umbrella lists pyspark/spark-shell/spark-submit and its sub-tasks
> are all resolved, so I want to confirm the right home.
> 2. Process: does this warrant a SPIP, or is it an incremental improvement
> completing the already-accepted Connect pattern? It is user-facing but adds
> no new public API (details below).
>
> Proposed approach (kept minimal):
> - A new SparkConnectSQLCLIDriver in sql/connect/client/jvm, selected
> on --remote / spark.api.mode=connect, mirroring how ConnectRepl
> backs spark-shell --remote (SPARK-48936).
> - Builds a remote session via SparkSession.builder().client(...),
> runs statements via spark.sql(...), and formats results client-side.
> - No change to the existing SparkSQLCLIDriver / sql/hive-thriftserver
> path; the legacy driver stays the default. No existing public API changes.
> - MVP first (-e/-f + interactive loop + basic output); follow-ups for
> full Hive-style output parity.
>
> I have a short design write-up I can share/attach to the JIRA.
>
> This is complementary to the draft "Add Unified Spark CLI" PR (#57313, cc
> @nchammas): that unifies the bin/ scripts and its sql subcommand currently
> wraps the existing spark-sql shell, so a Connect-native spark-sql would
> slot under it cleanly. Happy to coordinate to avoid duplicate CLI churn.
>
> Thanks,
> Dhruv Pratap (Netflix)
> Apache JIRA: dhruvpratap
> dpratap@netflix.com · dhruv.pratap@gmail.com
>
>
>
Re: [DISCUSS] Spark Connect support for the spark-sql CLI (completing SPARK-49194?)
Posted by Cheng Pan <pa...@gmail.com>.
Just wondering if you have noticed SPARK-54002 (https://github.com/apache/spark/pull/52706), which makes BeeLine support the Spark Connect JDBC driver and provides an SQL CLI to connect to the Connect Server, providing a similar experience to spark-sql.
Thanks,
Cheng Pan
> On Jul 21, 2026, at 00:50, Dhruv Pratap via dev <de...@spark.apache.org> wrote:
>
> Hi all,
>
> The Spark Connect entry-point umbrella (SPARK-49194) has landed Connect support for spark-shell (SPARK-48936), spark-submit (SPARK-48960), and pyspark. The one interactive entry point still missing it is the spark-sql CLI: there is currently no way to run spark-sql against a Connect server (spark-sql --remote sc://...).
>
> Before writing any code, two questions for the Connect maintainers:
>
> 1. Tracking: should this be a sub-task under SPARK-49194, or a new ticket? The umbrella lists pyspark/spark-shell/spark-submit and its sub-tasks are all resolved, so I want to confirm the right home.
> 2. Process: does this warrant a SPIP, or is it an incremental improvement completing the already-accepted Connect pattern? It is user-facing but adds no new public API (details below).
>
> Proposed approach (kept minimal):
> - A new SparkConnectSQLCLIDriver in sql/connect/client/jvm, selected on --remote / spark.api.mode=connect, mirroring how ConnectRepl backs spark-shell --remote (SPARK-48936).
> - Builds a remote session via SparkSession.builder().client(...), runs statements via spark.sql(...), and formats results client-side.
> - No change to the existing SparkSQLCLIDriver / sql/hive-thriftserver path; the legacy driver stays the default. No existing public API changes.
> - MVP first (-e/-f + interactive loop + basic output); follow-ups for full Hive-style output parity.
>
> I have a short design write-up I can share/attach to the JIRA.
>
> This is complementary to the draft "Add Unified Spark CLI" PR (#57313, cc @nchammas): that unifies the bin/ scripts and its sql subcommand currently wraps the existing spark-sql shell, so a Connect-native spark-sql would slot under it cleanly. Happy to coordinate to avoid duplicate CLI churn.
>
> Thanks,
> Dhruv Pratap (Netflix)
> Apache JIRA: dhruvpratap
> dpratap@netflix.com <ma...@netflix.com> · dhruv.pratap@gmail.com <ma...@gmail.com>