You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@zeppelin.apache.org by "dhama-shashank-meesho (via GitHub)" <gi...@apache.org> on 2026/08/30 18:04:23 UTC
[PR] Feature/spark connect interpreter [zeppelin]
dhama-shashank-meesho opened a new pull request, #5446:
URL: https://github.com/apache/zeppelin/pull/5446
This PR introduces a new Spark Connect interpreter group for Apache Zeppelin, enabling notebooks to execute against an external Spark Connect server via the official SparkSession.remote() gRPC API (Spark 3.4+).
Unlike the classic Spark interpreter, no Spark driver or SparkContext runs inside the Zeppelin JVM/pod. The interpreter operates as a thin client, while all computation is executed remotely on the Spark cluster.
------------------------------------------------------------
Key Features
------------------------------------------------------------
1) New Interpreter Bindings
Adds support for:
- %spark-connect (SparkConnectInterpreter)
- %spark-connect.sql (SparkConnectSqlInterpreter)
- %spark-connect.pyspark (PySparkConnectInterpreter)
- %spark-connect.ipyspark (IPySparkConnectInterpreter)
2) Remote Execution Model (gRPC)
Implements a remote-only execution model over Spark Connect gRPC, similar in deployment spirit to JDBC/Livy interpreters, but using Spark’s native Connect interface.
3) Session Management
- Supports per-user session limiting via zeppelin.spark.connect.maxSessionsPerUser
- open() is idempotent and safely reuses existing sessions
- Spark sessions are created using SparkSession.builder().remote(sc://...)
- close() cleanly terminates the gRPC session and releases user slot counters
4) Notebook-Level Fair Locking
Introduces NotebookLockManager to enforce FIFO paragraph execution ordering and prevent concurrent conflicts within the same note.
5) Optional Concurrent SQL Scheduling
Supports optional concurrent SQL execution via zeppelin.spark.concurrentSQL using a ParallelScheduler, while still enforcing notebook-level locking.
6) Streaming Result Mode
Adds zeppelin.spark.connect.streamResults to stream large result sets via iterator-based output.
7) PySpark Support (No Extra Spark Session)
Provides PySpark support using a Python subprocess + Py4j bridge, reusing the same Java Spark Connect session (avoids creating a separate Spark session in Python).
------------------------------------------------------------
Connection String Handling
------------------------------------------------------------
Connection URIs are built via SparkConnectUtils.buildConnectionString() and support token + SSL options. Sensitive values (e.g., token, user_id) are redacted in logs.
------------------------------------------------------------
Cancellation Support
------------------------------------------------------------
Stopping a paragraph triggers sparkSession.interruptAll(), cancelling active remote operations through Spark Connect.
------------------------------------------------------------
Configuration Properties
------------------------------------------------------------
Key configs include:
- spark.remote
- spark.connect.token
- spark.connect.use_ssl
- zeppelin.spark.connect.maxSessionsPerUser
- zeppelin.spark.connect.streamResults
- zeppelin.spark.concurrentSQL
- zeppelin.spark.concurrentSQL.max
- spark.connect.grpc.maxMessageSize
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/spark connect interpreter [zeppelin]
Posted by "jongyoul (via GitHub)" <gi...@apache.org>.
jongyoul commented on PR #5446:
URL: https://github.com/apache/zeppelin/pull/5446#issuecomment-5472042204
@dhama-shashank-meesho Hello, sorry for the late reply. I know we already had a spark-submit interpreter but I feel like we have several way to execute spark in Zeppelin. It's good itself but our codebase became conplicated. So I wonder if it's possible to merge all other type of spark interpreter into one way. I'm not 100% sure but spark-connect is the most recent way to execute spark qurey/script. Can you review the current way and merge them into one spark interpreter? WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/spark connect interpreter [zeppelin]
Posted by "dhama-shashank-meesho (via GitHub)" <gi...@apache.org>.
dhama-shashank-meesho commented on PR #5446:
URL: https://github.com/apache/zeppelin/pull/5446#issuecomment-5479630702
@jongyoul Agreed, there are too many spark paths right now.
What this one is for: point Zeppelin at one shared long-running Spark cluster
(Dataproc, or Spark on k8s) instead of every notebook starting its own Spark app.
Today each interpreter holds its own driver and executors, so memory sits
fragmented across many half-idle apps. With Connect there is no driver in
Zeppelin at all, and the cluster manages memory centrally for everyone.
On merging all of them into one, I'm for the goal but a few things block it now:
no Scala interpreter here yet (only SQL and PySpark), no SparkContext in Connect
so `sc`/RDD notebooks break, classic runs on `local[*]` with no server while
Connect needs one, and Connect needs Spark 3.4+ while we still build for 3.3.
This PR only adds a new module and doesn't touch the existing interpreters —
could we land it first and do the consolidation separately?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/spark connect interpreter [zeppelin]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #5446:
URL: https://github.com/apache/zeppelin/pull/5446#issuecomment-5524085042
I also doubt why this needs a new interpreter - Spark Connect tends to keep the public API compatible with classic Spark, ideally, you just need to replace the `master` from `local`/`yarn` to `sc://xxxx`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/spark connect interpreter [zeppelin]
Posted by "dhama-shashank-meesho (via GitHub)" <gi...@apache.org>.
dhama-shashank-meesho commented on PR #5446:
URL: https://github.com/apache/zeppelin/pull/5446#issuecomment-5559868291
@pan3793 True for the API — DataFrame/SQL code is source-compatible, and on
Spark 4.0 the Scala API is unified. But it's not just a master swap:
- `spark.master` doesn't accept `sc://`. Connect uses `spark.remote`, and Spark
errors if both are set. Zeppelin routes everything through `spark.master`.
- The interpreter uses SparkContext in a lot of places — `setJobGroup`,
`cancelJobGroup`, `SparkListener`, `uiWebUrl`. Connect has none of that, so
progress, cancel and the UI link need a different path.
So it's a refactor of the existing interpreter, not a config change. Happy to go
that route though — @jongyoul suggested something similar. Should I rework this
as a Connect backend inside the `spark` module instead of a separate one? Would
rather settle the shape before redoing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Feature/spark connect interpreter [zeppelin]
Posted by "dhama-shashank-meesho (via GitHub)" <gi...@apache.org>.
dhama-shashank-meesho commented on PR #5446:
URL: https://github.com/apache/zeppelin/pull/5446#issuecomment-6324436744
@pan3793 @jongyoul, could you please check this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org