You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "juliuszsompolski (via GitHub)" <gi...@apache.org> on 2025/12/08 14:56:03 UTC
[PR] [SPARK-54637][CONNECT] Add SQL API test helpers to SparkConnectServerTest [spark]
juliuszsompolski opened a new pull request, #53384:
URL: https://github.com/apache/spark/pull/53384
### What changes were proposed in this pull request?
Add testing helpers to SparkConnectServerTest to enable using connect Spark SQL APIs in tests using that helper.
### Why are the changes needed?
In Spark 3.5, a testing trait SparkConnectServerTest was introduced that helped test Spark Connect Service with a SparkConnectClient in the same JVM proccess, which tested real Spark Connect code paths (SparkConnectClient communicating with the server over actual connection to the localhost server). Before that, using RemoteSparkSession, server was started in a separate process.
It helped
testability: can trigger stuff from the client, then have verification code checking stuff server side. Can also do some more internal server side setup to test specific things.
debugging, as both client and server can be easily connected to by a debugger.
At that time, it was impossible to test Spark Connect client SQL APIs (SparkSession, Dataset) this way, because they were in the same namespace as server, and hence couldn't be classloaded together.
Since Spark 4.0, there is a new API layer that makes it possible for connect and classic implementation of the interfaces to coexist. With that, testing can be extended to use actual SparkSession and other APIs, instead of having to construct tests using more raw APIs.
### Does this PR introduce _any_ user-facing change?
No. It's testing only.
### How was this patch tested?
Added SparkConnectServerTestSuite showcasing the new APIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code opus 4.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54637][CONNECT][TESTS] Add SQL API test helpers to SparkConnectServerTest [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #53384:
URL: https://github.com/apache/spark/pull/53384#issuecomment-3633036726
Merged to master/4.1 for Apache Spark 4.1.0.
Thank you, @juliuszsompolski and @HyukjinKwon .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54637][CONNECT] Add SQL API test helpers to SparkConnectServerTest [spark]
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on PR #53384:
URL: https://github.com/apache/spark/pull/53384#issuecomment-3627360901
I would argue for putting it into 4.1 and 4.0, since it's testing only and may be useful for adding tests and debugging there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54637][CONNECT][TESTS] Add SQL API test helpers to SparkConnectServerTest [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #53384: [SPARK-54637][CONNECT][TESTS] Add SQL API test helpers to SparkConnectServerTest
URL: https://github.com/apache/spark/pull/53384
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54637][CONNECT] Add SQL API test helpers to SparkConnectServerTest [spark]
Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on PR #53384:
URL: https://github.com/apache/spark/pull/53384#issuecomment-3627328051
@hvanhovell @cloud-fan this is something I wanted to do back before Spark 3.5, but it was impossible without the API layer that was added in Spark 4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org