You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cdkrot (via GitHub)" <gi...@apache.org> on 2023/07/06 14:31:06 UTC
[GitHub] [spark] cdkrot opened a new pull request, #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
cdkrot opened a new pull request, #41880:
URL: https://github.com/apache/spark/pull/41880
### What changes were proposed in this pull request?
Extend SparkSession to allow custom interceptors.
This is similar to Python which allows custom channel builders (https://github.com/apache/spark/pull/40993)
### Why are the changes needed?
This can be used for different purposes, such as customized authentication mechanisms
### Does this PR introduce _any_ user-facing change?
SparkSession allows custom interceptors (builder.intercept method)
### How was this patch tested?
UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdkrot commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "cdkrot (via GitHub)" <gi...@apache.org>.
cdkrot commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1254533454
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala:
##########
@@ -533,10 +550,6 @@ object SparkConnectClient {
}
})
}
-
- override def thisUsesUnstableApi(): Unit = {
- // Marks this API is not stable. Left empty on purpose.
- }
Review Comment:
This API is no longer unstable (was stabilized) so this can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdkrot commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "cdkrot (via GitHub)" <gi...@apache.org>.
cdkrot commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1255576146
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/SparkConnectClientSuite.scala:
##########
@@ -123,6 +122,28 @@ class SparkConnectClientSuite extends ConnectFunSuite with BeforeAndAfterEach {
assert(df.plan === service.getAndClearLatestInputPlan())
}
+ test("CustomInterceptor") {
+ startDummyServer(0)
+ client = SparkConnectClient
+ .builder()
+ .connectionString(s"sc://localhost:${server.getPort}")
+ .interceptor(new ClientInterceptor {
+ override def interceptCall[ReqT, RespT](
+ methodDescriptor: MethodDescriptor[ReqT, RespT],
+ callOptions: CallOptions,
+ channel: Channel): ClientCall[ReqT, RespT] = {
+ throw new RuntimeException("Blocked")
+ }
+ })
+ .build()
+
+ val session = SparkSession.builder().client(client).create()
Review Comment:
Now there is an external api test as well (in SparkSessionSuite)
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/SparkConnectClientSuite.scala:
##########
@@ -123,6 +122,28 @@ class SparkConnectClientSuite extends ConnectFunSuite with BeforeAndAfterEach {
assert(df.plan === service.getAndClearLatestInputPlan())
}
+ test("CustomInterceptor") {
+ startDummyServer(0)
+ client = SparkConnectClient
+ .builder()
+ .connectionString(s"sc://localhost:${server.getPort}")
+ .interceptor(new ClientInterceptor {
+ override def interceptCall[ReqT, RespT](
+ methodDescriptor: MethodDescriptor[ReqT, RespT],
+ callOptions: CallOptions,
+ channel: Channel): ClientCall[ReqT, RespT] = {
+ throw new RuntimeException("Blocked")
+ }
+ })
+ .build()
+
+ val session = SparkSession.builder().client(client).create()
Review Comment:
Now there is an external api test as well (in SparkSessionSuite)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdkrot commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "cdkrot (via GitHub)" <gi...@apache.org>.
cdkrot commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1254533454
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala:
##########
@@ -533,10 +550,6 @@ object SparkConnectClient {
}
})
}
-
- override def thisUsesUnstableApi(): Unit = {
- // Marks this API is not stable. Left empty on purpose.
- }
Review Comment:
This API is no longer unstable (was stabilized) so this can be removed. See https://github.com/grpc/grpc-java/issues/1914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdkrot commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "cdkrot (via GitHub)" <gi...@apache.org>.
cdkrot commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1257960672
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##########
@@ -656,6 +654,16 @@ object SparkSession extends Logging {
this
}
+ /**
+ * Add an interceptor to be used during channel creation.
+ *
+ * Note that interceptors added last are executed first by gRPC.
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdkrot commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "cdkrot (via GitHub)" <gi...@apache.org>.
cdkrot commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1254845934
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/SparkConnectClientSuite.scala:
##########
@@ -123,6 +122,28 @@ class SparkConnectClientSuite extends ConnectFunSuite with BeforeAndAfterEach {
assert(df.plan === service.getAndClearLatestInputPlan())
}
+ test("CustomInterceptor") {
+ startDummyServer(0)
+ client = SparkConnectClient
+ .builder()
+ .connectionString(s"sc://localhost:${server.getPort}")
+ .interceptor(new ClientInterceptor {
+ override def interceptCall[ReqT, RespT](
+ methodDescriptor: MethodDescriptor[ReqT, RespT],
+ callOptions: CallOptions,
+ channel: Channel): ClientCall[ReqT, RespT] = {
+ throw new RuntimeException("Blocked")
+ }
+ })
+ .build()
+
+ val session = SparkSession.builder().client(client).create()
Review Comment:
The proposed usage is rather
`SparkSession.builder().interceptor(my interceptor).create()`
Perhaps I should add a test doing this exact line nearby
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdkrot commented on pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "cdkrot (via GitHub)" <gi...@apache.org>.
cdkrot commented on PR #41880:
URL: https://github.com/apache/spark/pull/41880#issuecomment-1623788682
cc @grundprinzip, @nija-at
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdkrot commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "cdkrot (via GitHub)" <gi...@apache.org>.
cdkrot commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1255630770
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/SparkSessionSuite.scala:
##########
@@ -16,12 +16,45 @@
*/
package org.apache.spark.sql
+import java.util.concurrent.TimeUnit
+
+import io.grpc.{CallOptions, Channel, ClientCall, ClientInterceptor, MethodDescriptor, Server}
+import io.grpc.netty.NettyServerBuilder
+import org.scalatest.BeforeAndAfterEach
+
+import org.apache.spark.sql.connect.client.DummySparkConnectService
import org.apache.spark.sql.connect.client.util.ConnectFunSuite
/**
* Tests for non-dataframe related SparkSession operations.
*/
-class SparkSessionSuite extends ConnectFunSuite {
+class SparkSessionSuite extends ConnectFunSuite with BeforeAndAfterEach {
+ private var service: DummySparkConnectService = _
+ private var server: Server = _
+
+ private def startDummyServer(port: Int): Unit = {
+ service = new DummySparkConnectService
+ server = NettyServerBuilder
+ .forPort(port)
+ .addService(service)
+ .build()
+ server.start()
+ }
Review Comment:
I guess you refer me to ClientE2ETestSuite? I will take a look. Here I was following how it's done in SparkConnectClientSuite. If there is a better way, we should replace how the server starts in both places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdkrot commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "cdkrot (via GitHub)" <gi...@apache.org>.
cdkrot commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1255576146
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/SparkConnectClientSuite.scala:
##########
@@ -123,6 +122,28 @@ class SparkConnectClientSuite extends ConnectFunSuite with BeforeAndAfterEach {
assert(df.plan === service.getAndClearLatestInputPlan())
}
+ test("CustomInterceptor") {
+ startDummyServer(0)
+ client = SparkConnectClient
+ .builder()
+ .connectionString(s"sc://localhost:${server.getPort}")
+ .interceptor(new ClientInterceptor {
+ override def interceptCall[ReqT, RespT](
+ methodDescriptor: MethodDescriptor[ReqT, RespT],
+ callOptions: CallOptions,
+ channel: Channel): ClientCall[ReqT, RespT] = {
+ throw new RuntimeException("Blocked")
+ }
+ })
+ .build()
+
+ val session = SparkSession.builder().client(client).create()
Review Comment:
Now there is an external api test as well (in SparkSessionSuite)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] grundprinzip commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1255578300
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala:
##########
@@ -509,12 +518,20 @@ object SparkConnectClient {
def createChannel(): ManagedChannel = {
val channelBuilder = Grpc.newChannelBuilderForAddress(host, port, credentials)
+
if (metadata.nonEmpty) {
channelBuilder.intercept(new MetadataHeaderClientInterceptor(metadata))
}
+
+ for (interceptor <- interceptors) {
+ channelBuilder.intercept(interceptor)
+ }
Review Comment:
```suggestion
interceptors.forEach(channelBuilder.intercept(_))
```
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/SparkSessionSuite.scala:
##########
@@ -16,12 +16,45 @@
*/
package org.apache.spark.sql
+import java.util.concurrent.TimeUnit
+
+import io.grpc.{CallOptions, Channel, ClientCall, ClientInterceptor, MethodDescriptor, Server}
+import io.grpc.netty.NettyServerBuilder
+import org.scalatest.BeforeAndAfterEach
+
+import org.apache.spark.sql.connect.client.DummySparkConnectService
import org.apache.spark.sql.connect.client.util.ConnectFunSuite
/**
* Tests for non-dataframe related SparkSession operations.
*/
-class SparkSessionSuite extends ConnectFunSuite {
+class SparkSessionSuite extends ConnectFunSuite with BeforeAndAfterEach {
+ private var service: DummySparkConnectService = _
+ private var server: Server = _
+
+ private def startDummyServer(port: Int): Unit = {
+ service = new DummySparkConnectService
+ server = NettyServerBuilder
+ .forPort(port)
+ .addService(service)
+ .build()
+ server.start()
+ }
Review Comment:
I think we have something called e2e client test for this that does properly start the service.
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##########
@@ -656,6 +654,11 @@ object SparkSession extends Logging {
this
}
+ def interceptor(interceptor: ClientInterceptor): Builder = {
Review Comment:
This is the public facing part of the interface and needs the most extensive documentation,.
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##########
@@ -656,6 +654,11 @@ object SparkSession extends Logging {
this
}
+ def interceptor(interceptor: ClientInterceptor): Builder = {
Review Comment:
doc please :)
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala:
##########
@@ -463,7 +460,18 @@ object SparkConnectClient {
this
}
- def build(): SparkConnectClient = new SparkConnectClient(_configuration)
+ /**
+ * Add an interceptor to be used during channel creation.
+ *
+ * Note that interceptors added last are executed first by grpc.
Review Comment:
```suggestion
* Note that interceptors added last are executed first by gRPC.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] grundprinzip commented on pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on PR #41880:
URL: https://github.com/apache/spark/pull/41880#issuecomment-1625210490
@cdkrot Please make the PR description very explicit, ideally with a code snippet of how to use it and why this is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #41880:
URL: https://github.com/apache/spark/pull/41880#issuecomment-1630390623
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
URL: https://github.com/apache/spark/pull/41880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdkrot commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "cdkrot (via GitHub)" <gi...@apache.org>.
cdkrot commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1255631767
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##########
@@ -656,6 +654,11 @@ object SparkSession extends Logging {
this
}
+ def interceptor(interceptor: ClientInterceptor): Builder = {
Review Comment:
Ok, good point :). I didn't do it because other public methods here are undocumented, but that's probably not a good idea either
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] grundprinzip commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1255633829
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/SparkSessionSuite.scala:
##########
@@ -16,12 +16,45 @@
*/
package org.apache.spark.sql
+import java.util.concurrent.TimeUnit
+
+import io.grpc.{CallOptions, Channel, ClientCall, ClientInterceptor, MethodDescriptor, Server}
+import io.grpc.netty.NettyServerBuilder
+import org.scalatest.BeforeAndAfterEach
+
+import org.apache.spark.sql.connect.client.DummySparkConnectService
import org.apache.spark.sql.connect.client.util.ConnectFunSuite
/**
* Tests for non-dataframe related SparkSession operations.
*/
-class SparkSessionSuite extends ConnectFunSuite {
+class SparkSessionSuite extends ConnectFunSuite with BeforeAndAfterEach {
+ private var service: DummySparkConnectService = _
+ private var server: Server = _
+
+ private def startDummyServer(port: Int): Unit = {
+ service = new DummySparkConnectService
+ server = NettyServerBuilder
+ .forPort(port)
+ .addService(service)
+ .build()
+ server.start()
+ }
Review Comment:
Check `ClientE2ETestSuite` which uses `RemoteSparkSession` which is probably what you want most.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] grundprinzip commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1257233751
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##########
@@ -656,6 +654,16 @@ object SparkSession extends Logging {
this
}
+ /**
+ * Add an interceptor to be used during channel creation.
+ *
+ * Note that interceptors added last are executed first by gRPC.
Review Comment:
Please check in other parts of spark for proper doc strings. It needs as well a since version marker.
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##########
@@ -20,13 +20,11 @@ import java.io.Closeable
import java.net.URI
import java.util.concurrent.TimeUnit._
import java.util.concurrent.atomic.AtomicLong
-
import scala.collection.JavaConverters._
import scala.reflect.runtime.universe.TypeTag
-
import com.google.common.cache.{CacheBuilder, CacheLoader}
+import io.grpc.ClientInterceptor
Review Comment:
I think there a bunch of Scala style issued Herr. Please use as well the automatic reformat for the connect code base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] grundprinzip commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1254843100
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/SparkConnectClientSuite.scala:
##########
@@ -123,6 +122,28 @@ class SparkConnectClientSuite extends ConnectFunSuite with BeforeAndAfterEach {
assert(df.plan === service.getAndClearLatestInputPlan())
}
+ test("CustomInterceptor") {
+ startDummyServer(0)
+ client = SparkConnectClient
+ .builder()
+ .connectionString(s"sc://localhost:${server.getPort}")
+ .interceptor(new ClientInterceptor {
+ override def interceptCall[ReqT, RespT](
+ methodDescriptor: MethodDescriptor[ReqT, RespT],
+ callOptions: CallOptions,
+ channel: Channel): ClientCall[ReqT, RespT] = {
+ throw new RuntimeException("Blocked")
+ }
+ })
+ .build()
+
+ val session = SparkSession.builder().client(client).create()
Review Comment:
The `client()` method is `private[sql]` for anyone providing any kinds of wrappers on top of the SparkSession for custom auth integrations it's not possible to use this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdkrot commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "cdkrot (via GitHub)" <gi...@apache.org>.
cdkrot commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1254845934
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/SparkConnectClientSuite.scala:
##########
@@ -123,6 +122,28 @@ class SparkConnectClientSuite extends ConnectFunSuite with BeforeAndAfterEach {
assert(df.plan === service.getAndClearLatestInputPlan())
}
+ test("CustomInterceptor") {
+ startDummyServer(0)
+ client = SparkConnectClient
+ .builder()
+ .connectionString(s"sc://localhost:${server.getPort}")
+ .interceptor(new ClientInterceptor {
+ override def interceptCall[ReqT, RespT](
+ methodDescriptor: MethodDescriptor[ReqT, RespT],
+ callOptions: CallOptions,
+ channel: Channel): ClientCall[ReqT, RespT] = {
+ throw new RuntimeException("Blocked")
+ }
+ })
+ .build()
+
+ val session = SparkSession.builder().client(client).create()
Review Comment:
The proposed usage is rather
`SparkSession.builder().interceptor(my interceptor).create()`, and this one is public.
Perhaps I should add a test doing this exact line nearby
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdkrot commented on a diff in pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "cdkrot (via GitHub)" <gi...@apache.org>.
cdkrot commented on code in PR #41880:
URL: https://github.com/apache/spark/pull/41880#discussion_r1256040988
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/SparkSessionSuite.scala:
##########
@@ -16,12 +16,45 @@
*/
package org.apache.spark.sql
+import java.util.concurrent.TimeUnit
+
+import io.grpc.{CallOptions, Channel, ClientCall, ClientInterceptor, MethodDescriptor, Server}
+import io.grpc.netty.NettyServerBuilder
+import org.scalatest.BeforeAndAfterEach
+
+import org.apache.spark.sql.connect.client.DummySparkConnectService
import org.apache.spark.sql.connect.client.util.ConnectFunSuite
/**
* Tests for non-dataframe related SparkSession operations.
*/
-class SparkSessionSuite extends ConnectFunSuite {
+class SparkSessionSuite extends ConnectFunSuite with BeforeAndAfterEach {
+ private var service: DummySparkConnectService = _
+ private var server: Server = _
+
+ private def startDummyServer(port: Int): Unit = {
+ service = new DummySparkConnectService
+ server = NettyServerBuilder
+ .forPort(port)
+ .addService(service)
+ .build()
+ server.start()
+ }
Review Comment:
Well I was going to complain that ClientE2ETestSuite is not a great fit to reuse, for some reasons such as it owns how the session is created and it lasts for the whole suite, but actually I realized that my test doesn't event need a server, so I dropped the server creation altogether
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdkrot commented on pull request #41880: [SPARK-44263][CONNECT] Custom Interceptors Support
Posted by "cdkrot (via GitHub)" <gi...@apache.org>.
cdkrot commented on PR #41880:
URL: https://github.com/apache/spark/pull/41880#issuecomment-1629003641
I fixed tests, and also this weird SQL<->Connect comparison errors. Hope we can get this merged. cc @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org