You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "WweiL (via GitHub)" <gi...@apache.org> on 2023/07/24 22:34:23 UTC
[GitHub] [spark] WweiL commented on a diff in pull request #42116: [SPARK-42941][SS][CONNECT] Python StreamingQueryListener
WweiL commented on code in PR #42116:
URL: https://github.com/apache/spark/pull/42116#discussion_r1272824535
##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:
##########
@@ -3047,18 +3047,29 @@ class SparkConnectPlanner(val sessionHolder: SessionHolder) extends Logging {
respBuilder.setResetTerminated(true)
case StreamingQueryManagerCommand.CommandCase.ADD_LISTENER =>
- val listenerPacket = Utils
- .deserialize[StreamingListenerPacket](
- command.getAddListener.getListenerPayload.toByteArray,
- Utils.getContextOrSparkClassLoader)
- val listener: StreamingQueryListener = listenerPacket.listener
- .asInstanceOf[StreamingQueryListener]
- val id: String = listenerPacket.id
- sessionHolder.cacheListenerById(id, listener)
+ val listener = if (command.getAddListener.hasListenerPayload) {
+ val listenerPacket = Utils
+ .deserialize[StreamingListenerPacket](
+ command.getAddListener.getListenerPayload.toByteArray,
+ Utils.getContextOrSparkClassLoader)
+ val listener: StreamingQueryListener = listenerPacket.listener
+ .asInstanceOf[StreamingQueryListener]
+ val id: String = listenerPacket.id
+ sessionHolder.cacheListenerById(id, listener)
+ listener
+ } else {
+ val listener = new PythonStreamingQueryListener(
+ transformPythonFunction(command.getAddListener.getPythonListenerPayload),
Review Comment:
Ah yea this is left deliberately. I thought I need to do the process termination as part of `removeListener` But then it seems that we already have a ticket filed for `foreachBatch`. [SPARK-44433](https://issues.apache.org/jira/browse/SPARK-44433) I'll just push the change for `removeListener`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org