You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/04/06 08:33:52 UTC

[GitHub] [spark] HyukjinKwon opened a new pull request, #36086: [SPARK-38804][PYTHON][SS] Add StreamingQueryManager.removeListener in PySpark

HyukjinKwon opened a new pull request, #36086:
URL: https://github.com/apache/spark/pull/36086

   ### What changes were proposed in this pull request?
   
   SPARK-38759 added `StreamingQueryManager.addListener`. This PR adds the ability to remove it back.
   
   ### Why are the changes needed?
   
   To be able to remove the listener added.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, now users can remove the listener added in PySpark via `spark.streams.removeListener(...)`
   
   ### How was this patch tested?
   
   Manually tested, and unittest was added.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon closed pull request #36086: [SPARK-38804][PYTHON][SS] Add StreamingQueryManager.removeListener in PySpark

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #36086: [SPARK-38804][PYTHON][SS] Add StreamingQueryManager.removeListener in PySpark
URL: https://github.com/apache/spark/pull/36086


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36086: [SPARK-38804][PYTHON][SS] Add StreamingQueryManager.removeListener in PySpark

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36086:
URL: https://github.com/apache/spark/pull/36086#discussion_r843663805


##########
python/pyspark/sql/streaming/listener.py:
##########
@@ -67,7 +67,7 @@ def onQueryStarted(self, event: "QueryStartedEvent") -> None:
         -----
         This is called synchronously with :py:meth:`~pyspark.sql.streaming.DataStreamWriter.start`,
         that is, `onQueryStart` will be called on all listeners before `DataStreamWriter.start()`
-        returns the corresponding :class:`~pyspark.streaming.StreamingQuery`.
+        returns the corresponding :class:`~pyspark.sql.streaming.StreamingQuery`.

Review Comment:
   I piggy-back some minor documentation fixes here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #36086: [SPARK-38804][PYTHON][SS] Add StreamingQueryManager.removeListener in PySpark

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #36086:
URL: https://github.com/apache/spark/pull/36086#issuecomment-1091890342

   Thanks all.
   
   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #36086: [SPARK-38804][PYTHON][SS] Add StreamingQueryManager.removeListener in PySpark

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #36086:
URL: https://github.com/apache/spark/pull/36086#issuecomment-1090001666

   cc @HeartSaVioR @alex-balikov @zsxwing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a diff in pull request #36086: [SPARK-38804][PYTHON][SS] Add StreamingQueryManager.removeListener in PySpark

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on code in PR #36086:
URL: https://github.com/apache/spark/pull/36086#discussion_r844218251


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -309,11 +309,22 @@ def addListener(self, listener: StreamingQueryListener) -> None:
         java_import(gw.jvm, "org.apache.spark.sql.streaming.*")
         ensure_callback_server_started(gw)
 
-        self._jsqm.addListener(
+        jlistener = (
             SparkContext._jvm.PythonStreamingQueryListenerWrapper(  # type: ignore[union-attr]
                 JStreamingQueryListener(listener)
             )
         )
+        listener._set_jlistener(jlistener)
+        self._jsqm.addListener(jlistener)
+
+    def removeListener(self, listener: StreamingQueryListener) -> None:
+        """
+        Deregister a :class:`StreamingQueryListener`.
+
+        .. versionadded:: 3.4.0
+        """
+        if hasattr(listener, "_jlistener"):

Review Comment:
   Out of curiosity, why don't we update the `_jlistener` attribute?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xinrong-databricks commented on a diff in pull request #36086: [SPARK-38804][PYTHON][SS] Add StreamingQueryManager.removeListener in PySpark

Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on code in PR #36086:
URL: https://github.com/apache/spark/pull/36086#discussion_r844211502


##########
python/pyspark/sql/streaming/listener.py:
##########
@@ -95,6 +95,13 @@ def onQueryTerminated(self, event: "QueryTerminatedEvent") -> None:
         """
         pass
 
+    def _set_jlistener(self, jlistener: JavaObject) -> None:
+        """
+        Used to keep the reference of JVM listener instance to be able to remove later viz

Review Comment:
   nit: ... to remove later vi**a**



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org