You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "WweiL (via GitHub)" <gi...@apache.org> on 2023/04/06 22:24:42 UTC

[GitHub] [spark] WweiL opened a new pull request, #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

WweiL opened a new pull request, #40691:
URL: https://github.com/apache/spark/pull/40691

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'core/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   Enable unit tests and doctests for streaming queries. A lot are skipped and needs to be un-skipped as the development goes on. 
   
   Note that I also separated the `{foreach, foreachBatch}` tests from the original test suite. Because currently they are not implemented and it seems unnecessary to manually add a skip for all of them in `StreamingParityTests`. Also it doesn't hurt to separate them I think as these tests are large enough to be separated.
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   More tests is always better than less.
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   No.
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   It's test itself.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162022638


##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -89,8 +92,8 @@ def format(self, source: str) -> "DataStreamReader":
 
         Examples
         --------
-        >>> spark.readStream.format("text")
-        <pyspark.sql.streaming.readwriter.DataStreamReader object ...>
+        >>> spark.readStream.format("text") # doctest: +ELLIPSIS

Review Comment:
   Hmm I believe you added these? https://github.com/apache/spark/pull/40586/files#r1157045341



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162081978


##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -89,8 +92,8 @@ def format(self, source: str) -> "DataStreamReader":
 
         Examples
         --------
-        >>> spark.readStream.format("text")
-        <pyspark.sql.streaming.readwriter.DataStreamReader object ...>
+        >>> spark.readStream.format("text") # doctest: +ELLIPSIS

Review Comment:
   Actually maybe it doesn't hurt to put them there..? We already have this flag set, e.g.:  https://github.com/apache/spark/blob/master/python/pyspark/sql/streaming/query.py#L222



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon closed pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming
URL: https://github.com/apache/spark/pull/40691


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162080735


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -330,7 +335,8 @@ def stop(self) -> None:
         Stop streaming query
 
         >>> sq.stop()
-        >>> sq.isActive
+
+        >>> sq.isActive # doctest: +SKIP

Review Comment:
   is this required? Why is it skipped?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1163147543


##########
python/pyspark/sql/tests/connect/streaming/test_parity_streaming.py:
##########
@@ -0,0 +1,68 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+
+from pyspark.sql.tests.streaming.test_streaming import StreamingTestsMixin
+from pyspark.testing.connectutils import ReusedConnectTestCase
+
+
+class StreamingParityTests(StreamingTestsMixin, ReusedConnectTestCase):

Review Comment:
   This file marks all tests that should be skipped currently because these functionalities are not supported yet in connect.
   
   It executes all tests from `StreamingTestsMixin`, except these below marked skipped.
   
   When anyone adds support to corresponding functions, they should delete the @unittest.skip and function, unless they want to do some change specific for connect.
   
   So ideally in the end, this class should only be like 
   ```
   class StreamingParityTests(StreamingTestsMixin, ReusedConnectTestCase):
       pass
   ```
   And then it runs all tests from StreamingTestsMixin



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1161970615


##########
python/pyspark/sql/tests/streaming/test_streaming_foreach_family.py:
##########
@@ -0,0 +1,369 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import tempfile
+import time
+
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class StreamingTestsForeachFamilyMixin:

Review Comment:
   Wondering why 'foreach family'. What are the other foreach() methods tested here? 



##########
python/pyspark/sql/tests/streaming/test_streaming.py:
##########
@@ -26,7 +26,39 @@
 from pyspark.testing.sqlutils import ReusedSQLTestCase
 
 
-class StreamingTests(ReusedSQLTestCase):
+class StreamingTestsMixin:
+    def test_streaming_query_functions_sanity(self):

Review Comment:
   Is this a new test?
   Could you add brief description? `test... sanity` does not sound good. May be `test_streaming_query_functions_basic()` ? 



##########
python/pyspark/sql/tests/streaming/test_streaming.py:
##########
@@ -77,8 +109,14 @@ def test_stream_read_options_overwrite(self):
                 .schema(bad_schema)
                 .load(path="python/test_support/sql/streaming", schema=schema, format="text")
             )
-        self.assertTrue(df.isStreaming)
-        self.assertEqual(df.schema.simpleString(), "struct<data:string>")
+            # TODO: Moving this outside of with block will trigger the following error,

Review Comment:
   It does not need to be a TODO. Testing it inside `with` is the right thing.
   Calls like `schema` are executed on server with the updated state of the conf. 



##########
python/pyspark/sql/tests/streaming/test_streaming_foreach_family.py:
##########
@@ -0,0 +1,369 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import tempfile
+import time
+
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class StreamingTestsForeachFamilyMixin:
+    class ForeachWriterTester:

Review Comment:
   When I tried these tests, I got a serialization error after moving this to 'Mixin'. We could handle it when run into it with connect tests. 



##########
python/pyspark/sql/tests/streaming/test_streaming.py:
##########
@@ -295,334 +333,6 @@ def test_query_manager_await_termination(self):
             q.stop()
             shutil.rmtree(tmpPath)
 
-    class ForeachWriterTester:

Review Comment:
   Moving foreach tests to a different files SGTM.



##########
python/pyspark/sql/streaming/query.py:
##########
@@ -188,7 +192,7 @@ def awaitTermination(self, timeout: Optional[int] = None) -> Optional[bool]:
 
         Return whether the query has terminated or not within 5 seconds
 
-        >>> sq.awaitTermination(5)
+        >>> sq.awaitTermination(5) # doctest: +SKIP

Review Comment:
   Why are these needed? This won't be tested in connect yet, right? 



##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -89,8 +92,8 @@ def format(self, source: str) -> "DataStreamReader":
 
         Examples
         --------
-        >>> spark.readStream.format("text")
-        <pyspark.sql.streaming.readwriter.DataStreamReader object ...>
+        >>> spark.readStream.format("text") # doctest: +ELLIPSIS

Review Comment:
   Is `# doctest: +ELLIPSIS` needed? I didn't add in my PR (#40586)



##########
python/pyspark/sql/tests/streaming/test_streaming_foreach_family.py:
##########
@@ -0,0 +1,369 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import tempfile
+import time
+
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class StreamingTestsForeachFamilyMixin:

Review Comment:
   Better not to have Mixin class yet. We don't have corresponding '_partity_' tests in connect for this. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1163446826


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -188,7 +192,7 @@ def awaitTermination(self, timeout: Optional[int] = None) -> Optional[bool]:
 
         Return whether the query has terminated or not within 5 seconds
 
-        >>> sq.awaitTermination(5)
+        >>> sq.awaitTermination(5) # doctest: +SKIP

Review Comment:
   Better solution is to comment out setting `__doc__` for awaitTermination() in connect's query.py.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] pengzhon-db commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "pengzhon-db (via GitHub)" <gi...@apache.org>.
pengzhon-db commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162341092


##########
python/pyspark/sql/tests/connect/streaming/test_parity_streaming.py:
##########
@@ -0,0 +1,68 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+
+from pyspark.sql.tests.streaming.test_streaming import StreamingTestsMixin
+from pyspark.testing.connectutils import ReusedConnectTestCase
+
+
+class StreamingParityTests(StreamingTestsMixin, ReusedConnectTestCase):

Review Comment:
   Not sure if I understand the intent of this file. Looks all tests in this file are skipped. Where do we test already supported spark streaming connect?



##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -1359,7 +1364,8 @@ def start(
         >>> q.name
         'this_query'
         >>> q.stop()
-        >>> q.isActive
+

Review Comment:
   nit: remove the space?



##########
python/pyspark/sql/dataframe.py:
##########
@@ -535,7 +535,7 @@ def writeStream(self) -> DataStreamWriter:
         >>> with tempfile.TemporaryDirectory() as d:
         ...     # Create a table with Rate source.
         ...     df.writeStream.toTable(
-        ...         "my_table", checkpointLocation=d) # doctest: +ELLIPSIS

Review Comment:
   curious what does `# doctest: +ELLIPSIS` mean?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162024767


##########
python/pyspark/sql/tests/streaming/test_streaming_foreach_family.py:
##########
@@ -0,0 +1,369 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import tempfile
+import time
+
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class StreamingTestsForeachFamilyMixin:
+    class ForeachWriterTester:

Review Comment:
   I get all tests passed by running ` python/run-tests --testnames pyspark.sql.tests.streaming.test_streaming_foreach_family`. I renamed the mixin part though in the new commit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162017055


##########
python/pyspark/sql/tests/streaming/test_streaming.py:
##########
@@ -77,8 +109,14 @@ def test_stream_read_options_overwrite(self):
                 .schema(bad_schema)
                 .load(path="python/test_support/sql/streaming", schema=schema, format="text")
             )
-        self.assertTrue(df.isStreaming)
-        self.assertEqual(df.schema.simpleString(), "struct<data:string>")
+            # TODO: Moving this outside of with block will trigger the following error,

Review Comment:
   I see, I'll remove the commetns. So this is a behavior change for migrating to connect?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1163147543


##########
python/pyspark/sql/tests/connect/streaming/test_parity_streaming.py:
##########
@@ -0,0 +1,68 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+
+from pyspark.sql.tests.streaming.test_streaming import StreamingTestsMixin
+from pyspark.testing.connectutils import ReusedConnectTestCase
+
+
+class StreamingParityTests(StreamingTestsMixin, ReusedConnectTestCase):

Review Comment:
   This file marks all tests that should be skipped currently because these functionality is not supported yet in connect.
   
   It executes all tests from `StreamingTestsMixin`, except these below marked skipped.
   
   When anyone adds support to corresponding functions, they should delete the @unittest.skip and function, unless they want to do some change specific for connect.
   
   So ideally in the end, this class should only be like 
   ```
   class StreamingParityTests(StreamingTestsMixin, ReusedConnectTestCase):
       pass
   ```
   And then it runs all tests from StreamingTestsMixin



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162087535


##########
python/pyspark/sql/tests/streaming/test_streaming_foreach_family.py:
##########
@@ -0,0 +1,369 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import tempfile
+import time
+
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class StreamingTestsForeachFamilyMixin:

Review Comment:
   Class is fine. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1160306325


##########
python/pyspark/sql/connect/streaming/readwriter.py:
##########
@@ -168,9 +168,75 @@ def json(
 

Review Comment:
   Please ignore the change in this file. They are added in https://github.com/apache/spark/pull/40689.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1160306325


##########
python/pyspark/sql/connect/streaming/readwriter.py:
##########
@@ -168,9 +168,75 @@ def json(
 

Review Comment:
   Please ignore this change. They are added in https://github.com/apache/spark/pull/40689.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162017055


##########
python/pyspark/sql/tests/streaming/test_streaming.py:
##########
@@ -77,8 +109,14 @@ def test_stream_read_options_overwrite(self):
                 .schema(bad_schema)
                 .load(path="python/test_support/sql/streaming", schema=schema, format="text")
             )
-        self.assertTrue(df.isStreaming)
-        self.assertEqual(df.schema.simpleString(), "struct<data:string>")
+            # TODO: Moving this outside of with block will trigger the following error,

Review Comment:
   I see. So this is a behavior change for migrating to connect?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162076008


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -188,7 +192,7 @@ def awaitTermination(self, timeout: Optional[int] = None) -> Optional[bool]:
 
         Return whether the query has terminated or not within 5 seconds
 
-        >>> sq.awaitTermination(5)
+        >>> sq.awaitTermination(5) # doctest: +SKIP

Review Comment:
   What is the pain? What fails if you remove this diff?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162084233


##########
python/pyspark/sql/tests/streaming/test_streaming_foreach_family.py:
##########
@@ -0,0 +1,369 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import tempfile
+import time
+
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class StreamingTestsForeachFamilyMixin:
+    class ForeachWriterTester:

Review Comment:
   I see! I could make another class for `foreach` then. The jira for `foreach` support is already added by me IIRC, in the epic



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162082952


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -188,7 +192,7 @@ def awaitTermination(self, timeout: Optional[int] = None) -> Optional[bool]:
 
         Return whether the query has terminated or not within 5 seconds
 
-        >>> sq.awaitTermination(5)
+        >>> sq.awaitTermination(5) # doctest: +SKIP

Review Comment:
   Nothing fails but we skips some tests for also non-connect scenario



##########
python/pyspark/sql/streaming/query.py:
##########
@@ -188,7 +192,7 @@ def awaitTermination(self, timeout: Optional[int] = None) -> Optional[bool]:
 
         Return whether the query has terminated or not within 5 seconds
 
-        >>> sq.awaitTermination(5)
+        >>> sq.awaitTermination(5) # doctest: +SKIP

Review Comment:
   Nothing fails but we skip some tests for also non-connect scenario



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162371303


##########
python/pyspark/sql/tests/connect/streaming/test_parity_streaming.py:
##########
@@ -0,0 +1,68 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+
+from pyspark.sql.tests.streaming.test_streaming import StreamingTestsMixin
+from pyspark.testing.connectutils import ReusedConnectTestCase
+
+
+class StreamingParityTests(StreamingTestsMixin, ReusedConnectTestCase):

Review Comment:
   uhoh, the tests here shouldn't be skipped.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162023412


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -188,7 +192,7 @@ def awaitTermination(self, timeout: Optional[int] = None) -> Optional[bool]:
 
         Return whether the query has terminated or not within 5 seconds
 
-        >>> sq.awaitTermination(5)
+        >>> sq.awaitTermination(5) # doctest: +SKIP

Review Comment:
   Right.. This actually won't be tested in both connect and non-connect I believe (@HyukjinKwon please correct me if I'm wrong) , just some temporary pain we have to bear with 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162023412


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -188,7 +192,7 @@ def awaitTermination(self, timeout: Optional[int] = None) -> Optional[bool]:
 
         Return whether the query has terminated or not within 5 seconds
 
-        >>> sq.awaitTermination(5)
+        >>> sq.awaitTermination(5) # doctest: +SKIP

Review Comment:
   Right.. This actually silents this line in the doctest for both connect and non-connect I believe (@HyukjinKwon please correct me if I'm wrong) , just some temporary pain we have to bear with as connect doesn't support this yet



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162079213


##########
python/pyspark/sql/tests/streaming/test_streaming_foreach_family.py:
##########
@@ -0,0 +1,369 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import tempfile
+import time
+
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class StreamingTestsForeachFamilyMixin:
+    class ForeachWriterTester:

Review Comment:
   Please add foreach() api to one of the jiras or a new jira. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1163460936


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -188,7 +192,7 @@ def awaitTermination(self, timeout: Optional[int] = None) -> Optional[bool]:
 
         Return whether the query has terminated or not within 5 seconds
 
-        >>> sq.awaitTermination(5)
+        >>> sq.awaitTermination(5) # doctest: +SKIP

Review Comment:
   Thanks! Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162075499


##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -89,8 +92,8 @@ def format(self, source: str) -> "DataStreamReader":
 
         Examples
         --------
-        >>> spark.readStream.format("text")
-        <pyspark.sql.streaming.readwriter.DataStreamReader object ...>
+        >>> spark.readStream.format("text") # doctest: +ELLIPSIS

Review Comment:
   I didn't add. E.g.:
   <img width="758" alt="image" src="https://user-images.githubusercontent.com/502522/230995675-b6d23836-5d49-42f3-967d-7cf04229a5f8.png">
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1160348218


##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -1150,6 +1150,7 @@ def foreach(self, f: Union[Callable[[Row], None], "SupportsProcess"]) -> "DataSt
         This API is evolving.
 
         Examples
+        TODO(SPARK-43054): remove the SKIP flags below

Review Comment:
   no biggie but should probably put this comment on the top of this method .. otherwise it'll be shown in API reference documentation ..



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162078515


##########
python/pyspark/sql/tests/streaming/test_streaming_foreach_family.py:
##########
@@ -0,0 +1,369 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import tempfile
+import time
+
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class StreamingTestsForeachFamilyMixin:
+    class ForeachWriterTester:

Review Comment:
   Can we move `foreach` back to its original place. Foreach() and foreachbath() are very different. The later takes a dataframe, the former is more like a UDF.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162087262


##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -89,8 +92,8 @@ def format(self, source: str) -> "DataStreamReader":
 
         Examples
         --------
-        >>> spark.readStream.format("text")
-        <pyspark.sql.streaming.readwriter.DataStreamReader object ...>
+        >>> spark.readStream.format("text") # doctest: +ELLIPSIS

Review Comment:
   No need to add them. It makes it the test less readable. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40691:
URL: https://github.com/apache/spark/pull/40691#issuecomment-1506121449

   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162023412


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -188,7 +192,7 @@ def awaitTermination(self, timeout: Optional[int] = None) -> Optional[bool]:
 
         Return whether the query has terminated or not within 5 seconds
 
-        >>> sq.awaitTermination(5)
+        >>> sq.awaitTermination(5) # doctest: +SKIP

Review Comment:
   Right.. This actually silents this line in the doctest for both connect and non-connect I believe (@HyukjinKwon please correct me if I'm wrong) , just some temporary pain we have to bear with 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162086693


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -330,7 +335,8 @@ def stop(self) -> None:
         Stop streaming query
 
         >>> sq.stop()
-        >>> sq.isActive
+
+        >>> sq.isActive # doctest: +SKIP

Review Comment:
   Is see, please add a TODO here with a brief comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1163651644


##########
python/pyspark/sql/tests/connect/streaming/test_parity_streaming.py:
##########
@@ -0,0 +1,68 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+
+from pyspark.sql.tests.streaming.test_streaming import StreamingTestsMixin
+from pyspark.testing.connectutils import ReusedConnectTestCase
+
+
+class StreamingParityTests(StreamingTestsMixin, ReusedConnectTestCase):

Review Comment:
   Each of the skipped tests here has a SPARK ticket associated with it. We will enable these soon. LGTM. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162175072


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -188,7 +192,7 @@ def awaitTermination(self, timeout: Optional[int] = None) -> Optional[bool]:
 
         Return whether the query has terminated or not within 5 seconds
 
-        >>> sq.awaitTermination(5)
+        >>> sq.awaitTermination(5) # doctest: +SKIP

Review Comment:
   If we're absolutely sure that we will enable this back very soon, I am fine. As @rangadi said, I tried my best to avoid skipping the tests, but I did few times when I am about to enable it back very soon.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on PR #40691:
URL: https://github.com/apache/spark/pull/40691#issuecomment-1502715640

   Hi @HyukjinKwon could you please take another look? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162080029


##########
python/pyspark/sql/streaming/readwriter.py:
##########
@@ -89,8 +92,8 @@ def format(self, source: str) -> "DataStreamReader":
 
         Examples
         --------
-        >>> spark.readStream.format("text")
-        <pyspark.sql.streaming.readwriter.DataStreamReader object ...>
+        >>> spark.readStream.format("text") # doctest: +ELLIPSIS

Review Comment:
   Ah I see, in line 538 there is one. I could remove all of the `# doctest: +ELLIPSIS` as we already defined them in the option: https://github.com/apache/spark/blob/17720b7121eee050a546126fad7ce8229fe6bda2/python/pyspark/sql/streaming/readwriter.py#L1504
   
   https://stackoverflow.com/questions/17092215/how-enable-ellipsis-when-calling-python-doctest



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1163151275


##########
python/pyspark/sql/dataframe.py:
##########
@@ -535,7 +535,7 @@ def writeStream(self) -> DataStreamWriter:
         >>> with tempfile.TemporaryDirectory() as d:
         ...     # Create a table with Rate source.
         ...     df.writeStream.toTable(
-        ...         "my_table", checkpointLocation=d) # doctest: +ELLIPSIS

Review Comment:
   My understanding is that in doctest the result will be kind of regex checked if this flag is set. Like the line below right now is ` <...streaming.query.StreamingQuery object at 0x...>`, but before it was ` <pyspark.sql.streaming.query.StreamingQuery object at 0x...>`, which would conflict with connect's test, which returns `<pyspark.sql.connect.streaming.query.StreamingQuery object at 0x...>`
   
   So to make this test works for both connect and non-connect, we enable this regex-like check and replace below with ...
   But it doesn't matter anyway, as we enabled the flag in test options in the `__main__` method below
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1160306325


##########
python/pyspark/sql/connect/streaming/readwriter.py:
##########
@@ -168,9 +168,75 @@ def json(
 

Review Comment:
   Please ignore the change in this file. They are added and to be reviewed in https://github.com/apache/spark/pull/40689.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on PR #40691:
URL: https://github.com/apache/spark/pull/40691#issuecomment-1500623778

   CC @rangadi @pengzhon-db


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162076519


##########
python/pyspark/sql/tests/streaming/test_streaming.py:
##########
@@ -77,8 +109,14 @@ def test_stream_read_options_overwrite(self):
                 .schema(bad_schema)
                 .load(path="python/test_support/sql/streaming", schema=schema, format="text")
             )
-        self.assertTrue(df.isStreaming)
-        self.assertEqual(df.schema.simpleString(), "struct<data:string>")
+            # TODO: Moving this outside of with block will trigger the following error,

Review Comment:
   Yes. Something like schema is evaluated with the conf at the time of `schema` call. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1161988877


##########
python/pyspark/sql/tests/streaming/test_streaming_foreach_family.py:
##########
@@ -0,0 +1,369 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import tempfile
+import time
+
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class StreamingTestsForeachFamilyMixin:

Review Comment:
   Right I put both `foreach` and `foreachBatch` to this class. My naming sense isn't that great so I'm very welcome to any suggestion on it's name here...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1161993951


##########
python/pyspark/sql/tests/streaming/test_streaming_foreach_family.py:
##########
@@ -0,0 +1,369 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import tempfile
+import time
+
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class StreamingTestsForeachFamilyMixin:

Review Comment:
   Sure I'll remove the mixin. Still better to create this class though, to avoid tons of `@unittest.skip` in the streaming parity test class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1163445676


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -188,7 +192,7 @@ def awaitTermination(self, timeout: Optional[int] = None) -> Optional[bool]:
 
         Return whether the query has terminated or not within 5 seconds
 
-        >>> sq.awaitTermination(5)
+        >>> sq.awaitTermination(5) # doctest: +SKIP

Review Comment:
   I see. I think I'll just remove all of the SKIP flags and disable the doctest for now. Some of the functionalities might not be supported that soon I think. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on PR #40691:
URL: https://github.com/apache/spark/pull/40691#issuecomment-1505663707

   @HyukjinKwon can you merge this? Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162133389


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -188,7 +192,7 @@ def awaitTermination(self, timeout: Optional[int] = None) -> Optional[bool]:
 
         Return whether the query has terminated or not within 5 seconds
 
-        >>> sq.awaitTermination(5)
+        >>> sq.awaitTermination(5) # doctest: +SKIP

Review Comment:
   Sure! I've moved the TODO to the top of these methods to avoid them showing up in the PySpark doc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162079902


##########
python/pyspark/sql/tests/streaming/test_streaming_foreach_family.py:
##########
@@ -0,0 +1,369 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import tempfile
+import time
+
+from pyspark.testing.sqlutils import ReusedSQLTestCase
+
+
+class StreamingTestsForeachFamilyMixin:
+    class ForeachWriterTester:

Review Comment:
   Or you could keep here. Either is ok. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] WweiL commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "WweiL (via GitHub)" <gi...@apache.org>.
WweiL commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162082565


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -330,7 +335,8 @@ def stop(self) -> None:
         Stop streaming query
 
         >>> sq.stop()
-        >>> sq.isActive
+
+        >>> sq.isActive # doctest: +SKIP

Review Comment:
   Calling isActive after the query is stopped will throw error right now before the better session management is implemented



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] rangadi commented on a diff in pull request #40691: [SPARK-43031] [SS] [Connect] Enable unit test and doctest for streaming

Posted by "rangadi (via GitHub)" <gi...@apache.org>.
rangadi commented on code in PR #40691:
URL: https://github.com/apache/spark/pull/40691#discussion_r1162100853


##########
python/pyspark/sql/streaming/query.py:
##########
@@ -188,7 +192,7 @@ def awaitTermination(self, timeout: Optional[int] = None) -> Optional[bool]:
 
         Return whether the query has terminated or not within 5 seconds
 
-        >>> sq.awaitTermination(5)
+        >>> sq.awaitTermination(5) # doctest: +SKIP

Review Comment:
   Lets not skip any tests. Any skip should have a TODO comment. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org