[PR] [Spark][#36841] Reactivate structured streaming state tests and prune obsolete DStream helpers [beam]

["tkaymak (via GitHub)" <gi...@apache.org>]

tkaymak opened a new pull request, #40130:
URL: https://github.com/apache/beam/pull/40130

   Part of #36841. Cleans up obsolete DStream test remnants in the Structured Streaming test suite and adds streaming lifecycle coverage for timeout handling.
   
   **Context**
   
   The shared StructuredStreamingPipelineStateTest retained four tests marked with @Ignore("TODO: Reactivate with streaming."). They used CreateStream, which is a legacy DStream concept not supported by Structured Streaming. Spark 4 now has dedicated streaming lifecycle tests in StreamingPipelineLifecycleTest covering RUNNING to DONE on idle, cancellation, and sibling query failure propagation.
   
   **What this changes**
   
   - runners/spark/src/test/.../StructuredStreamingPipelineStateTest.java: Removed the four obsolete @Ignore streaming test cases. Inlined Create.of("one", "two") at call sites and removed the obsolete getValues() and getStreamingOptions() helpers and CreateStream imports.
   - runners/spark/src/test/.../translation/streaming/SimpleSourceTest.java: Deleted this obsolete, ignored file.
   - runners/spark/4/src/test/.../translation/streaming/StreamingPipelineLifecycleTest.java: Added timeoutKeepsRunningState to assert that waitUntilFinish(Duration.millis(1)) preserves RUNNING state while the query remains active, with try, finally query cancellation to prevent leaks in the shared session.
   
   **Tests**
   
   - :runners:spark:3:test passed locally (StructuredStreamingPipelineStateTest).
   - :runners:spark:4:test passed locally (StreamingPipelineLifecycleTest, 4 passed).
   - checkstyle and spotless verified clean on Spark 3 and Spark 4.
   
   R: @Abacn


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [Spark][#36841] Reactivate structured streaming state tests and prune obsolete DStream helpers [beam]

["Abacn (via GitHub)" <gi...@apache.org>]

Abacn commented on PR #40130:
URL: https://github.com/apache/beam/pull/40130#issuecomment-5683113801

   classMethod (org.apache.beam.examples.complete.kafkatopubsub.KafkaToPubsubE2ETest) failed
   
   known flaky test, not related to this change


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [Spark][#36841] Reactivate structured streaming state tests and prune obsolete DStream helpers [beam]

["Abacn (via GitHub)" <gi...@apache.org>]

Abacn merged PR #40130:
URL: https://github.com/apache/beam/pull/40130


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org