[jira] [Created] (SPARK-35246) Disable intersects for all streaming queries

["Jose Torres (Jira)" <ji...@apache.org>]

Jose Torres created SPARK-35246:
-----------------------------------

             Summary: Disable intersects for all streaming queries
                 Key: SPARK-35246
                 URL: https://issues.apache.org/jira/browse/SPARK-35246
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 3.1.0, 3.0.0
            Reporter: Jose Torres
             Fix For: 3.2.0


The UnsupportedOperationChecker currently rejects streaming intersects only if both sides are streaming, but they don't work if even one side is streaming. The following simple test, for example, fails with a cryptic None.get error because the state store can't plan itself properly.
{code:java}
  test("intersect") {
    val input = MemoryStream[Long]
    val df = input.toDS().intersect(spark.range(10).as[Long])
    testStream(df) (
      AddData(input, 1L),
      CheckAnswer(1)
    )
  }
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org