You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael Armbrust (JIRA)" <ji...@apache.org> on 2017/05/03 00:13:04 UTC

[jira] [Created] (SPARK-20567) Failure to bind when using explode and collect_set in streaming

Michael Armbrust created SPARK-20567:
----------------------------------------

             Summary: Failure to bind when using explode and collect_set in streaming
                 Key: SPARK-20567
                 URL: https://issues.apache.org/jira/browse/SPARK-20567
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.2.0
            Reporter: Michael Armbrust
            Assignee: Michael Armbrust
            Priority: Critical


Here is a small test case:
{code}
  test("count distinct") {
    val inputData = MemoryStream[(Int, Seq[Int])]

    val aggregated =
      inputData.toDF()
        .select($"*", explode($"_2") as 'value)
        .groupBy($"_1")
        .agg(size(collect_set($"value")))
        .as[(Int, Int)]

    testStream(aggregated, Update)(
      AddData(inputData, (1, Seq(1, 2))),
      CheckLastBatch((1, 2))
    )
  }
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org