[jira] [Created] (SPARK-20432) Unioning two identical Streaming DataFrames fails during attribute resolution

["Burak Yavuz (JIRA)" <ji...@apache.org>]

Burak Yavuz created SPARK-20432:
-----------------------------------

             Summary: Unioning two identical Streaming DataFrames fails during attribute resolution
                 Key: SPARK-20432
                 URL: https://issues.apache.org/jira/browse/SPARK-20432
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 2.1.0
            Reporter: Burak Yavuz


To reproduce, try unioning two identical Kafka Streams:

{code}
df = spark.readStream.format("kafka")... \
  .select(from_json(col("value").cast("string"), simpleSchema).alias("parsed_value"))

df.union(df).writeStream...
{code}

Exception is confusing:

{code}
org.apache.spark.sql.AnalysisException: resolved attribute(s) value#526 missing from value#511,topic#512,partition#513,offset#514L,timestampType#516,key#510,timestamp#515 in operator !Project [jsontostructs(...) AS parsed_value#357];
{code}






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org