You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Burak Yavuz (JIRA)" <ji...@apache.org> on 2017/05/03 15:58:04 UTC
[jira] [Closed] (SPARK-20432) Unioning two identical Streaming
DataFrames fails during attribute resolution
[ https://issues.apache.org/jira/browse/SPARK-20432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Burak Yavuz closed SPARK-20432.
-------------------------------
Resolution: Duplicate
> Unioning two identical Streaming DataFrames fails during attribute resolution
> -----------------------------------------------------------------------------
>
> Key: SPARK-20432
> URL: https://issues.apache.org/jira/browse/SPARK-20432
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.1.0
> Reporter: Burak Yavuz
>
> To reproduce, try unioning two identical Kafka Streams:
> {code}
> df = spark.readStream.format("kafka")... \
> .select(from_json(col("value").cast("string"), simpleSchema).alias("parsed_value"))
> df.union(df).writeStream...
> {code}
> Exception is confusing:
> {code}
> org.apache.spark.sql.AnalysisException: resolved attribute(s) value#526 missing from value#511,topic#512,partition#513,offset#514L,timestampType#516,key#510,timestamp#515 in operator !Project [jsontostructs(...) AS parsed_value#357];
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org