You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2021/11/22 20:31:00 UTC
[jira] [Resolved] (SPARK-37440) org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;; despite of time-based window
[ https://issues.apache.org/jira/browse/SPARK-37440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim resolved SPARK-37440.
----------------------------------
Resolution: Duplicate
> org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;; despite of time-based window
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-37440
> URL: https://issues.apache.org/jira/browse/SPARK-37440
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.0.1
> Reporter: Ilya
> Priority: Major
>
> Initially posted here: [https://stackoverflow.com/questions/70062355/org-apache-spark-sql-analysisexception-non-time-based-windows-are-not-supported]
>
> 'm doing the window-based sorting for the Spark Structured Streaming:
>
> {{val filterWindow: WindowSpec = Window .partitionBy("key")
> .orderBy($"time")
> controlDataFrame=controlDataFrame.withColumn("Make Coffee", $"value").
> withColumn("datetime", date_trunc("second", current_timestamp())).
> withColumn("time", current_timestamp()).
> withColumn("temp_rank", rank().over(filterWindow))
> .filter(col("temp_rank") === 1)
> .drop("temp_rank").
> withColumn("digitalTwinId", lit(digitalTwinId)).
> withWatermark("datetime", "10 seconds")}}
> I'm obtaining {{time}} as {{current_timestamp()}} and in schemat I see its type as {{StructField(time,TimestampType,true)}}
> Why Spark 3.0 doesn't allow me to do the window operation based on it with the following exception, as the filed is clearly time-based?
>
> {{21/11/22 10:34:03 WARN SparkSession$Builder: Using an existing SparkSession; some spark core configurations may not take effect.
> org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;;Window [rank(time#163) windowspecdefinition(key#150, time#163 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS temp_rank#171], [key#150], [time#163 ASC NULLS FIRST]
> +- Project [key#150, value#151, Make Coffee#154, datetime#158, time#163]}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org