You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Artem Moskvin <ar...@cloud.upwork.com.INVALID> on 2018/03/19 14:08:34 UTC
[Spark Structured Streaming, Spark 2.3.0] Calling current_timestamp()
function within a streaming dataframe results in dataType error
Hi all,
There's probably a regression in Spark 2.3.0. Running the code below in
2.2.1 succeeds but in 2.3.0 results in error
`org.apache.spark.sql.streaming.StreamingQueryException: Invalid call to
dataType on unresolved object, tree: 'current_timestamp`.
```
import org.apache.spark.sql.functions._
import org.apache.spark.sql.streaming.Trigger
import scala.concurrent.duration._
val values = spark.
readStream.
format("rate").
load.
withColumn("current_timestamp", current_timestamp)
values.
writeStream.
format("console").
option("truncate", false).
trigger(Trigger.ProcessingTime(10.seconds)).
start().
awaitTermination()
```
Can anyone confirm the same behavior?
Respectfully,
Artem Moskvin