You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Artem Moskvin <ar...@cloud.upwork.com.INVALID> on 2018/03/19 14:08:34 UTC

[Spark Structured Streaming, Spark 2.3.0] Calling current_timestamp() function within a streaming dataframe results in dataType error

Hi all,

There's probably a regression in Spark 2.3.0. Running the code below in
2.2.1 succeeds but in 2.3.0 results in error
`org.apache.spark.sql.streaming.StreamingQueryException: Invalid call to
dataType on unresolved object, tree: 'current_timestamp`.

```
import org.apache.spark.sql.functions._
import org.apache.spark.sql.streaming.Trigger
import scala.concurrent.duration._

val values = spark.
  readStream.
  format("rate").
  load.
  withColumn("current_timestamp", current_timestamp)

values.
  writeStream.
  format("console").
  option("truncate", false).
  trigger(Trigger.ProcessingTime(10.seconds)).
  start().
  awaitTermination()
```

Can anyone confirm the same behavior?


Respectfully,
Artem Moskvin