You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by 张万新 <ke...@gmail.com> on 2017/09/11 10:03:18 UTC
[SS]How to add a column with custom system time?
Hi,
In structured streaming how can I add a column to a dataset with current
system time aligned with 15 minutes?
Thanks.
Re: [SS]How to add a column with custom system time?
Posted by Michael Armbrust <mi...@databricks.com>.
Can you show the explain() for the version that doesn't work? You might
just be hitting a bug.
On Tue, Sep 12, 2017 at 9:03 PM, 张万新 <ke...@gmail.com> wrote:
> It seems current_timestamp() cannot be used directly in window function?
> because after attempts I found that using
>
> *df.count.withColumn("pTime", current_timestamp).select(window($"pTime",
> "15 minutes"), $"count")*
>
> instead of
>
> *df.count.withColumn("window", window(current_timestamp(), "15 minutes"))*
>
> throws no exception and works fine. I don't know if this is a problem that
> needs improvement.
>
>
> 张万新 <ke...@gmail.com>于2017年9月13日周三 上午11:43写道:
>
>> and I use .withColumn("window", window(current_timestamp(), "15
>> minutes")) after count
>>
>> 张万新 <ke...@gmail.com>于2017年9月13日周三 上午11:32写道:
>>
>>> *Yes, my code is shown below*
>>> /**
>>> * input
>>> */
>>> val logs = spark
>>> .readStream
>>> .format("kafka")
>>> .option("kafka.bootstrap.servers", BROKER_SERVER)
>>> .option("subscribe", TOPIC)
>>> .option("startingOffset", "latest")
>>> .load()
>>>
>>> /**
>>> * process
>>> */
>>> val logValues = logs.selectExpr("CAST(value AS STRING)").as[(String)]
>>>
>>> val events = logValues
>>> .map(parseFunction)
>>> .select(
>>> $"_1".alias("date").cast("timestamp"),
>>> $"_2".alias("uuid").cast("string")
>>> )
>>>
>>> val results = events
>>> .withWatermark("date", "1 day")
>>> .dropDuplicates("uuid", "date")
>>> .groupBy($"date")
>>> .count()
>>> .withColumn("window", window(current_timestamp(), "15 minutes"))
>>>
>>> /**
>>> * output
>>> */
>>> val query = results
>>> .writeStream
>>> .outputMode("update")
>>> .format("console")
>>> .option("truncate", "false")
>>> .trigger(Trigger.ProcessingTime("1 seconds"))
>>> .start()
>>>
>>> query.awaitTermination()
>>>
>>> *and I use play json to parse input logs from kafka ,the parse function
>>> is like*
>>>
>>> def parseFunction(str: String): (Long, String) = {
>>> val json = Json.parse(str)
>>> val timestamp = (json \ "time").get.toString().toLong
>>> val date = (timestamp / (60 * 60 * 24) * 24 -8) * 60 * 60
>>> val uuid = (json \ "uuid").get.toString()
>>> (date, uuid)
>>> }
>>>
>>> Michael Armbrust <mi...@databricks.com>于2017年9月13日周三 上午2:36写道:
>>>
>>>> Can you show all the code? This works for me.
>>>>
>>>> On Tue, Sep 12, 2017 at 12:05 AM, 张万新 <ke...@gmail.com> wrote:
>>>>
>>>>> The spark version is 2.2.0
>>>>>
>>>>> Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 下午12:32写道:
>>>>>
>>>>>> Which version of spark?
>>>>>>
>>>>>> On Mon, Sep 11, 2017 at 8:27 PM, 张万新 <ke...@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks for reply, but using this method I got an exception:
>>>>>>>
>>>>>>> "Exception in thread "main" org.apache.spark.sql.streaming.StreamingQueryException:
>>>>>>> nondeterministic expressions are only allowed in
>>>>>>>
>>>>>>> Project, Filter, Aggregate or Window"
>>>>>>>
>>>>>>> Can you give more advice?
>>>>>>>
>>>>>>> Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 上午4:48写道:
>>>>>>>
>>>>>>>> import org.apache.spark.sql.functions._
>>>>>>>>
>>>>>>>> df.withColumn("window", window(current_timestamp(), "15 minutes"))
>>>>>>>>
>>>>>>>> On Mon, Sep 11, 2017 at 3:03 AM, 张万新 <ke...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> In structured streaming how can I add a column to a dataset with
>>>>>>>>> current system time aligned with 15 minutes?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
Re: [SS]How to add a column with custom system time?
Posted by 张万新 <ke...@gmail.com>.
It seems current_timestamp() cannot be used directly in window function?
because after attempts I found that using
*df.count.withColumn("pTime", current_timestamp).select(window($"pTime",
"15 minutes"), $"count")*
instead of
*df.count.withColumn("window", window(current_timestamp(), "15 minutes"))*
throws no exception and works fine. I don't know if this is a problem that
needs improvement.
张万新 <ke...@gmail.com>于2017年9月13日周三 上午11:43写道:
> and I use .withColumn("window", window(current_timestamp(), "15
> minutes")) after count
>
> 张万新 <ke...@gmail.com>于2017年9月13日周三 上午11:32写道:
>
>> *Yes, my code is shown below*
>> /**
>> * input
>> */
>> val logs = spark
>> .readStream
>> .format("kafka")
>> .option("kafka.bootstrap.servers", BROKER_SERVER)
>> .option("subscribe", TOPIC)
>> .option("startingOffset", "latest")
>> .load()
>>
>> /**
>> * process
>> */
>> val logValues = logs.selectExpr("CAST(value AS STRING)").as[(String)]
>>
>> val events = logValues
>> .map(parseFunction)
>> .select(
>> $"_1".alias("date").cast("timestamp"),
>> $"_2".alias("uuid").cast("string")
>> )
>>
>> val results = events
>> .withWatermark("date", "1 day")
>> .dropDuplicates("uuid", "date")
>> .groupBy($"date")
>> .count()
>> .withColumn("window", window(current_timestamp(), "15 minutes"))
>>
>> /**
>> * output
>> */
>> val query = results
>> .writeStream
>> .outputMode("update")
>> .format("console")
>> .option("truncate", "false")
>> .trigger(Trigger.ProcessingTime("1 seconds"))
>> .start()
>>
>> query.awaitTermination()
>>
>> *and I use play json to parse input logs from kafka ,the parse function
>> is like*
>>
>> def parseFunction(str: String): (Long, String) = {
>> val json = Json.parse(str)
>> val timestamp = (json \ "time").get.toString().toLong
>> val date = (timestamp / (60 * 60 * 24) * 24 -8) * 60 * 60
>> val uuid = (json \ "uuid").get.toString()
>> (date, uuid)
>> }
>>
>> Michael Armbrust <mi...@databricks.com>于2017年9月13日周三 上午2:36写道:
>>
>>> Can you show all the code? This works for me.
>>>
>>> On Tue, Sep 12, 2017 at 12:05 AM, 张万新 <ke...@gmail.com> wrote:
>>>
>>>> The spark version is 2.2.0
>>>>
>>>> Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 下午12:32写道:
>>>>
>>>>> Which version of spark?
>>>>>
>>>>> On Mon, Sep 11, 2017 at 8:27 PM, 张万新 <ke...@gmail.com> wrote:
>>>>>
>>>>>> Thanks for reply, but using this method I got an exception:
>>>>>>
>>>>>> "Exception in thread "main"
>>>>>> org.apache.spark.sql.streaming.StreamingQueryException: nondeterministic
>>>>>> expressions are only allowed in
>>>>>>
>>>>>> Project, Filter, Aggregate or Window"
>>>>>>
>>>>>> Can you give more advice?
>>>>>>
>>>>>> Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 上午4:48写道:
>>>>>>
>>>>>>> import org.apache.spark.sql.functions._
>>>>>>>
>>>>>>> df.withColumn("window", window(current_timestamp(), "15 minutes"))
>>>>>>>
>>>>>>> On Mon, Sep 11, 2017 at 3:03 AM, 张万新 <ke...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> In structured streaming how can I add a column to a dataset with
>>>>>>>> current system time aligned with 15 minutes?
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>
Re: [SS]How to add a column with custom system time?
Posted by 张万新 <ke...@gmail.com>.
and I use .withColumn("window", window(current_timestamp(), "15
minutes")) after
count
张万新 <ke...@gmail.com>于2017年9月13日周三 上午11:32写道:
> *Yes, my code is shown below*
> /**
> * input
> */
> val logs = spark
> .readStream
> .format("kafka")
> .option("kafka.bootstrap.servers", BROKER_SERVER)
> .option("subscribe", TOPIC)
> .option("startingOffset", "latest")
> .load()
>
> /**
> * process
> */
> val logValues = logs.selectExpr("CAST(value AS STRING)").as[(String)]
>
> val events = logValues
> .map(parseFunction)
> .select(
> $"_1".alias("date").cast("timestamp"),
> $"_2".alias("uuid").cast("string")
> )
>
> val results = events
> .withWatermark("date", "1 day")
> .dropDuplicates("uuid", "date")
> .groupBy($"date")
> .count()
> .withColumn("window", window(current_timestamp(), "15 minutes"))
>
> /**
> * output
> */
> val query = results
> .writeStream
> .outputMode("update")
> .format("console")
> .option("truncate", "false")
> .trigger(Trigger.ProcessingTime("1 seconds"))
> .start()
>
> query.awaitTermination()
>
> *and I use play json to parse input logs from kafka ,the parse function is
> like*
>
> def parseFunction(str: String): (Long, String) = {
> val json = Json.parse(str)
> val timestamp = (json \ "time").get.toString().toLong
> val date = (timestamp / (60 * 60 * 24) * 24 -8) * 60 * 60
> val uuid = (json \ "uuid").get.toString()
> (date, uuid)
> }
>
> Michael Armbrust <mi...@databricks.com>于2017年9月13日周三 上午2:36写道:
>
>> Can you show all the code? This works for me.
>>
>> On Tue, Sep 12, 2017 at 12:05 AM, 张万新 <ke...@gmail.com> wrote:
>>
>>> The spark version is 2.2.0
>>>
>>> Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 下午12:32写道:
>>>
>>>> Which version of spark?
>>>>
>>>> On Mon, Sep 11, 2017 at 8:27 PM, 张万新 <ke...@gmail.com> wrote:
>>>>
>>>>> Thanks for reply, but using this method I got an exception:
>>>>>
>>>>> "Exception in thread "main"
>>>>> org.apache.spark.sql.streaming.StreamingQueryException: nondeterministic
>>>>> expressions are only allowed in
>>>>>
>>>>> Project, Filter, Aggregate or Window"
>>>>>
>>>>> Can you give more advice?
>>>>>
>>>>> Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 上午4:48写道:
>>>>>
>>>>>> import org.apache.spark.sql.functions._
>>>>>>
>>>>>> df.withColumn("window", window(current_timestamp(), "15 minutes"))
>>>>>>
>>>>>> On Mon, Sep 11, 2017 at 3:03 AM, 张万新 <ke...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> In structured streaming how can I add a column to a dataset with
>>>>>>> current system time aligned with 15 minutes?
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>
Re: [SS]How to add a column with custom system time?
Posted by 张万新 <ke...@gmail.com>.
*Yes, my code is shown below*
/**
* input
*/
val logs = spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", BROKER_SERVER)
.option("subscribe", TOPIC)
.option("startingOffset", "latest")
.load()
/**
* process
*/
val logValues = logs.selectExpr("CAST(value AS STRING)").as[(String)]
val events = logValues
.map(parseFunction)
.select(
$"_1".alias("date").cast("timestamp"),
$"_2".alias("uuid").cast("string")
)
val results = events
.withWatermark("date", "1 day")
.dropDuplicates("uuid", "date")
.groupBy($"date")
.count()
.withColumn("window", window(current_timestamp(), "15 minutes"))
/**
* output
*/
val query = results
.writeStream
.outputMode("update")
.format("console")
.option("truncate", "false")
.trigger(Trigger.ProcessingTime("1 seconds"))
.start()
query.awaitTermination()
*and I use play json to parse input logs from kafka ,the parse function is
like*
def parseFunction(str: String): (Long, String) = {
val json = Json.parse(str)
val timestamp = (json \ "time").get.toString().toLong
val date = (timestamp / (60 * 60 * 24) * 24 -8) * 60 * 60
val uuid = (json \ "uuid").get.toString()
(date, uuid)
}
Michael Armbrust <mi...@databricks.com>于2017年9月13日周三 上午2:36写道:
> Can you show all the code? This works for me.
>
> On Tue, Sep 12, 2017 at 12:05 AM, 张万新 <ke...@gmail.com> wrote:
>
>> The spark version is 2.2.0
>>
>> Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 下午12:32写道:
>>
>>> Which version of spark?
>>>
>>> On Mon, Sep 11, 2017 at 8:27 PM, 张万新 <ke...@gmail.com> wrote:
>>>
>>>> Thanks for reply, but using this method I got an exception:
>>>>
>>>> "Exception in thread "main"
>>>> org.apache.spark.sql.streaming.StreamingQueryException: nondeterministic
>>>> expressions are only allowed in
>>>>
>>>> Project, Filter, Aggregate or Window"
>>>>
>>>> Can you give more advice?
>>>>
>>>> Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 上午4:48写道:
>>>>
>>>>> import org.apache.spark.sql.functions._
>>>>>
>>>>> df.withColumn("window", window(current_timestamp(), "15 minutes"))
>>>>>
>>>>> On Mon, Sep 11, 2017 at 3:03 AM, 张万新 <ke...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> In structured streaming how can I add a column to a dataset with
>>>>>> current system time aligned with 15 minutes?
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>
>>>>>
>>>
>
Re: [SS]How to add a column with custom system time?
Posted by Michael Armbrust <mi...@databricks.com>.
Can you show all the code? This works for me.
On Tue, Sep 12, 2017 at 12:05 AM, 张万新 <ke...@gmail.com> wrote:
> The spark version is 2.2.0
>
> Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 下午12:32写道:
>
>> Which version of spark?
>>
>> On Mon, Sep 11, 2017 at 8:27 PM, 张万新 <ke...@gmail.com> wrote:
>>
>>> Thanks for reply, but using this method I got an exception:
>>>
>>> "Exception in thread "main" org.apache.spark.sql.streaming.StreamingQueryException:
>>> nondeterministic expressions are only allowed in
>>>
>>> Project, Filter, Aggregate or Window"
>>>
>>> Can you give more advice?
>>>
>>> Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 上午4:48写道:
>>>
>>>> import org.apache.spark.sql.functions._
>>>>
>>>> df.withColumn("window", window(current_timestamp(), "15 minutes"))
>>>>
>>>> On Mon, Sep 11, 2017 at 3:03 AM, 张万新 <ke...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> In structured streaming how can I add a column to a dataset with
>>>>> current system time aligned with 15 minutes?
>>>>>
>>>>> Thanks.
>>>>>
>>>>
>>>>
>>
Re: [SS]How to add a column with custom system time?
Posted by 张万新 <ke...@gmail.com>.
The spark version is 2.2.0
Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 下午12:32写道:
> Which version of spark?
>
> On Mon, Sep 11, 2017 at 8:27 PM, 张万新 <ke...@gmail.com> wrote:
>
>> Thanks for reply, but using this method I got an exception:
>>
>> "Exception in thread "main"
>> org.apache.spark.sql.streaming.StreamingQueryException: nondeterministic
>> expressions are only allowed in
>>
>> Project, Filter, Aggregate or Window"
>>
>> Can you give more advice?
>>
>> Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 上午4:48写道:
>>
>>> import org.apache.spark.sql.functions._
>>>
>>> df.withColumn("window", window(current_timestamp(), "15 minutes"))
>>>
>>> On Mon, Sep 11, 2017 at 3:03 AM, 张万新 <ke...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> In structured streaming how can I add a column to a dataset with
>>>> current system time aligned with 15 minutes?
>>>>
>>>> Thanks.
>>>>
>>>
>>>
>
Re: [SS]How to add a column with custom system time?
Posted by Michael Armbrust <mi...@databricks.com>.
Which version of spark?
On Mon, Sep 11, 2017 at 8:27 PM, 张万新 <ke...@gmail.com> wrote:
> Thanks for reply, but using this method I got an exception:
>
> "Exception in thread "main" org.apache.spark.sql.streaming.StreamingQueryException:
> nondeterministic expressions are only allowed in
>
> Project, Filter, Aggregate or Window"
>
> Can you give more advice?
>
> Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 上午4:48写道:
>
>> import org.apache.spark.sql.functions._
>>
>> df.withColumn("window", window(current_timestamp(), "15 minutes"))
>>
>> On Mon, Sep 11, 2017 at 3:03 AM, 张万新 <ke...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> In structured streaming how can I add a column to a dataset with current
>>> system time aligned with 15 minutes?
>>>
>>> Thanks.
>>>
>>
>>
Re: [SS]How to add a column with custom system time?
Posted by 张万新 <ke...@gmail.com>.
Thanks for reply, but using this method I got an exception:
"Exception in thread "main"
org.apache.spark.sql.streaming.StreamingQueryException: nondeterministic
expressions are only allowed in
Project, Filter, Aggregate or Window"
Can you give more advice?
Michael Armbrust <mi...@databricks.com>于2017年9月12日周二 上午4:48写道:
> import org.apache.spark.sql.functions._
>
> df.withColumn("window", window(current_timestamp(), "15 minutes"))
>
> On Mon, Sep 11, 2017 at 3:03 AM, 张万新 <ke...@gmail.com> wrote:
>
>> Hi,
>>
>> In structured streaming how can I add a column to a dataset with current
>> system time aligned with 15 minutes?
>>
>> Thanks.
>>
>
>
Re: [SS]How to add a column with custom system time?
Posted by Michael Armbrust <mi...@databricks.com>.
import org.apache.spark.sql.functions._
df.withColumn("window", window(current_timestamp(), "15 minutes"))
On Mon, Sep 11, 2017 at 3:03 AM, 张万新 <ke...@gmail.com> wrote:
> Hi,
>
> In structured streaming how can I add a column to a dataset with current
> system time aligned with 15 minutes?
>
> Thanks.
>