Spark 1.6 ignoreNulls in first/last aggregate functions

[emlyn <em...@swiftkey.com>]

As I understand it, Spark 1.6 changes the behaviour of the first and last
aggregate functions to  take nulls into account
<https://issues.apache.org/jira/browse/SPARK-9740>   (where they were
ignored in 1.5). From SQL you can use "IGNORE NULLS" to get the old
behaviour back. How do I ignore nulls from the  Java API
<https://spark.apache.org/docs/1.6.0/api/java/org/apache/spark/sql/functions.html#first(org.apache.spark.sql.Column)> 
? I can't see any way to pass that option, so I suspect I may need to write
a user defined aggregate function, but I'd prefer not to if possible.

Thanks.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-6-ignoreNulls-in-first-last-aggregate-functions-tp26031.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Spark 1.6 ignoreNulls in first/last aggregate functions

[emlyn <em...@swiftkey.com>]

Turns out I can't use a user defined aggregate function, as they are not
supported in Window operations. There surely must be some way to do a
last_value with ignoreNulls enabled in Spark 1.6? Any ideas for workarounds?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-6-ignoreNulls-in-first-last-aggregate-functions-tp26031p26033.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org