You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by M Singh <ma...@yahoo.com.INVALID> on 2017/12/30 22:40:08 UTC

Apache Spark - Using withWatermark for DataSets

Hi:
I am working with DataSets so that I can use mapGroupsWithState for business logic and then use dropDuplicates over a set of fields.  I would like to use the withWatermark so that I can restrict the how much state is stored.  
From the API it looks like withWatermark takes a string - timestamp column name as argument.  Is it possible to use it with DataSets ?  If not, is there any alternative like withWatermark available for working with DataSets ?
Thanks
Mans