how to do incremental model updates using spark streaming and mllib

[vishnu <jo...@gmail.com>]

Hi,

Say I have created a clustering model using KMeans for 100million
transactions at time t1. I am using streaming and say for every 1 hour i
need to update my existing model. How do I do it. Should it include every
time all the data or can it be incrementally updated.

If I can do an incrementally updating , how do i do it.

Thanks,
Vishnu



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-do-incremental-model-updates-using-spark-streaming-and-mllib-tp20862.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: how to do incremental model updates using spark streaming and mllib

[Reza Zadeh <re...@databricks.com>]

As of Spark 1.2 you can do Streaming k-means, see examples here:
http://spark.apache.org/docs/latest/mllib-clustering.html#examples-1
Best,
Reza

On Fri, Dec 26, 2014 at 1:36 AM, vishnu <jo...@gmail.com> wrote:

> Hi,
>
> Say I have created a clustering model using KMeans for 100million
> transactions at time t1. I am using streaming and say for every 1 hour i
> need to update my existing model. How do I do it. Should it include every
> time all the data or can it be incrementally updated.
>
> If I can do an incrementally updating , how do i do it.
>
> Thanks,
> Vishnu
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-do-incremental-model-updates-using-spark-streaming-and-mllib-tp20862.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>