You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Debasish Ghosh <gh...@gmail.com> on 2016/09/27 13:08:02 UTC
Incremental model update
Hello -
I have a question on how to handle incremental model updation in Spark ..
We have a time series where we predict the future conditioned on the past.
We can train a model offline based on historical data and then use that
model during prediction.
But say, if the underlying process is non-stationary, the probability
distribution changes with time. In such cases we need to update the model
so as to reflect the current change in distribution.
We have 2 options -
a. retrain periodically
b. update the model incrementally
a. is expensive.
What about b. ? I think in Spark we have StreamingKMeans that takes care of
this incremental model update for this classifier. Is this true ?
But what about other classifiers that don't have these Streaming
counterparts ? How do we handle such incremental model changes with those
classifiers where the underlying distribution changes ?
regards.
--
Debasish Ghosh
http://manning.com/ghosh2
http://manning.com/ghosh
Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg