Re: Incremental model training in real time event processing

[Pat Ferrel <pa...@occamsmachete.com>]

No, lambda has nothing to do with how much data is used in training. Lambda just say that there is a batch/background process that account for changes in data in non-real-time. It is theoretically possible but not supported yet. However you may be training too often...

The Universal Recommender uses the entire accumulated dataset to train but uses real-time user history data to make queries for user-based recommendations. The training rule-of-thumb is to do it as often as your catalog of items changes. By training daily you are saying that you have new items every day? 

Also there is no reason to re-deploy. If the PredictionServer is deployed the model will be hot-swapped into it at the end of every successful `pio train`.


On Nov 29, 2016, at 1:21 AM, Saurav Sarkar <sa...@gmail.com> wrote:

Hi All,

We are using universal recommender for our use case.

We have both batch processing and event processing scenarios.

In this context i have the following question.

1. We have a batch processing setup, import the events in batch every day, train and deploy.

2. We also send real time events, train the model manually and deploy.

Can the second step be done by incremental training on the already existing model.
As per the lambda architecture setup it should be possible,

Best Regards,
Saurav