Train a model without stopping

[Saarthak Chandra <ch...@gmail.com>]

Hi,

Is there a way we can train a model without having to stop serving.
I mean, if I have an app deployed, can I add/post new data to the event
server and train the same app without stopping it?


Thanks!
-- 
Saarthak Chandra,
Masters in Computer Science,
Cornell University.

Re: Train a model without stopping

[Pat Ferrel <pa...@occamsmachete.com>]

The UR does this automatically. Once deployed you never have to deploy a second time. When a new `pio train` happens the new model is hot-swapped to replace the old, which is then erased, so there is no re-deploy and no downtime.

Yes, it uses Elasticsearch aliases but most other Templates do not use Elasticsearch for their model storage. However I believe that some could employ the same hot-swap method to re-deploy, they just weren’t written that way. You’d have to say which Template you are using.


On Sep 6, 2017, at 12:19 AM, Paul-Armand Verhaegen <pa...@gmail.com> wrote:

I believe there are 2 main methods:

1. stop serving a couple of seconds while deploying the newly trained model, this is supported from pio as is.
2. make a more flexible solution that can route traffic differently or cache results. We have a reverse proxy (openresty / nginx + lua) in front, so that we can do both if business requires it. 

When working with UR: Another solution would be to utilise ES aliases

I'm pretty sure other people have thought of other solutions, but it mostly depends on the exact use case.
I hope that helps.

Paul

> On 6 Sep 2017, at 03:34, Saarthak Chandra <ch...@gmail.com> wrote:
> 
> Hi,
> 
> Is there a way we can train a model without having to stop serving.
> I mean, if I have an app deployed, can I add/post new data to the event server and train the same app without stopping it?
> 
> 
> Thanks!
> -- 
> Saarthak Chandra,
> Masters in Computer Science,
> Cornell University.

-- 
You received this message because you are subscribed to the Google Groups "actionml-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionml-user+unsubscribe@googlegroups.com.
To post to this group, send email to actionml-user@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/0DECF939-9B44-40DD-A862-4E3C0AB9B6A1%40gmail.com <https://groups.google.com/d/msgid/actionml-user/0DECF939-9B44-40DD-A862-4E3C0AB9B6A1%40gmail.com>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.

Re: Train a model without stopping

[Paul-Armand Verhaegen <pa...@gmail.com>]

I believe there are 2 main methods:

1. stop serving a couple of seconds while deploying the newly trained model, this is supported from pio as is.
2. make a more flexible solution that can route traffic differently or cache results. We have a reverse proxy (openresty / nginx + lua) in front, so that we can do both if business requires it. 

When working with UR: Another solution would be to utilise ES aliases

I'm pretty sure other people have thought of other solutions, but it mostly depends on the exact use case.
I hope that helps.

Paul

> On 6 Sep 2017, at 03:34, Saarthak Chandra <ch...@gmail.com> wrote:
> 
> Hi,
> 
> Is there a way we can train a model without having to stop serving.
> I mean, if I have an app deployed, can I add/post new data to the event server and train the same app without stopping it?
> 
> 
> Thanks!
> -- 
> Saarthak Chandra,
> Masters in Computer Science,
> Cornell University.