Suggestion on tracking live engines and modifying their params

[Hasan Can Saral <ha...@gmail.com>]

Hi!

This question is more architectural and maybe outside of scope of PredictionIO, but I would appreciate any help on how to keep track of the deployed engines. I am considering PredictionIO for a use case where more than one engine would be deployed in a very distributed environment, and I would like to be able to list them and change their parameters.

What I have thought is that:

1- Storing dataSourceParams, preparatorParams, algorithmParams and ServingParams in EngineManifest instead of EngineInstance, reading it from meta store in pio train (Might there be a reason as to why I should not do this?)
2- Reading from meta store in DataSource, Preparator even in predict
3- Extending ServerConfig in CreateServer with hostname/IP in meta data store (I should have the IP of the machine running at this time, am I right?)
4- If the engine exits gracefully (with undeploy method in CreateServer), remove ServerConfig entry from meta data store,
5- If it exits for some other reason (crash, Ctrl+C. etc.), I am planning to write a command, e.g. pio cleanup that can be scheduled with cron to run say every 5 minutes, loop through hostnames/IPs stored in meta store and ssh/ping them on the deployment port with hostname (I can ping hostnames because the query server is containerized and hostname -i simply prints the IP in my bridge network) to see if it runs, if not delete them from meta data store.

I would appreciate a sanity check, and maybe some help on these:

- When does an engine get a new EngineManifest instance, i.e. an engineId stored in meta store?
- I will already be handling around 10m queries a day, and every query already hits db and I don’t want (5) above to generate any extra network traffic. Also, cron means delay. Would consider any alternatives to (5) - Do you think a ShutdownHook would help? Or a executing pio deploy in a try catch block in shell <http://stackoverflow.com/questions/15656492/linux-shell-try-catch-finally> so that I can call pio cleanup in catch?

Thanks in advance, hope someone comes with a suggestion that is a lot easier and embarrasses me :)
Hasan