[GitHub] [systemds] Baunsgaard commented on pull request #1154: [SYSTEMDS-2550] Parameter Server Validation and FedParamServ Statistics

[GitBox <gi...@apache.org>]

Baunsgaard commented on pull request #1154:
URL: https://github.com/apache/systemds/pull/1154#issuecomment-760756975


    
   > There is now a second createPS function in the ParamservBuiltinCPInstruction. If all the additional arguments are specified the parameter server is able to validate after each epoch. It will do so if LOG.info is enabled.
   > This feature is implemented for the federated parameter server ONLY, but is easily implemented for the other cases too.
   
   I can understand why this is done with the logging. But i think this is a slight misuse of the logging in the system, since ideally we (or at least me) want no difference in execution (or as close to) when we set different logging levels. This violates it since the validation in the parameter server can take many seconds of execution time.
   
   I would suggest adding a parameter some other way to enable / disable the validation. While it is still possible that we output the validation scores through LOG.info when that other "unrelated to logging level" setting is set.
   
   But maybe I'm the only one with this opinion? @mboehm7 ?
   
   
   
   furthermore designwise, you might want to consider doing this validation in parallel with the distribution of model parameters. since we know that the controller is doing nothing while the workers are training, you could do the validation on a separate thread at no cost to execution time.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org