[jira] [Created] (MADLIB-1268) Spike - CNN convergence, data parallel with merge

["Frank McQuillan (JIRA)" <ji...@apache.org>]

Frank McQuillan created MADLIB-1268:
---------------------------------------

             Summary: Spike - CNN convergence, data parallel with merge
                 Key: MADLIB-1268
                 URL: https://issues.apache.org/jira/browse/MADLIB-1268
             Project: Apache MADlib
          Issue Type: New Feature
            Reporter: Frank McQuillan
             Fix For: v2.0



Story

`As a MADlib developer`
I want investigate convergence behaviour when running a single distributed CNN model across the Greenplum cluster using Keras with a Tensorflow backend
`so that`
I can see if it converges in a predictable and expected way.


Details

* By "single distributed CNN model" I mean data parallel with merge (not model parallel).
* Does not need to use an aggregate for this spike, if that is too inconvenient, since performance is not the focus of this story.  It's about convergence.
* In defining the merge function, review [2] for single-server, multi-GPU merge function.  Perhaps we can do the exact same thing for multi-server?
* For dataset, consider using Pavan's CNN code and data set [3].  Another option is MNIST and/or CIFAR-10.
* See page 11 of [8] re synchronous data parallel in TF


Acceptance

1) Plot characteristic curves of loss vs. iteration number.  Compare with MADlib merge (this story) vs. without merge.
2) Define what the merge function is for CNN.  Is it the same as [3] or something else? Does it operate on weights only or does it need gradients?
3) What does the architecture look like?  Draw a diagram showing sync/merge step for distributed model training.
4) What tests do we need to do to convince ourselves that the architecture is valid?  
5) Do we need to write different merge functions, or have a different approach, for each different neural net type algorithm?  Or is there a general approach that we can use that will apply to this class of algorithms?
6) Anything to learn from pg-strom [2] ?
7) Anything to learn from H20 [3] ?  I don’t think they are doing distributed training, rather grid search and such.


References

[2] Check for “# Merge outputs under expected scope” section in the python program
 https://github.com/keras-team/keras/blob/bf1378f39d02b7d0b53ece5458f9275ac8208046/keras/utils/multi_gpu_utils.py

[3] Deep learning example for image segmentation
https://drive.google.com/drive/folders/1mgZPGuDP1JI1TUVaRndexDZTlSABLci9?usp=sharing

[4] Deep Learning & Greenplum Discussion (Pavan/Pivotal DE)
https://drive.google.com/file/d/1U808PAwMetNL38mrboPHdn8RKOrpQyBz/view?usp=sharing

[5] Single Machine Data Parallel multi GPU Training 
https://www.pyimagesearch.com/2017/10/30/how-to-multi-gpu-training-with-keras-python-and-deep-learning/

[6] Why are GPUs necessary for training Deep Learning models?
https://www.analyticsvidhya.com/blog/2017/05/gpus-necessary-for-deep-learning/

[7] Deep Learning vs Classical Machine Learning
https://towardsdatascience.com/deep-learning-vs-classical-machine-learning-9a42c6d48aa

[8] TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45166.pdf



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)