Sharing Survey Results

["Edward J. Yoon" <ed...@apache.org>]

Recently I've discussed little with co-workers, Greg (Pregel), and
Adam coates, and Here's few survey results:

In general, a large amount of memory is required by fully-connected
layers which are used in a lot of different architectures. In
Convolutional Neural Network, convolutional layers have a
computational bottleneck and fully connected layers have a memory
bottleneck (with matrix multiplication approach).

In multi-GPUs architecture, there're some communication overheads such
as the bottleneck of switching to the next mini-batch (copying next
set ofimages and parameters). So, I think neuron-based iterative
computing model on CPUs is very fit for model parallelism. As you
already know, the forward and backward passes are the essential
computations of a Neural Net. So, only few vertices of single layer of
Neural Net will be activated in a single superstep, if we follow the
Pregel style model. This is quite inefficient. So, instead of doing
like this, we sends training instance continuously at every superstep,
and then handles the information (forward messages of current training
instance) and error (backward messages of previous training instance)
at once. This approach is similar to scheme (b) of "One weird trick
for parallelizing convolutional neural network, Google 2014".

In conclusion, we're doing good.

-- 
Best Regards, Edward J. Yoon

Re: Sharing Survey Results

["Edward J. Yoon" <ed...@apache.org>]

P.S., http://www.cs.cornell.edu/~wenleix/paper/blockgrace_vldb2013.pdf

On Wed, Nov 25, 2015 at 4:52 PM, Edward J. Yoon <ed...@apache.org> wrote:
> Recently I've discussed little with co-workers, Greg (Pregel), and
> Adam coates, and Here's few survey results:
>
> In general, a large amount of memory is required by fully-connected
> layers which are used in a lot of different architectures. In
> Convolutional Neural Network, convolutional layers have a
> computational bottleneck and fully connected layers have a memory
> bottleneck (with matrix multiplication approach).
>
> In multi-GPUs architecture, there're some communication overheads such
> as the bottleneck of switching to the next mini-batch (copying next
> set ofimages and parameters). So, I think neuron-based iterative
> computing model on CPUs is very fit for model parallelism. As you
> already know, the forward and backward passes are the essential
> computations of a Neural Net. So, only few vertices of single layer of
> Neural Net will be activated in a single superstep, if we follow the
> Pregel style model. This is quite inefficient. So, instead of doing
> like this, we sends training instance continuously at every superstep,
> and then handles the information (forward messages of current training
> instance) and error (backward messages of previous training instance)
> at once. This approach is similar to scheme (b) of "One weird trick
> for parallelizing convolutional neural network, Google 2014".
>
> In conclusion, we're doing good.
>
> --
> Best Regards, Edward J. Yoon



-- 
Best Regards, Edward J. Yoon