You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@singa.apache.org by GitBox <gi...@apache.org> on 2020/04/04 10:16:55 UTC
[GitHub] [singa] chrishkchris opened a new pull request #651: [WIP] Simply
example APIs
chrishkchris opened a new pull request #651: [WIP] Simply example APIs
URL: https://github.com/apache/singa/pull/651
The changes recommended in
https://github.com/apache/singa-doc/pull/14
to simply the APIs of the examples
I have reduced the number of arguments in the train_mnist_cnn
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #651: [WIP ]Simply example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609045570
I suggest to merge the examples under `examples/autograd` into the following scheme
```
autograd
train.py
train_mpi.py
train_multiprocess.py
data # define the data loading and preprocessing
cifar10.py
mnist.py
model # define the model
cnn.py
resnet.py
xception.py
```
```python
# train.py
def run(max_epoch, rank, num_workers, model, data, sgd):
if model == 'resnet':
from model import resnet
model = resnet.create_model()
elif model == 'cnn':
model = cnn.create_model()
....
if data == 'cifar10':
from data import cifar10:
train_x, train_y, val_x, val_y = cifar10.load()
elif data == 'mnist':
....
train_x, train_y, val_x, val_y = partition(rank, num_workers, train_x, train_y, val_x, val_y)
# bp and sgd
if __name__ == '__main__':
# use argparse to get command config: max_epoch, model, data, etc. for single gpu training
sgd = # create sgd
run(0, 1, ..., sgd)
# train_mpi.py
if __name__ == '__main__':
# use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
sgd = # create sgd
dist_sgd = DistOpt(sgd...)
run(dist_sgd.rank, dist_sgd.world_size, ... dist_sgd)
# train_multiprocess.py
def run(rank, num_gpu, ...):
sgd = ...
dist_sgd = DistOpt(sgd)
train.run(rank, num_gpu, ... dist_sgd)
if __name__ == '__main__':
# use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
nccl_id = ...
process = []
for gpu_num in range(0, gpu_per_node):
process.append(multiprocessing.Process(target=run, args=(gpu_num, ... nccl_id)))
for p in process:
p.start()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609538714
can we run (cnn|resnet|xceptionnet) + (cifar10|mnist) ?
I think we can also add the mlp model?
then we can remove resnet_module.py, mlp_module.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609049842
> I suggest to merge the examples under `examples/autograd` into the follow structure.
>
> ```
> autograd
> train.py
> train_mpi.py
> train_multiprocess.py
> data # define the data loading and preprocessing
> cifar10.py
> mnist.py
> model # define the model
> cnn.py
> resnet.py
> xception.py
> ```
>
> The pseudo code of each file:
>
> ```python
> # train.py
> def run(max_epoch, worker_id, num_workers, model, data, sgd):
> if model == 'resnet':
> from model import resnet
> model = resnet.create_model()
> elif model == 'cnn':
> model = cnn.create_model()
> ....
> if data == 'cifar10':
> from data import cifar10:
> train_x, train_y, val_x, val_y = cifar10.load()
> elif data == 'mnist':
> ....
>
> train_x, train_y, val_x, val_y = partition(worker_id, num_workers, train_x, train_y, val_x, val_y)
>
> # bp and sgd
>
> if __name__ == '__main__':
> # use argparse to get command config: max_epoch, model, data, etc. for single gpu training
> sgd = # create sgd
> run(0, 1, ..., sgd)
>
> # train_mpi.py
> if __name__ == '__main__':
> # use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
> sgd = # create sgd
> dist_sgd = DistOpt(sgd...)
> run(dist_sgd.rank, dist_sgd.world_size, ... dist_sgd)
>
> # train_multiprocess.py
> def run(rank, num_gpu, ...):
> sgd = ...
> dist_sgd = DistOpt(sgd)
> train.run(rank, num_gpu, ... dist_sgd)
>
> if __name__ == '__main__':
> # use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
> nccl_id = ...
> process = []
> for worker_id in range(0, gpu_per_node):
> process.append(multiprocessing.Process(target=run, args=(worker_id, ... nccl_id)))
>
> for p in process:
> p.start()
> ```
thanks! I will work on it. Since seems a lot of work, to speed up may need Rulin to help
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris edited a comment on issue #651: [WIP ]Simply
example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris edited a comment on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609049842
> I suggest to merge the examples under `examples/autograd` into the follow structure.
>
> ```
> autograd
> train.py
> train_mpi.py
> train_multiprocess.py
> data # define the data loading and preprocessing
> cifar10.py
> mnist.py
> model # define the model
> cnn.py
> resnet.py
> xception.py
> ```
>
> The pseudo code of each file:
>
> ```python
> # train.py
> def run(max_epoch, worker_id, num_workers, model, data, sgd):
> if model == 'resnet':
> from model import resnet
> model = resnet.create_model()
> elif model == 'cnn':
> model = cnn.create_model()
> ....
> if data == 'cifar10':
> from data import cifar10:
> train_x, train_y, val_x, val_y = cifar10.load()
> elif data == 'mnist':
> ....
>
> train_x, train_y, val_x, val_y = partition(worker_id, num_workers, train_x, train_y, val_x, val_y)
>
> # bp and sgd
>
> if __name__ == '__main__':
> # use argparse to get command config: max_epoch, model, data, etc. for single gpu training
> sgd = # create sgd
> run(0, 1, ..., sgd)
>
> # train_mpi.py
> if __name__ == '__main__':
> # use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
> sgd = # create sgd
> dist_sgd = DistOpt(sgd...)
> run(dist_sgd.rank, dist_sgd.world_size, ... dist_sgd)
>
> # train_multiprocess.py
> def run(rank, num_gpu, ...):
> sgd = ...
> dist_sgd = DistOpt(sgd)
> train.run(rank, num_gpu, ... dist_sgd)
>
> if __name__ == '__main__':
> # use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
> nccl_id = ...
> process = []
> for worker_id in range(0, gpu_per_node):
> process.append(multiprocessing.Process(target=run, args=(worker_id, ... nccl_id)))
>
> for p in process:
> p.start()
> ```
thanks! I will work on it. Since seems a lot of work, to speed up may need Rulin to help
Seems the graph module cnn_module.py and resnet_module.py is not included, not sure how to include the graph module
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609419282
This pull request **introduces 5 alerts** and **fixes 4** when merging 66c7680e9db1a2570b43c61fff61b92677736b0b into 5c4d3db57630098d04d214b6e7ea2b1f88fc84fd - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-fdf94245a0cadcb72cf03835be4a329f61469ade)
**new alerts:**
* 4 for Unused import
* 1 for Unused local variable
**fixed alerts:**
* 4 for 'import \*' may pollute namespace
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #651: [WIP ]Simply example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609236505
we can use resnet_module.py to replace resnet.py?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: Simply example APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609041662
This pull request **introduces 3 alerts** when merging 68dcc49bb7321ff1d6bab85e389048673cb9205e into 182007508a904084bf551bfef9dbd8aa55a46d3b - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-c13f9d186eb24c4487edda45ff042309518f1f67)
**new alerts:**
* 2 for Unreachable code
* 1 for 'import \*' may pollute namespace
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609204054
This pull request **introduces 9 alerts** when merging 545378df098e8cd56e3d3016b95b9077ed646047 into a91de70a7a83c11c9aaf6947f944d6c088afbe3b - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-2550dbb51a0baa849388e4552ec63ace234b9368)
**new alerts:**
* 5 for Unused import
* 2 for Unreachable code
* 1 for Missing call to \_\_init\_\_ during object initialization
* 1 for 'import \*' may pollute namespace
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #651: Add new
example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403804778
##########
File path: examples/autograd/train.py
##########
@@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from singa import singa_wrap as singa
+from singa import opt
+from singa import device
+from singa import tensor
+import numpy as np
+import time
+import argparse
+from PIL import Image
+
+
+# Data Augmentation
+def augmentation(x, batch_size):
+ xpad = np.pad(x, [[0, 0], [0, 0], [4, 4], [4, 4]], 'symmetric')
+ for data_num in range(0, batch_size):
+ offset = np.random.randint(8, size=2)
+ x[data_num, :, :, :] = xpad[data_num, :,
+ offset[0]:offset[0] + x.shape[2],
+ offset[1]:offset[1] + x.shape[2]]
+ if_flip = np.random.randint(2)
+ if (if_flip):
+ x[data_num, :, :, :] = x[data_num, :, :, ::-1]
+ return x
+
+
+# Calculate Accuracy
+def accuracy(pred, target):
+ # y is binary array for ground truth
+ y = np.argmax(pred, axis=1)
+ # t is integer for ground truth, the dim of t is (batch, 1)
+ t = np.max(target, axis=1)
+ a = y == t
+ correct = np.array(a, "int").sum()
+ # print(correct)
+ return correct
+
+
+# Data partition according to the rank
+def partition(worker_id, num_workers, train_x, train_y, val_x, val_y):
+ # Partition training data
+ data_per_rank = train_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ train_x = train_x[idx_start:idx_end]
+ train_y = train_y[idx_start:idx_end]
+ # Partition evaluation data
+ data_per_rank = val_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ val_x = val_x[idx_start:idx_end]
+ val_y = val_y[idx_start:idx_end]
+ return train_x, train_y, val_x, val_y
+
+
+# Function to all reduce NUMPY Accuracy and Loss from Multiple Devices
+def reduce_variable(variable, dist_opt, reducer):
+ reducer.copy_from_numpy(variable)
+ dist_opt.all_reduce(reducer.data)
+ dist_opt.wait()
+ output = tensor.to_numpy(reducer)
+ return output
+
+
+def resize_dataset(x, image_size):
+ num_data = x.shape[0]
+ dim = x.shape[1]
+ X = np.zeros(shape=(num_data, dim, image_size, image_size), dtype=np.float32)
+ for n in range(0, num_data):
+ for d in range(0, dim):
+ X[n, d, :, :] = np.array(Image.fromarray(x[n, d, :, :]).resize(
+ (image_size, image_size), Image.BILINEAR),
+ dtype=np.float32)
+ return X
+
+
+def run(worker_id, num_workers, device_id, max_epoch, batch_size, model, data,
+ sgd):
+ dev = device.create_cuda_gpu_on(device_id)
+ dev.SetRandSeed(0)
+ np.random.seed(0)
+
+ if model == 'resnet':
+ from model import resnet
+ model = resnet.resnet50()
+ elif model == 'xceptionnet':
+ from model import xceptionnet
+ model = xceptionnet.create_model()
+ elif model == 'cnn':
+ from model import cnn
+ model = cnn.create_model()
+
+ if data == 'cifar10':
+ from data import cifar10
+ train_x, train_y, val_x, val_y = cifar10.load()
+ elif data == 'mnist':
+ from data import mnist
+ train_x, train_y, val_x, val_y = mnist.load()
+
+ if hasattr(sgd, "communicator"):
+ DIST = True
+ else:
+ DIST = False
+
+ if DIST:
+ train_x, train_y, val_x, val_y = partition(worker_id, num_workers,
+ train_x, train_y, val_x,
+ val_y)
+ '''
+ # check dataset shape correctness
+ if worker_id == 0:
+ print("Check the shape of dataset:")
+ print(train_x.shape)
+ print(train_y.shape)
+ '''
+
+ num_channels = train_x.shape[1]
+ image_size = train_x.shape[2]
+
+ tx = tensor.Tensor(
+ (batch_size, num_channels, model.input_size, model.input_size), dev,
+ tensor.float32)
+ ty = tensor.Tensor((batch_size,), dev, tensor.int32)
+ num_train_batch = train_x.shape[0] // batch_size
+ num_val_batch = val_x.shape[0] // batch_size
+ idx = np.arange(train_x.shape[0], dtype=np.int32)
+
+ # attached model to graph
+ graph = True
Review comment:
the graph variable is used once.
therefore, there is no need to define it. just pass True to `model.graph`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles edited a comment on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
nudles edited a comment on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609045570
I suggest to merge the examples under `examples/autograd` into the follow structure.
```
autograd
train.py
train_mpi.py
train_multiprocess.py
data # define the data loading and preprocessing
cifar10.py
mnist.py
model # define the model
cnn.py
resnet.py
xception.py
```
The pseudo code of each file:
```python
# train.py
def run(max_epoch, worker_id, num_workers, model, data, sgd):
if model == 'resnet':
from model import resnet
model = resnet.create_model()
elif model == 'cnn':
model = cnn.create_model()
....
if data == 'cifar10':
from data import cifar10:
train_x, train_y, val_x, val_y = cifar10.load()
elif data == 'mnist':
....
train_x, train_y, val_x, val_y = partition(worker_id, num_workers, train_x, train_y, val_x, val_y)
# bp and sgd
if __name__ == '__main__':
# use argparse to get command config: max_epoch, model, data, etc. for single gpu training
sgd = # create sgd
run(0, 1, ..., sgd)
# train_mpi.py
if __name__ == '__main__':
# use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
sgd = # create sgd
dist_sgd = DistOpt(sgd...)
run(dist_sgd.rank, dist_sgd.world_size, ... dist_sgd)
# train_multiprocess.py
def run(rank, num_gpu, ...):
sgd = ...
dist_sgd = DistOpt(sgd)
train.run(rank, num_gpu, ... dist_sgd)
if __name__ == '__main__':
# use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
nccl_id = ...
process = []
for worker_id in range(0, gpu_per_node):
process.append(multiprocessing.Process(target=run, args=(worker_id, ... nccl_id)))
for p in process:
p.start()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: Simply example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609011705
I have tested, ready for merge
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #651: [WIP ]Simply example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609263898
since graph version is more efficient, I think it is fine to run the examples using the graph.
we can still keep the current version in another folder
* imperative (for the current)
* declarative (for graph based)
we can do the graph based first and if no time, then just ignore the imperative version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609614006
> > xceptionnet error
> > 
>
> what's the size of the input to the model?
Input size of the network is 299. The first epoch can train so there may be other problem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609044475
This pull request **introduces 3 alerts** when merging 9e8badf7a394170c0d6aa8f2745ac3d482daf725 into 182007508a904084bf551bfef9dbd8aa55a46d3b - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-0297e363f9cb002f5fa4022bdcbd45ecaab15a26)
**new alerts:**
* 2 for Unreachable code
* 1 for 'import \*' may pollute namespace
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609434662
This pull request **fixes 4 alerts** when merging 1c011b746338abfb5ad209dda6ac56c3145821b4 into 5c4d3db57630098d04d214b6e7ea2b1f88fc84fd - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-1183f232d26e6ec82cd17d4b5fc837be4752ee7e)
**fixed alerts:**
* 4 for 'import \*' may pollute namespace
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris edited a comment on issue #651: Add new
example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris edited a comment on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609614006
> > xceptionnet error
> > 
>
> what's the size of the input to the model?
Input size of the network is 299. The first epoch can train so there may be other problem
From here https://keras.io/applications/#xception Xceptionnet input size is 299 so I use this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609229980
> 1. where is resnet_module.py? can we use the Module API (to replace Layer model) added by rulin for these examples and add a switch to turn on/off operation buffering at the command line?
> 2. i can handle the documentation.
The graph module is here.
https://github.com/apache/singa/blob/dev/examples/autograd/resnet_module.py
https://github.com/apache/singa/blob/dev/examples/autograd/cnn_module.py
https://github.com/apache/singa/blob/dev/examples/autograd/mlp_module.py
The on/off switch of buffering function is embedded in python/singa/module.py
https://github.com/apache/singa/blob/dev/python/singa/module.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: [WIP] Simply example
APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: [WIP] Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609007921
After testing, I will remove WIP
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] XJDKC edited a comment on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
XJDKC edited a comment on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609659740
> python3 train.py xceptionnet ci
I think it might be that the memory addresses of some tensors were changed during the evaluation phase, which caused the calculation graph to not be executed correctly. Because in one epoch we run the graph multiple times correctly, which means the graph execution is correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609049091
This pull request **introduces 3 alerts** when merging 40cbb534e51bd54a13eb5bd3be909ade8041db81 into 182007508a904084bf551bfef9dbd8aa55a46d3b - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-02d6e1c289c3df1f98b9d959ce7e493549ea04ef)
**new alerts:**
* 2 for Unreachable code
* 1 for 'import \*' may pollute namespace
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #651: [WIP ]Simply example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609215794
1. where is resnet_module.py? can we use the Module API (to replace Layer model) added by rulin for these examples and add a switch to turn on/off operation buffering at the command line?
2. i can handle the documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609383089
This pull request **introduces 5 alerts** and **fixes 4** when merging 6dcbac97a57e767bad2c87e08d661975aa64e438 into 5c4d3db57630098d04d214b6e7ea2b1f88fc84fd - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-f5733f2e8de5ee4480473d283905182e4b177cae)
**new alerts:**
* 5 for Unused import
**fixed alerts:**
* 4 for 'import \*' may pollute namespace
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] XJDKC commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
XJDKC commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609740189
> > can the original model run (not using the Module API)?
>
> Yes, I have turned the graph off (GRAPH=FALSE) and running, seems is okay I have just finished 2 rpochs, I will wait for more result. I think the different of resnet and xceptionnet is that it has multiple branch
>
> ```
> (singa) dcsysh@panda7:~/singa/examples/autograd$ python3 train.py xceptionnet ci
>
> Starting Epoch 0:
> Training loss = 11198.645508, training accuracy = 0.214420
> Evaluation accuracy = 0.309000, Elapsed Time = 606.547117s
> Starting Epoch 1:
> Training loss = 6354.611328, training accuracy = 0.381020
> Evaluation accuracy = 0.457300, Elapsed Time = 612.817129s
> Starting Epoch 2:
>
> ```
I know the problem. It's because some tensors were deallocated during the evaluation phase. I will fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #651:
Simply example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #651: Simply example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403477435
##########
File path: examples/autograd/doc_dist_train.py
##########
@@ -0,0 +1,254 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
Review comment:
not exactly, mnist_cnn.py mix multiple functions in the same file "single gpu, dist_train, asychronous training", also with data augmentation. Then doc_dist_train.py, only provide the dist train function and I removed the DIST variable for documentation use only, otherwise will be confusing if mixing all the thing into one example
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: Simply example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609019283
Just now I have added two examples specially for the dist-train.md documentation, the API for the train fuction becomes the same as that used in the docs
MPI train: doc_dist_train.py
Multiprocess train: doc_dist_multiprocess.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #651: Simply
example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #651: Simply example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403476030
##########
File path: examples/autograd/cifar10_multiprocess.py
##########
@@ -17,28 +17,16 @@
# under the License.
#
-from singa import opt
from resnet_cifar10 import *
import multiprocessing
-
-def data_partition(dataset_x, dataset_y, rank_in_global, world_size):
- data_per_rank = dataset_x.shape[0] // world_size
- idx_start = rank_in_global * data_per_rank
- idx_end = (rank_in_global + 1) * data_per_rank
- return dataset_x[idx_start:idx_end], dataset_y[idx_start:idx_end]
-
-
if __name__ == '__main__':
# Generate a NCCL ID to be used for collective communication
nccl_id = singa.NcclIdHolder()
- sgd = opt.SGD(lr=0.005, momentum=0.9, weight_decay=1e-5)
-
- gpu_per_node = 4
- max_epoch = 100
- batch_size = 32
+ # number of GPUs to be used
+ gpu_per_node = 2
Review comment:
can we use [argparse](https://github.com/apache/singa/blob/master/examples/mnist/train.py#L128) to get the number of gpus from the command line?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] XJDKC commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
XJDKC commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609659740
> python3 train.py xceptionnet ci
I think it might be that the memory addresses of some tensors were changed during the evaluation phase, which caused the calculation graph to not be executed correctly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #651:
Simply example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #651: Simply example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403479208
##########
File path: examples/autograd/cifar10_multiprocess.py
##########
@@ -17,28 +17,16 @@
# under the License.
#
-from singa import opt
from resnet_cifar10 import *
import multiprocessing
-
-def data_partition(dataset_x, dataset_y, rank_in_global, world_size):
- data_per_rank = dataset_x.shape[0] // world_size
- idx_start = rank_in_global * data_per_rank
- idx_end = (rank_in_global + 1) * data_per_rank
- return dataset_x[idx_start:idx_end], dataset_y[idx_start:idx_end]
-
-
if __name__ == '__main__':
# Generate a NCCL ID to be used for collective communication
nccl_id = singa.NcclIdHolder()
- sgd = opt.SGD(lr=0.005, momentum=0.9, weight_decay=1e-5)
-
- gpu_per_node = 4
- max_epoch = 100
- batch_size = 32
+ # number of GPUs to be used
+ gpu_per_node = 2
Review comment:
I added command line argument
```
root@877a3759b148:~/dcsysh/singa/examples/autograd# python3 mnist_multiprocess.py 2
Starting Epoch 0:
Training loss = 1029.400879, training accuracy = 0.628155
Evaluation accuracy = 0.882712, Elapsed Time = 1.923681s
Starting Epoch 1:
Training loss = 367.118927, training accuracy = 0.874215
Evaluation accuracy = 0.939804, Elapsed Time = 1.986267s
Starting Epoch 2:
Training loss = 270.464111, training accuracy = 0.908504
Evaluation accuracy = 0.935497, Elapsed Time = 1.985472s
Starting Epoch 3:
Training loss = 215.307678, training accuracy = 0.927017
Evaluation accuracy = 0.958333, Elapsed Time = 1.894766s
Starting Epoch 4:
Training loss = 179.509125, training accuracy = 0.940371
Evaluation accuracy = 0.965244, Elapsed Time = 1.889032s
Starting Epoch 5:
Training loss = 157.717468, training accuracy = 0.947282
Evaluation accuracy = 0.962640, Elapsed Time = 1.892571s
Starting Epoch 6:
Training loss = 143.918320, training accuracy = 0.952224
Evaluation accuracy = 0.968950, Elapsed Time = 1.910212s
Starting Epoch 7:
Training loss = 134.035339, training accuracy = 0.954911
Evaluation accuracy = 0.957332, Elapsed Time = 1.977645s
Starting Epoch 8:
Training loss = 121.463905, training accuracy = 0.959352
Evaluation accuracy = 0.970252, Elapsed Time = 1.944209s
Starting Epoch 9:
Training loss = 112.466797, training accuracy = 0.962457
Evaluation accuracy = 0.974259, Elapsed Time = 1.883536s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles edited a comment on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
nudles edited a comment on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609045570
I suggest to merge the examples under `examples/autograd` into the follow structure.
```
autograd
train.py
train_mpi.py
train_multiprocess.py
data # define the data loading and preprocessing
cifar10.py
mnist.py
model # define the model
cnn.py
resnet.py
xception.py
```
The pseudo code of each file:
```python
# train.py
def run(max_epoch, rank, num_workers, model, data, sgd):
if model == 'resnet':
from model import resnet
model = resnet.create_model()
elif model == 'cnn':
model = cnn.create_model()
....
if data == 'cifar10':
from data import cifar10:
train_x, train_y, val_x, val_y = cifar10.load()
elif data == 'mnist':
....
train_x, train_y, val_x, val_y = partition(rank, num_workers, train_x, train_y, val_x, val_y)
# bp and sgd
if __name__ == '__main__':
# use argparse to get command config: max_epoch, model, data, etc. for single gpu training
sgd = # create sgd
run(0, 1, ..., sgd)
# train_mpi.py
if __name__ == '__main__':
# use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
sgd = # create sgd
dist_sgd = DistOpt(sgd...)
run(dist_sgd.rank, dist_sgd.world_size, ... dist_sgd)
# train_multiprocess.py
def run(rank, num_gpu, ...):
sgd = ...
dist_sgd = DistOpt(sgd)
train.run(rank, num_gpu, ... dist_sgd)
if __name__ == '__main__':
# use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
nccl_id = ...
process = []
for gpu_num in range(0, gpu_per_node):
process.append(multiprocessing.Process(target=run, args=(gpu_num, ... nccl_id)))
for p in process:
p.start()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #651: Add new
example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403870271
##########
File path: src/api/dist_communicator.i
##########
@@ -44,7 +44,7 @@ public:
int totalMPIRanksInGlobal;
int MPIRankInLocal;
Communicator(int limit);
- Communicator(int gpu_num, int gpu_per_node, const NcclIdHolder &holder, int limit);
+ Communicator(int gpu_num, int num_gpus, const NcclIdHolder &holder, int limit);
Review comment:
ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609800631
This pull request **introduces 2 alerts** when merging e3c51d3bccd996c7057afae1cc5f0a7ccfd2ee26 into 793dcace41eaf3569c8222e0f4ca667cb9ecf412 - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-ca57aa6707d9a5d6b4ca57e00d04b7e8a19c96c1)
**new alerts:**
* 2 for Unreachable code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609545036
>
>
> > can we run (cnn|resnet|xceptionnet) + (cifar10|mnist) ?
> > I think we can also add the mlp model?
> > then we can remove resnet_module.py, mlp_module.py
>
> There are 4 points, I guess the issue most difficult to solve is the no.1 below
>
> 1. xceptionnet has some problem, the running stopped after 1 epoch, maybe the model has some issues. And this model is very large I can only run batch size=16 (not 32)
what is the error? if you put it on colab, we can debug together.
>
> 2. The current mlp is 2d input, which is not CNN problem which has 4d input (batch, channel/RGB, height, width). Maybe need to change to 4d
when we load the data, pass an argument to specify the dimension (is_2d=True/False).
then reshape the data according to the argument inside the `load_data` function.
>
> 3. The CNN has only one input channel, in order to support cifar10, maybe I need to adjust the number of input channel of CNN as variable
>
change the CNN model to accept 4d input
> 4. mnist is not RGB, so I has only one colour channel, if I used it for resnet I can see if:
> (i) change the input channel of resnet to a variable so it support 1 channel
> (ii) replicate one channel of mnist dataset to three
reshape it into 4d inside the `load_data` function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609605379
This pull request **introduces 2 alerts** when merging c7c2afd127c3890e3963019be63f52a8f4f0f3c6 into ec5f8b223d052b8eadd512bf193166c0270b31c8 - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-352d7ed3e60ea2a18a1f5c3ad1b66ce13aeda4c1)
**new alerts:**
* 2 for Unreachable code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris edited a comment on issue #651: Add new
example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris edited a comment on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609758264
I added two args in train_mpi,py and train_multiprocess.py to support all the distributed training options, but train.py don't need the argument because it doesn't have DistOpt
Below are four examples for the use of arguments:
```
root@71ac539cda77:~/dcsysh/singa/examples/autograd# mpiexec -np 2 python3 train_mpi.py cnn mnist --op fp16 --lr 0.01
Starting Epoch 0:
Training loss = 625.136963, training accuracy = 0.776893
Evaluation accuracy = 0.939704, Elapsed Time = 1.619492s
Starting Epoch 1:
Training loss = 235.280151, training accuracy = 0.920339
Evaluation accuracy = 0.947115, Elapsed Time = 1.517764s
Starting Epoch 2:
Training loss = 171.311310, training accuracy = 0.942508
Evaluation accuracy = 0.970252, Elapsed Time = 1.557606s
Starting Epoch 3:
Training loss = 139.594086, training accuracy = 0.953342
Evaluation accuracy = 0.972356, Elapsed Time = 1.541372s
Starting Epoch 4:
Training loss = 120.380058, training accuracy = 0.959852
Evaluation accuracy = 0.971655, Elapsed Time = 1.560016s
Starting Epoch 5:
Training loss = 104.767105, training accuracy = 0.965345
Evaluation accuracy = 0.979467, Elapsed Time = 1.525147s
Starting Epoch 6:
Training loss = 98.995010, training accuracy = 0.966730
Evaluation accuracy = 0.976963, Elapsed Time = 1.528173s
Starting Epoch 7:
Training loss = 88.012024, training accuracy = 0.970202
Evaluation accuracy = 0.977865, Elapsed Time = 1.515204s
Starting Epoch 8:
Training loss = 86.540741, training accuracy = 0.970519
Evaluation accuracy = 0.975361, Elapsed Time = 1.604774s
Starting Epoch 9:
Training loss = 80.653885, training accuracy = 0.973024
Evaluation accuracy = 0.982071, Elapsed Time = 1.633480s
root@71ac539cda77:~/dcsysh/singa/examples/autograd# mpiexec -np 2 python3 train_mpi.py cnn mnist --op partialUpdate --lr 0.01
Starting Epoch 0:
Training loss = 623.692139, training accuracy = 0.777694
Evaluation accuracy = 0.939804, Elapsed Time = 1.678683s
Starting Epoch 1:
Training loss = 235.369232, training accuracy = 0.920690
Evaluation accuracy = 0.948017, Elapsed Time = 1.567066s
Starting Epoch 2:
Training loss = 171.335693, training accuracy = 0.942157
Evaluation accuracy = 0.971855, Elapsed Time = 1.595140s
Starting Epoch 3:
Training loss = 139.396088, training accuracy = 0.953592
Evaluation accuracy = 0.971755, Elapsed Time = 1.661490s
Starting Epoch 4:
Training loss = 120.173752, training accuracy = 0.960019
Evaluation accuracy = 0.970753, Elapsed Time = 1.571710s
Starting Epoch 5:
Training loss = 105.472672, training accuracy = 0.965061
Evaluation accuracy = 0.978065, Elapsed Time = 1.570710s
Starting Epoch 6:
Training loss = 99.055389, training accuracy = 0.966930
Evaluation accuracy = 0.977163, Elapsed Time = 1.561047s
Starting Epoch 7:
Training loss = 88.166275, training accuracy = 0.970002
Evaluation accuracy = 0.976663, Elapsed Time = 1.665272s
Starting Epoch 8:
Training loss = 85.920563, training accuracy = 0.970469
Evaluation accuracy = 0.976162, Elapsed Time = 1.579652s
Starting Epoch 9:
Training loss = 79.568375, training accuracy = 0.973040
Evaluation accuracy = 0.982472, Elapsed Time = 1.605414s
root@71ac539cda77:~/dcsysh/singa/examples/autograd# mpiexec -np 2 python3 train_mpi.py cnn mnist --op sparseTopK --spars 0.05 --lr 0.01
Starting Epoch 0:
Training loss = 1011.997559, training accuracy = 0.637720
Evaluation accuracy = 0.890124, Elapsed Time = 2.135889s
Starting Epoch 1:
Training loss = 359.958496, training accuracy = 0.876920
Evaluation accuracy = 0.931390, Elapsed Time = 1.748328s
Starting Epoch 2:
Training loss = 259.234558, training accuracy = 0.912660
Evaluation accuracy = 0.953826, Elapsed Time = 1.694958s
Starting Epoch 3:
Training loss = 206.229309, training accuracy = 0.930939
Evaluation accuracy = 0.954327, Elapsed Time = 1.627520s
Starting Epoch 4:
Training loss = 176.526505, training accuracy = 0.940138
Evaluation accuracy = 0.958634, Elapsed Time = 1.618325s
Starting Epoch 5:
Training loss = 158.930298, training accuracy = 0.947349
Evaluation accuracy = 0.962941, Elapsed Time = 1.619869s
Starting Epoch 6:
Training loss = 145.359879, training accuracy = 0.951289
Evaluation accuracy = 0.966446, Elapsed Time = 1.621834s
Starting Epoch 7:
Training loss = 129.951813, training accuracy = 0.955746
Evaluation accuracy = 0.972055, Elapsed Time = 1.712546s
Starting Epoch 8:
Training loss = 117.017708, training accuracy = 0.961405
Evaluation accuracy = 0.973257, Elapsed Time = 1.863692s
Starting Epoch 9:
Training loss = 110.554947, training accuracy = 0.963241
Evaluation accuracy = 0.974659, Elapsed Time = 1.614416s
root@71ac539cda77:~/dcsysh/singa/examples/autograd# mpiexec -np 2 python3 train_mpi.py cnn mnist --op sparseThreshold --spars 0.05 --lr 0.01
Starting Epoch 0:
Training loss = 823.523926, training accuracy = 0.701756
Evaluation accuracy = 0.883413, Elapsed Time = 1.945497s
Starting Epoch 1:
Training loss = 278.808777, training accuracy = 0.906333
Evaluation accuracy = 0.950421, Elapsed Time = 1.600112s
Starting Epoch 2:
Training loss = 186.442337, training accuracy = 0.937250
Evaluation accuracy = 0.967748, Elapsed Time = 1.615348s
Starting Epoch 3:
Training loss = 145.515228, training accuracy = 0.951305
Evaluation accuracy = 0.964944, Elapsed Time = 1.599972s
Starting Epoch 4:
Training loss = 119.955498, training accuracy = 0.959502
Evaluation accuracy = 0.968650, Elapsed Time = 1.601941s
Starting Epoch 5:
Training loss = 105.754234, training accuracy = 0.965077
Evaluation accuracy = 0.976562, Elapsed Time = 1.601124s
Starting Epoch 6:
Training loss = 97.148003, training accuracy = 0.967732
Evaluation accuracy = 0.975661, Elapsed Time = 1.600486s
Starting Epoch 7:
Training loss = 88.760986, training accuracy = 0.969351
Evaluation accuracy = 0.982672, Elapsed Time = 1.597402s
Starting Epoch 8:
Training loss = 81.591782, training accuracy = 0.973124
Evaluation accuracy = 0.979868, Elapsed Time = 1.584163s
Starting Epoch 9:
Training loss = 76.670799, training accuracy = 0.974426
Evaluation accuracy = 0.979868, Elapsed Time = 1.564388s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #651:
Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r404088768
##########
File path: examples/autograd/train.py
##########
@@ -0,0 +1,261 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from singa import singa_wrap as singa
+from singa import opt
+from singa import device
+from singa import tensor
+import numpy as np
+import time
+import argparse
+from PIL import Image
+
+
+# Data Augmentation
+def augmentation(x, batch_size):
+ xpad = np.pad(x, [[0, 0], [0, 0], [4, 4], [4, 4]], 'symmetric')
+ for data_num in range(0, batch_size):
+ offset = np.random.randint(8, size=2)
+ x[data_num, :, :, :] = xpad[data_num, :,
+ offset[0]:offset[0] + x.shape[2],
+ offset[1]:offset[1] + x.shape[2]]
+ if_flip = np.random.randint(2)
+ if (if_flip):
+ x[data_num, :, :, :] = x[data_num, :, :, ::-1]
+ return x
+
+
+# Calculate Accuracy
+def accuracy(pred, target):
+ # y is binary array for ground truth
+ y = np.argmax(pred, axis=1)
+ # t is integer for ground truth, the dim of t is (batch, 1)
+ t = np.max(target, axis=1)
+ a = y == t
+ correct = np.array(a, "int").sum()
+ # print(correct)
+ return correct
+
+
+# Data partition according to the rank
+def partition(global_rank, world_size, train_x, train_y, val_x, val_y):
+ # Partition training data
+ data_per_rank = train_x.shape[0] // world_size
+ idx_start = global_rank * data_per_rank
+ idx_end = (global_rank + 1) * data_per_rank
+ train_x = train_x[idx_start:idx_end]
+ train_y = train_y[idx_start:idx_end]
+ # Partition evaluation data
+ data_per_rank = val_x.shape[0] // world_size
+ idx_start = global_rank * data_per_rank
+ idx_end = (global_rank + 1) * data_per_rank
+ val_x = val_x[idx_start:idx_end]
+ val_y = val_y[idx_start:idx_end]
+ return train_x, train_y, val_x, val_y
+
+
+# Function to all reduce NUMPY Accuracy and Loss from Multiple Devices
+def reduce_variable(variable, dist_opt, reducer):
+ reducer.copy_from_numpy(variable)
+ dist_opt.all_reduce(reducer.data)
+ dist_opt.wait()
+ output = tensor.to_numpy(reducer)
+ return output
+
+
+def resize_dataset(x, image_size):
+ num_data = x.shape[0]
+ dim = x.shape[1]
+ X = np.zeros(shape=(num_data, dim, image_size, image_size), dtype=np.float32)
+ for n in range(0, num_data):
+ for d in range(0, dim):
+ X[n, d, :, :] = np.array(Image.fromarray(x[n, d, :, :]).resize(
+ (image_size, image_size), Image.BILINEAR),
+ dtype=np.float32)
+ return X
+
+
+def run(global_rank, world_size, local_rank, max_epoch, batch_size, model, data,
+ sgd):
+ dev = device.create_cuda_gpu_on(local_rank)
+ dev.SetRandSeed(0)
+ np.random.seed(0)
+
+ if data == 'cifar10':
+ from data import cifar10
+ train_x, train_y, val_x, val_y = cifar10.load()
+ elif data == 'mnist':
+ from data import mnist
+ train_x, train_y, val_x, val_y = mnist.load()
+
+ num_channels = train_x.shape[1]
+ image_size = train_x.shape[2]
+
+ print(num_channels)
+
+ if model == 'resnet':
+ from model import resnet
+ model = resnet.resnet50(num_channels=num_channels)
+ elif model == 'xceptionnet':
+ from model import xceptionnet
+ model = xceptionnet.create_model(num_channels=num_channels)
+ elif model == 'cnn':
+ from model import cnn
+ model = cnn.create_model(num_channels=num_channels)
+
+ # For distributed training, sequential gives better performance
+ if hasattr(sgd, "communicator"):
+ DIST = True
+ sequential = True
+ else:
+ DIST = False
+ sequential = False
+
+ if DIST:
+ train_x, train_y, val_x, val_y = partition(global_rank, world_size,
+ train_x, train_y, val_x,
+ val_y)
+ '''
+ # check dataset shape correctness
+ if global_rank == 0:
+ print("Check the shape of dataset:")
+ print(train_x.shape)
+ print(train_y.shape)
+ '''
+
+ tx = tensor.Tensor(
+ (batch_size, num_channels, model.input_size, model.input_size), dev,
+ tensor.float32)
+ ty = tensor.Tensor((batch_size,), dev, tensor.int32)
+ num_train_batch = train_x.shape[0] // batch_size
+ num_val_batch = val_x.shape[0] // batch_size
+ idx = np.arange(train_x.shape[0], dtype=np.int32)
+
+ # attached model to graph
+ model.on_device(dev)
+ model.set_optimizer(sgd)
+ model.graph(True, sequential)
+
+ # Training and Evaluation Loop
+ for epoch in range(max_epoch):
+ start_time = time.time()
+ np.random.shuffle(idx)
+
+ if global_rank == 0:
+ print('Starting Epoch %d:' % (epoch))
+
+ # Training Phase
+ train_correct = np.zeros(shape=[1], dtype=np.float32)
+ test_correct = np.zeros(shape=[1], dtype=np.float32)
+ train_loss = np.zeros(shape=[1], dtype=np.float32)
+
+ model.train()
+ for b in range(num_train_batch):
+ # Generate the patch data in this iteration
+ x = train_x[idx[b * batch_size:(b + 1) * batch_size]]
+ x = augmentation(x, batch_size)
+ if (image_size != model.input_size):
+ x = resize_dataset(x, model.input_size)
+ y = train_y[idx[b * batch_size:(b + 1) * batch_size]]
+
+ # Copy the patch data into input tensors
+ tx.copy_from_numpy(x)
+ ty.copy_from_numpy(y)
+
+ # Train the model
+ out = model(tx)
+ loss = model.loss(out, ty)
+ model.optim(loss)
+ train_correct += accuracy(tensor.to_numpy(out),y)
+ train_loss += tensor.to_numpy(loss)[0]
+
+ if DIST:
+ # Reduce the Evaluation Accuracy and Loss from Multiple Devices
+ reducer = tensor.Tensor((1,), dev, tensor.float32)
+ train_correct = reduce_variable(train_correct, sgd, reducer)
+ train_loss = reduce_variable(train_loss, sgd, reducer)
+
+ if global_rank == 0:
+ print('Training loss = %f, training accuracy = %f' %
+ (train_loss, train_correct /
+ (num_train_batch * batch_size * world_size)),
+ flush=True)
+
+ # Evaluation Phase
+ model.eval()
+ for b in range(num_val_batch):
+ x = val_x[b * batch_size:(b + 1) * batch_size]
+ if (image_size != model.input_size):
+ x = resize_dataset(x, model.input_size)
+ y = val_y[b * batch_size:(b + 1) * batch_size]
+ tx.copy_from_numpy(x)
+ ty.copy_from_numpy(y)
+ out_test = model(tx)
+ test_correct += accuracy(tensor.to_numpy(out_test),y)
+
+ if DIST:
+ # Reduce the Evaulation Accuracy from Multiple Devices
+ test_correct = reduce_variable(test_correct, sgd, reducer)
+
+ # Output the Evaluation Accuracy
+ if global_rank == 0:
+ print('Evaluation accuracy = %f, Elapsed Time = %fs' %
+ (test_correct / (num_val_batch * batch_size * world_size),
+ time.time() - start_time),
+ flush=True)
+
+
+if __name__ == '__main__':
+ # use argparse to get command config: max_epoch, model, data, etc. for single gpu training
+ parser = argparse.ArgumentParser(
+ description='Training using the autograd and graph.')
+ parser.add_argument('model',
+ choices=['resnet', 'xceptionnet', 'cnn'],
+ default='cnn')
+ parser.add_argument('data', choices=['cifar10', 'mnist'], default='mnist')
+ parser.add_argument('--epoch',
+ '--max-epoch',
+ default=10,
+ type=int,
+ help='maximum epochs',
+ dest='max_epoch')
+ parser.add_argument('--bs',
+ '--batch-size',
+ default=64,
+ type=int,
+ help='batch size',
+ dest='batch_size')
+ parser.add_argument('--lr',
+ '--learning-rate',
+ default=0.005,
+ type=float,
+ help='initial learning rate',
+ dest='lr')
+ # determine which gpu to use in gpu
+ parser.add_argument('--id',
Review comment:
addressed in the new comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609783520
This pull request **introduces 2 alerts** when merging 036d82f3cf9a87c8fa34e630b09da3dc4b28d0c4 into 793dcace41eaf3569c8222e0f4ca667cb9ecf412 - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-73dcae9c133b28c3fea040a9c511c5bf36d4d8dc)
**new alerts:**
* 2 for Unreachable code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #651:
Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403880363
##########
File path: examples/autograd/train.py
##########
@@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from singa import singa_wrap as singa
+from singa import opt
+from singa import device
+from singa import tensor
+import numpy as np
+import time
+import argparse
+from PIL import Image
+
+
+# Data Augmentation
+def augmentation(x, batch_size):
+ xpad = np.pad(x, [[0, 0], [0, 0], [4, 4], [4, 4]], 'symmetric')
+ for data_num in range(0, batch_size):
+ offset = np.random.randint(8, size=2)
+ x[data_num, :, :, :] = xpad[data_num, :,
+ offset[0]:offset[0] + x.shape[2],
+ offset[1]:offset[1] + x.shape[2]]
+ if_flip = np.random.randint(2)
+ if (if_flip):
+ x[data_num, :, :, :] = x[data_num, :, :, ::-1]
+ return x
+
+
+# Calculate Accuracy
+def accuracy(pred, target):
+ # y is binary array for ground truth
+ y = np.argmax(pred, axis=1)
+ # t is integer for ground truth, the dim of t is (batch, 1)
+ t = np.max(target, axis=1)
+ a = y == t
+ correct = np.array(a, "int").sum()
+ # print(correct)
+ return correct
+
+
+# Data partition according to the rank
+def partition(worker_id, num_workers, train_x, train_y, val_x, val_y):
+ # Partition training data
+ data_per_rank = train_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ train_x = train_x[idx_start:idx_end]
+ train_y = train_y[idx_start:idx_end]
+ # Partition evaluation data
+ data_per_rank = val_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ val_x = val_x[idx_start:idx_end]
+ val_y = val_y[idx_start:idx_end]
+ return train_x, train_y, val_x, val_y
+
+
+# Function to all reduce NUMPY Accuracy and Loss from Multiple Devices
+def reduce_variable(variable, dist_opt, reducer):
+ reducer.copy_from_numpy(variable)
+ dist_opt.all_reduce(reducer.data)
+ dist_opt.wait()
+ output = tensor.to_numpy(reducer)
+ return output
+
+
+def resize_dataset(x, image_size):
+ num_data = x.shape[0]
+ dim = x.shape[1]
+ X = np.zeros(shape=(num_data, dim, image_size, image_size), dtype=np.float32)
+ for n in range(0, num_data):
+ for d in range(0, dim):
+ X[n, d, :, :] = np.array(Image.fromarray(x[n, d, :, :]).resize(
+ (image_size, image_size), Image.BILINEAR),
+ dtype=np.float32)
+ return X
+
+
+def run(worker_id, num_workers, device_id, max_epoch, batch_size, model, data,
+ sgd):
+ dev = device.create_cuda_gpu_on(device_id)
+ dev.SetRandSeed(0)
+ np.random.seed(0)
+
+ if model == 'resnet':
+ from model import resnet
+ model = resnet.resnet50()
Review comment:
eval accuracy can achieve around 90% on resnet50, I will try resent 18
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609423668
I followed the new design of API, now can run (i) cnn+mnist, (ii) resnet+cifar10
```
root@877a3759b148:~/dcsysh/singa/examples/autograd# python3 train_multiprocess.py cnn mnist --lr 0.01
Starting Epoch 0:
Training loss = 625.124268, training accuracy = 0.777227
Evaluation accuracy = 0.941206, Elapsed Time = 1.721721s
Starting Epoch 1:
Training loss = 235.566132, training accuracy = 0.920757
Evaluation accuracy = 0.945413, Elapsed Time = 1.685381s
Starting Epoch 2:
Training loss = 171.600082, training accuracy = 0.942258
Evaluation accuracy = 0.969852, Elapsed Time = 1.703893s
Starting Epoch 3:
Training loss = 139.339203, training accuracy = 0.953476
Evaluation accuracy = 0.972857, Elapsed Time = 1.695866s
Starting Epoch 4:
Training loss = 120.622467, training accuracy = 0.959118
Evaluation accuracy = 0.971254, Elapsed Time = 1.703533s
Starting Epoch 5:
Training loss = 105.304459, training accuracy = 0.964777
Evaluation accuracy = 0.978466, Elapsed Time = 1.702236s
Starting Epoch 6:
Training loss = 99.502411, training accuracy = 0.966964
Evaluation accuracy = 0.975761, Elapsed Time = 1.694606s
Starting Epoch 7:
Training loss = 88.000076, training accuracy = 0.969985
Evaluation accuracy = 0.977364, Elapsed Time = 1.700657s
Starting Epoch 8:
Training loss = 85.234161, training accuracy = 0.971004
Evaluation accuracy = 0.976262, Elapsed Time = 1.690182s
Starting Epoch 9:
Training loss = 79.724716, training accuracy = 0.973591
Evaluation accuracy = 0.983073, Elapsed Time = 1.701022s
root@877a3759b148:~/dcsysh/singa/examples/autograd# python3 train_multiprocess.py resnet cifar10 --lr 0.01 --bs 32
Check the shape of dataset:
Starting Epoch 0:
Training loss = 3112.784912, training accuracy = 0.317702
Evaluation accuracy = 0.464543, Elapsed Time = 151.415575s
Starting Epoch 1:
Training loss = 2140.442383, training accuracy = 0.500840
Evaluation accuracy = 0.568710, Elapsed Time = 156.812310s
Starting Epoch 2:
Training loss = 1680.163208, training accuracy = 0.616997
Evaluation accuracy = 0.681891, Elapsed Time = 157.775971s
Starting Epoch 3:
Training loss = 1395.033936, training accuracy = 0.688500
Evaluation accuracy = 0.711939, Elapsed Time = 157.391661s
Starting Epoch 4:
Training loss = 1174.264160, training accuracy = 0.738916
Evaluation accuracy = 0.756711, Elapsed Time = 157.423720s
Starting Epoch 5:
Training loss = 1028.127197, training accuracy = 0.773007
Evaluation accuracy = 0.779948, Elapsed Time = 157.615884s
Starting Epoch 6:
Training loss = 910.041748, training accuracy = 0.796935
Evaluation accuracy = 0.803886, Elapsed Time = 157.282300s
Starting Epoch 7:
Training loss = 829.032227, training accuracy = 0.817882
Evaluation accuracy = 0.810296, Elapsed Time = 157.427150s
Starting Epoch 8:
Training loss = 761.830078, training accuracy = 0.832686
Evaluation accuracy = 0.825621, Elapsed Time = 157.431677s
Starting Epoch 9:
Training loss = 703.944153, training accuracy = 0.846791
Evaluation accuracy = 0.825421, Elapsed Time = 157.413234s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #651: Add new
example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403804988
##########
File path: examples/autograd/train.py
##########
@@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from singa import singa_wrap as singa
+from singa import opt
+from singa import device
+from singa import tensor
+import numpy as np
+import time
+import argparse
+from PIL import Image
+
+
+# Data Augmentation
+def augmentation(x, batch_size):
+ xpad = np.pad(x, [[0, 0], [0, 0], [4, 4], [4, 4]], 'symmetric')
+ for data_num in range(0, batch_size):
+ offset = np.random.randint(8, size=2)
+ x[data_num, :, :, :] = xpad[data_num, :,
+ offset[0]:offset[0] + x.shape[2],
+ offset[1]:offset[1] + x.shape[2]]
+ if_flip = np.random.randint(2)
+ if (if_flip):
+ x[data_num, :, :, :] = x[data_num, :, :, ::-1]
+ return x
+
+
+# Calculate Accuracy
+def accuracy(pred, target):
+ # y is binary array for ground truth
+ y = np.argmax(pred, axis=1)
+ # t is integer for ground truth, the dim of t is (batch, 1)
+ t = np.max(target, axis=1)
+ a = y == t
+ correct = np.array(a, "int").sum()
+ # print(correct)
+ return correct
+
+
+# Data partition according to the rank
+def partition(worker_id, num_workers, train_x, train_y, val_x, val_y):
+ # Partition training data
+ data_per_rank = train_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ train_x = train_x[idx_start:idx_end]
+ train_y = train_y[idx_start:idx_end]
+ # Partition evaluation data
+ data_per_rank = val_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ val_x = val_x[idx_start:idx_end]
+ val_y = val_y[idx_start:idx_end]
+ return train_x, train_y, val_x, val_y
+
+
+# Function to all reduce NUMPY Accuracy and Loss from Multiple Devices
+def reduce_variable(variable, dist_opt, reducer):
+ reducer.copy_from_numpy(variable)
+ dist_opt.all_reduce(reducer.data)
+ dist_opt.wait()
+ output = tensor.to_numpy(reducer)
+ return output
+
+
+def resize_dataset(x, image_size):
+ num_data = x.shape[0]
+ dim = x.shape[1]
+ X = np.zeros(shape=(num_data, dim, image_size, image_size), dtype=np.float32)
+ for n in range(0, num_data):
+ for d in range(0, dim):
+ X[n, d, :, :] = np.array(Image.fromarray(x[n, d, :, :]).resize(
+ (image_size, image_size), Image.BILINEAR),
+ dtype=np.float32)
+ return X
+
+
+def run(worker_id, num_workers, device_id, max_epoch, batch_size, model, data,
+ sgd):
+ dev = device.create_cuda_gpu_on(device_id)
+ dev.SetRandSeed(0)
+ np.random.seed(0)
+
+ if model == 'resnet':
+ from model import resnet
+ model = resnet.resnet50()
Review comment:
can we run resnet50 over mnist and cifar10? the input size is very small.
after many pooling/conv layers, the feature map size may be equal or smaller than 1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: Simply example APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609012455
This pull request **introduces 2 alerts** when merging 9fb6580a4914b07f50681f334635df57ff05ffd4 into 182007508a904084bf551bfef9dbd8aa55a46d3b - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-864271850bb685a0c793c11ee6b119712f9c32e8)
**new alerts:**
* 2 for Unreachable code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] XJDKC edited a comment on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
XJDKC edited a comment on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609659740
> python3 train.py xceptionnet ci
I think it might be that the memory addresses of some tensors were changed during the evaluation phase, which caused the calculation graph to not be executed correctly. Because in one epoch we run the graph multiple times correctly, which means the graph execution is correct.
After commenting out the code of evaluation, I got this result.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609632911
can the original model run (not using the Module API)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609587049
I have already unified terms to <local_rank, global_rank, world_size> in the singa code (last commit)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609758264
I added two args in train_mpi,py and train_multiprocess.py to support all the distributed training options, but train.py don't need the argument because it doesn't have DistOpt
Below are two examples for the use of arguments:
```
root@71ac539cda77:~/dcsysh/singa/examples/autograd# mpiexec -np 2 python3 train_mpi.py cnn mnist --op fp16 --lr 0.01
Starting Epoch 0:
Training loss = 625.136963, training accuracy = 0.776893
Evaluation accuracy = 0.939704, Elapsed Time = 1.619492s
Starting Epoch 1:
Training loss = 235.280151, training accuracy = 0.920339
Evaluation accuracy = 0.947115, Elapsed Time = 1.517764s
Starting Epoch 2:
Training loss = 171.311310, training accuracy = 0.942508
Evaluation accuracy = 0.970252, Elapsed Time = 1.557606s
Starting Epoch 3:
Training loss = 139.594086, training accuracy = 0.953342
Evaluation accuracy = 0.972356, Elapsed Time = 1.541372s
Starting Epoch 4:
Training loss = 120.380058, training accuracy = 0.959852
Evaluation accuracy = 0.971655, Elapsed Time = 1.560016s
Starting Epoch 5:
Training loss = 104.767105, training accuracy = 0.965345
Evaluation accuracy = 0.979467, Elapsed Time = 1.525147s
Starting Epoch 6:
Training loss = 98.995010, training accuracy = 0.966730
Evaluation accuracy = 0.976963, Elapsed Time = 1.528173s
Starting Epoch 7:
Training loss = 88.012024, training accuracy = 0.970202
Evaluation accuracy = 0.977865, Elapsed Time = 1.515204s
Starting Epoch 8:
Training loss = 86.540741, training accuracy = 0.970519
Evaluation accuracy = 0.975361, Elapsed Time = 1.604774s
Starting Epoch 9:
Training loss = 80.653885, training accuracy = 0.973024
Evaluation accuracy = 0.982071, Elapsed Time = 1.633480s
root@71ac539cda77:~/dcsysh/singa/examples/autograd# mpiexec -np 2 python3 train_mpi.py cnn mnist --op partialUpdate --lr 0.01
Starting Epoch 0:
Training loss = 623.692139, training accuracy = 0.777694
Evaluation accuracy = 0.939804, Elapsed Time = 1.678683s
Starting Epoch 1:
Training loss = 235.369232, training accuracy = 0.920690
Evaluation accuracy = 0.948017, Elapsed Time = 1.567066s
Starting Epoch 2:
Training loss = 171.335693, training accuracy = 0.942157
Evaluation accuracy = 0.971855, Elapsed Time = 1.595140s
Starting Epoch 3:
Training loss = 139.396088, training accuracy = 0.953592
Evaluation accuracy = 0.971755, Elapsed Time = 1.661490s
Starting Epoch 4:
Training loss = 120.173752, training accuracy = 0.960019
Evaluation accuracy = 0.970753, Elapsed Time = 1.571710s
Starting Epoch 5:
Training loss = 105.472672, training accuracy = 0.965061
Evaluation accuracy = 0.978065, Elapsed Time = 1.570710s
Starting Epoch 6:
Training loss = 99.055389, training accuracy = 0.966930
Evaluation accuracy = 0.977163, Elapsed Time = 1.561047s
Starting Epoch 7:
Training loss = 88.166275, training accuracy = 0.970002
Evaluation accuracy = 0.976663, Elapsed Time = 1.665272s
Starting Epoch 8:
Training loss = 85.920563, training accuracy = 0.970469
Evaluation accuracy = 0.976162, Elapsed Time = 1.579652s
Starting Epoch 9:
Training loss = 79.568375, training accuracy = 0.973040
Evaluation accuracy = 0.982472, Elapsed Time = 1.605414s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #651:
Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403807582
##########
File path: examples/autograd/train.py
##########
@@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from singa import singa_wrap as singa
+from singa import opt
+from singa import device
+from singa import tensor
+import numpy as np
+import time
+import argparse
+from PIL import Image
+
+
+# Data Augmentation
+def augmentation(x, batch_size):
+ xpad = np.pad(x, [[0, 0], [0, 0], [4, 4], [4, 4]], 'symmetric')
+ for data_num in range(0, batch_size):
+ offset = np.random.randint(8, size=2)
+ x[data_num, :, :, :] = xpad[data_num, :,
+ offset[0]:offset[0] + x.shape[2],
+ offset[1]:offset[1] + x.shape[2]]
+ if_flip = np.random.randint(2)
+ if (if_flip):
+ x[data_num, :, :, :] = x[data_num, :, :, ::-1]
+ return x
+
+
+# Calculate Accuracy
+def accuracy(pred, target):
+ # y is binary array for ground truth
+ y = np.argmax(pred, axis=1)
+ # t is integer for ground truth, the dim of t is (batch, 1)
+ t = np.max(target, axis=1)
+ a = y == t
+ correct = np.array(a, "int").sum()
+ # print(correct)
+ return correct
+
+
+# Data partition according to the rank
+def partition(worker_id, num_workers, train_x, train_y, val_x, val_y):
+ # Partition training data
+ data_per_rank = train_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ train_x = train_x[idx_start:idx_end]
+ train_y = train_y[idx_start:idx_end]
+ # Partition evaluation data
+ data_per_rank = val_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ val_x = val_x[idx_start:idx_end]
+ val_y = val_y[idx_start:idx_end]
+ return train_x, train_y, val_x, val_y
+
+
+# Function to all reduce NUMPY Accuracy and Loss from Multiple Devices
+def reduce_variable(variable, dist_opt, reducer):
+ reducer.copy_from_numpy(variable)
+ dist_opt.all_reduce(reducer.data)
+ dist_opt.wait()
+ output = tensor.to_numpy(reducer)
+ return output
+
+
+def resize_dataset(x, image_size):
+ num_data = x.shape[0]
+ dim = x.shape[1]
+ X = np.zeros(shape=(num_data, dim, image_size, image_size), dtype=np.float32)
+ for n in range(0, num_data):
+ for d in range(0, dim):
+ X[n, d, :, :] = np.array(Image.fromarray(x[n, d, :, :]).resize(
+ (image_size, image_size), Image.BILINEAR),
+ dtype=np.float32)
+ return X
+
+
+def run(worker_id, num_workers, device_id, max_epoch, batch_size, model, data,
+ sgd):
+ dev = device.create_cuda_gpu_on(device_id)
+ dev.SetRandSeed(0)
+ np.random.seed(0)
+
+ if model == 'resnet':
+ from model import resnet
+ model = resnet.resnet50()
Review comment:
can run resnet 50 on cifar10, I rescaled to 224 (see the resize function after data augmentation)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris edited a comment on issue #651: [WIP ]Simply
example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris edited a comment on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609423668
I followed the new design of API, now can run (i) cnn+mnist, (ii) resnet+cifar10
```
root@877a3759b148:~/dcsysh/singa/examples/autograd# python3 train_multiprocess.py cnn mnist --lr 0.01
Starting Epoch 0:
Training loss = 625.124268, training accuracy = 0.777227
Evaluation accuracy = 0.941206, Elapsed Time = 1.721721s
Starting Epoch 1:
Training loss = 235.566132, training accuracy = 0.920757
Evaluation accuracy = 0.945413, Elapsed Time = 1.685381s
Starting Epoch 2:
Training loss = 171.600082, training accuracy = 0.942258
Evaluation accuracy = 0.969852, Elapsed Time = 1.703893s
Starting Epoch 3:
Training loss = 139.339203, training accuracy = 0.953476
Evaluation accuracy = 0.972857, Elapsed Time = 1.695866s
Starting Epoch 4:
Training loss = 120.622467, training accuracy = 0.959118
Evaluation accuracy = 0.971254, Elapsed Time = 1.703533s
Starting Epoch 5:
Training loss = 105.304459, training accuracy = 0.964777
Evaluation accuracy = 0.978466, Elapsed Time = 1.702236s
Starting Epoch 6:
Training loss = 99.502411, training accuracy = 0.966964
Evaluation accuracy = 0.975761, Elapsed Time = 1.694606s
Starting Epoch 7:
Training loss = 88.000076, training accuracy = 0.969985
Evaluation accuracy = 0.977364, Elapsed Time = 1.700657s
Starting Epoch 8:
Training loss = 85.234161, training accuracy = 0.971004
Evaluation accuracy = 0.976262, Elapsed Time = 1.690182s
Starting Epoch 9:
Training loss = 79.724716, training accuracy = 0.973591
Evaluation accuracy = 0.983073, Elapsed Time = 1.701022s
root@877a3759b148:~/dcsysh/singa/examples/autograd# python3 train_multiprocess.py resnet cifar10 --lr 0.01 --bs 32
Starting Epoch 0:
Training loss = 3112.784912, training accuracy = 0.317702
Evaluation accuracy = 0.464543, Elapsed Time = 151.415575s
Starting Epoch 1:
Training loss = 2140.442383, training accuracy = 0.500840
Evaluation accuracy = 0.568710, Elapsed Time = 156.812310s
Starting Epoch 2:
Training loss = 1680.163208, training accuracy = 0.616997
Evaluation accuracy = 0.681891, Elapsed Time = 157.775971s
Starting Epoch 3:
Training loss = 1395.033936, training accuracy = 0.688500
Evaluation accuracy = 0.711939, Elapsed Time = 157.391661s
Starting Epoch 4:
Training loss = 1174.264160, training accuracy = 0.738916
Evaluation accuracy = 0.756711, Elapsed Time = 157.423720s
Starting Epoch 5:
Training loss = 1028.127197, training accuracy = 0.773007
Evaluation accuracy = 0.779948, Elapsed Time = 157.615884s
Starting Epoch 6:
Training loss = 910.041748, training accuracy = 0.796935
Evaluation accuracy = 0.803886, Elapsed Time = 157.282300s
Starting Epoch 7:
Training loss = 829.032227, training accuracy = 0.817882
Evaluation accuracy = 0.810296, Elapsed Time = 157.427150s
Starting Epoch 8:
Training loss = 761.830078, training accuracy = 0.832686
Evaluation accuracy = 0.825621, Elapsed Time = 157.431677s
Starting Epoch 9:
Training loss = 703.944153, training accuracy = 0.846791
Evaluation accuracy = 0.825421, Elapsed Time = 157.413234s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles merged pull request #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
nudles merged pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: [WIP] Simply example
APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: [WIP] Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609008149
This pull request **introduces 6 alerts** when merging 5293b9a7709d5d5b2c020be826c89a29cbb809e7 into 182007508a904084bf551bfef9dbd8aa55a46d3b - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-1dc3ebbce7e94b2b0ed3788010e1c52cf9c441b0)
**new alerts:**
* 4 for Unused import
* 2 for Unreachable code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #651: Add new
example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403804641
##########
File path: examples/autograd/train.py
##########
@@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from singa import singa_wrap as singa
+from singa import opt
+from singa import device
+from singa import tensor
+import numpy as np
+import time
+import argparse
+from PIL import Image
+
+
+# Data Augmentation
+def augmentation(x, batch_size):
+ xpad = np.pad(x, [[0, 0], [0, 0], [4, 4], [4, 4]], 'symmetric')
+ for data_num in range(0, batch_size):
+ offset = np.random.randint(8, size=2)
+ x[data_num, :, :, :] = xpad[data_num, :,
+ offset[0]:offset[0] + x.shape[2],
+ offset[1]:offset[1] + x.shape[2]]
+ if_flip = np.random.randint(2)
+ if (if_flip):
+ x[data_num, :, :, :] = x[data_num, :, :, ::-1]
+ return x
+
+
+# Calculate Accuracy
+def accuracy(pred, target):
+ # y is binary array for ground truth
+ y = np.argmax(pred, axis=1)
+ # t is integer for ground truth, the dim of t is (batch, 1)
+ t = np.max(target, axis=1)
+ a = y == t
+ correct = np.array(a, "int").sum()
+ # print(correct)
+ return correct
+
+
+# Data partition according to the rank
+def partition(worker_id, num_workers, train_x, train_y, val_x, val_y):
+ # Partition training data
+ data_per_rank = train_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ train_x = train_x[idx_start:idx_end]
+ train_y = train_y[idx_start:idx_end]
+ # Partition evaluation data
+ data_per_rank = val_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ val_x = val_x[idx_start:idx_end]
+ val_y = val_y[idx_start:idx_end]
+ return train_x, train_y, val_x, val_y
+
+
+# Function to all reduce NUMPY Accuracy and Loss from Multiple Devices
+def reduce_variable(variable, dist_opt, reducer):
+ reducer.copy_from_numpy(variable)
+ dist_opt.all_reduce(reducer.data)
+ dist_opt.wait()
+ output = tensor.to_numpy(reducer)
+ return output
+
+
+def resize_dataset(x, image_size):
+ num_data = x.shape[0]
+ dim = x.shape[1]
+ X = np.zeros(shape=(num_data, dim, image_size, image_size), dtype=np.float32)
+ for n in range(0, num_data):
+ for d in range(0, dim):
+ X[n, d, :, :] = np.array(Image.fromarray(x[n, d, :, :]).resize(
+ (image_size, image_size), Image.BILINEAR),
+ dtype=np.float32)
+ return X
+
+
+def run(worker_id, num_workers, device_id, max_epoch, batch_size, model, data,
+ sgd):
+ dev = device.create_cuda_gpu_on(device_id)
+ dev.SetRandSeed(0)
+ np.random.seed(0)
+
+ if model == 'resnet':
+ from model import resnet
+ model = resnet.resnet50()
+ elif model == 'xceptionnet':
+ from model import xceptionnet
+ model = xceptionnet.create_model()
+ elif model == 'cnn':
+ from model import cnn
+ model = cnn.create_model()
+
+ if data == 'cifar10':
+ from data import cifar10
+ train_x, train_y, val_x, val_y = cifar10.load()
+ elif data == 'mnist':
+ from data import mnist
+ train_x, train_y, val_x, val_y = mnist.load()
+
+ if hasattr(sgd, "communicator"):
+ DIST = True
+ else:
+ DIST = False
+
+ if DIST:
+ train_x, train_y, val_x, val_y = partition(worker_id, num_workers,
+ train_x, train_y, val_x,
+ val_y)
+ '''
+ # check dataset shape correctness
+ if worker_id == 0:
+ print("Check the shape of dataset:")
+ print(train_x.shape)
+ print(train_y.shape)
+ '''
+
+ num_channels = train_x.shape[1]
+ image_size = train_x.shape[2]
+
+ tx = tensor.Tensor(
+ (batch_size, num_channels, model.input_size, model.input_size), dev,
+ tensor.float32)
+ ty = tensor.Tensor((batch_size,), dev, tensor.int32)
+ num_train_batch = train_x.shape[0] // batch_size
+ num_val_batch = val_x.shape[0] // batch_size
+ idx = np.arange(train_x.shape[0], dtype=np.int32)
+
+ # attached model to graph
+ graph = True
+
+ # For distributed training, sequential gives better performance
+ if DIST:
Review comment:
can merge the following lines into line 117?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #651: Add new
example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403952052
##########
File path: examples/autograd/train.py
##########
@@ -0,0 +1,261 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from singa import singa_wrap as singa
+from singa import opt
+from singa import device
+from singa import tensor
+import numpy as np
+import time
+import argparse
+from PIL import Image
+
+
+# Data Augmentation
+def augmentation(x, batch_size):
+ xpad = np.pad(x, [[0, 0], [0, 0], [4, 4], [4, 4]], 'symmetric')
+ for data_num in range(0, batch_size):
+ offset = np.random.randint(8, size=2)
+ x[data_num, :, :, :] = xpad[data_num, :,
+ offset[0]:offset[0] + x.shape[2],
+ offset[1]:offset[1] + x.shape[2]]
+ if_flip = np.random.randint(2)
+ if (if_flip):
+ x[data_num, :, :, :] = x[data_num, :, :, ::-1]
+ return x
+
+
+# Calculate Accuracy
+def accuracy(pred, target):
+ # y is binary array for ground truth
+ y = np.argmax(pred, axis=1)
+ # t is integer for ground truth, the dim of t is (batch, 1)
+ t = np.max(target, axis=1)
+ a = y == t
+ correct = np.array(a, "int").sum()
+ # print(correct)
+ return correct
+
+
+# Data partition according to the rank
+def partition(global_rank, world_size, train_x, train_y, val_x, val_y):
+ # Partition training data
+ data_per_rank = train_x.shape[0] // world_size
+ idx_start = global_rank * data_per_rank
+ idx_end = (global_rank + 1) * data_per_rank
+ train_x = train_x[idx_start:idx_end]
+ train_y = train_y[idx_start:idx_end]
+ # Partition evaluation data
+ data_per_rank = val_x.shape[0] // world_size
+ idx_start = global_rank * data_per_rank
+ idx_end = (global_rank + 1) * data_per_rank
+ val_x = val_x[idx_start:idx_end]
+ val_y = val_y[idx_start:idx_end]
+ return train_x, train_y, val_x, val_y
+
+
+# Function to all reduce NUMPY Accuracy and Loss from Multiple Devices
+def reduce_variable(variable, dist_opt, reducer):
+ reducer.copy_from_numpy(variable)
+ dist_opt.all_reduce(reducer.data)
+ dist_opt.wait()
+ output = tensor.to_numpy(reducer)
+ return output
+
+
+def resize_dataset(x, image_size):
+ num_data = x.shape[0]
+ dim = x.shape[1]
+ X = np.zeros(shape=(num_data, dim, image_size, image_size), dtype=np.float32)
+ for n in range(0, num_data):
+ for d in range(0, dim):
+ X[n, d, :, :] = np.array(Image.fromarray(x[n, d, :, :]).resize(
+ (image_size, image_size), Image.BILINEAR),
+ dtype=np.float32)
+ return X
+
+
+def run(global_rank, world_size, local_rank, max_epoch, batch_size, model, data,
+ sgd):
+ dev = device.create_cuda_gpu_on(local_rank)
+ dev.SetRandSeed(0)
+ np.random.seed(0)
+
+ if data == 'cifar10':
+ from data import cifar10
+ train_x, train_y, val_x, val_y = cifar10.load()
+ elif data == 'mnist':
+ from data import mnist
+ train_x, train_y, val_x, val_y = mnist.load()
+
+ num_channels = train_x.shape[1]
+ image_size = train_x.shape[2]
+
+ print(num_channels)
+
+ if model == 'resnet':
+ from model import resnet
+ model = resnet.resnet50(num_channels=num_channels)
+ elif model == 'xceptionnet':
+ from model import xceptionnet
+ model = xceptionnet.create_model(num_channels=num_channels)
+ elif model == 'cnn':
+ from model import cnn
+ model = cnn.create_model(num_channels=num_channels)
+
+ # For distributed training, sequential gives better performance
+ if hasattr(sgd, "communicator"):
+ DIST = True
+ sequential = True
+ else:
+ DIST = False
+ sequential = False
+
+ if DIST:
+ train_x, train_y, val_x, val_y = partition(global_rank, world_size,
+ train_x, train_y, val_x,
+ val_y)
+ '''
+ # check dataset shape correctness
+ if global_rank == 0:
+ print("Check the shape of dataset:")
+ print(train_x.shape)
+ print(train_y.shape)
+ '''
+
+ tx = tensor.Tensor(
+ (batch_size, num_channels, model.input_size, model.input_size), dev,
+ tensor.float32)
+ ty = tensor.Tensor((batch_size,), dev, tensor.int32)
+ num_train_batch = train_x.shape[0] // batch_size
+ num_val_batch = val_x.shape[0] // batch_size
+ idx = np.arange(train_x.shape[0], dtype=np.int32)
+
+ # attached model to graph
+ model.on_device(dev)
+ model.set_optimizer(sgd)
+ model.graph(True, sequential)
+
+ # Training and Evaluation Loop
+ for epoch in range(max_epoch):
+ start_time = time.time()
+ np.random.shuffle(idx)
+
+ if global_rank == 0:
+ print('Starting Epoch %d:' % (epoch))
+
+ # Training Phase
+ train_correct = np.zeros(shape=[1], dtype=np.float32)
+ test_correct = np.zeros(shape=[1], dtype=np.float32)
+ train_loss = np.zeros(shape=[1], dtype=np.float32)
+
+ model.train()
+ for b in range(num_train_batch):
+ # Generate the patch data in this iteration
+ x = train_x[idx[b * batch_size:(b + 1) * batch_size]]
+ x = augmentation(x, batch_size)
+ if (image_size != model.input_size):
+ x = resize_dataset(x, model.input_size)
+ y = train_y[idx[b * batch_size:(b + 1) * batch_size]]
+
+ # Copy the patch data into input tensors
+ tx.copy_from_numpy(x)
+ ty.copy_from_numpy(y)
+
+ # Train the model
+ out = model(tx)
+ loss = model.loss(out, ty)
+ model.optim(loss)
+ train_correct += accuracy(tensor.to_numpy(out),y)
+ train_loss += tensor.to_numpy(loss)[0]
+
+ if DIST:
+ # Reduce the Evaluation Accuracy and Loss from Multiple Devices
+ reducer = tensor.Tensor((1,), dev, tensor.float32)
+ train_correct = reduce_variable(train_correct, sgd, reducer)
+ train_loss = reduce_variable(train_loss, sgd, reducer)
+
+ if global_rank == 0:
+ print('Training loss = %f, training accuracy = %f' %
+ (train_loss, train_correct /
+ (num_train_batch * batch_size * world_size)),
+ flush=True)
+
+ # Evaluation Phase
+ model.eval()
+ for b in range(num_val_batch):
+ x = val_x[b * batch_size:(b + 1) * batch_size]
+ if (image_size != model.input_size):
+ x = resize_dataset(x, model.input_size)
+ y = val_y[b * batch_size:(b + 1) * batch_size]
+ tx.copy_from_numpy(x)
+ ty.copy_from_numpy(y)
+ out_test = model(tx)
+ test_correct += accuracy(tensor.to_numpy(out_test),y)
+
+ if DIST:
+ # Reduce the Evaulation Accuracy from Multiple Devices
+ test_correct = reduce_variable(test_correct, sgd, reducer)
+
+ # Output the Evaluation Accuracy
+ if global_rank == 0:
+ print('Evaluation accuracy = %f, Elapsed Time = %fs' %
+ (test_correct / (num_val_batch * batch_size * world_size),
+ time.time() - start_time),
+ flush=True)
+
+
+if __name__ == '__main__':
+ # use argparse to get command config: max_epoch, model, data, etc. for single gpu training
+ parser = argparse.ArgumentParser(
+ description='Training using the autograd and graph.')
+ parser.add_argument('model',
+ choices=['resnet', 'xceptionnet', 'cnn'],
+ default='cnn')
+ parser.add_argument('data', choices=['cifar10', 'mnist'], default='mnist')
+ parser.add_argument('--epoch',
+ '--max-epoch',
+ default=10,
+ type=int,
+ help='maximum epochs',
+ dest='max_epoch')
+ parser.add_argument('--bs',
+ '--batch-size',
+ default=64,
+ type=int,
+ help='batch size',
+ dest='batch_size')
+ parser.add_argument('--lr',
+ '--learning-rate',
+ default=0.005,
+ type=float,
+ help='initial learning rate',
+ dest='lr')
+ # determine which gpu to use in gpu
+ parser.add_argument('--id',
Review comment:
I think we can add the sparsification argument and pass it to `run()` function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #651: Add new
example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403870470
##########
File path: examples/autograd/train.py
##########
@@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from singa import singa_wrap as singa
+from singa import opt
+from singa import device
+from singa import tensor
+import numpy as np
+import time
+import argparse
+from PIL import Image
+
+
+# Data Augmentation
+def augmentation(x, batch_size):
+ xpad = np.pad(x, [[0, 0], [0, 0], [4, 4], [4, 4]], 'symmetric')
+ for data_num in range(0, batch_size):
+ offset = np.random.randint(8, size=2)
+ x[data_num, :, :, :] = xpad[data_num, :,
+ offset[0]:offset[0] + x.shape[2],
+ offset[1]:offset[1] + x.shape[2]]
+ if_flip = np.random.randint(2)
+ if (if_flip):
+ x[data_num, :, :, :] = x[data_num, :, :, ::-1]
+ return x
+
+
+# Calculate Accuracy
+def accuracy(pred, target):
+ # y is binary array for ground truth
+ y = np.argmax(pred, axis=1)
+ # t is integer for ground truth, the dim of t is (batch, 1)
+ t = np.max(target, axis=1)
+ a = y == t
+ correct = np.array(a, "int").sum()
+ # print(correct)
+ return correct
+
+
+# Data partition according to the rank
+def partition(worker_id, num_workers, train_x, train_y, val_x, val_y):
+ # Partition training data
+ data_per_rank = train_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ train_x = train_x[idx_start:idx_end]
+ train_y = train_y[idx_start:idx_end]
+ # Partition evaluation data
+ data_per_rank = val_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ val_x = val_x[idx_start:idx_end]
+ val_y = val_y[idx_start:idx_end]
+ return train_x, train_y, val_x, val_y
+
+
+# Function to all reduce NUMPY Accuracy and Loss from Multiple Devices
+def reduce_variable(variable, dist_opt, reducer):
+ reducer.copy_from_numpy(variable)
+ dist_opt.all_reduce(reducer.data)
+ dist_opt.wait()
+ output = tensor.to_numpy(reducer)
+ return output
+
+
+def resize_dataset(x, image_size):
+ num_data = x.shape[0]
+ dim = x.shape[1]
+ X = np.zeros(shape=(num_data, dim, image_size, image_size), dtype=np.float32)
+ for n in range(0, num_data):
+ for d in range(0, dim):
+ X[n, d, :, :] = np.array(Image.fromarray(x[n, d, :, :]).resize(
+ (image_size, image_size), Image.BILINEAR),
+ dtype=np.float32)
+ return X
+
+
+def run(worker_id, num_workers, device_id, max_epoch, batch_size, model, data,
+ sgd):
+ dev = device.create_cuda_gpu_on(device_id)
+ dev.SetRandSeed(0)
+ np.random.seed(0)
+
+ if model == 'resnet':
+ from model import resnet
+ model = resnet.resnet50()
Review comment:
does it affect the accuracy?
we can use resnet18 I think, which is also available.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609262829
> > we can use resnet_module.py to replace resnet.py?
>
> yes, should be okay, in this case maybe run everything with graph
note that in this setting we give up the non-graph mode, but I guess it will be okay to run all the example
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #651:
Simply example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #651: Simply example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403477723
##########
File path: examples/autograd/cifar10_multiprocess.py
##########
@@ -17,28 +17,16 @@
# under the License.
#
-from singa import opt
from resnet_cifar10 import *
import multiprocessing
-
-def data_partition(dataset_x, dataset_y, rank_in_global, world_size):
- data_per_rank = dataset_x.shape[0] // world_size
- idx_start = rank_in_global * data_per_rank
- idx_end = (rank_in_global + 1) * data_per_rank
- return dataset_x[idx_start:idx_end], dataset_y[idx_start:idx_end]
-
-
if __name__ == '__main__':
# Generate a NCCL ID to be used for collective communication
nccl_id = singa.NcclIdHolder()
- sgd = opt.SGD(lr=0.005, momentum=0.9, weight_decay=1e-5)
-
- gpu_per_node = 4
- max_epoch = 100
- batch_size = 32
+ # number of GPUs to be used
+ gpu_per_node = 2
Review comment:
yes, okay I will add command line input argument and change to num_gpus
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris edited a comment on issue #651: [WIP ]Simply
example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris edited a comment on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609049842
> I suggest to merge the examples under `examples/autograd` into the follow structure.
>
> ```
> autograd
> train.py
> train_mpi.py
> train_multiprocess.py
> data # define the data loading and preprocessing
> cifar10.py
> mnist.py
> model # define the model
> cnn.py
> resnet.py
> xception.py
> ```
>
> The pseudo code of each file:
>
> ```python
> # train.py
> def run(max_epoch, worker_id, num_workers, model, data, sgd):
> if model == 'resnet':
> from model import resnet
> model = resnet.create_model()
> elif model == 'cnn':
> model = cnn.create_model()
> ....
> if data == 'cifar10':
> from data import cifar10:
> train_x, train_y, val_x, val_y = cifar10.load()
> elif data == 'mnist':
> ....
>
> train_x, train_y, val_x, val_y = partition(worker_id, num_workers, train_x, train_y, val_x, val_y)
>
> # bp and sgd
>
> if __name__ == '__main__':
> # use argparse to get command config: max_epoch, model, data, etc. for single gpu training
> sgd = # create sgd
> run(0, 1, ..., sgd)
>
> # train_mpi.py
> if __name__ == '__main__':
> # use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
> sgd = # create sgd
> dist_sgd = DistOpt(sgd...)
> run(dist_sgd.rank, dist_sgd.world_size, ... dist_sgd)
>
> # train_multiprocess.py
> def run(rank, num_gpu, ...):
> sgd = ...
> dist_sgd = DistOpt(sgd)
> train.run(rank, num_gpu, ... dist_sgd)
>
> if __name__ == '__main__':
> # use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
> nccl_id = ...
> process = []
> for worker_id in range(0, gpu_per_node):
> process.append(multiprocessing.Process(target=run, args=(worker_id, ... nccl_id)))
>
> for p in process:
> p.start()
> ```
thanks! I am working on it. There may be three issues that need to think about:
(i) maybe difficult to include graph module (please see resnet_module.py to see the different between layer model and module model). In this case, we may consider
(ii) don't know how to show this example in the doc dist-train.md. So maybe in the doc we can still use doc_dist_train.py for examples
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #651: Simply
example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #651: Simply example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403476068
##########
File path: examples/autograd/cifar10_multiprocess.py
##########
@@ -17,28 +17,16 @@
# under the License.
#
-from singa import opt
from resnet_cifar10 import *
import multiprocessing
-
-def data_partition(dataset_x, dataset_y, rank_in_global, world_size):
- data_per_rank = dataset_x.shape[0] // world_size
- idx_start = rank_in_global * data_per_rank
- idx_end = (rank_in_global + 1) * data_per_rank
- return dataset_x[idx_start:idx_end], dataset_y[idx_start:idx_end]
-
-
if __name__ == '__main__':
# Generate a NCCL ID to be used for collective communication
nccl_id = singa.NcclIdHolder()
- sgd = opt.SGD(lr=0.005, momentum=0.9, weight_decay=1e-5)
-
- gpu_per_node = 4
- max_epoch = 100
- batch_size = 32
+ # number of GPUs to be used
+ gpu_per_node = 2
Review comment:
rename gpu_per_node --> num_gpus?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609888894
This pull request **introduces 2 alerts** when merging db914e10ec23498f37813a408432fd08cb7a63bc into 10499c77640fa3655f9f4a74185ef060558d532c - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-cd3c5476e4dfa25f75a76d81f3835da32af81e27)
**new alerts:**
* 2 for Unreachable code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: Simply example APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609021605
This pull request **introduces 3 alerts** when merging ac1b51ac68a005e0877c104b06bc04484a2de8b7 into 182007508a904084bf551bfef9dbd8aa55a46d3b - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-f43e9a080505e57618867c97640b8fdeff1f9506)
**new alerts:**
* 2 for Unreachable code
* 1 for 'import \*' may pollute namespace
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #651:
Simply example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #651: Simply example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403481122
##########
File path: examples/autograd/cifar10_multiprocess.py
##########
@@ -17,28 +17,16 @@
# under the License.
#
-from singa import opt
from resnet_cifar10 import *
import multiprocessing
-
-def data_partition(dataset_x, dataset_y, rank_in_global, world_size):
- data_per_rank = dataset_x.shape[0] // world_size
- idx_start = rank_in_global * data_per_rank
- idx_end = (rank_in_global + 1) * data_per_rank
- return dataset_x[idx_start:idx_end], dataset_y[idx_start:idx_end]
-
-
if __name__ == '__main__':
# Generate a NCCL ID to be used for collective communication
nccl_id = singa.NcclIdHolder()
- sgd = opt.SGD(lr=0.005, momentum=0.9, weight_decay=1e-5)
-
- gpu_per_node = 4
- max_epoch = 100
- batch_size = 32
+ # number of GPUs to be used
+ gpu_per_node = 2
Review comment:
changed gpu_per_node to num_gpus
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris edited a comment on issue #651: [WIP ]Simply
example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris edited a comment on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609049842
> I suggest to merge the examples under `examples/autograd` into the follow structure.
>
> ```
> autograd
> train.py
> train_mpi.py
> train_multiprocess.py
> data # define the data loading and preprocessing
> cifar10.py
> mnist.py
> model # define the model
> cnn.py
> resnet.py
> xception.py
> ```
>
> The pseudo code of each file:
>
> ```python
> # train.py
> def run(max_epoch, worker_id, num_workers, model, data, sgd):
> if model == 'resnet':
> from model import resnet
> model = resnet.create_model()
> elif model == 'cnn':
> model = cnn.create_model()
> ....
> if data == 'cifar10':
> from data import cifar10:
> train_x, train_y, val_x, val_y = cifar10.load()
> elif data == 'mnist':
> ....
>
> train_x, train_y, val_x, val_y = partition(worker_id, num_workers, train_x, train_y, val_x, val_y)
>
> # bp and sgd
>
> if __name__ == '__main__':
> # use argparse to get command config: max_epoch, model, data, etc. for single gpu training
> sgd = # create sgd
> run(0, 1, ..., sgd)
>
> # train_mpi.py
> if __name__ == '__main__':
> # use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
> sgd = # create sgd
> dist_sgd = DistOpt(sgd...)
> run(dist_sgd.rank, dist_sgd.world_size, ... dist_sgd)
>
> # train_multiprocess.py
> def run(rank, num_gpu, ...):
> sgd = ...
> dist_sgd = DistOpt(sgd)
> train.run(rank, num_gpu, ... dist_sgd)
>
> if __name__ == '__main__':
> # use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
> nccl_id = ...
> process = []
> for worker_id in range(0, gpu_per_node):
> process.append(multiprocessing.Process(target=run, args=(worker_id, ... nccl_id)))
>
> for p in process:
> p.start()
> ```
thanks! I am working on it. There may be three issues that need to think about:
(i) maybe difficult to include graph module (please see resnet_module.py to see the different between layer model and module model). In this case, we may consider retain resnet_module.py, cnn_module.py, mlp_module.py indpendently, or to make another run.py function specially for graph.
(ii) don't know how to show this example in the doc dist-train.md. So maybe in the doc we can still use doc_dist_train.py for examples
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609763150
This pull request **introduces 2 alerts** when merging 1b3be23bc9c39d641cbf4b235c16beef56bae6a2 into 793dcace41eaf3569c8222e0f4ca667cb9ecf412 - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-f7ecac4367a2592cc3b81b98c2ac7c04a7e1a970)
**new alerts:**
* 2 for Unreachable code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609592626
This pull request **introduces 2 alerts** when merging 615e6b5e76d473711febcb961c03d1c07d75d204 into ec5f8b223d052b8eadd512bf193166c0270b31c8 - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-903913d919b912a38953212cc27196470a60b838)
**new alerts:**
* 2 for Unreachable code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: [WIP]Add new example
APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: [WIP]Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609864659
now it support mlp also:
```
root@71ac539cda77:~/dcsysh/singa/examples/autograd# python3 train.py mlp mnist
Starting Epoch 0:
Training loss = 485.473358, training accuracy = 0.851771
Evaluation accuracy = 0.911358, Elapsed Time = 0.615843s
Starting Epoch 1:
Training loss = 267.414551, training accuracy = 0.918657
Evaluation accuracy = 0.928986, Elapsed Time = 0.623670s
Starting Epoch 2:
Training loss = 219.956863, training accuracy = 0.932998
Evaluation accuracy = 0.937400, Elapsed Time = 0.618422s
Starting Epoch 3:
Training loss = 188.373474, training accuracy = 0.942519
Evaluation accuracy = 0.946715, Elapsed Time = 0.617576s
Starting Epoch 4:
Training loss = 164.958145, training accuracy = 0.949223
Evaluation accuracy = 0.951723, Elapsed Time = 0.618342s
Starting Epoch 5:
Training loss = 146.759750, training accuracy = 0.954442
Evaluation accuracy = 0.956330, Elapsed Time = 0.619707s
Starting Epoch 6:
Training loss = 132.402222, training accuracy = 0.959178
Evaluation accuracy = 0.958534, Elapsed Time = 0.620100s
Starting Epoch 7:
Training loss = 120.471268, training accuracy = 0.963114
Evaluation accuracy = 0.961639, Elapsed Time = 0.620941s
Starting Epoch 8:
Training loss = 110.338516, training accuracy = 0.966832
Evaluation accuracy = 0.963642, Elapsed Time = 0.619157s
Starting Epoch 9:
Training loss = 101.753929, training accuracy = 0.968450
Evaluation accuracy = 0.966947, Elapsed Time = 0.619542s
root@71ac539cda77:~/dcsysh/singa/examples/autograd# python3 train.py mlp cifar10
Starting Epoch 0:
Training loss = 1560.984497, training accuracy = 0.323504
Evaluation accuracy = 0.354868, Elapsed Time = 0.689808s
Starting Epoch 1:
Training loss = 1337.684814, training accuracy = 0.396327
Evaluation accuracy = 0.398037, Elapsed Time = 0.693844s
Starting Epoch 2:
Training loss = 1263.678955, training accuracy = 0.430498
Evaluation accuracy = 0.412760, Elapsed Time = 0.688408s
Starting Epoch 3:
Training loss = 1214.546997, training accuracy = 0.453585
Evaluation accuracy = 0.427885, Elapsed Time = 0.686060s
Starting Epoch 4:
Training loss = 1174.774658, training accuracy = 0.469310
Evaluation accuracy = 0.441106, Elapsed Time = 0.688051s
Starting Epoch 5:
Training loss = 1142.419312, training accuracy = 0.483395
Evaluation accuracy = 0.437200, Elapsed Time = 0.688847s
Starting Epoch 6:
Training loss = 1112.198486, training accuracy = 0.497759
Evaluation accuracy = 0.430889, Elapsed Time = 0.685957s
Starting Epoch 7:
Training loss = 1082.584961, training accuracy = 0.510563
Evaluation accuracy = 0.447516, Elapsed Time = 0.685680s
Starting Epoch 8:
Training loss = 1060.778320, training accuracy = 0.521627
Evaluation accuracy = 0.453125, Elapsed Time = 0.686706s
Starting Epoch 9:
Training loss = 1039.045410, training accuracy = 0.529289
Evaluation accuracy = 0.449920, Elapsed Time = 0.687589s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609866901
I think this PR is ready for merge, Rulin can submit another PR to fix xceptionnet
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609184415
I am restructing all the examples and APIs in autograd example folder,
for the first commit I have created the model and data folders
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609266367
> since graph version is more efficient, I think it is fine to run the examples using the graph.
> we can still keep the current version in another folder
>
> * imperative (for the current)
> * declarative (for graph based)
>
> we can do the graph based first and if no time, then just ignore the imperative version.
OK, we can provide both, but I focus on graph based first
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #651:
Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403807239
##########
File path: src/api/dist_communicator.i
##########
@@ -44,7 +44,7 @@ public:
int totalMPIRanksInGlobal;
int MPIRankInLocal;
Communicator(int limit);
- Communicator(int gpu_num, int gpu_per_node, const NcclIdHolder &holder, int limit);
+ Communicator(int gpu_num, int num_gpus, const NcclIdHolder &holder, int limit);
Review comment:
how about I change all the code including python and C++ to <rank, local_rank, world_size> ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #651:
Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403820643
##########
File path: examples/autograd/train.py
##########
@@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from singa import singa_wrap as singa
+from singa import opt
+from singa import device
+from singa import tensor
+import numpy as np
+import time
+import argparse
+from PIL import Image
+
+
+# Data Augmentation
+def augmentation(x, batch_size):
+ xpad = np.pad(x, [[0, 0], [0, 0], [4, 4], [4, 4]], 'symmetric')
+ for data_num in range(0, batch_size):
+ offset = np.random.randint(8, size=2)
+ x[data_num, :, :, :] = xpad[data_num, :,
+ offset[0]:offset[0] + x.shape[2],
+ offset[1]:offset[1] + x.shape[2]]
+ if_flip = np.random.randint(2)
+ if (if_flip):
+ x[data_num, :, :, :] = x[data_num, :, :, ::-1]
+ return x
+
+
+# Calculate Accuracy
+def accuracy(pred, target):
+ # y is binary array for ground truth
+ y = np.argmax(pred, axis=1)
+ # t is integer for ground truth, the dim of t is (batch, 1)
+ t = np.max(target, axis=1)
+ a = y == t
+ correct = np.array(a, "int").sum()
+ # print(correct)
+ return correct
+
+
+# Data partition according to the rank
+def partition(worker_id, num_workers, train_x, train_y, val_x, val_y):
+ # Partition training data
+ data_per_rank = train_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ train_x = train_x[idx_start:idx_end]
+ train_y = train_y[idx_start:idx_end]
+ # Partition evaluation data
+ data_per_rank = val_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ val_x = val_x[idx_start:idx_end]
+ val_y = val_y[idx_start:idx_end]
+ return train_x, train_y, val_x, val_y
+
+
+# Function to all reduce NUMPY Accuracy and Loss from Multiple Devices
+def reduce_variable(variable, dist_opt, reducer):
+ reducer.copy_from_numpy(variable)
+ dist_opt.all_reduce(reducer.data)
+ dist_opt.wait()
+ output = tensor.to_numpy(reducer)
+ return output
+
+
+def resize_dataset(x, image_size):
+ num_data = x.shape[0]
+ dim = x.shape[1]
+ X = np.zeros(shape=(num_data, dim, image_size, image_size), dtype=np.float32)
+ for n in range(0, num_data):
+ for d in range(0, dim):
+ X[n, d, :, :] = np.array(Image.fromarray(x[n, d, :, :]).resize(
+ (image_size, image_size), Image.BILINEAR),
+ dtype=np.float32)
+ return X
+
+
+def run(worker_id, num_workers, device_id, max_epoch, batch_size, model, data,
+ sgd):
+ dev = device.create_cuda_gpu_on(device_id)
+ dev.SetRandSeed(0)
+ np.random.seed(0)
+
+ if model == 'resnet':
+ from model import resnet
+ model = resnet.resnet50()
+ elif model == 'xceptionnet':
+ from model import xceptionnet
+ model = xceptionnet.create_model()
+ elif model == 'cnn':
+ from model import cnn
+ model = cnn.create_model()
+
+ if data == 'cifar10':
+ from data import cifar10
+ train_x, train_y, val_x, val_y = cifar10.load()
+ elif data == 'mnist':
+ from data import mnist
+ train_x, train_y, val_x, val_y = mnist.load()
+
+ if hasattr(sgd, "communicator"):
+ DIST = True
+ else:
+ DIST = False
+
+ if DIST:
+ train_x, train_y, val_x, val_y = partition(worker_id, num_workers,
+ train_x, train_y, val_x,
+ val_y)
+ '''
+ # check dataset shape correctness
+ if worker_id == 0:
+ print("Check the shape of dataset:")
+ print(train_x.shape)
+ print(train_y.shape)
+ '''
+
+ num_channels = train_x.shape[1]
+ image_size = train_x.shape[2]
+
+ tx = tensor.Tensor(
+ (batch_size, num_channels, model.input_size, model.input_size), dev,
+ tensor.float32)
+ ty = tensor.Tensor((batch_size,), dev, tensor.int32)
+ num_train_batch = train_x.shape[0] // batch_size
+ num_val_batch = val_x.shape[0] // batch_size
+ idx = np.arange(train_x.shape[0], dtype=np.int32)
+
+ # attached model to graph
+ graph = True
+
+ # For distributed training, sequential gives better performance
+ if DIST:
Review comment:
yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #651: Add new
example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403952306
##########
File path: examples/autograd/train.py
##########
@@ -0,0 +1,261 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from singa import singa_wrap as singa
+from singa import opt
+from singa import device
+from singa import tensor
+import numpy as np
+import time
+import argparse
+from PIL import Image
+
+
+# Data Augmentation
+def augmentation(x, batch_size):
+ xpad = np.pad(x, [[0, 0], [0, 0], [4, 4], [4, 4]], 'symmetric')
+ for data_num in range(0, batch_size):
+ offset = np.random.randint(8, size=2)
+ x[data_num, :, :, :] = xpad[data_num, :,
+ offset[0]:offset[0] + x.shape[2],
+ offset[1]:offset[1] + x.shape[2]]
+ if_flip = np.random.randint(2)
+ if (if_flip):
+ x[data_num, :, :, :] = x[data_num, :, :, ::-1]
+ return x
+
+
+# Calculate Accuracy
+def accuracy(pred, target):
+ # y is binary array for ground truth
+ y = np.argmax(pred, axis=1)
+ # t is integer for ground truth, the dim of t is (batch, 1)
+ t = np.max(target, axis=1)
+ a = y == t
+ correct = np.array(a, "int").sum()
+ # print(correct)
+ return correct
+
+
+# Data partition according to the rank
+def partition(global_rank, world_size, train_x, train_y, val_x, val_y):
+ # Partition training data
+ data_per_rank = train_x.shape[0] // world_size
+ idx_start = global_rank * data_per_rank
+ idx_end = (global_rank + 1) * data_per_rank
+ train_x = train_x[idx_start:idx_end]
+ train_y = train_y[idx_start:idx_end]
+ # Partition evaluation data
+ data_per_rank = val_x.shape[0] // world_size
+ idx_start = global_rank * data_per_rank
+ idx_end = (global_rank + 1) * data_per_rank
+ val_x = val_x[idx_start:idx_end]
+ val_y = val_y[idx_start:idx_end]
+ return train_x, train_y, val_x, val_y
+
+
+# Function to all reduce NUMPY Accuracy and Loss from Multiple Devices
+def reduce_variable(variable, dist_opt, reducer):
+ reducer.copy_from_numpy(variable)
+ dist_opt.all_reduce(reducer.data)
+ dist_opt.wait()
+ output = tensor.to_numpy(reducer)
+ return output
+
+
+def resize_dataset(x, image_size):
+ num_data = x.shape[0]
+ dim = x.shape[1]
+ X = np.zeros(shape=(num_data, dim, image_size, image_size), dtype=np.float32)
+ for n in range(0, num_data):
+ for d in range(0, dim):
+ X[n, d, :, :] = np.array(Image.fromarray(x[n, d, :, :]).resize(
+ (image_size, image_size), Image.BILINEAR),
+ dtype=np.float32)
+ return X
+
+
+def run(global_rank, world_size, local_rank, max_epoch, batch_size, model, data,
+ sgd):
+ dev = device.create_cuda_gpu_on(local_rank)
+ dev.SetRandSeed(0)
+ np.random.seed(0)
+
+ if data == 'cifar10':
+ from data import cifar10
+ train_x, train_y, val_x, val_y = cifar10.load()
+ elif data == 'mnist':
+ from data import mnist
+ train_x, train_y, val_x, val_y = mnist.load()
+
+ num_channels = train_x.shape[1]
+ image_size = train_x.shape[2]
+
+ print(num_channels)
+
+ if model == 'resnet':
+ from model import resnet
+ model = resnet.resnet50(num_channels=num_channels)
+ elif model == 'xceptionnet':
+ from model import xceptionnet
+ model = xceptionnet.create_model(num_channels=num_channels)
+ elif model == 'cnn':
+ from model import cnn
+ model = cnn.create_model(num_channels=num_channels)
+
+ # For distributed training, sequential gives better performance
+ if hasattr(sgd, "communicator"):
+ DIST = True
+ sequential = True
+ else:
+ DIST = False
+ sequential = False
+
+ if DIST:
+ train_x, train_y, val_x, val_y = partition(global_rank, world_size,
+ train_x, train_y, val_x,
+ val_y)
+ '''
+ # check dataset shape correctness
+ if global_rank == 0:
+ print("Check the shape of dataset:")
+ print(train_x.shape)
+ print(train_y.shape)
+ '''
+
+ tx = tensor.Tensor(
+ (batch_size, num_channels, model.input_size, model.input_size), dev,
+ tensor.float32)
+ ty = tensor.Tensor((batch_size,), dev, tensor.int32)
+ num_train_batch = train_x.shape[0] // batch_size
+ num_val_batch = val_x.shape[0] // batch_size
+ idx = np.arange(train_x.shape[0], dtype=np.int32)
+
+ # attached model to graph
+ model.on_device(dev)
+ model.set_optimizer(sgd)
+ model.graph(True, sequential)
+
+ # Training and Evaluation Loop
+ for epoch in range(max_epoch):
+ start_time = time.time()
+ np.random.shuffle(idx)
+
+ if global_rank == 0:
+ print('Starting Epoch %d:' % (epoch))
+
+ # Training Phase
+ train_correct = np.zeros(shape=[1], dtype=np.float32)
+ test_correct = np.zeros(shape=[1], dtype=np.float32)
+ train_loss = np.zeros(shape=[1], dtype=np.float32)
+
+ model.train()
+ for b in range(num_train_batch):
+ # Generate the patch data in this iteration
+ x = train_x[idx[b * batch_size:(b + 1) * batch_size]]
+ x = augmentation(x, batch_size)
+ if (image_size != model.input_size):
+ x = resize_dataset(x, model.input_size)
+ y = train_y[idx[b * batch_size:(b + 1) * batch_size]]
+
+ # Copy the patch data into input tensors
+ tx.copy_from_numpy(x)
+ ty.copy_from_numpy(y)
+
+ # Train the model
+ out = model(tx)
+ loss = model.loss(out, ty)
+ model.optim(loss)
+ train_correct += accuracy(tensor.to_numpy(out),y)
+ train_loss += tensor.to_numpy(loss)[0]
+
+ if DIST:
+ # Reduce the Evaluation Accuracy and Loss from Multiple Devices
+ reducer = tensor.Tensor((1,), dev, tensor.float32)
+ train_correct = reduce_variable(train_correct, sgd, reducer)
+ train_loss = reduce_variable(train_loss, sgd, reducer)
+
+ if global_rank == 0:
+ print('Training loss = %f, training accuracy = %f' %
+ (train_loss, train_correct /
+ (num_train_batch * batch_size * world_size)),
+ flush=True)
+
+ # Evaluation Phase
+ model.eval()
+ for b in range(num_val_batch):
+ x = val_x[b * batch_size:(b + 1) * batch_size]
+ if (image_size != model.input_size):
+ x = resize_dataset(x, model.input_size)
+ y = val_y[b * batch_size:(b + 1) * batch_size]
+ tx.copy_from_numpy(x)
+ ty.copy_from_numpy(y)
+ out_test = model(tx)
+ test_correct += accuracy(tensor.to_numpy(out_test),y)
+
+ if DIST:
+ # Reduce the Evaulation Accuracy from Multiple Devices
+ test_correct = reduce_variable(test_correct, sgd, reducer)
+
+ # Output the Evaluation Accuracy
+ if global_rank == 0:
+ print('Evaluation accuracy = %f, Elapsed Time = %fs' %
+ (test_correct / (num_val_batch * batch_size * world_size),
+ time.time() - start_time),
+ flush=True)
+
+
+if __name__ == '__main__':
+ # use argparse to get command config: max_epoch, model, data, etc. for single gpu training
+ parser = argparse.ArgumentParser(
+ description='Training using the autograd and graph.')
+ parser.add_argument('model',
+ choices=['resnet', 'xceptionnet', 'cnn'],
+ default='cnn')
+ parser.add_argument('data', choices=['cifar10', 'mnist'], default='mnist')
+ parser.add_argument('--epoch',
+ '--max-epoch',
+ default=10,
+ type=int,
+ help='maximum epochs',
+ dest='max_epoch')
+ parser.add_argument('--bs',
+ '--batch-size',
+ default=64,
+ type=int,
+ help='batch size',
+ dest='batch_size')
+ parser.add_argument('--lr',
+ '--learning-rate',
+ default=0.005,
+ type=float,
+ help='initial learning rate',
+ dest='lr')
+ # determine which gpu to use in gpu
+ parser.add_argument('--id',
Review comment:
the same for train_mpi and train_multiprocess.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609540962
> can we run (cnn|resnet|xceptionnet) + (cifar10|mnist) ?
> I think we can also add the mlp model?
> then we can remove resnet_module.py, mlp_module.py
There are 4 points, I guess the issue most difficult to solve is the no.1 below
1. xceptionnet has some problem, the running stopped after 1 epoch, maybe the model has some issues. And this model is very large I can only run batch size=16 (not 32)
2. The current mlp is 2d input, which is not CNN problem which has 4d input (batch, channel/RGB, height, width). Maybe need to change to 4d
3. The CNN has only one input channel, in order to support cifar10, maybe I need to adjust the number of input channel of CNN as variable
4. mnist is not RGB, so I has only one colour channel, if I used it for resnet I can see if:
(i) change the input channel of resnet to a variable so it support 1 channel
(ii) replicate one channel of mnist dataset to three
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609424548
This pull request **fixes 4 alerts** when merging 233d033abe04a29828a7268818bb4a82c990f514 into 5c4d3db57630098d04d214b6e7ea2b1f88fc84fd - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-8c32e9fa2d5e40bc56cbff9fab03f8b258e05fa9)
**fixed alerts:**
* 4 for 'import \*' may pollute namespace
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: [WIP]Add new example
APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: [WIP]Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609831174
adding mlp
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: Simply example APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609019865
This pull request **introduces 4 alerts** when merging 21310302a5cd64f98b4bf2b6ec5b0e8c093b8886 into 182007508a904084bf551bfef9dbd8aa55a46d3b - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-36388584600ecd20b5c5a29cae0d4b7aa396f562)
**new alerts:**
* 2 for Unreachable code
* 1 for Unused import
* 1 for 'import \*' may pollute namespace
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] lgtm-com[bot] commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609441889
This pull request **introduces 2 alerts** when merging 1762a07c245d9924b32711c7bcb0301b585521b3 into ec5f8b223d052b8eadd512bf193166c0270b31c8 - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-319a2b39c262c79931d7e2f3793ebf03b8a7c71d)
**new alerts:**
* 2 for Unreachable code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #651: Add new
example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403806157
##########
File path: src/api/dist_communicator.i
##########
@@ -44,7 +44,7 @@ public:
int totalMPIRanksInGlobal;
int MPIRankInLocal;
Communicator(int limit);
- Communicator(int gpu_num, int gpu_per_node, const NcclIdHolder &holder, int limit);
+ Communicator(int gpu_num, int num_gpus, const NcclIdHolder &holder, int limit);
Review comment:
I think we can unify the terms, e.g., using <worker_id, num_workers> or <rank, local_rank and world_size>
the first set is for distributed training; the second set is for communication. maybe the second set is better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris edited a comment on issue #651: [WIP ]Simply
example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris edited a comment on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609049842
> I suggest to merge the examples under `examples/autograd` into the follow structure.
>
> ```
> autograd
> train.py
> train_mpi.py
> train_multiprocess.py
> data # define the data loading and preprocessing
> cifar10.py
> mnist.py
> model # define the model
> cnn.py
> resnet.py
> xception.py
> ```
>
> The pseudo code of each file:
>
> ```python
> # train.py
> def run(max_epoch, worker_id, num_workers, model, data, sgd):
> if model == 'resnet':
> from model import resnet
> model = resnet.create_model()
> elif model == 'cnn':
> model = cnn.create_model()
> ....
> if data == 'cifar10':
> from data import cifar10:
> train_x, train_y, val_x, val_y = cifar10.load()
> elif data == 'mnist':
> ....
>
> train_x, train_y, val_x, val_y = partition(worker_id, num_workers, train_x, train_y, val_x, val_y)
>
> # bp and sgd
>
> if __name__ == '__main__':
> # use argparse to get command config: max_epoch, model, data, etc. for single gpu training
> sgd = # create sgd
> run(0, 1, ..., sgd)
>
> # train_mpi.py
> if __name__ == '__main__':
> # use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
> sgd = # create sgd
> dist_sgd = DistOpt(sgd...)
> run(dist_sgd.rank, dist_sgd.world_size, ... dist_sgd)
>
> # train_multiprocess.py
> def run(rank, num_gpu, ...):
> sgd = ...
> dist_sgd = DistOpt(sgd)
> train.run(rank, num_gpu, ... dist_sgd)
>
> if __name__ == '__main__':
> # use argparse to get command args: max_epoch, model, data, etc. for multi-gpu training
> nccl_id = ...
> process = []
> for worker_id in range(0, gpu_per_node):
> process.append(multiprocessing.Process(target=run, args=(worker_id, ... nccl_id)))
>
> for p in process:
> p.start()
> ```
thanks! I will work on it. Since seems a lot of work, to speed up may need Rulin to help
Seems the graph module cnn_module.py and resnet_module.py is not included, not sure how to include the graph module because the graph model and unbuffered model are different
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-610168094
now it supports cifar 100 as well, we can train (mlp|cnn|resnet) + (cifar10|cifar100|mnist)
```
root@71ac539cda77:~/dcsysh/singa/examples/autograd# mpiexec -np 2 python3 train_mpi.py resnet cifar100 --lr 0.1
Starting Epoch 0:
Training loss = 3123.622803, training accuracy = 0.094952
Evaluation accuracy = 0.151743, Elapsed Time = 71.773229s
Starting Epoch 1:
Training loss = 2569.560547, training accuracy = 0.197817
Evaluation accuracy = 0.273538, Elapsed Time = 72.339331s
Starting Epoch 2:
Training loss = 2142.290039, training accuracy = 0.299519
Evaluation accuracy = 0.367688, Elapsed Time = 72.516621s
Starting Epoch 3:
Training loss = 1766.168091, training accuracy = 0.395593
Evaluation accuracy = 0.455929, Elapsed Time = 72.540777s
Starting Epoch 4:
Training loss = 1488.601074, training accuracy = 0.478446
Evaluation accuracy = 0.503806, Elapsed Time = 72.693376s
Starting Epoch 5:
Training loss = 1302.042725, training accuracy = 0.535337
Evaluation accuracy = 0.536859, Elapsed Time = 72.012540s
Starting Epoch 6:
Training loss = 1148.359131, training accuracy = 0.583213
Evaluation accuracy = 0.583133, Elapsed Time = 72.194807s
Starting Epoch 7:
Training loss = 1035.075928, training accuracy = 0.620353
Evaluation accuracy = 0.606370, Elapsed Time = 72.698941s
Starting Epoch 8:
Training loss = 924.950745, training accuracy = 0.654067
Evaluation accuracy = 0.621194, Elapsed Time = 72.009050s
Starting Epoch 9:
Training loss = 834.806763, training accuracy = 0.685857
Evaluation accuracy = 0.643129, Elapsed Time = 72.860034s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: [WIP ]Simply example
APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609246334
> we can use resnet_module.py to replace resnet.py?
yes, should be okay, in this case maybe run everything with graph
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609601411
It is now able to support (cnn|resnet) + (cifar10|mnist)
```
root@877a3759b148:~/dcsysh/singa/examples/autograd# python3 train.py cnn cifar10
3
Starting Epoch 0:
Training loss = 1334.210571, training accuracy = 0.383543
Evaluation accuracy = 0.497997, Elapsed Time = 17.269876s
Starting Epoch 1:
Training loss = 1080.154541, training accuracy = 0.507542
Evaluation accuracy = 0.568810, Elapsed Time = 17.324370s
Starting Epoch 2:
Training loss = 974.208618, training accuracy = 0.555598
Evaluation accuracy = 0.612680, Elapsed Time = 17.230355s
Starting Epoch 3:
Training loss = 907.781128, training accuracy = 0.585147
Evaluation accuracy = 0.625901, Elapsed Time = 17.264113s
Starting Epoch 4:
Training loss = 858.995056, training accuracy = 0.609435
Evaluation accuracy = 0.650841, Elapsed Time = 17.296929s
Starting Epoch 5:
Training loss = 819.569641, training accuracy = 0.630642
Evaluation accuracy = 0.663862, Elapsed Time = 17.185191s
Starting Epoch 6:
Training loss = 791.844910, training accuracy = 0.643606
Evaluation accuracy = 0.681991, Elapsed Time = 17.057992s
Starting Epoch 7:
Training loss = 763.051758, training accuracy = 0.657690
Evaluation accuracy = 0.688301, Elapsed Time = 16.876048s
Starting Epoch 8:
Training loss = 735.888611, training accuracy = 0.671195
Evaluation accuracy = 0.686999, Elapsed Time = 16.930443s
Starting Epoch 9:
Training loss = 715.715027, training accuracy = 0.680778
Evaluation accuracy = 0.704928, Elapsed Time = 16.901450s
root@877a3759b148:~/dcsysh/singa/examples/autograd# python3 train.py resnet mnist --bs 32
Starting Epoch 0:
Training loss = 682.914490, training accuracy = 0.892883
Evaluation accuracy = 0.958233, Elapsed Time = 302.701322s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609610122
>
>
> xceptionnet error
> 
what's the size of the input to the model?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #651:
Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403807759
##########
File path: examples/autograd/train.py
##########
@@ -0,0 +1,257 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from singa import singa_wrap as singa
+from singa import opt
+from singa import device
+from singa import tensor
+import numpy as np
+import time
+import argparse
+from PIL import Image
+
+
+# Data Augmentation
+def augmentation(x, batch_size):
+ xpad = np.pad(x, [[0, 0], [0, 0], [4, 4], [4, 4]], 'symmetric')
+ for data_num in range(0, batch_size):
+ offset = np.random.randint(8, size=2)
+ x[data_num, :, :, :] = xpad[data_num, :,
+ offset[0]:offset[0] + x.shape[2],
+ offset[1]:offset[1] + x.shape[2]]
+ if_flip = np.random.randint(2)
+ if (if_flip):
+ x[data_num, :, :, :] = x[data_num, :, :, ::-1]
+ return x
+
+
+# Calculate Accuracy
+def accuracy(pred, target):
+ # y is binary array for ground truth
+ y = np.argmax(pred, axis=1)
+ # t is integer for ground truth, the dim of t is (batch, 1)
+ t = np.max(target, axis=1)
+ a = y == t
+ correct = np.array(a, "int").sum()
+ # print(correct)
+ return correct
+
+
+# Data partition according to the rank
+def partition(worker_id, num_workers, train_x, train_y, val_x, val_y):
+ # Partition training data
+ data_per_rank = train_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ train_x = train_x[idx_start:idx_end]
+ train_y = train_y[idx_start:idx_end]
+ # Partition evaluation data
+ data_per_rank = val_x.shape[0] // num_workers
+ idx_start = worker_id * data_per_rank
+ idx_end = (worker_id + 1) * data_per_rank
+ val_x = val_x[idx_start:idx_end]
+ val_y = val_y[idx_start:idx_end]
+ return train_x, train_y, val_x, val_y
+
+
+# Function to all reduce NUMPY Accuracy and Loss from Multiple Devices
+def reduce_variable(variable, dist_opt, reducer):
+ reducer.copy_from_numpy(variable)
+ dist_opt.all_reduce(reducer.data)
+ dist_opt.wait()
+ output = tensor.to_numpy(reducer)
+ return output
+
+
+def resize_dataset(x, image_size):
+ num_data = x.shape[0]
+ dim = x.shape[1]
+ X = np.zeros(shape=(num_data, dim, image_size, image_size), dtype=np.float32)
+ for n in range(0, num_data):
+ for d in range(0, dim):
+ X[n, d, :, :] = np.array(Image.fromarray(x[n, d, :, :]).resize(
+ (image_size, image_size), Image.BILINEAR),
+ dtype=np.float32)
+ return X
+
+
+def run(worker_id, num_workers, device_id, max_epoch, batch_size, model, data,
+ sgd):
+ dev = device.create_cuda_gpu_on(device_id)
+ dev.SetRandSeed(0)
+ np.random.seed(0)
+
+ if model == 'resnet':
+ from model import resnet
+ model = resnet.resnet50()
+ elif model == 'xceptionnet':
+ from model import xceptionnet
+ model = xceptionnet.create_model()
+ elif model == 'cnn':
+ from model import cnn
+ model = cnn.create_model()
+
+ if data == 'cifar10':
+ from data import cifar10
+ train_x, train_y, val_x, val_y = cifar10.load()
+ elif data == 'mnist':
+ from data import mnist
+ train_x, train_y, val_x, val_y = mnist.load()
+
+ if hasattr(sgd, "communicator"):
+ DIST = True
+ else:
+ DIST = False
+
+ if DIST:
+ train_x, train_y, val_x, val_y = partition(worker_id, num_workers,
+ train_x, train_y, val_x,
+ val_y)
+ '''
+ # check dataset shape correctness
+ if worker_id == 0:
+ print("Check the shape of dataset:")
+ print(train_x.shape)
+ print(train_y.shape)
+ '''
+
+ num_channels = train_x.shape[1]
+ image_size = train_x.shape[2]
+
+ tx = tensor.Tensor(
+ (batch_size, num_channels, model.input_size, model.input_size), dev,
+ tensor.float32)
+ ty = tensor.Tensor((batch_size,), dev, tensor.int32)
+ num_train_batch = train_x.shape[0] // batch_size
+ num_val_batch = val_x.shape[0] // batch_size
+ idx = np.arange(train_x.shape[0], dtype=np.int32)
+
+ # attached model to graph
+ graph = True
Review comment:
Yes, I see
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #651: Simply
example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #651: Simply example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403476442
##########
File path: examples/autograd/doc_dist_train.py
##########
@@ -0,0 +1,254 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
Review comment:
the same as mnist_cnn.py?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609637262
> can the original model run (not using the Module API)?
Yes, I have turned the graph off (GRAPH=FALSE) and running, seems is okay I have just finished 2 rpochs, I will wait for more result. I think the different of resnet and xceptionnet is that it has multiple branch
```
(singa) dcsysh@panda7:~/singa/examples/autograd$ python3 train.py xceptionnet ci
Starting Epoch 0:
Training loss = 11198.645508, training accuracy = 0.214420
Evaluation accuracy = 0.309000, Elapsed Time = 606.547117s
Starting Epoch 1:
Training loss = 6354.611328, training accuracy = 0.381020
Evaluation accuracy = 0.457300, Elapsed Time = 612.817129s
Starting Epoch 2:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #651: Add new example APIs
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #651: Add new example APIs
URL: https://github.com/apache/singa/pull/651#issuecomment-609562016
xceptionnet error
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #651: [WIP
]Simply example APIs
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #651: [WIP ]Simply example APIs
URL: https://github.com/apache/singa/pull/651#discussion_r403642255
##########
File path: examples/autograd/doc_dist_train.py
##########
@@ -0,0 +1,254 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
Review comment:
I think there is no need to create an extra doc_dist_train.py.
In the documentation, we just show some code snippet instead of the full example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services