You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@singa.apache.org by GitBox <gi...@apache.org> on 2020/06/10 04:28:44 UTC
[GitHub] [singa] chrishkchris opened a new pull request #728: Fix create_cuda_gpu
chrishkchris opened a new pull request #728:
URL: https://github.com/apache/singa/pull/728
It fixes the problem that the original function could occupy multiple GPUs memory
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] XJDKC commented on pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
XJDKC commented on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-647857959
No, we just use the cop device directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on a change in pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #728:
URL: https://github.com/apache/singa/pull/728#discussion_r438196529
##########
File path: python/singa/device.py
##########
@@ -106,17 +106,21 @@ def create_cuda_gpus(num):
return singa.Platform.CreateCudaGPUs(num)
-def create_cuda_gpu(set_default=False):
+Device.default_gpu_device = None
Review comment:
Thanks! I have added the `global default_gpu_device` and it is working now. I have combined three commits into one in this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] XJDKC commented on pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
XJDKC commented on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-648847004
> I have deleted the python device, thanks
Thanks, Chris.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on a change in pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #728:
URL: https://github.com/apache/singa/pull/728#discussion_r438166926
##########
File path: python/singa/device.py
##########
@@ -106,17 +106,21 @@ def create_cuda_gpus(num):
return singa.Platform.CreateCudaGPUs(num)
-def create_cuda_gpu(set_default=False):
+Device.default_gpu_device = None
Review comment:
When I change from `Device.default_gpu_device` to `default_gpu_device`
```
default_gpu_device = None
def create_cuda_gpu():
'''Create a single CudaGPU device.
Returns:
a swig converted CudaGPU device.
'''
assert singa.USE_CUDA, 'SINGA has not been compiled with CUDA enabled.'
if default_gpu_device is None:
default_gpu_device = create_cuda_gpu_on(0)
return default_gpu_device
```
When I run the script `simple_test_device.py` to create device
```
from singa import device
if __name__ == '__main__':
dev=device.create_cuda_gpu()
```
It returns:
```
root@d05828f767ee:~/dcsysh/singa/examples/cnn# python3 simple_test_device.py
Traceback (most recent call last):
File "test_device.py", line 4, in <module>
dev=device.create_cuda_gpu()
File "/root/dcsysh/singa/build/python/singa/device.py", line 120, in create_cuda_gpu
if default_gpu_device is None:
UnboundLocalError: local variable 'default_gpu_device' referenced before assignment
```
Seems if I don't store the variable `default_gpu_device` in a class, the variable `default_gpu_device` is lost
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] nudles commented on a change in pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #728:
URL: https://github.com/apache/singa/pull/728#discussion_r438182223
##########
File path: python/singa/device.py
##########
@@ -106,17 +106,21 @@ def create_cuda_gpus(num):
return singa.Platform.CreateCudaGPUs(num)
-def create_cuda_gpu(set_default=False):
+Device.default_gpu_device = None
Review comment:
add
`global default_gpu_device` inside the function `create_cuda_gpu():`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on pull request #728: Fix create_cuda_gpu and linting error
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-644748994
> this pr is not ready for merge?
> I think the 3 issues are not resolved yet.
Not yet ready for merge. The last commit was to rebase to dev branch. Still working towards it, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-648846427
I have deleted the python device, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] XJDKC commented on pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
XJDKC commented on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-648886213
I have tested training cnn twice on the same device and found no problems.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on a change in pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #728:
URL: https://github.com/apache/singa/pull/728#discussion_r437997370
##########
File path: python/singa/device.py
##########
@@ -113,10 +113,8 @@ def create_cuda_gpu(set_default=False):
a swig converted CudaGPU device.
'''
assert singa.USE_CUDA, 'SINGA has not been compiled with CUDA enabled.'
- devices = singa.Platform.CreateCudaGPUs(1)
- if set_default:
- set_default_device(devices[0])
- return devices[0]
+ device = create_cuda_gpu_on(0, set_default)
Review comment:
I added the default_gpu_device
Moreover, I also removed some unused code `set_default_device(device)`, which was used to set default device for graph operation. It was used to prevent the buffering of `to_device` operation, which repeats the `to_device` every iteration. However, since the new layer API do the initialization separately, the `set_default_device(device)` is no longer necessary. I have tested the resnet cifar10 the training is ok
```
root@56142bc34887:~/dcsysh/singa/examples/cnn# mpiexec -np 8 python3 train_mpi.py resnet cifar10 -b 32 -l 0.04
Starting Epoch 0:
Training loss = 3952.119385, training accuracy = 0.216567
Evaluation accuracy = 0.342648, Elapsed Time = 52.988312s
Starting Epoch 1:
Training loss = 2519.932373, training accuracy = 0.399439
Evaluation accuracy = 0.467849, Elapsed Time = 52.414376s
Starting Epoch 2:
Training loss = 2165.224854, training accuracy = 0.497937
Evaluation accuracy = 0.560998, Elapsed Time = 52.504168s
Starting Epoch 3:
Training loss = 1884.613525, training accuracy = 0.565605
Evaluation accuracy = 0.596755, Elapsed Time = 52.721652s
Starting Epoch 4:
Training loss = 1682.462158, training accuracy = 0.617188
Evaluation accuracy = 0.643429, Elapsed Time = 52.857880s
Starting Epoch 5:
Training loss = 1514.762329, training accuracy = 0.654888
Evaluation accuracy = 0.689002, Elapsed Time = 52.902534s
Starting Epoch 6:
Training loss = 1372.399536, training accuracy = 0.689283
Evaluation accuracy = 0.708434, Elapsed Time = 53.047120s
Starting Epoch 7:
Training loss = 1220.632446, training accuracy = 0.726362
Evaluation accuracy = 0.743389, Elapsed Time = 53.035040s
Starting Epoch 8:
Training loss = 1110.090942, training accuracy = 0.751182
Evaluation accuracy = 0.761919, Elapsed Time = 53.105462s
Starting Epoch 9:
Training loss = 1006.458618, training accuracy = 0.775160
Evaluation accuracy = 0.772436, Elapsed Time = 53.146286s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on a change in pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #728:
URL: https://github.com/apache/singa/pull/728#discussion_r437997370
##########
File path: python/singa/device.py
##########
@@ -113,10 +113,8 @@ def create_cuda_gpu(set_default=False):
a swig converted CudaGPU device.
'''
assert singa.USE_CUDA, 'SINGA has not been compiled with CUDA enabled.'
- devices = singa.Platform.CreateCudaGPUs(1)
- if set_default:
- set_default_device(devices[0])
- return devices[0]
+ device = create_cuda_gpu_on(0, set_default)
Review comment:
I added the default_gpu_device
Moreover, I also removed some unused code `set_default_device(device)`, which was used to set default device for graph operation. It was used to prevent the buffering of `to_device` operation, which repeats the `to_device` every iteration. However, since the new layer API do the initialization separately, the `set_default_device(device)` is no longer necessary.
I have tested the (i) cnn mnist, amd (ii) resnet cifar10, the training is ok
(i) CNN MNIST
```
root@56142bc34887:~/dcsysh/singa/examples/cnn# mpiexec -np 8 python3 train_mpi.py cnn mnist -l 0.04
Starting Epoch 0:
Training loss = 834.012695, training accuracy = 0.701389
Evaluation accuracy = 0.921258, Elapsed Time = 0.849777s
Starting Epoch 1:
Training loss = 262.023132, training accuracy = 0.910857
Evaluation accuracy = 0.963405, Elapsed Time = 0.775921s
Starting Epoch 2:
Training loss = 175.929840, training accuracy = 0.941256
Evaluation accuracy = 0.962068, Elapsed Time = 0.784108s
Starting Epoch 3:
Training loss = 138.893051, training accuracy = 0.953609
Evaluation accuracy = 0.971937, Elapsed Time = 0.796294s
Starting Epoch 4:
Training loss = 119.912170, training accuracy = 0.959602
Evaluation accuracy = 0.974609, Elapsed Time = 0.788538s
Starting Epoch 5:
Training loss = 106.647232, training accuracy = 0.964159
Evaluation accuracy = 0.976460, Elapsed Time = 0.791224s
Starting Epoch 6:
Training loss = 100.382210, training accuracy = 0.966763
Evaluation accuracy = 0.978516, Elapsed Time = 0.804543s
Starting Epoch 7:
Training loss = 86.349213, training accuracy = 0.971137
Evaluation accuracy = 0.976562, Elapsed Time = 0.798764s
Starting Epoch 8:
Training loss = 82.797058, training accuracy = 0.972055
Evaluation accuracy = 0.982833, Elapsed Time = 0.746563s
Starting Epoch 9:
Training loss = 77.220978, training accuracy = 0.974376
Evaluation accuracy = 0.977796, Elapsed Time = 0.738852s
```
(ii) RESNET CIFAR10
```
root@56142bc34887:~/dcsysh/singa/examples/cnn# mpiexec -np 8 python3 train_mpi.py resnet cifar10 -b 32 -l 0.04
Starting Epoch 0:
Training loss = 3952.119385, training accuracy = 0.216567
Evaluation accuracy = 0.342648, Elapsed Time = 52.988312s
Starting Epoch 1:
Training loss = 2519.932373, training accuracy = 0.399439
Evaluation accuracy = 0.467849, Elapsed Time = 52.414376s
Starting Epoch 2:
Training loss = 2165.224854, training accuracy = 0.497937
Evaluation accuracy = 0.560998, Elapsed Time = 52.504168s
Starting Epoch 3:
Training loss = 1884.613525, training accuracy = 0.565605
Evaluation accuracy = 0.596755, Elapsed Time = 52.721652s
Starting Epoch 4:
Training loss = 1682.462158, training accuracy = 0.617188
Evaluation accuracy = 0.643429, Elapsed Time = 52.857880s
Starting Epoch 5:
Training loss = 1514.762329, training accuracy = 0.654888
Evaluation accuracy = 0.689002, Elapsed Time = 52.902534s
Starting Epoch 6:
Training loss = 1372.399536, training accuracy = 0.689283
Evaluation accuracy = 0.708434, Elapsed Time = 53.047120s
Starting Epoch 7:
Training loss = 1220.632446, training accuracy = 0.726362
Evaluation accuracy = 0.743389, Elapsed Time = 53.035040s
Starting Epoch 8:
Training loss = 1110.090942, training accuracy = 0.751182
Evaluation accuracy = 0.761919, Elapsed Time = 53.105462s
Starting Epoch 9:
Training loss = 1006.458618, training accuracy = 0.775160
Evaluation accuracy = 0.772436, Elapsed Time = 53.146286s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] XJDKC edited a comment on pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
XJDKC edited a comment on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-647857959
No, we just use the cpp device directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-648889131
ready for merge, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-648883912
Some test after delete unused code:
```
root@71ac539cda77:~/dcsysh/singa/examples/cnn# python3 train_cnn.py cnn mnist
Starting Epoch 0:
Training loss = 564.004028, training accuracy = 0.801411
Evaluation accuracy = 0.936198, Elapsed Time = 3.381418s
Starting Epoch 1:
Training loss = 230.022369, training accuracy = 0.923159
Evaluation accuracy = 0.960537, Elapsed Time = 3.394230s
Starting Epoch 2:
Training loss = 162.285233, training accuracy = 0.946088
Evaluation accuracy = 0.969551, Elapsed Time = 3.409970s
Starting Epoch 3:
Training loss = 136.865356, training accuracy = 0.953859
Evaluation accuracy = 0.972356, Elapsed Time = 3.413537s
Starting Epoch 4:
Training loss = 117.966805, training accuracy = 0.960245
Evaluation accuracy = 0.974659, Elapsed Time = 3.394174s
Starting Epoch 5:
Training loss = 104.174706, training accuracy = 0.965115
Evaluation accuracy = 0.978466, Elapsed Time = 3.368983s
Starting Epoch 6:
Training loss = 93.883789, training accuracy = 0.968783
Evaluation accuracy = 0.981370, Elapsed Time = 3.387374s
Starting Epoch 7:
Training loss = 87.536545, training accuracy = 0.970468
Evaluation accuracy = 0.977364, Elapsed Time = 3.380798s
Starting Epoch 8:
Training loss = 80.650063, training accuracy = 0.972502
Evaluation accuracy = 0.979968, Elapsed Time = 3.378383s
Starting Epoch 9:
Training loss = 77.312569, training accuracy = 0.974086
Evaluation accuracy = 0.981270, Elapsed Time = 3.379956s
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] XJDKC commented on pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
XJDKC commented on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-647435613
Ready to merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] nudles commented on a change in pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #728:
URL: https://github.com/apache/singa/pull/728#discussion_r438144122
##########
File path: python/singa/device.py
##########
@@ -106,17 +106,21 @@ def create_cuda_gpus(num):
return singa.Platform.CreateCudaGPUs(num)
-def create_cuda_gpu(set_default=False):
+Device.default_gpu_device = None
Review comment:
in fact, I think we are not using the Device class?
shall we just remove it and make default_gpu_device an attribute of the device module?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] nudles commented on pull request #728: Fix create_cuda_gpu and linting error
Posted by GitBox <gi...@apache.org>.
nudles commented on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-644744892
this pr is not ready for merge?
I think the 3 issues are not resolved yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] lgtm-com[bot] commented on pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-649059113
This pull request **introduces 1 alert** when merging 21be0caf1d6e679d62c94d99319eabcb88d2f92e into 4fa7ea0efe04f37c4bb164e094dc10376ca70eba - [view on LGTM.com](https://lgtm.com/projects/g/apache/singa/rev/pr-9dca5f84dc1c8091daba6018f1bb94200c00e217)
**new alerts:**
* 1 for Unused import
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] nudles commented on pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
nudles commented on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-647856948
Are we using https://github.com/apache/singa/blob/master/python/singa/device.py#L29 this class?
It seems we are using the cpp device class exported by swig directly?
Then there is no need to have this py class?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] nudles commented on pull request #728: Fix create_cuda_gpu and linting error
Posted by GitBox <gi...@apache.org>.
nudles commented on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-643555875
How about this solution:
1. one gpu --- one device instance --- one graph.
if we want to reuse the device for another model, we have to reset the device instance to clear all existing data structures on the device (including both tensors and ops)
```cpp
// device.h
Class Device(){
void reset() ;
}
```
2. override the del function to call the reset function only.
3. create a list of device instances
```python
# device.py
global_device_list = {}
def create_gpu_device():
create the gpu device and put it into global_device_list or return global_device_list[0]
def create_gpu_on(idx):
create the gpu device and put it into global_device_list or return global_device_list[idx]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on pull request #728:
URL: https://github.com/apache/singa/pull/728#issuecomment-643140742
Originally, the CI linting test report "no-member" error on examples/cnn/train.py.
```
************* Module train_mpi
examples/cnn/train_mpi.py:82:4: E1101: Module 'train' has no 'run' member (no-member)
************* Module train_multiprocess
examples/cnn/train_multiprocess.py:30:4: E1101: Module 'train' has no 'run' member (no-member)
```
It think it is due to the repeat of name with examples/qabot/train.py. Therefore, I renamed train.py to train_cnn.py in order to pass the CI linting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] nudles merged pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
nudles merged pull request #728:
URL: https://github.com/apache/singa/pull/728
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] nudles commented on a change in pull request #728: Fix create_cuda_gpu
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #728:
URL: https://github.com/apache/singa/pull/728#discussion_r437931959
##########
File path: python/singa/device.py
##########
@@ -113,10 +113,8 @@ def create_cuda_gpu(set_default=False):
a swig converted CudaGPU device.
'''
assert singa.USE_CUDA, 'SINGA has not been compiled with CUDA enabled.'
- devices = singa.Platform.CreateCudaGPUs(1)
- if set_default:
- set_default_device(devices[0])
- return devices[0]
+ device = create_cuda_gpu_on(0, set_default)
Review comment:
I think we should create a default_gpu_device attribute for the device module
```python
default_gpu_device = None
def create_cuda_gpu():
assert ...
if default_gpu_device is None:
default_gpu_device = create_cuda_gpu_on(0)
return default_gpu_device
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org