You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@singa.apache.org by GitBox <gi...@apache.org> on 2020/05/22 17:20:39 UTC
[GitHub] [singa] chrishkchris opened a new pull request #708: SINGA-510 Improvement of Time Profiling Function
chrishkchris opened a new pull request #708:
URL: https://github.com/apache/singa/pull/708
There are several improvement in the time profiling function
1. Greatly reduced the measurement overhead: In the new version, we synchronize the cudastream only one time at the end of iteration. The previous version needs to synchronize cudastream whenever it measures each operation, so it introduced time overhead.
2. Can skip the initial iterations for time profiling. It can be set using the function SetSkipIteration, while the default is 5 iterations.
3. Modify the benchmark.py for resnet 50 benchmark time profiling.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on a change in pull request #708: SINGA-510 Improvement of Time Profiling Function
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #708:
URL: https://github.com/apache/singa/pull/708#discussion_r429513576
##########
File path: src/core/scheduler/scheduler.cc
##########
@@ -325,9 +337,10 @@ void Graph::RunGraph() {
}
}
+ EvaluateTimeElapsed();
+
// increment iteration counter
step();
Review comment:
Addressed
Added iteration_=0 in Graph::reset, the elapsed time will be cleared in Graph::rest together with node as well
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on a change in pull request #708: SINGA-510 Improvement of Time Profiling Function
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #708:
URL: https://github.com/apache/singa/pull/708#discussion_r429513662
##########
File path: src/core/scheduler/scheduler.cc
##########
@@ -325,9 +337,10 @@ void Graph::RunGraph() {
}
}
+ EvaluateTimeElapsed();
Review comment:
Addressed
In the revised version, we call the step function before EvaluateTimeElapsed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] XJDKC commented on a change in pull request #708: SINGA-510 Improvement of Time Profiling Function
Posted by GitBox <gi...@apache.org>.
XJDKC commented on a change in pull request #708:
URL: https://github.com/apache/singa/pull/708#discussion_r429511478
##########
File path: src/core/scheduler/scheduler.cc
##########
@@ -325,9 +337,10 @@ void Graph::RunGraph() {
}
}
+ EvaluateTimeElapsed();
Review comment:
I think we should call the step function before EvaluateTimeElapsed. Otherwise, it may have problems when dividing the time if the skip_iteration is zero.
##########
File path: src/core/scheduler/scheduler.cc
##########
@@ -325,9 +337,10 @@ void Graph::RunGraph() {
}
}
+ EvaluateTimeElapsed();
+
// increment iteration counter
step();
Review comment:
Remember to clear the elapsed time and the iteration when resetting the graph.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] nudles merged pull request #708: SINGA-510 Improvement of Time Profiling Function
Posted by GitBox <gi...@apache.org>.
nudles merged pull request #708:
URL: https://github.com/apache/singa/pull/708
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on pull request #708: SINGA-510 Improvement of Time Profiling Function
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on pull request #708:
URL: https://github.com/apache/singa/pull/708#issuecomment-632817048
Results:
```
root@71ac539cda77:~/dcsysh/singa/examples/cnn# python3 benchmark.py --verbosity 1
100%|##################################################################################################################################################| 100/100 [00:15<00:00, 6.42it/s]
Throughput = 205.34910581511633 per second
TotalTime=15.583218574523926
Total=0.15583218574523927
Time Profiling:
Forward Propagation Time : 0.0456643 sec
Backward Propagation Time : 0.104244 sec
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on pull request #708: SINGA-510 Improvement of Time Profiling Function
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on pull request #708:
URL: https://github.com/apache/singa/pull/708#issuecomment-632984166
Addressed the comments, a test:
```
root@71ac539cda77:~/dcsysh/singa/examples/cnn# python3 benchmark.py --verbosity 1
100%|##################################################################################################################################################| 100/100 [00:15<00:00, 6.62it/s]
Throughput = 211.74177390107678 per second
TotalTime=15.11274766921997
Total=0.1511274766921997
Time Profiling:
Forward Propagation Time : 0.0442915 sec
Backward Propagation Time : 0.101003 sec
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] XJDKC commented on pull request #708: SINGA-510 Improvement of Time Profiling Function
Posted by GitBox <gi...@apache.org>.
XJDKC commented on pull request #708:
URL: https://github.com/apache/singa/pull/708#issuecomment-632978715
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on a change in pull request #708: SINGA-510 Improvement of Time Profiling Function
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #708:
URL: https://github.com/apache/singa/pull/708#discussion_r429513576
##########
File path: src/core/scheduler/scheduler.cc
##########
@@ -325,9 +337,10 @@ void Graph::RunGraph() {
}
}
+ EvaluateTimeElapsed();
+
// increment iteration counter
step();
Review comment:
Addressed
Added iteration_=0 in Graph::reset, the elapsed time will be cleared in Graph::Reset together with node as well
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] nudles commented on pull request #708: SINGA-510 Improvement of Time Profiling Function
Posted by GitBox <gi...@apache.org>.
nudles commented on pull request #708:
URL: https://github.com/apache/singa/pull/708#issuecomment-632977682
@XJDKC pls help review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org