[GitHub] Lizonghang opened a new issue #12640: Time bottleneck in forward_backward and next(train_iter)

[GitBox <gi...@apache.org>]

Lizonghang opened a new issue #12640: Time bottleneck in forward_backward and next(train_iter)
URL: https://github.com/apache/incubator-mxnet/issues/12640
 
 
   ## Description
   
   In BaseModule.fit function, <code>forward_backward</code> only needs 0.00055s, <code>update</code> only needs 0.079953s, <code>next(data_iter)</code> only needs 0.000265s. However, we run <code>forward_backward</code> and <code>next(data_iter)</code> together and spend 2.919867s. Could you please tell me what is the reason of this time bottleneck?
   
   ## Environment info (Required)
   
   Platform: MacOS
   Language: Python 2.7
   Platform Version: MXNet 1.2.0
   Device: 2.9 GHz Intel Core i5 (The test runs using single CPU)
   Memory: 8 GB 1867 MHz DDR3
   Install Tool: pip
   
   ## Steps to reproduce (In BaseModule.fit function)
   
   * Run <code>forward_backward</code> only:
   
   ```
   while True:
       data_batch = next_data_batch
       calc_start = time.time()
       self.forward_backward(data_batch)
       calc_end = time.time()
       self.logger.info("[CALC TIME] %f", calc_end - calc_start)
   ```
   
   result: [CALC TIME] 0.000551	
   
   * Run <code>next(data_iter)</code> only:
   
   ```
   while True:
       data_iter_start = time.time()
       next(data_iter)
       data_iter_end = time.time()
       self.logger.info("[DATAITER TIME] %f", data_iter_end - data_iter_start)
   ```
   
   result: [DATAITER TIME] 0.000265
   
   * Run <code>forward_backward</code> and <code>next(data_iter)</code> together:
   
   ```
   while True:
       data_batch = next_data_batch
       calc_iter_start = time.time()
       self.forward_backward(data_batch)
       next_data_batch = next(data_iter)
       calc_iter_end = time.time()
       self.logger.info("[CALC AND DATAITER TIME] %f", calc_iter_end - calc_iter_start)
   ```
   
   result: [CALC AND DATAITER TIME] 2.919867	
   
   ## What have you tried to solve it?
   
   1. I read the source code of <code>forward_backward</code> and <code>ImageRecordIter</code> but have no idea.
   2. I found little modify in <code>data_batch</code> variable would also cause the time bottleneck in the loop of <code>forward_backward</code>.
   
   Thanks a lot for your kind !
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services