[GitHub] vishaalkapoor commented on issue #7637: Strange Validation and Training Losses at epoch change

[GitBox <gi...@apache.org>]

vishaalkapoor commented on issue #7637: Strange Validation and Training Losses at epoch change
URL: https://github.com/apache/incubator-mxnet/issues/7637#issuecomment-406021948
 
 
   Hi,
   
   Because you're tracking the evaluation error after each minibatch rather than after each epoch, the variance you're seeing at the beginning of each epoch is consistent (and also appears to be) due to the fact that you're averaging accuracies over a small number of minibatches. The smaller number of accuracies you're averaging over, the more variance you'll see, and that's why you see more variance at the start of each epoch and then things level out.
   
   Otherwise, there are no indications in the code that anything else is going on, so this seems to be what you're seeing.
   
   In more detail, from metric.py
   
       def update(self, labels, preds):
           """Updates the internal evaluation result.
   
           Parameters
           ----------
           labels : list of `NDArray`
               The labels of the data.
   
           preds : list of `NDArray`
               Predicted values.
           """
           labels, preds = check_label_shapes(labels, preds, True)
   
           for label, pred in zip(labels, preds):
               label = label.asnumpy()
               pred = pred.asnumpy()
   
               if len(label.shape) == 1:
                   label = label.reshape(label.shape[0], 1)
               if len(pred.shape) == 1:
                   pred = pred.reshape(pred.shape[0], 1)
   
     A)        self.sum_metric += ((label - pred)**2.0).mean()
     B)        self.num_inst += 1 # numpy.prod(label.shape)
   
   Line A computes a running sum of averages and line B keeps track of the number of summands in that running sum (equal to the minibatch number in the epoch).
   
   After each epoch, sum_metric and num_inst are reset, so your averaging starts fresh with only a few number of samples.
   
   In general, the variance of the average of n independent and identically distributed random variables is inversely proportional to the number of random variables (the sample size). Thus as you increase the number of samples, you'll decrease the variance.
   
   https://www.quora.com/How-is-variance-inversely-proportional-to-sample-size
   https://stats.stackexchange.com/questions/160103/why-does-the-variance-decrease-when-averaging-zero-mean-gaussians
   
   There is a related issue on convergence that I'm including just for interest, but it's unrelated to this phenomenon:
   https://stats.stackexchange.com/questions/229301/why-does-the-accuracy-of-my-cnn-periodically-spike-down
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services