[GitHub] [incubator-mxnet] smukund28 opened a new issue #16096: FixedBucketSampler needs last_batch = discard feature

[GitBox <gi...@apache.org>]

smukund28 opened a new issue #16096: FixedBucketSampler needs last_batch = discard feature 
URL: https://github.com/apache/incubator-mxnet/issues/16096
 
 
   Feature Request :
   
   When DataLoader uses a Sampler, one cannot rely on the last_batch='discard' feature. It would be nice to have this in place for Sampler as well.
   
   What have you tried to solve it?
   One way to overcome this error is to pad the samples and load the data. Another way is to simply check for the batch size in the loop and discard if the sample size does not match the batch size.. 
   
   Example:
   
       for batch_id, (token_ids, valid_length, segment_ids, label) in enumerate(bert_dataloader):
           
           if len(token_ids) != batch_size: continue
   
           token_ids1 = gluon.utils.split_and_load(token_ids, ctx, even_split=False )
           valid_length1 = gluon.utils.split_and_load(valid_length, ctx, even_split=False )
           segment_ids1 = gluon.utils.split_and_load(segment_ids, ctx, even_split=False)
           label1 = gluon.utils.split_and_load(label, ctx, even_split=False)
               

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services