[GitHub] leezu commented on issue #13447: Rewrite dataloader, improves responsiveness and reliability

[GitBox <gi...@apache.org>]

leezu commented on issue #13447: Rewrite dataloader, improves responsiveness and reliability
URL: https://github.com/apache/incubator-mxnet/pull/13447#issuecomment-442730074
 
 
   @ThomasDelteil regarding point 1, this is currently available in GluonNLP with `PrefetchingStream`. The current API is as follows: Consider a some Stream (generalization of Dataset that does not support random access but only sequential) of batches, we can wrap it in a `PrefetchingStream` as `data = PrefetchingStream(streamofbatches)`. When calling `iter(data)` will begin prefetching. It is currently used as follows
   
   ```
       prefetched_iters = []
       for _ in range(min(args.num_prefetch_epoch, args.epochs)):
           prefetched_iters.append(iter(data))
       for epoch in range(args.epochs):
           if epoch + len(prefetched_iters) < args.epochs:
               prefetched_iters.append(iter(data))
           data_iter = prefetched_iters.pop(0)
   ```
   
   The usage-pattern may be made more user-friendly, but the basic idea may also be applicable to Dataset API?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services