You are viewing a plain text version of this content. The canonical link for it is here.

Posted to discuss-archive@mxnet.apache.org by Stacey via MXNet Forum <mx...@discoursemail.com.INVALID> on 2021/01/21 08:16:53 UTC

[MXNet Forum] [Performance] Data loading from .rec with multiprocessing?


I am working on cpu with 80 virtual cores(what cpu_count() tells me) and 6 nvidia1080 gpus. 
I used mx.io.ImageRecordIter to load .rec and .idx, which contains about 3 million images. During training process, the cpu usage goes up to 3000%. I figured out if I use smaller batches and less preprocessing_threads, the usage could be back to like 600%, however that also reduces the speed of training. Is there a way to load my training data in .rec and .idx files with multiprocessing so that I could separate the workload?





---
[Visit Topic](https://discuss.mxnet.apache.org/t/data-loading-from-rec-with-multiprocessing/6826/1) or reply to this email to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/47be13a60e0dfdf94beccb468e96a8873fcc703d4f99c0dbfb881a71fb043907).