You are viewing a plain text version of this content. The canonical link for it is here.

Posted to discuss-archive@mxnet.apache.org by Lou via MXNet Forum <mx...@discoursemail.com.INVALID> on 2021/05/25 10:05:24 UTC

[MXNet Forum] Combining .rec files from im2rec's "chunk" option


I ran into some performance issues packaging about 2 million images into .rec files -- after about 200k, the time to pack each 1,000 images got really high. As a workaround, I used the `--chunks` option for im2rec, which resulted in about 10 .rec files with around 200k images each.

That was nice, but I don't see an easy way to combine them. I'm using gluon's ImageRecordDataset, which only accepts a single .rec and idx file.

Is there an easy way to combine these .rec I've generated? I'd be OK combining them into one file for passing to ImageRecordDataset, or having a dataset support multiple input files.





---
[Visit Topic](https://discuss.mxnet.apache.org/t/combining-rec-files-from-im2recs-chunk-option/6969/1) or reply to this email to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/a27e5f6758228de37310f38db13d36ba7781f665be52fbb3230e79ef405f11b8).