[GitHub] houkai commented on issue #7450: reporting bugs: pbegin_ <= pend_. Two thread conflicts.

[gi...@git.apache.org]

houkai commented on issue #7450:  reporting bugs: pbegin_ <= pend_. Two thread conflicts.
URL: https://github.com/apache/incubator-mxnet/issues/7450#issuecomment-322362334
 
 
   @ptrendx 
   In iter_image_recordio_2.cc , source_ is a ptr which from the object created by InputSplitShuffle.
   ```
       if (num_shuffle_parts > 1) {
         source_.reset(dmlc::InputSplitShuffle::Create(
             param_.path_imgrec.c_str(), param_.part_index,
             param_.num_parts, "recordio", num_shuffle_parts, param_.shuffle_chunk_seed));
       }
   ```
   InputSplitShuffle::create function run the code in io.cc as below:
   ```
   #if DMLC_ENABLE_STD_THREAD
     if (spec.cache_file.length() == 0) {
       return new ThreadedInputSplit(split);
     } else {
       return new CachedInputSplit(split, spec.cache_file.c_str());
     }
   #
   ```
   so,I think they point same object.
   In ThreadedInputSplit, it has created a thread to read data by fs_. 
   From the outside, we can read code in iter_image_recordio_2.cc.
   ```
       virtual void Init(const std::vector<std::pair<std::string, std::string> >& kwargs) {
         prefetch_param_.InitAllowUnknown(kwargs);
         /*create one thread in the function*/
         parser_.Init(kwargs); 
         // maximum prefetch threaded iter internal size
         const int kMaxPrefetchBuffer = 16;
         // init thread iter
         iter_.set_max_capacity(kMaxPrefetchBuffer);
         // init thread iter
         /*another one*/
         iter_.Init([this](DataBatch **dptr) {
             if (*dptr == nullptr) {
               *dptr = new DataBatch();
             }
             return parser_.ParseNext(*dptr);
             },
             [this]() { parser_.BeforeFirst(); });
       }
   ```
   Anyway, I comment ?return new ThreadedInputSplit(split)? and 'return split' in io.cc, so stop first thread. The question never came back. ~
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services