[GitHub] [incubator-pinot] afilipchik opened a new issue #4916: Thoughts on possible refactoring of SegmentIndexCreationDriverImpl

[GitBox <gi...@apache.org>]

afilipchik opened a new issue #4916: Thoughts on possible refactoring of SegmentIndexCreationDriverImpl
URL: https://github.com/apache/incubator-pinot/issues/4916
 
 
   I was working on creating Flink based  job for generating Offline segments and I realized that it is a bit tricky to SegmentIndexCreationDriverImpl when input is a flow of data and not a file. 
   
   Currently it uses RecordReader that acts as an iterator. to make it work with Flink dataflow model I had to create a bridge between a stream and an iterator using an in-memory buffer. 
   
   In general, it looks like using it for anything in motion will require some kind of adapter and probably multithreading which is error prone. SegmentIndexCreationDriverImpl can be refactored so it don't depend on RecordReader and let outside caller to feed data into it. Example of the usage can be:
   `
   SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl(config)
   driver.init();
   for (int i = 0; i < n; i++) {
     PinotGenericRow row = createRow();
     driver.addRow(row)
   }
   
   driver.flush()
   driver.close()
   `
   
   Any thoughts? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org