[GitHub] carbondata pull request #2672: [HOTFIX] improve sdk multi-thread performance

[jackylk <gi...@git.apache.org>]

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2672#discussion_r214509355
  
    --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java ---
    @@ -460,27 +461,32 @@ public CarbonLoadModel getLoadModel() {
     
         private CarbonOutputIteratorWrapper[] iterators;
     
    -    private int counter;
    +    // keep counts of number of writes called
    +    // and it is used to load balance each write call to one iterator.
    +    private AtomicLong counter;
     
         CarbonMultiRecordWriter(CarbonOutputIteratorWrapper[] iterators,
             DataLoadExecutor dataLoadExecutor, CarbonLoadModel loadModel, Future future,
             ExecutorService executorService) {
           super(null, dataLoadExecutor, loadModel, future, executorService);
           this.iterators = iterators;
    +      counter = new AtomicLong(0);
         }
     
    -    @Override public synchronized void write(NullWritable aVoid, ObjectArrayWritable objects)
    +    @Override public void write(NullWritable aVoid, ObjectArrayWritable objects)
             throws InterruptedException {
    -      iterators[counter].write(objects.get());
    -      if (++counter == iterators.length) {
    -        //round robin reset
    -        counter = 0;
    +      int iteratorLength = iterators.length;
    +      int iteratorNum = (int) (counter.incrementAndGet() % iteratorLength);
    +      synchronized (iterators[iteratorNum]) {
    --- End diff --
    
    If we want to let each SDK user thread write to its own iterator, why not create multiple CarbonWriter (each has one iterator) which shared one SortDataRows and let user call each CarbonWriter to write. Thus no synchronization is required. 


---