[jira] [Commented] (TEZ-2918) Make progress notifications in IOs

["Rajesh Balamohan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984592#comment-14984592 ] 

Rajesh Balamohan commented on TEZ-2918:
---------------------------------------

Patch introduces volatile updates in inner loops of the runtime.  Is it possible to avoid that as it would hurt performance.  For instance, pipelinedsorter does not have any sync or volatile in its core loop. In other places like merge, this is one of the reasons for introducing TEZ_RUNTIME_RECORDS_BEFORE_PROGRESS so that progress is updated every 10000 records or so.  TEZ-1491, HADOOP-11029 refer to counter stats on volatile and sync. Places where this can hurt are

- DefaultSorter's collect code path as it would get executed for every record.
- DefaultSorter's sortAndSpill, as quick sort would internally invoke progress() lots of times. Instead "progressable.progress()" can be invoked just before calling sort in sortAndSpill?
- MRReaderMapred.next(), MRReaderMapReduce.next(), MROutput.KeyValueWriter.write(), OrderedGroupedKVInput.getReader().next(), PipelinedSorter.collect(), UnorderedKVInput.getReader().next(), UnorderedKVReader.next(). If needed, should it make use of TEZ_RUNTIME_RECORDS_BEFORE_PROGRESS as they are hot code paths?

>> In UnorderedPartitionedKVWriter, {code} "writer.append(key, value); // ???? Why is outputrecordscounter not updated here?" {code} 
- Writer already has reference to outputRecordsCounter. So when it gets closed, outputRecordsCounter is automatically adjusted.

>> In PipelinedSorter, {code} // ??? why are events not being sent here? {code} 
- Final event is sent out by OrderedPartitionedKVOutput itself. So in cases where only one output is available, there is no need to send any events from within sorter itself.


> Make progress notifications in IOs
> ----------------------------------
>
>                 Key: TEZ-2918
>                 URL: https://issues.apache.org/jira/browse/TEZ-2918
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Bikas Saha
>            Assignee: Bikas Saha
>         Attachments: TEZ-2918.1.patch, TEZ-2918.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)