[jira] [Comment Edited] (TEZ-4075) Tez: Reimplement tez.runtime.transfer.data-via-events.enabled

["Rajesh Balamohan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-4075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933951#comment-16933951 ] 

Rajesh Balamohan edited comment on TEZ-4075 at 9/20/19 2:19 AM:
----------------------------------------------------------------

Attaching {{TEZ-4075.10.patch}} . This is based out of 0.8 patch you attached.

1. No IFile changes. Caching in IFile (e.g 512 bytes) can be introduced later, as IFile is always in hotpath. {{outputFileHandler.getOutputFileForWrite()}} anyways incurs disk hit, as it has to check dir validity. Caching can be looked into after potentially fixing LocalDirAllocator issues.
 2. Fixed test cases and have tried with other Hive queries.
 3. Refactored DME event generation


was (Author: rajesh.balamohan):
Attaching {{TEZ-4075.10.patch}}.  This is based out of 0.8 patch you attached.

1. No IFile changes. Caching in IFile (e.g 512 bytes) can be introduced later, as IFile is always in hotpath. {{outputFileHandler.getOutputFileForWrite()}} anyways incurs disk hit, as it has to check dir validity. Caching can be looked into after potentially fixing LocalDirAllocator issues.
2. Fixed test cases and have tried with other Hive queries.
3. Refactored DME event generation

> Tez: Reimplement tez.runtime.transfer.data-via-events.enabled
> -------------------------------------------------------------
>
>                 Key: TEZ-4075
>                 URL: https://issues.apache.org/jira/browse/TEZ-4075
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Gopal V
>            Assignee: Richard Zhang
>            Priority: Major
>         Attachments: TEZ-4075.10.patch, Tez-4075.5.patch, Tez-4075.8.patch
>
>
> This was factored out by TEZ-2196, which does skip buffers for 1-partition data exchanges (therefore goes to disk directly).
> {code}
>     if (shufflePayload.hasData()) {	    shuffleManager.addKnownInput(shufflePayload.getHost(),
>       DataProto dataProto = shufflePayload.getData();	        shufflePayload.getPort(), srcAttemptIdentifier, srcIndex);
>       FetchedInput fetchedInput = inputAllocator.allocate(dataProto.getRawLength(),	
>           dataProto.getCompressedLength(), srcAttemptIdentifier);	
>       moveDataToFetchedInput(dataProto, fetchedInput, hostIdentifier);	
>       shuffleManager.addCompletedInputWithData(srcAttemptIdentifier, fetchedInput);	
>     } else {	
>       shuffleManager.addKnownInput(shufflePayload.getHost(),	
>           shufflePayload.getPort(), srcAttemptIdentifier, srcIndex);	
>     }	
> {code}
> got removed in 
> https://github.com/apache/tez/commit/1ba1f927c16a1d7c273b6cd1a8553e5269d1541a
> It would be better to buffer up the 512Byte limit for the event size before writing to disk, since creating a new file always incurs disk traffic, even if the file is eventually being served out of the buffer cache.
> The total overhead of receiving an event, then firing an HTTP call to fetch the data etc adds approx 100-150ms to a query - the data xfer through the event will skip the disk entirely for this & also remove the extra IOPS incurred.
> This channel is not suitable for large-scale event transport, but specifically the workload here deals with 1-row control tables which consume more bandwidth with HTTP headers and hostnames than the 93 byte payload.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)