[jira] [Commented] (TEZ-3222) Reduce messaging overhead for auto-reduce parallelism case

["Jonathan Eagles (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667599#comment-15667599 ] 

Jonathan Eagles commented on TEZ-3222:
--------------------------------------

{quote}
return CompositeEventRouteMetadata.create(1, sourceTaskIndex, 0);
{quote}

This call is only called once per source, this reduces the memory footprint while having the same CPU footprint.

{quote}
+message CompositeRoutedDataMovementEventProto {
+  optional int32 source_index = 1;
+  optional int32 target_index = 2;
+  optional int32 count = 3;
+  optional bytes user_payload = 4;
+  optional int32 version = 5;
+}
{quote}

[~bikassaha], I looked at adding a CompositeRouteMeta as proto and having CompositeRoutedDataMovementEventProto contain this new structure in place of meta fields. There were several downsides of using this technique including nearly doubling the cpu needed to use this events. If we can leverage the benefits of protobuf and keep them in the top level it will be much easier to use and faster. The downside of the evolution of the event can be handled using protobuf optional fields. When it becomes too unwieldy, perhaps that is the time to redesign. If you are ok with that I would prefer to push that change to a later date.

> Reduce messaging overhead for auto-reduce parallelism case
> ----------------------------------------------------------
>
>                 Key: TEZ-3222
>                 URL: https://issues.apache.org/jira/browse/TEZ-3222
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>         Attachments: TEZ-3222.1.patch, TEZ-3222.2.patch, TEZ-3222.3.patch, TEZ-3222.4.patch, TEZ-3222.5.patch, TEZ-3222.6.patch
>
>
> A dag with 15k x 1000k vertex may auto-reduce to 15k x 1. And while the data  size is appropriate for 1 task attempt, this results in an increase in task attempt message processing of 1000x.
> This jira aims to reduce the message processing in the auto-reduced task while keeping the amount of message processing in the AM the same or less.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)