You are viewing a plain text version of this content. The canonical link for it is here.

Posted to yarn-dev@hadoop.apache.org by "Roni Burd (JIRA)" <ji...@apache.org> on 2017/04/01 00:11:41 UTC

[jira] [Created] (YARN-6426) Compress ZK YARN keys to scale up (especially AppStateData

Roni Burd created YARN-6426:
-------------------------------

             Summary: Compress ZK YARN keys to scale up (especially AppStateData
                 Key: YARN-6426
                 URL: https://issues.apache.org/jira/browse/YARN-6426
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: yarn
            Reporter: Roni Burd
            Assignee: Roni Burd
             Fix For: 2.8.0, 3.0.0-alpha2


ZK today stores the protobuf files uncompressed. This is not an issue except that if a customer job has thousands of files, AppStateData will store the user context as a string with multiple URLs and it is easy to get to 1MB or more. 

This can put unnecessary strain on ZK and make the process slow. 

The proposal is to simply compress protobufs before sending them to ZK



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org