You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "chaoli (Jira)" <ji...@apache.org> on 2019/12/25 18:06:00 UTC
[jira] [Comment Edited] (YARN-2962) ZKRMStateStore: Limit the
number of znodes under a znode
[ https://issues.apache.org/jira/browse/YARN-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17003353#comment-17003353 ]
chaoli edited comment on YARN-2962 at 12/25/19 6:05 PM:
--------------------------------------------------------
I'm not quite agreed with this patch. I recently read zookeeper source code, and I found that the response of *getChildren* will be splited into multi packet（each size limit is *4MB*).
Since *GetChildrenResponse* is nested object, therefore In zookeeper server(*NettyServerCnxn),* the method ** *sendResponse* calls *output.write* iteratively * *which makes each packet size hardly exceed the default limit*. * **
NettyServerCnxn.sendResponse
{code:java}
// Make space for length
BinaryOutputArchive bos = BinaryOutputArchive.getArchive(baos);
try {
baos.write(fourBytes);
bos.writeRecord(h, "header");
if (r != null) {
bos.writeRecord(r, tag);
}
baos.close();
} catch (IOException e) {
LOG.error("Error serializing response");
}
{code}
GetChildrenResponse.serialize
{code:java}
public void serialize(OutputArchive a_, String tag) throws IOException {
a_.startRecord(this, tag);
a_.startVector(this.children, "children");
if (this.children != null) {
int len1 = this.children.size();
for(int vidx1 = 0; vidx1 < len1; ++vidx1) {
String e1 = (String)this.children.get(vidx1);
a_.writeString(e1, "e1");
}
}
a_.endVector(this.children, "children");
a_.endRecord(this, tag);
}
{code}
was (Author: lichaojacobs):
I'm not quite agreed with this patch. I recently read zookeeper source code, and I found that the response of *getChildren* will be splited into multi packet（each size limit is *4MB*).
Since *GetChildrenResponse* is nested object, therefore In zookeeper server(*NettyServerCnxn),* the method ** *sendResponse* calls *output.write* iteratively * *which makes each packet size hardly exceed the default limit*. ** *
NettyServerCnxn.sendResponse
{code:java}
// code placeholder
// Make space for length
BinaryOutputArchive bos = BinaryOutputArchive.getArchive(baos);
try {
baos.write(fourBytes);
bos.writeRecord(h, "header");
if (r != null) {
bos.writeRecord(r, tag);
}
baos.close();
} catch (IOException e) {
LOG.error("Error serializing response");
}
{code}
GetChildrenResponse.serialize
{code:java}
public void serialize(OutputArchive a_, String tag) throws IOException {
a_.startRecord(this, tag);
a_.startVector(this.children, "children");
if (this.children != null) {
int len1 = this.children.size();
for(int vidx1 = 0; vidx1 < len1; ++vidx1) {
String e1 = (String)this.children.get(vidx1);
a_.writeString(e1, "e1");
}
}
a_.endVector(this.children, "children");
a_.endRecord(this, tag);
}
{code}
> ZKRMStateStore: Limit the number of znodes under a znode
> --------------------------------------------------------
>
> Key: YARN-2962
> URL: https://issues.apache.org/jira/browse/YARN-2962
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
> Affects Versions: 2.6.0
> Reporter: Karthik Kambatla
> Assignee: Varun Saxena
> Priority: Critical
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: YARN-2962.006.patch, YARN-2962.007.patch, YARN-2962.008.patch, YARN-2962.008.patch, YARN-2962.009.patch, YARN-2962.01.patch, YARN-2962.010.patch, YARN-2962.011.patch, YARN-2962.04.patch, YARN-2962.05.patch, YARN-2962.2.patch, YARN-2962.3.patch
>
>
> We ran into this issue where we were hitting the default ZK server message size configs, primarily because the message had too many znodes even though they individually they were all small.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org