You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Shawn Heisey <ap...@elyograg.org> on 2018/04/04 19:50:42 UTC
Is the current max packet length available via the API?
Is it possible to get the current max packet length from the API?
(version 3.4.x)
If not, I'm guessing that I need to look for the jute.maxbuffer system
property and fallback to ZkClientConfig.CLIENT_MAX_PACKET_LENGTH_DEFAULT
if it's not defined.
What I'm trying to do is log a useful error message in Solr if somebody
tries to upload a file that's too big for what's allowed. The error
that they get currently is not helpful, and figuring out what went wrong
seems to require looking at the server log.
Side note: I can see in current code (and the 3.5.2 programmer's guide)
that the default max packet length is 4MB, but the administrators guide
(even the 3.5.3 version) still says 1MB.
Thanks,
Shawn
Re: Is the current max packet length available via the API?
Posted by Andor Molnar <an...@cloudera.com>.
Martin's instructions about JMX are pretty much accurate. JMX is a standard
Java protocol to retrieve metrics and interact with a running application.
I'm sure you can find plenty of information on the internet and existing
client libraries to retrieve data.
ZooKeeper's admin documentation also gives you information about JMX
support in ZK: http://zookeeper.apache.org/doc/r3.4.11/zookeeperJMX.html
That's the only I'm aware of to ask ZK about current jute.maxbuffer
setting. I don't think there's support for that in ZK cli at the moment.
Jute buffer must have enough room to contain the entire packet which sent
over the network (in other words, the payload of TCP packet). So yes, you
should subtract the size of the header (2x int) and the length of path.
(this is just off the top of my head and refers to create request only, you
should verify it)
The const CLIENT_MAX_PACKET_LENGTH_DEFAULT is 3.5 only, but apart from
that, the default is 4MB in 3.4 too.
Error messages are "len error" for server-server and "Packet len" for
client-server communication.
Hope that helps.
Regards,
Andor
On Thu, Apr 5, 2018 at 5:58 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 4/5/2018 3:44 AM, Andor Molnar wrote:
>
>> You can get the current jute.maxbuffer setting from a running ZooKeeper
>> instance by querying ZooKeeperServerBean via JMX.
>>
>
> I'm not sure how I would do that in a client program. It might be
> trivial, but it's not something I've ever done.
>
> Currently there're 2 usage of the setting in ZK: 1) server-client
>> communication which is by default 4MB, 2) server-server communication which
>> is by default 1MB. They can't be set individually, but can be overriden
>> with the jute.maxbuffer system property.
>>
>
> I'm looking for a way to ask the ZK client to give me the value it is
> currently using as its max packet length. I'm only going to be logging a
> warning to inform the user about which file may have caused a problem due
> to size, not preventing the attempt at uploading the file, so I'm not
> opposed to falling back to a hard-coded value if I can't figure it out. I
> can look for the jute.maxbuffer sysprop, but if ZK will tell me what it's
> actually using, I'd prefer that.
>
> Does the max packet length cover ONLY the size of the znode data, or does
> the znode name get included in that? Asked another way: Should I subtract
> a little bit from the max packet length (maybe 128 or 256) before I compare
> the file size, or just compare the unchanged value?
>
> I did discover that the ZkClientConfig.CLIENT_MAX_PACKET_LENGTH_DEFAULT
> field I mentioned before is not available in 3.4.x, it seems to have been
> added to a 3.5 version. Since Solr uses 3.4.x and won't upgrade until
> there is a stable 3.5 release, I can't use that.
>
> I do think that the ZK client should log something useful when the max
> packet length is exceeded -- if that's even possible. The user in this
> scenario is running the latest version of Solr that was available at the
> time, which includes ZK 3.4.10 for its client. The error message indicated
> socket problems, but didn't have any information about the cause.
>
> When running under java 9, they got this as the error:
>
> WARN - 2018-04-04 09:05:28.194; org.apache.zookeeper.ClientCnxn$SendThread;
> Session 0x100244e8ffb0004 for server localhost/127.0.0.1:2181, unexpected
> error, closing socket connection and attempting reconnect
> java.io.IOException: Connection reset by peer
>
> With Java 8, they got this:
>
> WARN - 2018-04-04 09:10:11.879; org.apache.zookeeper.ClientCnxn$SendThread;
> Session 0x10024db7e280002 for server localhost/0:0:0:0:0:0:0:1:2181,
> unexpected error, closing socket connection and attempting reconnect
> java.io.IOException: Protocol wrong type for socket
>
> In both cases, the stacktrace listed a bunch of sun classes and then a
> couple of methods in zookeeper's ClientCnxnSocketNIO class.
>
> When I asked them what their ZK server log said, that's when I figured out
> the problem:
>
> 2018-04-04 14:06:01,361 [myid:] - WARN [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - Exception causing close of
> session 0x10024db7e280006: Len error 5327937
>
> Do I understand correctly that Solr uploads file to ZooKeeper?
>>
>
> Solr *itself* won't typically be uploading data to ZK that can exceed the
> max packet size. It is typically done either with a separate commandline
> program (the ZkCLI class the commandline program uses is included in Solr),
> or by a client program using the SolrJ library (which is part of Solr like
> ZkCLI, but usable by itself). The action being performed is an upload of a
> configuration for a Solr index.
>
> Solr does sometimes run into the problem described in ZOOKEEPER-1162, but
> this is due to the number of children in a znode, where each one has
> minimal data.
>
> Thanks,
> Shawn
>
>
Re: Is the current max packet length available via the API?
Posted by Andor Molnar <an...@cloudera.com>.
If the ZK client is part of your application, you can probably read the
jute.maxbuffer setting.
However on the server side - as you mentioned - the setting should be
consistent with the client. I believe it would make sense to expose the
effective jute maxbuffer size via the "conf" 4lw command if that's any help
to you.
Would you like to contribute and submit a pull request for that on the
master branch?
Regards,
Andor
On Sat, Apr 7, 2018 at 8:13 PM, Shawn Heisey <el...@elyograg.org> wrote:
> On 4/6/2018 6:46 AM, Martin Gainty wrote:
>
> ZOOMAIN="-Dcom.sun.management.jmxremote
>> -Dcom.sun.management.jmxremote.port=$JMXPORT
>> -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH
>> -Dcom.sun.management.jmxremote.ssl=$JMXSSL -Dzookeeper.jmx.log4j.disable=$JMXLOG4J
>> org.apache.zookeeper.server.quorum.QuorumPeerMain"
>> fi
>> else
>> echo "JMX disabled by user request" >&2
>> ZOOMAIN="org.apache.zookeeper.server.quorum.QuorumPeerMain"
>> fi
>> MG>everything you need to setup JMX is located MG>athttp://
>> java.sun.com/javase/6/docs/technotes/guides/management/agent.html
>>
>
> I know how to enable remote JMX when running a java program. I also know
> how to connect a JMX client like jconsole to that program.
>
> What I was saying that I didn't know how to do was access data from JMX in
> a Java program that I've written myself.
>
> Remote JMX seems like overkill, when all I want to know is what the ZK
> client that's part of my application is currently using as a maximum packet
> length. Reading ClientCnxn.packetLen gets me what I was after. It would
> have taken an extensive code review for me to find that particular field,
> so thank you for letting me know about it.
>
> If the client has an increased jute.maxbuffer but the server doesn't, then
> there could still be a problem with writing data, but I'm not going to try
> and detect that. In any case, I'm not going to abort the upload, I'm just
> logging a warning.
>
> Thanks,
> Shawn
>
>
Re: Is the current max packet length available via the API?
Posted by Shawn Heisey <el...@elyograg.org>.
On 4/6/2018 6:46 AM, Martin Gainty wrote:
> ZOOMAIN="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=$JMXPORT -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH -Dcom.sun.management.jmxremote.ssl=$JMXSSL -Dzookeeper.jmx.log4j.disable=$JMXLOG4J org.apache.zookeeper.server.quorum.QuorumPeerMain"
> fi
> else
> echo "JMX disabled by user request" >&2
> ZOOMAIN="org.apache.zookeeper.server.quorum.QuorumPeerMain"
> fi
> MG>everything you need to setup JMX is located MG>athttp://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
I know how to enable remote JMX when running a java program. I also know
how to connect a JMX client like jconsole to that program.
What I was saying that I didn't know how to do was access data from JMX
in a Java program that I've written myself.
Remote JMX seems like overkill, when all I want to know is what the ZK
client that's part of my application is currently using as a maximum
packet length. Reading ClientCnxn.packetLen gets me what I was after.
It would have taken an extensive code review for me to find that
particular field, so thank you for letting me know about it.
If the client has an increased jute.maxbuffer but the server doesn't,
then there could still be a problem with writing data, but I'm not going
to try and detect that. In any case, I'm not going to abort the upload,
I'm just logging a warning.
Thanks,
Shawn
Re: Is the current max packet length available via the API?
Posted by Martin Gainty <mg...@hotmail.com>.
MG>confirm for andor below (with instructions for JMX)
________________________________
From: Shawn Heisey <ap...@elyograg.org>
Sent: Thursday, April 5, 2018 11:58 AM
To: user@zookeeper.apache.org
Subject: Re: Is the current max packet length available via the API?
On 4/5/2018 3:44 AM, Andor Molnar wrote:
> You can get the current jute.maxbuffer setting from a running
> ZooKeeper instance by querying ZooKeeperServerBean via JMX.
I'm not sure how I would do that in a client program. It might be
trivial, but it's not something I've ever done.
MG>explained here in zkserver.sh
# See the following page for extensive details on setting
# up the JVM to accept JMX remote management:
# http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
# by default we allow local JMX connections
if [ "x$JMXLOCALONLY" = "x" ]
then
JMXLOCALONLY=false
fi
#JMXDISABLE must not be null
if [ "x$JMXDISABLE" = "x" ]
then
echo "ZooKeeper JMX enabled by default" >&2
if [ "x$JMXPORT" = "x" ]
then
# for some reason these two options are necessary on jdk6 on Ubuntu
# accord to the docs they are not necessary, but otw jconsole cannot
# do a local attach
ZOOMAIN="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=$JMXLOCALONLY org.apache.zookeeper.server.quorum.QuorumPeerMain"
else
#AUTH credentials go here
if [ "x$JMXAUTH" = "x" ]
then
JMXAUTH=false
fi
#FOR SSL/TLS
if [ "x$JMXSSL" = "x" ]
then
JMXSSL=false
fi
#suggest use default of JMXLOG4J=true
if [ "x$JMXLOG4J" = "x" ]
then
JMXLOG4J=true
fi
echo "ZooKeeper remote JMX Port set to $JMXPORT" >&2
echo "ZooKeeper remote JMX authenticate set to $JMXAUTH" >&2
echo "ZooKeeper remote JMX ssl set to $JMXSSL" >&2
echo "ZooKeeper remote JMX log4j set to $JMXLOG4J" >&2
ZOOMAIN="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=$JMXPORT -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH -Dcom.sun.management.jmxremote.ssl=$JMXSSL -Dzookeeper.jmx.log4j.disable=$JMXLOG4J org.apache.zookeeper.server.quorum.QuorumPeerMain"
fi
else
echo "JMX disabled by user request" >&2
ZOOMAIN="org.apache.zookeeper.server.quorum.QuorumPeerMain"
fi
MG>everything you need to setup JMX is located MG>at http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
> Currently there're 2 usage of the setting in ZK: 1) server-client
> communication which is by default 4MB, 2) server-server communication
> which is by default 1MB. They can't be set individually, but can be
> overriden with the jute.maxbuffer system property.
I'm looking for a way to ask the ZK client to give me the value it is
currently using as its max packet length. I'm only going to be logging
a warning to inform the user about which file may have caused a problem
due to size, not preventing the attempt at uploading the file, so I'm
not opposed to falling back to a hard-coded value if I can't figure it
out. I can look for the jute.maxbuffer sysprop, but if ZK will tell me
what it's actually using, I'd prefer that.
MG>andor is correct packetLen is set in org.apache.zookeeper.ClientCnxn
MG>
public static final int packetLen = Integer.getInteger("jute.maxbuffer",
0496 * 1024);
MG>as packetLen is final... it cannot be overidden to any other value
Does the max packet length cover ONLY the size of the znode data, or
does the znode name get included in that? Asked another way: Should I
subtract a little bit from the max packet length (maybe 128 or 256)
before I compare the file size, or just compare the unchanged value?
I did discover that the ZkClientConfig.CLIENT_MAX_PACKET_LENGTH_DEFAULT
field I mentioned before is not available in 3.4.x, it seems to have
been added to a 3.5 version. Since Solr uses 3.4.x and won't upgrade
until there is a stable 3.5 release, I can't use that.
MG>confirmed CLIENT_MAX_PACKET_LENGTH_DEFAULT is not avail in 3.4.x
I do think that the ZK client should log something useful when the max
packet length is exceeded -- if that's even possible.
MG>happens here in org.apache.zookeeper.ClientCnxnSocket
protected void readLength() throws IOException {
int len = incomingBuffer.getInt();
if (len < 0 || len >= ClientCnxn.packetLen) {
throw new IOException("Packet len" + len + " is out of range!");
MG>
The user in this
scenario is running the latest version of Solr that was available at the
time, which includes ZK 3.4.10 for its client. The error message
indicated socket problems, but didn't have any information about the cause.
When running under java 9, they got this as the error:
WARN - 2018-04-04 09:05:28.194;
org.apache.zookeeper.ClientCnxn$SendThread; Session 0x100244e8ffb0004
for server localhost/127.0.0.1:2181, unexpected error, closing socket
connection and attempting reconnect java.io.IOException: Connection
reset by peer
With Java 8, they got this:
WARN - 2018-04-04 09:10:11.879;
org.apache.zookeeper.ClientCnxn$SendThread; Session 0x10024db7e280002
for server localhost/0:0:0:0:0:0:0:1:2181, unexpected error, closing
socket connection and attempting reconnect java.io.IOException: Protocol
wrong type for socket
In both cases, the stacktrace listed a bunch of sun classes and then a
couple of methods in zookeeper's ClientCnxnSocketNIO class.
When I asked them what their ZK server log said, that's when I figured
out the problem:
2018-04-04 14:06:01,361 [myid:] - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - Exception causing close of
session 0x10024db7e280006: Len error 5327937
> Do I understand correctly that Solr uploads file to ZooKeeper?
Solr *itself* won't typically be uploading data to ZK that can exceed
the max packet size. It is typically done either with a separate
commandline program (the ZkCLI class the commandline program uses is
included in Solr), or by a client program using the SolrJ library (which
is part of Solr like ZkCLI, but usable by itself). The action being
performed is an upload of a configuration for a Solr index.
Solr does sometimes run into the problem described in ZOOKEEPER-1162,
but this is due to the number of children in a znode, where each one has
minimal data.
Thanks,
Shawn
Re: Is the current max packet length available via the API?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 4/5/2018 3:44 AM, Andor Molnar wrote:
> You can get the current jute.maxbuffer setting from a running
> ZooKeeper instance by querying ZooKeeperServerBean via JMX.
I'm not sure how I would do that in a client program. It might be
trivial, but it's not something I've ever done.
> Currently there're 2 usage of the setting in ZK: 1) server-client
> communication which is by default 4MB, 2) server-server communication
> which is by default 1MB. They can't be set individually, but can be
> overriden with the jute.maxbuffer system property.
I'm looking for a way to ask the ZK client to give me the value it is
currently using as its max packet length. I'm only going to be logging
a warning to inform the user about which file may have caused a problem
due to size, not preventing the attempt at uploading the file, so I'm
not opposed to falling back to a hard-coded value if I can't figure it
out. I can look for the jute.maxbuffer sysprop, but if ZK will tell me
what it's actually using, I'd prefer that.
Does the max packet length cover ONLY the size of the znode data, or
does the znode name get included in that? Asked another way: Should I
subtract a little bit from the max packet length (maybe 128 or 256)
before I compare the file size, or just compare the unchanged value?
I did discover that the ZkClientConfig.CLIENT_MAX_PACKET_LENGTH_DEFAULT
field I mentioned before is not available in 3.4.x, it seems to have
been added to a 3.5 version. Since Solr uses 3.4.x and won't upgrade
until there is a stable 3.5 release, I can't use that.
I do think that the ZK client should log something useful when the max
packet length is exceeded -- if that's even possible. The user in this
scenario is running the latest version of Solr that was available at the
time, which includes ZK 3.4.10 for its client. The error message
indicated socket problems, but didn't have any information about the cause.
When running under java 9, they got this as the error:
WARN - 2018-04-04 09:05:28.194;
org.apache.zookeeper.ClientCnxn$SendThread; Session 0x100244e8ffb0004
for server localhost/127.0.0.1:2181, unexpected error, closing socket
connection and attempting reconnect java.io.IOException: Connection
reset by peer
With Java 8, they got this:
WARN - 2018-04-04 09:10:11.879;
org.apache.zookeeper.ClientCnxn$SendThread; Session 0x10024db7e280002
for server localhost/0:0:0:0:0:0:0:1:2181, unexpected error, closing
socket connection and attempting reconnect java.io.IOException: Protocol
wrong type for socket
In both cases, the stacktrace listed a bunch of sun classes and then a
couple of methods in zookeeper's ClientCnxnSocketNIO class.
When I asked them what their ZK server log said, that's when I figured
out the problem:
2018-04-04 14:06:01,361 [myid:] - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@383] - Exception causing close of
session 0x10024db7e280006: Len error 5327937
> Do I understand correctly that Solr uploads file to ZooKeeper?
Solr *itself* won't typically be uploading data to ZK that can exceed
the max packet size. It is typically done either with a separate
commandline program (the ZkCLI class the commandline program uses is
included in Solr), or by a client program using the SolrJ library (which
is part of Solr like ZkCLI, but usable by itself). The action being
performed is an upload of a configuration for a Solr index.
Solr does sometimes run into the problem described in ZOOKEEPER-1162,
but this is due to the number of children in a znode, where each one has
minimal data.
Thanks,
Shawn
Re: Is the current max packet length available via the API?
Posted by Andor Molnar <an...@cloudera.com>.
Hi Shawn,
You can get the current jute.maxbuffer setting from a running ZooKeeper
instance by querying ZooKeeperServerBean via JMX.
Currently there're 2 usage of the setting in ZK:
1) server-client communication which is by default 4MB,
2) server-server communication which is by default 1MB.
They can't be set individually, but can be overriden with the
jute.maxbuffer system property.
Do I understand correctly that Solr uploads file to ZooKeeper?
Regards,
Andor
On Wed, Apr 4, 2018 at 9:50 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> Is it possible to get the current max packet length from the API?
> (version 3.4.x)
>
> If not, I'm guessing that I need to look for the jute.maxbuffer system
> property and fallback to ZkClientConfig.CLIENT_MAX_PACKET_LENGTH_DEFAULT
> if it's not defined.
>
> What I'm trying to do is log a useful error message in Solr if somebody
> tries to upload a file that's too big for what's allowed. The error
> that they get currently is not helpful, and figuring out what went wrong
> seems to require looking at the server log.
>
> Side note: I can see in current code (and the 3.5.2 programmer's guide)
> that the default max packet length is 4MB, but the administrators guide
> (even the 3.5.3 version) still says 1MB.
>
> Thanks,
> Shawn
>
>