You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by li li <li...@gmail.com> on 2010/03/31 11:57:38 UTC
the error
Dear developer,
when I use the zookeeper in a cluster of two machine.I find problem when I
make the over 17 threads read data in one znode.
I will get the err info as follows.I will be greatly appreciated if you can
tell the reason why this happy.
2010-03-31 17:46:59,718 - WARN [main-SendThread:ClientCnxn$SendThread@967]
- Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@3a953a95
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
2010-03-31 17:46:59,734 - WARN [main-SendThread:ClientCnxn$SendThread@967]
- Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@3dce3dce
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
I am looking forward your reply.Thanks for your reading
Re: the error
Posted by Patrick Hunt <ph...@apache.org>.
Hi Li, when you say 17 threads reading a znode, do you mean that you
have 17 threads each creating a session and using that session to read a
znode? If so it's probably due to this:
http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
see the parameter "maxClientCnxns", this defaults to 10. Try setting the
value to 0 (no limit) in your server config, restart server, and see if
that helps.
Patrick
li li wrote:
> Dear developer,
> when I use the zookeeper in a cluster of two machine.I find problem when I
> make the over 17 threads read data in one znode.
> I will get the err info as follows.I will be greatly appreciated if you can
> tell the reason why this happy.
>
> 2010-03-31 17:46:59,718 - WARN [main-SendThread:ClientCnxn$SendThread@967]
> - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@3a953a95
> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> lim=4 cap=4]
> at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
> 2010-03-31 17:46:59,734 - WARN [main-SendThread:ClientCnxn$SendThread@967]
> - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@3dce3dce
> java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
> lim=4 cap=4]
> at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
>
> I am looking forward your reply.Thanks for your reading
>
Re: the error
Posted by Ted Dunning <te...@gmail.com>.
Suppose a machine has probability of soft-failure p_1 and catastrophic p_2
<< p_1. Assume that two machines have independent failure modes.
Probably of soft failure of a one machine cluster = p_1, two machine cluster
= probability of soft failure of 1 or 2 machines + probability of one
machine hard failure = 1-(1-p_1)^2 + 2 p_2 (1-p_1-p_2) \approx 2 p_1
Probability of hard failure of one machine cluster = p_2, two machine
cluster = probability of double hard failure = p_2 ^ 2
So I should have said that the probability of soft failures increases
significantly and the probability of hard failures drops dramatically.
Standard measures can be used to decrease the single machine catastrophic
failure rate substantially. The net is higher failure rate for the two
machine cluster.
On the other hand, the soft failure rate for a three machine cluster is
roughly p_1^2 and the hard failure rate is about p_2^3. These are
dramatically better than either the one or two machine case.
On Wed, Mar 31, 2010 at 1:06 PM, David Rosenstrauch <da...@darose.net>wrote:
> Using two machines running ZK will actually decrease your reliability
>> compared to using a single machine. Consider using one machine or three.
>>
>
> ?
>
> Not meaning to pull the thread off-topic, but I don't understand why this
> should be the case. Can you elaborate?
Re: the error
Posted by Ted Dunning <te...@gmail.com>.
As I pointed out in my response, you should distinguish hard and soft
failures. If one machine fails even catastrophically, you can provide a new
machine to replace it, thus converting a hard failure into a soft one.
The conclusion is the same. Three machines is vastly better than one or
two.
On Wed, Mar 31, 2010 at 1:12 PM, Henry Robinson <he...@cloudera.com> wrote:
> With majority-based quorums, a 2 node ensemble will fail if either machine
> fails. This is the same as a 1 node ensemble - if one machine fails, the
> ensemble fails.
>
> But with the 2 nodes, you have (roughly) double the probability of
> experiencing a single failure, with the same failure profile.
>
Re: the error
Posted by Henry Robinson <he...@cloudera.com>.
Using two machines running ZK will actually decrease your reliability
>> compared to using a single machine. Consider using one machine or three.
>>
>
> ?
>
> Not meaning to pull the thread off-topic, but I don't understand why this
> should be the case. Can you elaborate?
>
>
With majority-based quorums, a 2 node ensemble will fail if either machine
fails. This is the same as a 1 node ensemble - if one machine fails, the
ensemble fails.
But with the 2 nodes, you have (roughly) double the probability of
experiencing a single failure, with the same failure profile.
cheers,
Henry
> Thanks,
>
> DR
>
--
Henry Robinson
Software Engineer
Cloudera
415-994-6679
Re: the error
Posted by David Rosenstrauch <da...@darose.net>.
On 03/31/2010 02:10 PM, Ted Dunning wrote:
> To add to Patrick's comments, I hope you mean that you are connecting to ZK
> from a cluster of two machines rather than having only two machines that
> form a ZK cluster.
>
> Using two machines running ZK will actually decrease your reliability
> compared to using a single machine. Consider using one machine or three.
?
Not meaning to pull the thread off-topic, but I don't understand why
this should be the case. Can you elaborate?
Thanks,
DR
Re: the error
Posted by Ted Dunning <te...@gmail.com>.
To add to Patrick's comments, I hope you mean that you are connecting to ZK
from a cluster of two machines rather than having only two machines that
form a ZK cluster.
Using two machines running ZK will actually decrease your reliability
compared to using a single machine. Consider using one machine or three.
On Wed, Mar 31, 2010 at 2:57 AM, li li <li...@gmail.com> wrote:
> when I use the zookeeper in a cluster of two machine.I find problem when I