You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Patrick Hunt <ph...@gmail.com> on 2016/06/01 22:02:25 UTC
Re: Use-case with lots of child nodes
On Thu, May 26, 2016 at 2:22 AM, Szekeres, Zoltan <
Zoltan.Szekeres@morganstanley.com> wrote:
> Thanks for the response.
> I'm trying to understand what operational stability issues would there be.
> I'm also planning to do a latency test for requests at the time I'm
> requesting getChildren on "/someparentNode".
>
> To give more detail I have a primary and secondary use-case:
> My primary use-case includes having watchers on the children of
> "/someparentNode" and requesting getChildren for
> "/someparentNode/LotsOfChildNodesHere-N" (which only has a couple child
> nodes).
> My secondary use-case would be requesting the children of
> "/someparentNode", which would be only occasionally for reporting purposes
> (which has a lot of child nodes and probably won't be as much as 70k nodes,
> but I hit the limit there).
>
> I'm looking for answers for the following questions:
> What are the stability issues that you think might occur having lots of
> nodes under one node, even if we read them rarely?
> Can I reliable use the "jute.maxbuffer" system property on the client in
> the future?
>
I don't see why not. We have it there as a safety mechanism and to corral
folks towards using it "the right way". Although opinions on what "the
right way" means may differ, which is probably why it's configurable. ;-)
> Looking for answers whether the asymmetry of the default value on client
> side and on server side is accidental or intentional.
>
See this jira, it's currently being discussed:
https://issues.apache.org/jira/browse/ZOOKEEPER-2430
Patrick
>
> Any advice is much appreciated.
>
> Thanks,
> Zoltan Szekeres
>
> -----Original Message-----
> From: Tom Crayford [mailto:tcrayford@heroku.com]
> Sent: Wednesday, May 25, 2016 12:06 PM
> To: user@zookeeper.apache.org
> Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure)
> Subject: Re: Use-case with lots of child nodes
>
> Hi,
>
> I'd recommend rethinking your use case. Zookeeper isn't really great with
> large amounts of data, nor does it handle high volumes of changes that well.
>
> It looks like setting that system property will work for now, but I'd
> expect trying to use such a high volume of child nodes would have severe
> consequences for operational stability.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Wednesday, 25 May 2016, Szekeres, Zoltan <
> Zoltan.Szekeres@morganstanley.com> wrote:
>
> > Hi ZooKeeper users,
> >
> > I have a use-case where I need to create a very large number (~70,000)
> > of child nodes under a parent. These nodes themselves contain no data
> > and will only have a handful of child nodes themselves.
> > e.g.
> > /someparentNode/LotsOfChildNodesHere-1/ACoupleofNodesAtThisLevel
> > /someparentNode/LotsOfChildNodesHere-2/ACoupleofNodesAtThisLevel
> > ...
> > /someparentNode/LotsOfChildNodesHere-70000/ACoupleofNodesAtThisLevel
> >
> > I've read
> > (https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html)
> > there is a limit of 1 MB. But I hit the limit for the getChildren
> > operation around 4 MB. I'm interested in what's causing the difference
> in the limit.
> > I was able to increase the 4MB limit by setting the "jute.maxbuffer"
> > system property at the client. Can I reliably use this system property
> > in the future to set the limit?
> >
> > Any advice is appreciated.
> >
> > Thank you,
> > Zoltan Szekeres
> >
> >
> >
> > ________________________________
> >
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do not
> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
> > Wall Street Reform and Consumer Protection Act. If you have received
> > this communication in error, please destroy all electronic and paper
> > copies; do not disclose, use or act upon the information; and notify
> > the sender immediately. Mistransmission is not intended to waive
> > confidentiality or privilege. Morgan Stanley reserves the right, to
> > the extent permitted under applicable law, to monitor electronic
> > communications. This message is subject to terms available at the
> following link:
> > http://www.morganstanley.com/disclaimers If you cannot access these
> > links, please notify us by reply message and we will send the contents
> > to you. By messaging with Morgan Stanley you consent to the foregoing.
> >
>
>
>
> --------------------------------------------------------------------------------
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. If you have received this
> communication in error, please destroy all electronic and paper copies; do
> not disclose, use or act upon the information; and notify the sender
> immediately. Mistransmission is not intended to waive confidentiality or
> privilege. Morgan Stanley reserves the right, to the extent permitted under
> applicable law, to monitor electronic communications. This message is
> subject to terms available at the following link:
> http://www.morganstanley.com/disclaimers. If you cannot access these
> links, please notify us by reply message and we will send the contents to
> you. By messaging with Morgan Stanley you consent to the foregoing.
RE: Use-case with lots of child nodes
Posted by "Szekeres, Zoltan" <Zo...@morganstanley.com>.
Thanks Camille for the response and checking it in the SVN history. I think what was confusing me that the same system property is used.
Thanks,
Zoltan
-----Original Message-----
From: Camille Fournier [mailto:camille@apache.org]
Sent: Wednesday, June 08, 2016 11:08 PM
To: user@zookeeper.apache.org
Cc: Patrick Hunt; Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure)
Subject: Re: Use-case with lots of child nodes
Ah, yes. So for whatever reason packetLen in the client is set up as 4096 *
1024 if there's no value for jute.maxBuffer set in the client. It's been this way since time immemorial (or rather, I got tired of going back through SVN history but since 2008 anyway). The packetLen is clearly different than the jute maxBuffer because you aren't hitting this limitation in the server (no one node has too much data), you're just hitting this because there are so many children the response size is too big. These seem like somewhat conflated values, it's not clear that maxBuffer should be the same thing as packetLen, and we could probably fix it to have a different variable name if folks want to make it more clear.
C
On Tue, Jun 7, 2016 at 1:13 PM, Szekeres, Zoltan < Zoltan.Szekeres@morganstanley.com> wrote:
> Hi Camille,
>
> Thanks for taking the time looking into this.
>
> An IOException is thrown in
> org.apache.zookeeper.ClientCnxnSocket#readLength(), when receiving the
> response for the getChildren. The default 4MB is set in
> ClientCnxn.packetLen (in zookeeper 3.4.8). I hit the limit with 80k
> nodes where each node name's length is 50 bytes. This does seem only a
> client side check, because w/o change on server side and setting the
> "jute.maxBuffer" system property to 5 MB at the client, I was able to
> get the 80k nodes with the getChildren.
>
> Thanks,
> Zoltan
>
> -----Original Message-----
> From: Camille Fournier [mailto:camille@apache.org]
> Sent: Monday, June 06, 2016 10:39 PM
> To: user@zookeeper.apache.org
> Cc: Patrick Hunt; Gupta, Abhishek (IST); Hejj, Botond (Enterprise
> Infrastructure)
> Subject: Re: Use-case with lots of child nodes
>
> Where exactly are you hitting the limit? Where in the stack is it
> throwing the ioexception with "Unreasonable length"?
> I believe that the way jute checks for this limit is when it
> serializes/deserializes various elements of the data structure, and
> glancing at the code it looks like the deserialization of the children
> actually happens child by child and so isn't counted, but possibly
> some other element of the response is hitting the limit. I have been
> unable to reproduce this myself creating 70K child notes and then
> calling "getChildren" on a normal config so I'm wondering what 4MB
> limit you are hitting.
>
> On Fri, Jun 3, 2016 at 12:48 PM, Szekeres, Zoltan <
> Zoltan.Szekeres@morganstanley.com> wrote:
>
> > Thanks Patrick for the response. We’ll make use of the system
> > property if needed.
> >
> > What’s not clear for me from the jira, I’m not sure if I’ve missed
> > something, is the reason for the asymmetry in the default allowed
> > length of the incoming message on client side and on server side. It
> > is 4MB on the client side and 1MB on the server side. Is the
> > asymmetry intentional or accidental?
> >
> > Thanks,
> > Zoltan Szekeres
> >
> > From: Patrick Hunt [mailto:phunt1@gmail.com]
> > Sent: Thursday, June 02, 2016 12:02 AM
> > To: UserZooKeeper
> > Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure);
> > Szekeres, Zoltan (IST)
> > Subject: Re: Use-case with lots of child nodes
> >
> >
> > On Thu, May 26, 2016 at 2:22 AM, Szekeres, Zoltan <
> > Zoltan.Szekeres@morganstanley.com<mailto:Zoltan.Szekeres@morganstanl
> > ey
> > .com>>
> > wrote:
> > Thanks for the response.
> > I'm trying to understand what operational stability issues would
> > there
> be.
> > I'm also planning to do a latency test for requests at the time I'm
> > requesting getChildren on "/someparentNode".
> >
> > To give more detail I have a primary and secondary use-case:
> > My primary use-case includes having watchers on the children of
> > "/someparentNode" and requesting getChildren for
> > "/someparentNode/LotsOfChildNodesHere-N" (which only has a couple
> > child nodes).
> > My secondary use-case would be requesting the children of
> > "/someparentNode", which would be only occasionally for reporting
> > purposes (which has a lot of child nodes and probably won't be as
> > much as 70k nodes, but I hit the limit there).
> >
> > I'm looking for answers for the following questions:
> > What are the stability issues that you think might occur having lots
> > of nodes under one node, even if we read them rarely?
> > Can I reliable use the "jute.maxbuffer" system property on the
> > client in the future?
> >
> > I don't see why not. We have it there as a safety mechanism and to
> > corral folks towards using it "the right way". Although opinions on
> > what "the right way" means may differ, which is probably why it's
> > configurable. ;-)
> >
> > Looking for answers whether the asymmetry of the default value on
> > client side and on server side is accidental or intentional.
> >
> > See this jira, it's currently being discussed:
> > https://issues.apache.org/jira/browse/ZOOKEEPER-2430
> >
> > Patrick
> >
> >
> > Any advice is much appreciated.
> >
> > Thanks,
> > Zoltan Szekeres
> >
> > -----Original Message-----
> > From: Tom Crayford [mailto:tcrayford@heroku.com<mailto:
> > tcrayford@heroku.com>]
> > Sent: Wednesday, May 25, 2016 12:06 PM
> > To: user@zookeeper.apache.org<ma...@zookeeper.apache.org>
> > Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure)
> > Subject: Re: Use-case with lots of child nodes
> >
> > Hi,
> >
> > I'd recommend rethinking your use case. Zookeeper isn't really great
> > with large amounts of data, nor does it handle high volumes of
> > changes
> that well.
> >
> > It looks like setting that system property will work for now, but
> > I'd expect trying to use such a high volume of child nodes would
> > have severe consequences for operational stability.
> >
> > Thanks
> >
> > Tom Crayford
> > Heroku Kafka
> >
> > On Wednesday, 25 May 2016, Szekeres, Zoltan <
> > Zoltan.Szekeres@morganstanley.com<mailto:Zoltan.Szekeres@morganstanl
> > ey
> > .com>>
> > wrote:
> >
> > > Hi ZooKeeper users,
> > >
> > > I have a use-case where I need to create a very large number
> > > (~70,000) of child nodes under a parent. These nodes themselves
> > > contain no data and will only have a handful of child nodes themselves.
> > > e.g.
> > > /someparentNode/LotsOfChildNodesHere-1/ACoupleofNodesAtThisLevel
> > > /someparentNode/LotsOfChildNodesHere-2/ACoupleofNodesAtThisLevel
> > > ...
> > > /someparentNode/LotsOfChildNodesHere-70000/ACoupleofNodesAtThisLev
> > > el
> > >
> > > I've read
> > > (https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html)
> > > there is a limit of 1 MB. But I hit the limit for the getChildren
> > > operation around 4 MB. I'm interested in what's causing the
> > > difference
> > in the limit.
> > > I was able to increase the 4MB limit by setting the "jute.maxbuffer"
> > > system property at the client. Can I reliably use this system
> > > property in the future to set the limit?
> > >
> > > Any advice is appreciated.
> > >
> > > Thank you,
> > > Zoltan Szekeres
> > >
> > >
> > >
> > > ________________________________
> > >
> > > NOTICE: Morgan Stanley is not acting as a municipal advisor and
> > > the opinions or views contained herein are not intended to be, and
> > > do not constitute, advice within the meaning of Section 975 of the
> > > Dodd-Frank Wall Street Reform and Consumer Protection Act. If you
> > > have received this communication in error, please destroy all
> > > electronic and paper copies; do not disclose, use or act upon the
> > > information; and notify the sender immediately. Mistransmission is
> > > not intended to waive confidentiality or privilege. Morgan Stanley
> > > reserves the right, to the extent permitted under applicable law,
> > > to monitor electronic communications. This message is subject to
> > > terms available at the
> > following link:
> > > http://www.morganstanley.com/disclaimers If you cannot access
> > > these links, please notify us by reply message and we will send
> > > the contents to you. By messaging with Morgan Stanley you consent
> > > to the
> foregoing.
> > >
> >
> >
> > --------------------------------------------------------------------
> > --
> > ----------
> >
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do
> > not constitute, advice within the meaning of Section 975 of the
> > Dodd-Frank Wall Street Reform and Consumer Protection Act. If you
> > have received this communication in error, please destroy all
> > electronic and paper copies; do not disclose, use or act upon the
> > information; and notify the sender immediately. Mistransmission is
> > not intended to waive confidentiality or privilege. Morgan Stanley
> > reserves the right, to the extent permitted under applicable law, to
> > monitor electronic communications. This message is subject to terms
> > available at the
> following link:
> > http://www.morganstanley.com/disclaimers. If you cannot access these
> > links, please notify us by reply message and we will send the
> > contents to you. By messaging with Morgan Stanley you consent to the foregoing.
> >
> >
> >
> > ________________________________
> >
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do
> > not constitute, advice within the meaning of Section 975 of the
> > Dodd-Frank Wall Street Reform and Consumer Protection Act. If you
> > have received this communication in error, please destroy all
> > electronic and paper copies; do not disclose, use or act upon the
> > information; and notify the sender immediately. Mistransmission is
> > not intended to waive confidentiality or privilege. Morgan Stanley
> > reserves the right, to the extent permitted under applicable law, to
> > monitor electronic communications. This message is subject to terms
> > available at the
> following link:
> > http://www.morganstanley.com/disclaimers If you cannot access these
> > links, please notify us by reply message and we will send the
> > contents to you. By messaging with Morgan Stanley you consent to the foregoing.
> >
>
>
>
> ----------------------------------------------------------------------
> ----------
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank
> Wall Street Reform and Consumer Protection Act. If you have received
> this communication in error, please destroy all electronic and paper
> copies; do not disclose, use or act upon the information; and notify
> the sender immediately. Mistransmission is not intended to waive
> confidentiality or privilege. Morgan Stanley reserves the right, to
> the extent permitted under applicable law, to monitor electronic
> communications. This message is subject to terms available at the following link:
> http://www.morganstanley.com/disclaimers. If you cannot access these
> links, please notify us by reply message and we will send the contents
> to you. By messaging with Morgan Stanley you consent to the foregoing.
>
--------------------------------------------------------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
Re: Use-case with lots of child nodes
Posted by Camille Fournier <ca...@apache.org>.
Ah, yes. So for whatever reason packetLen in the client is set up as 4096 *
1024 if there's no value for jute.maxBuffer set in the client. It's been
this way since time immemorial (or rather, I got tired of going back
through SVN history but since 2008 anyway). The packetLen is clearly
different than the jute maxBuffer because you aren't hitting this
limitation in the server (no one node has too much data), you're just
hitting this because there are so many children the response size is too
big. These seem like somewhat conflated values, it's not clear that
maxBuffer should be the same thing as packetLen, and we could probably fix
it to have a different variable name if folks want to make it more clear.
C
On Tue, Jun 7, 2016 at 1:13 PM, Szekeres, Zoltan <
Zoltan.Szekeres@morganstanley.com> wrote:
> Hi Camille,
>
> Thanks for taking the time looking into this.
>
> An IOException is thrown in
> org.apache.zookeeper.ClientCnxnSocket#readLength(), when receiving the
> response for the getChildren. The default 4MB is set in
> ClientCnxn.packetLen (in zookeeper 3.4.8). I hit the limit with 80k nodes
> where each node name's length is 50 bytes. This does seem only a client
> side check, because w/o change on server side and setting the
> "jute.maxBuffer" system property to 5 MB at the client, I was able to get
> the 80k nodes with the getChildren.
>
> Thanks,
> Zoltan
>
> -----Original Message-----
> From: Camille Fournier [mailto:camille@apache.org]
> Sent: Monday, June 06, 2016 10:39 PM
> To: user@zookeeper.apache.org
> Cc: Patrick Hunt; Gupta, Abhishek (IST); Hejj, Botond (Enterprise
> Infrastructure)
> Subject: Re: Use-case with lots of child nodes
>
> Where exactly are you hitting the limit? Where in the stack is it throwing
> the ioexception with "Unreasonable length"?
> I believe that the way jute checks for this limit is when it
> serializes/deserializes various elements of the data structure, and
> glancing at the code it looks like the deserialization of the children
> actually happens child by child and so isn't counted, but possibly some
> other element of the response is hitting the limit. I have been unable to
> reproduce this myself creating 70K child notes and then calling
> "getChildren" on a normal config so I'm wondering what 4MB limit you are
> hitting.
>
> On Fri, Jun 3, 2016 at 12:48 PM, Szekeres, Zoltan <
> Zoltan.Szekeres@morganstanley.com> wrote:
>
> > Thanks Patrick for the response. We’ll make use of the system property
> > if needed.
> >
> > What’s not clear for me from the jira, I’m not sure if I’ve missed
> > something, is the reason for the asymmetry in the default allowed
> > length of the incoming message on client side and on server side. It
> > is 4MB on the client side and 1MB on the server side. Is the asymmetry
> > intentional or accidental?
> >
> > Thanks,
> > Zoltan Szekeres
> >
> > From: Patrick Hunt [mailto:phunt1@gmail.com]
> > Sent: Thursday, June 02, 2016 12:02 AM
> > To: UserZooKeeper
> > Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure);
> > Szekeres, Zoltan (IST)
> > Subject: Re: Use-case with lots of child nodes
> >
> >
> > On Thu, May 26, 2016 at 2:22 AM, Szekeres, Zoltan <
> > Zoltan.Szekeres@morganstanley.com<mailto:Zoltan.Szekeres@morganstanley
> > .com>>
> > wrote:
> > Thanks for the response.
> > I'm trying to understand what operational stability issues would there
> be.
> > I'm also planning to do a latency test for requests at the time I'm
> > requesting getChildren on "/someparentNode".
> >
> > To give more detail I have a primary and secondary use-case:
> > My primary use-case includes having watchers on the children of
> > "/someparentNode" and requesting getChildren for
> > "/someparentNode/LotsOfChildNodesHere-N" (which only has a couple
> > child nodes).
> > My secondary use-case would be requesting the children of
> > "/someparentNode", which would be only occasionally for reporting
> > purposes (which has a lot of child nodes and probably won't be as much
> > as 70k nodes, but I hit the limit there).
> >
> > I'm looking for answers for the following questions:
> > What are the stability issues that you think might occur having lots
> > of nodes under one node, even if we read them rarely?
> > Can I reliable use the "jute.maxbuffer" system property on the client
> > in the future?
> >
> > I don't see why not. We have it there as a safety mechanism and to
> > corral folks towards using it "the right way". Although opinions on
> > what "the right way" means may differ, which is probably why it's
> > configurable. ;-)
> >
> > Looking for answers whether the asymmetry of the default value on
> > client side and on server side is accidental or intentional.
> >
> > See this jira, it's currently being discussed:
> > https://issues.apache.org/jira/browse/ZOOKEEPER-2430
> >
> > Patrick
> >
> >
> > Any advice is much appreciated.
> >
> > Thanks,
> > Zoltan Szekeres
> >
> > -----Original Message-----
> > From: Tom Crayford [mailto:tcrayford@heroku.com<mailto:
> > tcrayford@heroku.com>]
> > Sent: Wednesday, May 25, 2016 12:06 PM
> > To: user@zookeeper.apache.org<ma...@zookeeper.apache.org>
> > Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure)
> > Subject: Re: Use-case with lots of child nodes
> >
> > Hi,
> >
> > I'd recommend rethinking your use case. Zookeeper isn't really great
> > with large amounts of data, nor does it handle high volumes of changes
> that well.
> >
> > It looks like setting that system property will work for now, but I'd
> > expect trying to use such a high volume of child nodes would have
> > severe consequences for operational stability.
> >
> > Thanks
> >
> > Tom Crayford
> > Heroku Kafka
> >
> > On Wednesday, 25 May 2016, Szekeres, Zoltan <
> > Zoltan.Szekeres@morganstanley.com<mailto:Zoltan.Szekeres@morganstanley
> > .com>>
> > wrote:
> >
> > > Hi ZooKeeper users,
> > >
> > > I have a use-case where I need to create a very large number
> > > (~70,000) of child nodes under a parent. These nodes themselves
> > > contain no data and will only have a handful of child nodes themselves.
> > > e.g.
> > > /someparentNode/LotsOfChildNodesHere-1/ACoupleofNodesAtThisLevel
> > > /someparentNode/LotsOfChildNodesHere-2/ACoupleofNodesAtThisLevel
> > > ...
> > > /someparentNode/LotsOfChildNodesHere-70000/ACoupleofNodesAtThisLevel
> > >
> > > I've read
> > > (https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html)
> > > there is a limit of 1 MB. But I hit the limit for the getChildren
> > > operation around 4 MB. I'm interested in what's causing the
> > > difference
> > in the limit.
> > > I was able to increase the 4MB limit by setting the "jute.maxbuffer"
> > > system property at the client. Can I reliably use this system
> > > property in the future to set the limit?
> > >
> > > Any advice is appreciated.
> > >
> > > Thank you,
> > > Zoltan Szekeres
> > >
> > >
> > >
> > > ________________________________
> > >
> > > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > > opinions or views contained herein are not intended to be, and do
> > > not constitute, advice within the meaning of Section 975 of the
> > > Dodd-Frank Wall Street Reform and Consumer Protection Act. If you
> > > have received this communication in error, please destroy all
> > > electronic and paper copies; do not disclose, use or act upon the
> > > information; and notify the sender immediately. Mistransmission is
> > > not intended to waive confidentiality or privilege. Morgan Stanley
> > > reserves the right, to the extent permitted under applicable law, to
> > > monitor electronic communications. This message is subject to terms
> > > available at the
> > following link:
> > > http://www.morganstanley.com/disclaimers If you cannot access these
> > > links, please notify us by reply message and we will send the
> > > contents to you. By messaging with Morgan Stanley you consent to the
> foregoing.
> > >
> >
> >
> > ----------------------------------------------------------------------
> > ----------
> >
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do not
> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
> > Wall Street Reform and Consumer Protection Act. If you have received
> > this communication in error, please destroy all electronic and paper
> > copies; do not disclose, use or act upon the information; and notify
> > the sender immediately. Mistransmission is not intended to waive
> > confidentiality or privilege. Morgan Stanley reserves the right, to
> > the extent permitted under applicable law, to monitor electronic
> > communications. This message is subject to terms available at the
> following link:
> > http://www.morganstanley.com/disclaimers. If you cannot access these
> > links, please notify us by reply message and we will send the contents
> > to you. By messaging with Morgan Stanley you consent to the foregoing.
> >
> >
> >
> > ________________________________
> >
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do not
> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
> > Wall Street Reform and Consumer Protection Act. If you have received
> > this communication in error, please destroy all electronic and paper
> > copies; do not disclose, use or act upon the information; and notify
> > the sender immediately. Mistransmission is not intended to waive
> > confidentiality or privilege. Morgan Stanley reserves the right, to
> > the extent permitted under applicable law, to monitor electronic
> > communications. This message is subject to terms available at the
> following link:
> > http://www.morganstanley.com/disclaimers If you cannot access these
> > links, please notify us by reply message and we will send the contents
> > to you. By messaging with Morgan Stanley you consent to the foregoing.
> >
>
>
>
> --------------------------------------------------------------------------------
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. If you have received this
> communication in error, please destroy all electronic and paper copies; do
> not disclose, use or act upon the information; and notify the sender
> immediately. Mistransmission is not intended to waive confidentiality or
> privilege. Morgan Stanley reserves the right, to the extent permitted under
> applicable law, to monitor electronic communications. This message is
> subject to terms available at the following link:
> http://www.morganstanley.com/disclaimers. If you cannot access these
> links, please notify us by reply message and we will send the contents to
> you. By messaging with Morgan Stanley you consent to the foregoing.
>
RE: Use-case with lots of child nodes
Posted by "Szekeres, Zoltan" <Zo...@morganstanley.com>.
Hi Camille,
Thanks for taking the time looking into this.
An IOException is thrown in org.apache.zookeeper.ClientCnxnSocket#readLength(), when receiving the response for the getChildren. The default 4MB is set in ClientCnxn.packetLen (in zookeeper 3.4.8). I hit the limit with 80k nodes where each node name's length is 50 bytes. This does seem only a client side check, because w/o change on server side and setting the "jute.maxBuffer" system property to 5 MB at the client, I was able to get the 80k nodes with the getChildren.
Thanks,
Zoltan
-----Original Message-----
From: Camille Fournier [mailto:camille@apache.org]
Sent: Monday, June 06, 2016 10:39 PM
To: user@zookeeper.apache.org
Cc: Patrick Hunt; Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure)
Subject: Re: Use-case with lots of child nodes
Where exactly are you hitting the limit? Where in the stack is it throwing the ioexception with "Unreasonable length"?
I believe that the way jute checks for this limit is when it serializes/deserializes various elements of the data structure, and glancing at the code it looks like the deserialization of the children actually happens child by child and so isn't counted, but possibly some other element of the response is hitting the limit. I have been unable to reproduce this myself creating 70K child notes and then calling "getChildren" on a normal config so I'm wondering what 4MB limit you are hitting.
On Fri, Jun 3, 2016 at 12:48 PM, Szekeres, Zoltan < Zoltan.Szekeres@morganstanley.com> wrote:
> Thanks Patrick for the response. We’ll make use of the system property
> if needed.
>
> What’s not clear for me from the jira, I’m not sure if I’ve missed
> something, is the reason for the asymmetry in the default allowed
> length of the incoming message on client side and on server side. It
> is 4MB on the client side and 1MB on the server side. Is the asymmetry
> intentional or accidental?
>
> Thanks,
> Zoltan Szekeres
>
> From: Patrick Hunt [mailto:phunt1@gmail.com]
> Sent: Thursday, June 02, 2016 12:02 AM
> To: UserZooKeeper
> Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure);
> Szekeres, Zoltan (IST)
> Subject: Re: Use-case with lots of child nodes
>
>
> On Thu, May 26, 2016 at 2:22 AM, Szekeres, Zoltan <
> Zoltan.Szekeres@morganstanley.com<mailto:Zoltan.Szekeres@morganstanley
> .com>>
> wrote:
> Thanks for the response.
> I'm trying to understand what operational stability issues would there be.
> I'm also planning to do a latency test for requests at the time I'm
> requesting getChildren on "/someparentNode".
>
> To give more detail I have a primary and secondary use-case:
> My primary use-case includes having watchers on the children of
> "/someparentNode" and requesting getChildren for
> "/someparentNode/LotsOfChildNodesHere-N" (which only has a couple
> child nodes).
> My secondary use-case would be requesting the children of
> "/someparentNode", which would be only occasionally for reporting
> purposes (which has a lot of child nodes and probably won't be as much
> as 70k nodes, but I hit the limit there).
>
> I'm looking for answers for the following questions:
> What are the stability issues that you think might occur having lots
> of nodes under one node, even if we read them rarely?
> Can I reliable use the "jute.maxbuffer" system property on the client
> in the future?
>
> I don't see why not. We have it there as a safety mechanism and to
> corral folks towards using it "the right way". Although opinions on
> what "the right way" means may differ, which is probably why it's
> configurable. ;-)
>
> Looking for answers whether the asymmetry of the default value on
> client side and on server side is accidental or intentional.
>
> See this jira, it's currently being discussed:
> https://issues.apache.org/jira/browse/ZOOKEEPER-2430
>
> Patrick
>
>
> Any advice is much appreciated.
>
> Thanks,
> Zoltan Szekeres
>
> -----Original Message-----
> From: Tom Crayford [mailto:tcrayford@heroku.com<mailto:
> tcrayford@heroku.com>]
> Sent: Wednesday, May 25, 2016 12:06 PM
> To: user@zookeeper.apache.org<ma...@zookeeper.apache.org>
> Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure)
> Subject: Re: Use-case with lots of child nodes
>
> Hi,
>
> I'd recommend rethinking your use case. Zookeeper isn't really great
> with large amounts of data, nor does it handle high volumes of changes that well.
>
> It looks like setting that system property will work for now, but I'd
> expect trying to use such a high volume of child nodes would have
> severe consequences for operational stability.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Wednesday, 25 May 2016, Szekeres, Zoltan <
> Zoltan.Szekeres@morganstanley.com<mailto:Zoltan.Szekeres@morganstanley
> .com>>
> wrote:
>
> > Hi ZooKeeper users,
> >
> > I have a use-case where I need to create a very large number
> > (~70,000) of child nodes under a parent. These nodes themselves
> > contain no data and will only have a handful of child nodes themselves.
> > e.g.
> > /someparentNode/LotsOfChildNodesHere-1/ACoupleofNodesAtThisLevel
> > /someparentNode/LotsOfChildNodesHere-2/ACoupleofNodesAtThisLevel
> > ...
> > /someparentNode/LotsOfChildNodesHere-70000/ACoupleofNodesAtThisLevel
> >
> > I've read
> > (https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html)
> > there is a limit of 1 MB. But I hit the limit for the getChildren
> > operation around 4 MB. I'm interested in what's causing the
> > difference
> in the limit.
> > I was able to increase the 4MB limit by setting the "jute.maxbuffer"
> > system property at the client. Can I reliably use this system
> > property in the future to set the limit?
> >
> > Any advice is appreciated.
> >
> > Thank you,
> > Zoltan Szekeres
> >
> >
> >
> > ________________________________
> >
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do
> > not constitute, advice within the meaning of Section 975 of the
> > Dodd-Frank Wall Street Reform and Consumer Protection Act. If you
> > have received this communication in error, please destroy all
> > electronic and paper copies; do not disclose, use or act upon the
> > information; and notify the sender immediately. Mistransmission is
> > not intended to waive confidentiality or privilege. Morgan Stanley
> > reserves the right, to the extent permitted under applicable law, to
> > monitor electronic communications. This message is subject to terms
> > available at the
> following link:
> > http://www.morganstanley.com/disclaimers If you cannot access these
> > links, please notify us by reply message and we will send the
> > contents to you. By messaging with Morgan Stanley you consent to the foregoing.
> >
>
>
> ----------------------------------------------------------------------
> ----------
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank
> Wall Street Reform and Consumer Protection Act. If you have received
> this communication in error, please destroy all electronic and paper
> copies; do not disclose, use or act upon the information; and notify
> the sender immediately. Mistransmission is not intended to waive
> confidentiality or privilege. Morgan Stanley reserves the right, to
> the extent permitted under applicable law, to monitor electronic
> communications. This message is subject to terms available at the following link:
> http://www.morganstanley.com/disclaimers. If you cannot access these
> links, please notify us by reply message and we will send the contents
> to you. By messaging with Morgan Stanley you consent to the foregoing.
>
>
>
> ________________________________
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank
> Wall Street Reform and Consumer Protection Act. If you have received
> this communication in error, please destroy all electronic and paper
> copies; do not disclose, use or act upon the information; and notify
> the sender immediately. Mistransmission is not intended to waive
> confidentiality or privilege. Morgan Stanley reserves the right, to
> the extent permitted under applicable law, to monitor electronic
> communications. This message is subject to terms available at the following link:
> http://www.morganstanley.com/disclaimers If you cannot access these
> links, please notify us by reply message and we will send the contents
> to you. By messaging with Morgan Stanley you consent to the foregoing.
>
--------------------------------------------------------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
Re: Use-case with lots of child nodes
Posted by Camille Fournier <ca...@apache.org>.
Where exactly are you hitting the limit? Where in the stack is it throwing
the ioexception with "Unreasonable length"?
I believe that the way jute checks for this limit is when it
serializes/deserializes various elements of the data structure, and
glancing at the code it looks like the deserialization of the children
actually happens child by child and so isn't counted, but possibly some
other element of the response is hitting the limit. I have been unable to
reproduce this myself creating 70K child notes and then calling
"getChildren" on a normal config so I'm wondering what 4MB limit you are
hitting.
On Fri, Jun 3, 2016 at 12:48 PM, Szekeres, Zoltan <
Zoltan.Szekeres@morganstanley.com> wrote:
> Thanks Patrick for the response. We’ll make use of the system property if
> needed.
>
> What’s not clear for me from the jira, I’m not sure if I’ve missed
> something, is the reason for the asymmetry in the default allowed length of
> the incoming message on client side and on server side. It is 4MB on the
> client side and 1MB on the server side. Is the asymmetry intentional or
> accidental?
>
> Thanks,
> Zoltan Szekeres
>
> From: Patrick Hunt [mailto:phunt1@gmail.com]
> Sent: Thursday, June 02, 2016 12:02 AM
> To: UserZooKeeper
> Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure);
> Szekeres, Zoltan (IST)
> Subject: Re: Use-case with lots of child nodes
>
>
> On Thu, May 26, 2016 at 2:22 AM, Szekeres, Zoltan <
> Zoltan.Szekeres@morganstanley.com<ma...@morganstanley.com>>
> wrote:
> Thanks for the response.
> I'm trying to understand what operational stability issues would there be.
> I'm also planning to do a latency test for requests at the time I'm
> requesting getChildren on "/someparentNode".
>
> To give more detail I have a primary and secondary use-case:
> My primary use-case includes having watchers on the children of
> "/someparentNode" and requesting getChildren for
> "/someparentNode/LotsOfChildNodesHere-N" (which only has a couple child
> nodes).
> My secondary use-case would be requesting the children of
> "/someparentNode", which would be only occasionally for reporting purposes
> (which has a lot of child nodes and probably won't be as much as 70k nodes,
> but I hit the limit there).
>
> I'm looking for answers for the following questions:
> What are the stability issues that you think might occur having lots of
> nodes under one node, even if we read them rarely?
> Can I reliable use the "jute.maxbuffer" system property on the client in
> the future?
>
> I don't see why not. We have it there as a safety mechanism and to corral
> folks towards using it "the right way". Although opinions on what "the
> right way" means may differ, which is probably why it's configurable. ;-)
>
> Looking for answers whether the asymmetry of the default value on client
> side and on server side is accidental or intentional.
>
> See this jira, it's currently being discussed:
> https://issues.apache.org/jira/browse/ZOOKEEPER-2430
>
> Patrick
>
>
> Any advice is much appreciated.
>
> Thanks,
> Zoltan Szekeres
>
> -----Original Message-----
> From: Tom Crayford [mailto:tcrayford@heroku.com<mailto:
> tcrayford@heroku.com>]
> Sent: Wednesday, May 25, 2016 12:06 PM
> To: user@zookeeper.apache.org<ma...@zookeeper.apache.org>
> Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure)
> Subject: Re: Use-case with lots of child nodes
>
> Hi,
>
> I'd recommend rethinking your use case. Zookeeper isn't really great with
> large amounts of data, nor does it handle high volumes of changes that well.
>
> It looks like setting that system property will work for now, but I'd
> expect trying to use such a high volume of child nodes would have severe
> consequences for operational stability.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Wednesday, 25 May 2016, Szekeres, Zoltan <
> Zoltan.Szekeres@morganstanley.com<ma...@morganstanley.com>>
> wrote:
>
> > Hi ZooKeeper users,
> >
> > I have a use-case where I need to create a very large number (~70,000)
> > of child nodes under a parent. These nodes themselves contain no data
> > and will only have a handful of child nodes themselves.
> > e.g.
> > /someparentNode/LotsOfChildNodesHere-1/ACoupleofNodesAtThisLevel
> > /someparentNode/LotsOfChildNodesHere-2/ACoupleofNodesAtThisLevel
> > ...
> > /someparentNode/LotsOfChildNodesHere-70000/ACoupleofNodesAtThisLevel
> >
> > I've read
> > (https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html)
> > there is a limit of 1 MB. But I hit the limit for the getChildren
> > operation around 4 MB. I'm interested in what's causing the difference
> in the limit.
> > I was able to increase the 4MB limit by setting the "jute.maxbuffer"
> > system property at the client. Can I reliably use this system property
> > in the future to set the limit?
> >
> > Any advice is appreciated.
> >
> > Thank you,
> > Zoltan Szekeres
> >
> >
> >
> > ________________________________
> >
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do not
> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
> > Wall Street Reform and Consumer Protection Act. If you have received
> > this communication in error, please destroy all electronic and paper
> > copies; do not disclose, use or act upon the information; and notify
> > the sender immediately. Mistransmission is not intended to waive
> > confidentiality or privilege. Morgan Stanley reserves the right, to
> > the extent permitted under applicable law, to monitor electronic
> > communications. This message is subject to terms available at the
> following link:
> > http://www.morganstanley.com/disclaimers If you cannot access these
> > links, please notify us by reply message and we will send the contents
> > to you. By messaging with Morgan Stanley you consent to the foregoing.
> >
>
>
> --------------------------------------------------------------------------------
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. If you have received this
> communication in error, please destroy all electronic and paper copies; do
> not disclose, use or act upon the information; and notify the sender
> immediately. Mistransmission is not intended to waive confidentiality or
> privilege. Morgan Stanley reserves the right, to the extent permitted under
> applicable law, to monitor electronic communications. This message is
> subject to terms available at the following link:
> http://www.morganstanley.com/disclaimers. If you cannot access these
> links, please notify us by reply message and we will send the contents to
> you. By messaging with Morgan Stanley you consent to the foregoing.
>
>
>
> ________________________________
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. If you have received this
> communication in error, please destroy all electronic and paper copies; do
> not disclose, use or act upon the information; and notify the sender
> immediately. Mistransmission is not intended to waive confidentiality or
> privilege. Morgan Stanley reserves the right, to the extent permitted under
> applicable law, to monitor electronic communications. This message is
> subject to terms available at the following link:
> http://www.morganstanley.com/disclaimers If you cannot access these
> links, please notify us by reply message and we will send the contents to
> you. By messaging with Morgan Stanley you consent to the foregoing.
>
RE: Use-case with lots of child nodes
Posted by "Szekeres, Zoltan" <Zo...@morganstanley.com>.
Thanks Patrick for the response. We’ll make use of the system property if needed.
What’s not clear for me from the jira, I’m not sure if I’ve missed something, is the reason for the asymmetry in the default allowed length of the incoming message on client side and on server side. It is 4MB on the client side and 1MB on the server side. Is the asymmetry intentional or accidental?
Thanks,
Zoltan Szekeres
From: Patrick Hunt [mailto:phunt1@gmail.com]
Sent: Thursday, June 02, 2016 12:02 AM
To: UserZooKeeper
Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure); Szekeres, Zoltan (IST)
Subject: Re: Use-case with lots of child nodes
On Thu, May 26, 2016 at 2:22 AM, Szekeres, Zoltan <Zo...@morganstanley.com>> wrote:
Thanks for the response.
I'm trying to understand what operational stability issues would there be. I'm also planning to do a latency test for requests at the time I'm requesting getChildren on "/someparentNode".
To give more detail I have a primary and secondary use-case:
My primary use-case includes having watchers on the children of "/someparentNode" and requesting getChildren for "/someparentNode/LotsOfChildNodesHere-N" (which only has a couple child nodes).
My secondary use-case would be requesting the children of "/someparentNode", which would be only occasionally for reporting purposes (which has a lot of child nodes and probably won't be as much as 70k nodes, but I hit the limit there).
I'm looking for answers for the following questions:
What are the stability issues that you think might occur having lots of nodes under one node, even if we read them rarely?
Can I reliable use the "jute.maxbuffer" system property on the client in the future?
I don't see why not. We have it there as a safety mechanism and to corral folks towards using it "the right way". Although opinions on what "the right way" means may differ, which is probably why it's configurable. ;-)
Looking for answers whether the asymmetry of the default value on client side and on server side is accidental or intentional.
See this jira, it's currently being discussed: https://issues.apache.org/jira/browse/ZOOKEEPER-2430
Patrick
Any advice is much appreciated.
Thanks,
Zoltan Szekeres
-----Original Message-----
From: Tom Crayford [mailto:tcrayford@heroku.com<ma...@heroku.com>]
Sent: Wednesday, May 25, 2016 12:06 PM
To: user@zookeeper.apache.org<ma...@zookeeper.apache.org>
Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure)
Subject: Re: Use-case with lots of child nodes
Hi,
I'd recommend rethinking your use case. Zookeeper isn't really great with large amounts of data, nor does it handle high volumes of changes that well.
It looks like setting that system property will work for now, but I'd expect trying to use such a high volume of child nodes would have severe consequences for operational stability.
Thanks
Tom Crayford
Heroku Kafka
On Wednesday, 25 May 2016, Szekeres, Zoltan < Zoltan.Szekeres@morganstanley.com<ma...@morganstanley.com>> wrote:
> Hi ZooKeeper users,
>
> I have a use-case where I need to create a very large number (~70,000)
> of child nodes under a parent. These nodes themselves contain no data
> and will only have a handful of child nodes themselves.
> e.g.
> /someparentNode/LotsOfChildNodesHere-1/ACoupleofNodesAtThisLevel
> /someparentNode/LotsOfChildNodesHere-2/ACoupleofNodesAtThisLevel
> ...
> /someparentNode/LotsOfChildNodesHere-70000/ACoupleofNodesAtThisLevel
>
> I've read
> (https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html)
> there is a limit of 1 MB. But I hit the limit for the getChildren
> operation around 4 MB. I'm interested in what's causing the difference in the limit.
> I was able to increase the 4MB limit by setting the "jute.maxbuffer"
> system property at the client. Can I reliably use this system property
> in the future to set the limit?
>
> Any advice is appreciated.
>
> Thank you,
> Zoltan Szekeres
>
>
>
> ________________________________
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank
> Wall Street Reform and Consumer Protection Act. If you have received
> this communication in error, please destroy all electronic and paper
> copies; do not disclose, use or act upon the information; and notify
> the sender immediately. Mistransmission is not intended to waive
> confidentiality or privilege. Morgan Stanley reserves the right, to
> the extent permitted under applicable law, to monitor electronic
> communications. This message is subject to terms available at the following link:
> http://www.morganstanley.com/disclaimers If you cannot access these
> links, please notify us by reply message and we will send the contents
> to you. By messaging with Morgan Stanley you consent to the foregoing.
>
--------------------------------------------------------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
________________________________
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.