You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Joshua Tuberville (JIRA)" <ji...@apache.org> on 2009/01/13 19:16:01 UTC
[jira] Created: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
getChildren can fail for large numbers of children
--------------------------------------------------
Key: ZOOKEEPER-272
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
Project: Zookeeper
Issue Type: Bug
Reporter: Joshua Tuberville
Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated ZOOKEEPER-272:
------------------------------------
Attachment: ZOOKEEPER-272.patch
this patch allows tthe user to set java property jute.maxbuffer at the client. This value is used at the client to throw an error if the read len of value exceeds the set value. By default its set to 4MB.
if the user wants to cahnge this value, he can do it via setting java property
-Djute.maxbuffer=someintvalue
> getChildren can fail for large numbers of children
> --------------------------------------------------
>
> Key: ZOOKEEPER-272
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Joshua Tuberville
> Assignee: Mahadev konar
> Fix For: 3.1.0
>
> Attachments: ZOOKEEPER-272.patch
>
>
> Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663422#action_12663422 ]
Mahadev konar commented on ZOOKEEPER-272:
-----------------------------------------
returning an offset for getchildren is a good suggestion.... should we just fix the limit in this jira and open another one for getchildren returning an offset ? fixing the limit would be easir to do and can be a part of 3.1 release.
> getChildren can fail for large numbers of children
> --------------------------------------------------
>
> Key: ZOOKEEPER-272
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Joshua Tuberville
>
> Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated ZOOKEEPER-272:
------------------------------------
Hadoop Flags: [Reviewed]
> getChildren can fail for large numbers of children
> --------------------------------------------------
>
> Key: ZOOKEEPER-272
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Joshua Tuberville
> Assignee: Mahadev konar
> Fix For: 3.1.0
>
> Attachments: ZOOKEEPER-272.patch
>
>
> Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668652#action_12668652 ]
Benjamin Reed commented on ZOOKEEPER-272:
-----------------------------------------
This is a good comment, but it should be tracked in a separate Jira.
> getChildren can fail for large numbers of children
> --------------------------------------------------
>
> Key: ZOOKEEPER-272
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Joshua Tuberville
> Assignee: Mahadev konar
> Fix For: 3.1.0
>
> Attachments: ZOOKEEPER-272.patch
>
>
> Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar reassigned ZOOKEEPER-272:
---------------------------------------
Assignee: Mahadev konar
> getChildren can fail for large numbers of children
> --------------------------------------------------
>
> Key: ZOOKEEPER-272
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Joshua Tuberville
> Assignee: Mahadev konar
> Fix For: 3.1.0
>
>
> Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated ZOOKEEPER-272:
------------------------------------
Fix Version/s: 3.1.0
i think the limit should be fixed in 3.1 release and we can open a new jira for getchildren api....
> getChildren can fail for large numbers of children
> --------------------------------------------------
>
> Key: ZOOKEEPER-272
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Joshua Tuberville
> Fix For: 3.1.0
>
>
> Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668373#action_12668373 ]
Mahadev konar commented on ZOOKEEPER-272:
-----------------------------------------
sorry i should have mentioned this earlier. I planned to open a jira for exactly what you mentioned above. Thsi fix is a quick fix for the upcoming release.
> getChildren can fail for large numbers of children
> --------------------------------------------------
>
> Key: ZOOKEEPER-272
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Joshua Tuberville
> Assignee: Mahadev konar
> Fix For: 3.1.0
>
> Attachments: ZOOKEEPER-272.patch
>
>
> Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated ZOOKEEPER-272:
------------------------------------
Status: Patch Available (was: Open)
> getChildren can fail for large numbers of children
> --------------------------------------------------
>
> Key: ZOOKEEPER-272
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Joshua Tuberville
> Assignee: Mahadev konar
> Fix For: 3.1.0
>
> Attachments: ZOOKEEPER-272.patch
>
>
> Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12669570#action_12669570 ]
Hudson commented on ZOOKEEPER-272:
----------------------------------
Integrated in ZooKeeper-trunk #217 (See [http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/217/])
> getChildren can fail for large numbers of children
> --------------------------------------------------
>
> Key: ZOOKEEPER-272
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Joshua Tuberville
> Assignee: Mahadev konar
> Fix For: 3.1.0
>
> Attachments: ZOOKEEPER-272.patch
>
>
> Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
Posted by "Joshua Tuberville (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668532#action_12668532 ]
Joshua Tuberville commented on ZOOKEEPER-272:
---------------------------------------------
Consider returning Iterable<String> instead of Iterator<String> so you can take advantage of Java 5 foreach idiom
for (String child : foo.getChildren()) {
// Something done with child
}
http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Iterable.html
> getChildren can fail for large numbers of children
> --------------------------------------------------
>
> Key: ZOOKEEPER-272
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Joshua Tuberville
> Assignee: Mahadev konar
> Fix For: 3.1.0
>
> Attachments: ZOOKEEPER-272.patch
>
>
> Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668371#action_12668371 ]
Nitay Joffe commented on ZOOKEEPER-272:
---------------------------------------
What about having getChildren() return an Iterator<String> instead of a List<String>? Using such an abstraction allows us to put the implementation details of what "walking the list children" mean inside ZooKeeper instead of client code. With an Iterator we can make the next() call handle the multiple offset getChildren() calls under the hood, rather than put this burden on the client. Furthermore, if we have, say, some compressed protocol, a lot of space/cpu can be saved by only decompressing/storing the current child being worked on.
> getChildren can fail for large numbers of children
> --------------------------------------------------
>
> Key: ZOOKEEPER-272
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Joshua Tuberville
> Assignee: Mahadev konar
> Fix For: 3.1.0
>
> Attachments: ZOOKEEPER-272.patch
>
>
> Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated ZOOKEEPER-272:
------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
i just committed this. Ill open a new jira for the above discussion.
> getChildren can fail for large numbers of children
> --------------------------------------------------
>
> Key: ZOOKEEPER-272
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Joshua Tuberville
> Assignee: Mahadev konar
> Fix For: 3.1.0
>
> Attachments: ZOOKEEPER-272.patch
>
>
> Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-272) getChildren can fail for large
numbers of children
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663418#action_12663418 ]
Benjamin Reed commented on ZOOKEEPER-272:
-----------------------------------------
I think there are two stages to fixing this problem. First we should make that line of code use BinaryInputArchive.maxBuffer, so that it can be configured. Then we should probably add a version of getChildren that takes and returns an offset, so that getChildren could be called multiple times to get all of the children.
> getChildren can fail for large numbers of children
> --------------------------------------------------
>
> Key: ZOOKEEPER-272
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-272
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Joshua Tuberville
>
> Zookeeper allows creation of an abritrary number of children, yet if the String array of children names exceeds 4,194,304 bytes a getChildren will fail because ClientCnxn$SendThread.readLength() throws an exception on line 490. Mahadev Konar questioned this byte limit's need. In any case consistency of create children, get children should exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.