You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by Michael Dürig <md...@apache.org> on 2011/12/15 22:40:09 UTC
[j3 microkernel] support for large child node lists (svn commit:
r1214921 - /jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java)
Hi,
Starting with this revision there is some experimental support for large
child node lists in spi2microkernel and jcr2spi (jackrabbit-mk branch in
the sandbox). The Microkernel itself is not yet optimized for large
child node list but AFAIK Stefan is working on this.
To support large child node lists, I basically defer loading of child
entries as much as possible. Furthermore child nodes are loaded in
chunks of max. 512 entries at a time. Loaded child nodes are kept
internally by the ChildNodeEntries class until cleared by the garbage
collector. Using the garbage collector to control the child entries is
not optimal and may lead to memory thrashing when clients fully iterate
over large child node lists. Within the current design of the jcr2spi
layer there does not seem to be a better solution though.
Disclaimer: while the test cases pass, I did not yet test the
effectiveness of this solution.
Michael
-------- Original Message --------
Subject: svn commit: r1214921 -
/jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
Date: Thu, 15 Dec 2011 19:25:18 -0000
From: mduerig@apache.org
Reply-To: dev@jackrabbit.apache.org
To: commits@jackrabbit.apache.org
Author: mduerig
Date: Thu Dec 15 19:25:17 2011
New Revision: 1214921
URL: http://svn.apache.org/viewvc?rev=1214921&view=rev
Log:
Microkernel based Jackrabbit prototype (WIP)
batch loading of child nodes
Modified:
jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
Modified:
jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
URL:
http://svn.apache.org/viewvc/jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java?rev=1214921&r1=1214920&r2=1214921&view=diff
==============================================================================
---
jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
(original)
+++
jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
Thu Dec 15 19:25:17 2011
@@ -301,7 +301,7 @@ public class RepositoryServiceImpl exten
@Override
protected void checkWorkspace(String workspaceName) throws
NoSuchWorkspaceException {
- if (workspaceName == null || "".equals(workspaceName)) {
+ if (workspaceName == null || workspaceName.isEmpty()) {
workspaceName = defaultWorkspace;
}
else if (workspaceName.contains("/")) {
@@ -339,7 +339,7 @@ public class RepositoryServiceImpl exten
throw new PathNotFoundException(itemPath.toString());
}
- String json = microKernel.getNodes(mkPath, rev);
+ String json = microKernel.getNodes(mkPath, rev, 0, 0, 0);
return buildItemInfos(nodePath, itemPath, json,
readFromDataStore);
}
catch (MicroKernelException e) {
@@ -352,18 +352,65 @@ public class RepositoryServiceImpl exten
public Iterator<ChildInfo> getChildInfos(SessionInfo sessionInfo,
NodeId parentId) throws RepositoryException {
try {
String wspName = sessionInfo.getWorkspaceName();
- String rev = getRevision(sessionInfo);
- Path path = getPath(parentId, wspName, rev);
- String mkPath = Paths.pathToString(wspName, path);
+ final String rev = getRevision(sessionInfo);
+ final Path path = getPath(parentId, wspName, rev);
+ final String mkPath = Paths.pathToString(wspName, path);
if (!microKernel.nodeExists(mkPath, rev)) {
throw new PathNotFoundException(path.toString());
}
- String json = microKernel.getNodes(mkPath, rev);
- Iterator<? extends ItemInfo> infos = buildItemInfos(path,
path, json, readFromDataStore);
- NodeInfo info = (NodeInfo) infos.next();
- return info.getChildInfos();
+ return new Iterator<ChildInfo>() {
+ private static final int chunkSize = 512;
+
+ private int pos;
+ private Iterator<ChildInfo> currentChunk;
+
+ {
+ nextChunk();
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (currentChunk.hasNext()) {
+ return true;
+ }
+ else {
+ nextChunk();
+ return currentChunk.hasNext();
+ }
+ }
+
+ @Override
+ public ChildInfo next() {
+ if (currentChunk.hasNext()) {
+ return currentChunk.next();
+ }
+ else {
+ nextChunk();
+ return currentChunk.next();
+ }
+ }
+
+ @Override
+ public void remove() {
+ throw new UnsupportedOperationException("remove");
+ }
+
+ private void nextChunk() {
+ try {
+ String json = microKernel.getNodes(mkPath, rev,
0, pos, chunkSize);
+ pos += chunkSize;
+ Iterator<? extends ItemInfo> infos =
buildItemInfos(path, path, json, readFromDataStore);
+ NodeInfo info = (NodeInfo) infos.next();
+ currentChunk = info.getChildInfos();
+ }
+ catch (RepositoryException e) {
+ // fixme think of a better way to handle this
+ log.error(e.getMessage(), e);
+ }
+ }
+ };
}
catch (MicroKernelException e) {
log.error(e.getMessage(), e);
Re: [j3 microkernel] support for large child node lists (svn
commit: r1214921 -
/jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java)
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
By the way, the MemoryKernelImpl supports an optimization for large child
node lists, but it is disabled by default. To enable it, call:
mk.commit("/:root/head/config", "^ \"maxMemoryChildren\":" + max,
head, "");
See also org.apache.jackrabbit.mk.large.LargeNodeTest
Regards,
Thomas
On 12/15/11 10:40 PM, "Michael Dürig" <md...@apache.org> wrote:
>
>Hi,
>
>Starting with this revision there is some experimental support for large
>child node lists in spi2microkernel and jcr2spi (jackrabbit-mk branch in
>the sandbox). The Microkernel itself is not yet optimized for large
>child node list but AFAIK Stefan is working on this.
>
>To support large child node lists, I basically defer loading of child
>entries as much as possible. Furthermore child nodes are loaded in
>chunks of max. 512 entries at a time. Loaded child nodes are kept
>internally by the ChildNodeEntries class until cleared by the garbage
>collector. Using the garbage collector to control the child entries is
>not optimal and may lead to memory thrashing when clients fully iterate
>over large child node lists. Within the current design of the jcr2spi
>layer there does not seem to be a better solution though.
>
>Disclaimer: while the test cases pass, I did not yet test the
>effectiveness of this solution.
>
>Michael
>
>
>-------- Original Message --------
>Subject: svn commit: r1214921 -
>/jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java
>/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
>Date: Thu, 15 Dec 2011 19:25:18 -0000
>From: mduerig@apache.org
>Reply-To: dev@jackrabbit.apache.org
>To: commits@jackrabbit.apache.org
>
>Author: mduerig
>Date: Thu Dec 15 19:25:17 2011
>New Revision: 1214921
>
>URL: http://svn.apache.org/viewvc?rev=1214921&view=rev
>Log:
>Microkernel based Jackrabbit prototype (WIP)
>batch loading of child nodes
>
>Modified:
>
>jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/
>org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
>
>Modified:
>jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/
>org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
>URL:
>http://svn.apache.org/viewvc/jackrabbit/sandbox/jackrabbit-mk/jackrabbit-s
>pi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/Reposit
>oryServiceImpl.java?rev=1214921&r1=1214920&r2=1214921&view=diff
>==========================================================================
>====
>---
>jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/
>org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
>(original)
>+++
>jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/
>org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
>Thu Dec 15 19:25:17 2011
>@@ -301,7 +301,7 @@ public class RepositoryServiceImpl exten
>
> @Override
> protected void checkWorkspace(String workspaceName) throws
>NoSuchWorkspaceException {
>- if (workspaceName == null || "".equals(workspaceName)) {
>+ if (workspaceName == null || workspaceName.isEmpty()) {
> workspaceName = defaultWorkspace;
> }
> else if (workspaceName.contains("/")) {
>@@ -339,7 +339,7 @@ public class RepositoryServiceImpl exten
> throw new PathNotFoundException(itemPath.toString());
> }
>
>- String json = microKernel.getNodes(mkPath, rev);
>+ String json = microKernel.getNodes(mkPath, rev, 0, 0, 0);
> return buildItemInfos(nodePath, itemPath, json,
>readFromDataStore);
> }
> catch (MicroKernelException e) {
>@@ -352,18 +352,65 @@ public class RepositoryServiceImpl exten
> public Iterator<ChildInfo> getChildInfos(SessionInfo sessionInfo,
>NodeId parentId) throws RepositoryException {
> try {
> String wspName = sessionInfo.getWorkspaceName();
>- String rev = getRevision(sessionInfo);
>- Path path = getPath(parentId, wspName, rev);
>- String mkPath = Paths.pathToString(wspName, path);
>+ final String rev = getRevision(sessionInfo);
>+ final Path path = getPath(parentId, wspName, rev);
>+ final String mkPath = Paths.pathToString(wspName, path);
>
> if (!microKernel.nodeExists(mkPath, rev)) {
> throw new PathNotFoundException(path.toString());
> }
>
>- String json = microKernel.getNodes(mkPath, rev);
>- Iterator<? extends ItemInfo> infos = buildItemInfos(path,
>path, json, readFromDataStore);
>- NodeInfo info = (NodeInfo) infos.next();
>- return info.getChildInfos();
>+ return new Iterator<ChildInfo>() {
>+ private static final int chunkSize = 512;
>+
>+ private int pos;
>+ private Iterator<ChildInfo> currentChunk;
>+
>+ {
>+ nextChunk();
>+ }
>+
>+ @Override
>+ public boolean hasNext() {
>+ if (currentChunk.hasNext()) {
>+ return true;
>+ }
>+ else {
>+ nextChunk();
>+ return currentChunk.hasNext();
>+ }
>+ }
>+
>+ @Override
>+ public ChildInfo next() {
>+ if (currentChunk.hasNext()) {
>+ return currentChunk.next();
>+ }
>+ else {
>+ nextChunk();
>+ return currentChunk.next();
>+ }
>+ }
>+
>+ @Override
>+ public void remove() {
>+ throw new UnsupportedOperationException("remove");
>+ }
>+
>+ private void nextChunk() {
>+ try {
>+ String json = microKernel.getNodes(mkPath, rev,
>0, pos, chunkSize);
>+ pos += chunkSize;
>+ Iterator<? extends ItemInfo> infos =
>buildItemInfos(path, path, json, readFromDataStore);
>+ NodeInfo info = (NodeInfo) infos.next();
>+ currentChunk = info.getChildInfos();
>+ }
>+ catch (RepositoryException e) {
>+ // fixme think of a better way to handle this
>+ log.error(e.getMessage(), e);
>+ }
>+ }
>+ };
> }
> catch (MicroKernelException e) {
> log.error(e.getMessage(), e);
>
>
>