You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@jackrabbit.apache.org by md...@apache.org on 2013/10/24 12:42:03 UTC
svn commit: r1535337 [10/11] - in /jackrabbit/site/live/oak/docs: ./ css/ js/
Modified: jackrabbit/site/live/oak/docs/query.html
URL: http://svn.apache.org/viewvc/jackrabbit/site/live/oak/docs/query.html?rev=1535337&r1=1535336&r2=1535337&view=diff
==============================================================================
--- jackrabbit/site/live/oak/docs/query.html (original)
+++ jackrabbit/site/live/oak/docs/query.html Thu Oct 24 10:42:02 2013
@@ -1,507 +1,507 @@
-<!DOCTYPE html>
-<!--
- | Generated by Apache Maven Doxia at 2013-10-15
- | Rendered using Apache Maven Fluido Skin 1.3.0
--->
-<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
- <head>
- <meta charset="UTF-8" />
- <meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20131015" />
- <meta http-equiv="Content-Language" content="en" />
- <title>Jackrabbit Oak - </title>
- <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
- <link rel="stylesheet" href="./css/site.css" />
- <link rel="stylesheet" href="./css/print.css" media="print" />
-
-
- <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script>
-
-
- </head>
- <body class="topBarEnabled">
-
-
-
-
-
-
- <a href="http://github.com/apache/jackrabbit-oak">
- <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
- src="https://s3.amazonaws.com/github/ribbons/forkme_right_red_aa0000.png"
- alt="Fork me on GitHub">
- </a>
-
-
-
-
-
- <div id="topbar" class="navbar navbar-fixed-top ">
- <div class="navbar-inner">
- <div class="container-fluid">
- <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </a>
-
- <ul class="nav">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Overview <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
- <li> <a href="index.html" title="Jackrabbit Oak">Jackrabbit Oak</a>
-</li>
-
- <li> <a href="license.html" title="License">License</a>
-</li>
-
- <li> <a href="downloads.html" title="Downloads">Downloads</a>
-</li>
-
- <li> <a href="from_here.html" title="From here">From here</a>
-</li>
- </ul>
- </li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Concepts and architecture <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
- <li> <a href="overview.html" title="Overview">Overview</a>
-</li>
-
- <li> <a href="nodestate.html" title="Understanding the node state model">Understanding the node state model</a>
-</li>
-
- <li> <a href="microkernel.html" title="Microkernel">Microkernel</a>
-</li>
-
- <li> <a href="query.html" title="Query">Query</a>
-</li>
-
- <li> <a href="blobstore.html" title="BlobStore">BlobStore</a>
-</li>
- </ul>
- </li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Using Oak <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
- <li> <a href="use_getting_started.html" title="Getting Started">Getting Started</a>
-</li>
-
- <li> <a href="differences.html" title="Differences to Jackrabbit 2">Differences to Jackrabbit 2</a>
-</li>
-
- <li> <a href="dos_and_donts.html" title="Dos and don'ts">Dos and don'ts</a>
-</li>
-
- <li> <a href="when_things_go_wrong.html" title="When things go wrong">When things go wrong</a>
-</li>
- </ul>
- </li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developing Oak <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
- <li> <a href="dev_getting_started.html" title="Getting Started">Getting Started</a>
-</li>
-
- <li> <a href="participating.html" title="Participating">Participating</a>
-</li>
-
- <li> <a href="apidocs/index.html" title="API docs">API docs</a>
-</li>
- </ul>
- </li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Links <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
- <li> <a href="http://jackrabbit.apache.org/oak" title="Apache Jackrabbit Oak">Apache Jackrabbit Oak</a>
-</li>
-
- <li> <a href="http://jackrabbit.apache.org/" title="Apache Jackrabbit">Apache Jackrabbit</a>
-</li>
- </ul>
- </li>
- </ul>
-
-
-
-
- </div>
-
- </div>
- </div>
- </div>
-
- <div class="container-fluid">
- <div id="banner">
- <div class="pull-left">
- <div id="bannerLeft">
- <h2>Oak Documentation</h2>
- </div>
- </div>
- <div class="pull-right"> </div>
- <div class="clear"><hr/></div>
- </div>
-
- <div id="breadcrumbs">
- <ul class="breadcrumb">
-
-
- <li id="publishDate">Last Published: 2013-10-15</li>
- <li class="divider">|</li> <li id="projectVersion">Version: 0.11-SNAPSHOT</li>
-
-
-
-
- </ul>
- </div>
-
-
- <div class="row-fluid">
- <div id="leftColumn" class="span3">
- <div class="well sidebar-nav">
-
-
- <ul class="nav nav-list">
- <li class="nav-header">Overview</li>
-
- <li>
-
- <a href="index.html" title="Jackrabbit Oak">
- <i class="none"></i>
- Jackrabbit Oak</a>
- </li>
-
- <li>
-
- <a href="license.html" title="License">
- <i class="none"></i>
- License</a>
- </li>
-
- <li>
-
- <a href="downloads.html" title="Downloads">
- <i class="none"></i>
- Downloads</a>
- </li>
-
- <li>
-
- <a href="from_here.html" title="From here">
- <i class="none"></i>
- From here</a>
- </li>
- <li class="nav-header">Concepts and architecture</li>
-
- <li>
-
- <a href="overview.html" title="Overview">
- <i class="none"></i>
- Overview</a>
- </li>
-
- <li>
-
- <a href="nodestate.html" title="Understanding the node state model">
- <i class="none"></i>
- Understanding the node state model</a>
- </li>
-
- <li>
-
- <a href="microkernel.html" title="Microkernel">
- <i class="none"></i>
- Microkernel</a>
- </li>
-
- <li class="active">
-
- <a href="#"><i class="none"></i>Query</a>
- </li>
-
- <li>
-
- <a href="blobstore.html" title="BlobStore">
- <i class="none"></i>
- BlobStore</a>
- </li>
- <li class="nav-header">Using Oak</li>
-
- <li>
-
- <a href="use_getting_started.html" title="Getting Started">
- <i class="none"></i>
- Getting Started</a>
- </li>
-
- <li>
-
- <a href="differences.html" title="Differences to Jackrabbit 2">
- <i class="none"></i>
- Differences to Jackrabbit 2</a>
- </li>
-
- <li>
-
- <a href="dos_and_donts.html" title="Dos and don'ts">
- <i class="none"></i>
- Dos and don'ts</a>
- </li>
-
- <li>
-
- <a href="when_things_go_wrong.html" title="When things go wrong">
- <i class="none"></i>
- When things go wrong</a>
- </li>
- <li class="nav-header">Developing Oak</li>
-
- <li>
-
- <a href="dev_getting_started.html" title="Getting Started">
- <i class="none"></i>
- Getting Started</a>
- </li>
-
- <li>
-
- <a href="participating.html" title="Participating">
- <i class="none"></i>
- Participating</a>
- </li>
-
- <li>
-
- <a href="apidocs/index.html" title="API docs">
- <i class="none"></i>
- API docs</a>
- </li>
- <li class="nav-header">Links</li>
-
- <li>
-
- <a href="http://jackrabbit.apache.org/oak" class="externalLink" title="Apache Jackrabbit Oak">
- <i class="none"></i>
- Apache Jackrabbit Oak</a>
- </li>
-
- <li>
-
- <a href="http://jackrabbit.apache.org/" class="externalLink" title="Apache Jackrabbit">
- <i class="none"></i>
- Apache Jackrabbit</a>
- </li>
- </ul>
-
-
-
- <hr class="divider" />
-
- <div id="poweredBy">
-
- <script type="text/javascript" src="https://apis.google.com/js/plusone.js"></script>
-
-
- <div class="g-plusone" data-href="http://jackrabbit.apache.org/oak-doc/" data-size="tall" ></div>
-
- <div class="clear"></div>
- <div class="clear"></div>
- <div class="clear"></div>
- <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
- <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" />
- </a>
- </div>
- </div>
- </div>
-
-
- <div id="bodyColumn" class="span9" >
-
- <!-- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. --><div class="section">
-<h2>The Query Engine<a name="The_Query_Engine"></a></h2>
-<p>Oak does not index content by default as does Jackrabbit 2. You need to create custom indexes when necessary, much like in traditional RDBMSs. If there is no index for a specific query, then the repository will be traversed. That is, the query will still work but probably be very slow.</p>
-<p>Query Indices are defined under the <tt>oak:index</tt> node.</p>
-<div class="section">
-<h3>XPath to SQL2 Transformation<a name="XPath_to_SQL2_Transformation"></a></h3>
-<p>To support the XPath query language, such queries are internally converted to SQL2. </p>
-<p>Every conversion is logged in <tt>debug</tt> level under the <tt>org.apache.jackrabbit.oak.query.QueryEngineImpl</tt> logger:</p>
-
-<div class="source">
-<pre>org.apache.jackrabbit.oak.query.QueryEngineImpl Parsing xpath statement:
- //element(*)[@sling:resourceType = 'slingevent:Lock')]
-org.apache.jackrabbit.oak.query.QueryEngineImpl XPath > SQL2:
- select [jcr:path], [jcr:score], * from [nt:base] as a
- where [sling:resourceType] = 'slingevent:Lock'
- /* xpath: //element(*)[@sling:resourceType = 'slingevent:Lock'
- and @lock.created < xs:dateTime('2013-09-02T15:44:05.920+02:00')] */
-</pre></div>
-<p><i>Each transformed SQL2 query contains the original XPath query as a comment.</i></p></div>
-<div class="section">
-<h3>Query Processing<a name="Query_Processing"></a></h3>
-<p>Internally, the query engine uses a cost based query optimizer that asks all the available query indexes for the estimated cost to process the query. It then uses the index with the lowest cost.</p>
-<p>By default, the following indexes are available:</p>
-
-<ul>
-
-<li>A property index for each indexed property.</li>
-
-<li>A full-text index which is based on Apache Lucene / Solr.</li>
-
-<li>A node type index (which is based on an property index for the properties jcr:primaryType and jcr:mixins).</li>
-
-<li>A traversal index that iterates over a subtree.</li>
-</ul>
-<p>If no index can efficiently process the filter condition, the nodes in the repository are traversed at the given subtree.</p>
-<p>Usually, data is read from the index and repository while traversing over the query result. There are exceptions however, where all data is read in memory when the query is executed: when using a full-text index, and when using an “order by” clause.</p></div>
-<div class="section">
-<h3>The Property Index<a name="The_Property_Index"></a></h3>
-<p>Is useful whenever there is a query with a property constraint that is not full-text:</p>
-
-<div class="source">
-<pre>SELECT * FROM [nt:base] WHERE [jcr:uuid] = $id
-</pre></div>
-<p>To define a property index on a subtree you have to add an index definition node that:</p>
-
-<ul>
-
-<li>must be of type <tt>oak:queryIndexDefinition</tt></li>
-
-<li>must have the <tt>type</tt> property set to <b><tt>property</tt></b></li>
-
-<li>contains the <tt>propertyNames</tt> property that indicates what properties will be stored in the index. <tt>propertyNames</tt> can be a list of properties, and it is optional.in case it is missing, the node name will be used as a property name reference value</li>
-</ul>
-<p><i>Optionally</i> you can specify</p>
-
-<ul>
-
-<li>a uniqueness constraint on a property index by setting the <tt>unique</tt> flag to <tt>true</tt></li>
-
-<li>that the property index only applies to a certain node type by setting the <tt>declaringNodeTypes</tt> property</li>
-
-<li>the <tt>reindex</tt> flag which when set to <tt>true</tt>, triggers a full content re-index.</li>
-</ul>
-<p>Example:</p>
-
-<div class="source">
-<pre>{
- NodeBuilder index = root.child("oak:index");
- index.child("uuid")
- .setProperty("jcr:primaryType", "oak:queryIndexDefinition", Type.NAME)
- .setProperty("type", "property")
- .setProperty("propertyNames", "jcr:uuid")
- .setProperty("declaringNodeTypes", "mix:referenceable")
- .setProperty("unique", true)
- .setProperty("reindex", true);
-}
-</pre></div>
-<p>or to simplify you can use one of the existing <tt>IndexUtils#createIndexDefinition</tt> helper methods:</p>
-
-<div class="source">
-<pre>{
- NodeBuilder index = IndexUtils.getOrCreateOakIndex(root);
- IndexUtils.createIndexDefinition(index, "myProp", true, false, ImmutableList.of("myProp"), null);
-}
-</pre></div></div>
-<div class="section">
-<h3>The Lucene Full-Text Index<a name="The_Lucene_Full-Text_Index"></a></h3>
-<p>The full-text index handles only the ‘contains’ type of queries.</p>
-
-<div class="source">
-<pre>//*[jcr:contains(., 'text')]
-</pre></div>
-<p>Not having a full-text index means that the full-text queries will not be able to work properly. Currently the query engine has a basic verification in place for full-text conditions, but that is brittle and can miss hits.</p>
-<p>The full-text index update is asynchronous via a background thread, see <tt>Oak#withAsyncIndexing</tt>. This means that some full-text searches will not work for a small window of time: the background thread runs every 5 seconds, plus the time is takes to run the diff and to run the text-extraction process. </p>
-<p>The async update status is now reflected on the <tt>oak:index</tt> node with the help of a few properties, see <a class="externalLink" href="https://issues.apache.org/jira/browse/OAK-980">OAK-980</a></p>
-<p>TODO Node aggregation <a class="externalLink" href="https://issues.apache.org/jira/browse/OAK-828">OAK-828</a></p>
-<p>The index definition node for a lucene-based full-text index:</p>
-
-<ul>
-
-<li>must be of type <tt>oak:queryIndexDefinition</tt></li>
-
-<li>must have the <tt>type</tt> property set to <b><tt>lucene</tt></b></li>
-
-<li>must contain the <tt>async</tt> property set to the value <tt>async</tt>, this is what sends the index update process to a background thread</li>
-</ul>
-<p><i>Optionally</i> you can add</p>
-
-<ul>
-
-<li>what subset of property types to be included in the index via the <tt>includePropertyTypes</tt> property</li>
-
-<li>a blacklist of property names: what property to be excluded from the index via the <tt>excludePropertyNames</tt> property</li>
-
-<li>the <tt>reindex</tt> flag which when set to <tt>true</tt>, triggers a full content re-index.</li>
-</ul>
-<p>Example:</p>
-
-<div class="source">
-<pre>{
- NodeBuilder index = root.child("oak:index");
- index.child("lucene")
- .setProperty("jcr:primaryType", "oak:queryIndexDefinition", Type.NAME)
- .setProperty("type", "lucene")
- .setProperty("async", "async")
- .setProperty(PropertyStates.createProperty("includePropertyTypes", ImmutableSet.of(
- PropertyType.TYPENAME_STRING, PropertyType.TYPENAME_BINARY), Type.STRINGS))
- .setProperty(PropertyStates.createProperty("excludePropertyNames", ImmutableSet.of(
- "jcr:createdBy", "jcr:lastModifiedBy"), Type.STRINGS))
- .setProperty("reindex", true);
-}
-</pre></div></div>
-<div class="section">
-<h3>The Solr Full-Text Index<a name="The_Solr_Full-Text_Index"></a></h3>
-<p><tt>TODO</tt></p></div>
-<div class="section">
-<h3>The Node Type Index<a name="The_Node_Type_Index"></a></h3>
-<p>The <tt>NodeTypeIndex</tt> implements a <tt>QueryIndex</tt> using <tt>PropertyIndexLookup</tt>s on <tt>jcr:primaryType</tt> <tt>jcr:mixinTypes</tt> to evaluate a node type restriction on the filter. The cost for this index is the sum of the costs of the <tt>PropertyIndexLookup</tt> for queries on <tt>jcr:primaryType</tt> and <tt>jcr:mixinTypes</tt>.</p></div>
-<div class="section">
-<h3>Cost Calculation<a name="Cost_Calculation"></a></h3>
-<p>Each query index is expected to estimate the worst-case cost to query with the given filter. The returned value is between 1 (very fast; lookup of a unique node) and the estimated number of entries to traverse, if the cursor would be fully read, and if there could in theory be one network round-trip or disk read operation per node (this method may return a lower number if the data is known to be fully in memory).</p>
-<p>The returned value is supposed to be an estimate and doesn’t have to be very accurate. Please note this method is called on each index whenever a query is run, so the method should be reasonably fast (not read any data itself, or at least not read too much data).</p>
-<p>If an index implementation can not query the data, it has to return <tt>Double.POSITIVE_INFINITY</tt>.</p>
-<p>TODO Traversal warnings</p></div></div>
- </div>
- </div>
- </div>
-
- <hr/>
-
- <footer>
- <div class="container-fluid">
- <div class="row span12">Copyright © 2012-2013
- <a href="http://www.apache.org/">The Apache Software Foundation</a>.
- All Rights Reserved.
-
- </div>
-
-
-
-
-
-
- <div id="ohloh" class="pull-right">
- <script type="text/javascript" src="http://www.ohloh.net/p/jackrabbit-oak/widgets/project_users_logo.js"></script>
- </div>
- </div>
- </footer>
- </body>
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2013-10-24
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20131024" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Jackrabbit Oak - </title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+
+ <a href="http://github.com/apache/jackrabbit-oak">
+ <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
+ src="https://s3.amazonaws.com/github/ribbons/forkme_right_red_aa0000.png"
+ alt="Fork me on GitHub">
+ </a>
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container-fluid">
+ <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </a>
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Overview <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="index.html" title="Jackrabbit Oak">Jackrabbit Oak</a>
+</li>
+
+ <li> <a href="license.html" title="License">License</a>
+</li>
+
+ <li> <a href="downloads.html" title="Downloads">Downloads</a>
+</li>
+
+ <li> <a href="from_here.html" title="From here">From here</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Concepts and architecture <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="overview.html" title="Overview">Overview</a>
+</li>
+
+ <li> <a href="nodestate.html" title="Understanding the node state model">Understanding the node state model</a>
+</li>
+
+ <li> <a href="microkernel.html" title="Microkernel">Microkernel</a>
+</li>
+
+ <li> <a href="query.html" title="Query">Query</a>
+</li>
+
+ <li> <a href="blobstore.html" title="BlobStore">BlobStore</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Using Oak <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="use_getting_started.html" title="Getting Started">Getting Started</a>
+</li>
+
+ <li> <a href="differences.html" title="Differences to Jackrabbit 2">Differences to Jackrabbit 2</a>
+</li>
+
+ <li> <a href="dos_and_donts.html" title="Dos and don'ts">Dos and don'ts</a>
+</li>
+
+ <li> <a href="when_things_go_wrong.html" title="When things go wrong">When things go wrong</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developing Oak <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="dev_getting_started.html" title="Getting Started">Getting Started</a>
+</li>
+
+ <li> <a href="participating.html" title="Participating">Participating</a>
+</li>
+
+ <li> <a href="apidocs/index.html" title="API docs">API docs</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Links <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="http://jackrabbit.apache.org/oak" title="Apache Jackrabbit Oak">Apache Jackrabbit Oak</a>
+</li>
+
+ <li> <a href="http://jackrabbit.apache.org/" title="Apache Jackrabbit">Apache Jackrabbit</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <div id="bannerLeft">
+ <h2>Oak Documentation</h2>
+ </div>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li id="publishDate">Last Published: 2013-10-24</li>
+ <li class="divider">|</li> <li id="projectVersion">Version: 0.11-SNAPSHOT</li>
+
+
+
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Overview</li>
+
+ <li>
+
+ <a href="index.html" title="Jackrabbit Oak">
+ <i class="none"></i>
+ Jackrabbit Oak</a>
+ </li>
+
+ <li>
+
+ <a href="license.html" title="License">
+ <i class="none"></i>
+ License</a>
+ </li>
+
+ <li>
+
+ <a href="downloads.html" title="Downloads">
+ <i class="none"></i>
+ Downloads</a>
+ </li>
+
+ <li>
+
+ <a href="from_here.html" title="From here">
+ <i class="none"></i>
+ From here</a>
+ </li>
+ <li class="nav-header">Concepts and architecture</li>
+
+ <li>
+
+ <a href="overview.html" title="Overview">
+ <i class="none"></i>
+ Overview</a>
+ </li>
+
+ <li>
+
+ <a href="nodestate.html" title="Understanding the node state model">
+ <i class="none"></i>
+ Understanding the node state model</a>
+ </li>
+
+ <li>
+
+ <a href="microkernel.html" title="Microkernel">
+ <i class="none"></i>
+ Microkernel</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Query</a>
+ </li>
+
+ <li>
+
+ <a href="blobstore.html" title="BlobStore">
+ <i class="none"></i>
+ BlobStore</a>
+ </li>
+ <li class="nav-header">Using Oak</li>
+
+ <li>
+
+ <a href="use_getting_started.html" title="Getting Started">
+ <i class="none"></i>
+ Getting Started</a>
+ </li>
+
+ <li>
+
+ <a href="differences.html" title="Differences to Jackrabbit 2">
+ <i class="none"></i>
+ Differences to Jackrabbit 2</a>
+ </li>
+
+ <li>
+
+ <a href="dos_and_donts.html" title="Dos and don'ts">
+ <i class="none"></i>
+ Dos and don'ts</a>
+ </li>
+
+ <li>
+
+ <a href="when_things_go_wrong.html" title="When things go wrong">
+ <i class="none"></i>
+ When things go wrong</a>
+ </li>
+ <li class="nav-header">Developing Oak</li>
+
+ <li>
+
+ <a href="dev_getting_started.html" title="Getting Started">
+ <i class="none"></i>
+ Getting Started</a>
+ </li>
+
+ <li>
+
+ <a href="participating.html" title="Participating">
+ <i class="none"></i>
+ Participating</a>
+ </li>
+
+ <li>
+
+ <a href="apidocs/index.html" title="API docs">
+ <i class="none"></i>
+ API docs</a>
+ </li>
+ <li class="nav-header">Links</li>
+
+ <li>
+
+ <a href="http://jackrabbit.apache.org/oak" class="externalLink" title="Apache Jackrabbit Oak">
+ <i class="none"></i>
+ Apache Jackrabbit Oak</a>
+ </li>
+
+ <li>
+
+ <a href="http://jackrabbit.apache.org/" class="externalLink" title="Apache Jackrabbit">
+ <i class="none"></i>
+ Apache Jackrabbit</a>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+
+ <script type="text/javascript" src="https://apis.google.com/js/plusone.js"></script>
+
+
+ <div class="g-plusone" data-href="http://jackrabbit.apache.org/oak-doc/" data-size="tall" ></div>
+
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <!-- Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. --><div class="section">
+<h2>The Query Engine<a name="The_Query_Engine"></a></h2>
+<p>Oak does not index content by default as does Jackrabbit 2. You need to create custom indexes when necessary, much like in traditional RDBMSs. If there is no index for a specific query, then the repository will be traversed. That is, the query will still work but probably be very slow.</p>
+<p>Query Indices are defined under the <tt>oak:index</tt> node.</p>
+<div class="section">
+<h3>XPath to SQL2 Transformation<a name="XPath_to_SQL2_Transformation"></a></h3>
+<p>To support the XPath query language, such queries are internally converted to SQL2. </p>
+<p>Every conversion is logged in <tt>debug</tt> level under the <tt>org.apache.jackrabbit.oak.query.QueryEngineImpl</tt> logger:</p>
+
+<div class="source">
+<pre>org.apache.jackrabbit.oak.query.QueryEngineImpl Parsing xpath statement:
+ //element(*)[@sling:resourceType = 'slingevent:Lock')]
+org.apache.jackrabbit.oak.query.QueryEngineImpl XPath > SQL2:
+ select [jcr:path], [jcr:score], * from [nt:base] as a
+ where [sling:resourceType] = 'slingevent:Lock'
+ /* xpath: //element(*)[@sling:resourceType = 'slingevent:Lock'
+ and @lock.created < xs:dateTime('2013-09-02T15:44:05.920+02:00')] */
+</pre></div>
+<p><i>Each transformed SQL2 query contains the original XPath query as a comment.</i></p></div>
+<div class="section">
+<h3>Query Processing<a name="Query_Processing"></a></h3>
+<p>Internally, the query engine uses a cost based query optimizer that asks all the available query indexes for the estimated cost to process the query. It then uses the index with the lowest cost.</p>
+<p>By default, the following indexes are available:</p>
+
+<ul>
+
+<li>A property index for each indexed property.</li>
+
+<li>A full-text index which is based on Apache Lucene / Solr.</li>
+
+<li>A node type index (which is based on an property index for the properties jcr:primaryType and jcr:mixins).</li>
+
+<li>A traversal index that iterates over a subtree.</li>
+</ul>
+<p>If no index can efficiently process the filter condition, the nodes in the repository are traversed at the given subtree.</p>
+<p>Usually, data is read from the index and repository while traversing over the query result. There are exceptions however, where all data is read in memory when the query is executed: when using a full-text index, and when using an “order by” clause.</p></div>
+<div class="section">
+<h3>The Property Index<a name="The_Property_Index"></a></h3>
+<p>Is useful whenever there is a query with a property constraint that is not full-text:</p>
+
+<div class="source">
+<pre>SELECT * FROM [nt:base] WHERE [jcr:uuid] = $id
+</pre></div>
+<p>To define a property index on a subtree you have to add an index definition node that:</p>
+
+<ul>
+
+<li>must be of type <tt>oak:queryIndexDefinition</tt></li>
+
+<li>must have the <tt>type</tt> property set to <b><tt>property</tt></b></li>
+
+<li>contains the <tt>propertyNames</tt> property that indicates what properties will be stored in the index. <tt>propertyNames</tt> can be a list of properties, and it is optional.in case it is missing, the node name will be used as a property name reference value</li>
+</ul>
+<p><i>Optionally</i> you can specify</p>
+
+<ul>
+
+<li>a uniqueness constraint on a property index by setting the <tt>unique</tt> flag to <tt>true</tt></li>
+
+<li>that the property index only applies to a certain node type by setting the <tt>declaringNodeTypes</tt> property</li>
+
+<li>the <tt>reindex</tt> flag which when set to <tt>true</tt>, triggers a full content re-index.</li>
+</ul>
+<p>Example:</p>
+
+<div class="source">
+<pre>{
+ NodeBuilder index = root.child("oak:index");
+ index.child("uuid")
+ .setProperty("jcr:primaryType", "oak:queryIndexDefinition", Type.NAME)
+ .setProperty("type", "property")
+ .setProperty("propertyNames", "jcr:uuid")
+ .setProperty("declaringNodeTypes", "mix:referenceable")
+ .setProperty("unique", true)
+ .setProperty("reindex", true);
+}
+</pre></div>
+<p>or to simplify you can use one of the existing <tt>IndexUtils#createIndexDefinition</tt> helper methods:</p>
+
+<div class="source">
+<pre>{
+ NodeBuilder index = IndexUtils.getOrCreateOakIndex(root);
+ IndexUtils.createIndexDefinition(index, "myProp", true, false, ImmutableList.of("myProp"), null);
+}
+</pre></div></div>
+<div class="section">
+<h3>The Lucene Full-Text Index<a name="The_Lucene_Full-Text_Index"></a></h3>
+<p>The full-text index handles only the ‘contains’ type of queries.</p>
+
+<div class="source">
+<pre>//*[jcr:contains(., 'text')]
+</pre></div>
+<p>Not having a full-text index means that the full-text queries will not be able to work properly. Currently the query engine has a basic verification in place for full-text conditions, but that is brittle and can miss hits.</p>
+<p>The full-text index update is asynchronous via a background thread, see <tt>Oak#withAsyncIndexing</tt>. This means that some full-text searches will not work for a small window of time: the background thread runs every 5 seconds, plus the time is takes to run the diff and to run the text-extraction process. </p>
+<p>The async update status is now reflected on the <tt>oak:index</tt> node with the help of a few properties, see <a class="externalLink" href="https://issues.apache.org/jira/browse/OAK-980">OAK-980</a></p>
+<p>TODO Node aggregation <a class="externalLink" href="https://issues.apache.org/jira/browse/OAK-828">OAK-828</a></p>
+<p>The index definition node for a lucene-based full-text index:</p>
+
+<ul>
+
+<li>must be of type <tt>oak:queryIndexDefinition</tt></li>
+
+<li>must have the <tt>type</tt> property set to <b><tt>lucene</tt></b></li>
+
+<li>must contain the <tt>async</tt> property set to the value <tt>async</tt>, this is what sends the index update process to a background thread</li>
+</ul>
+<p><i>Optionally</i> you can add</p>
+
+<ul>
+
+<li>what subset of property types to be included in the index via the <tt>includePropertyTypes</tt> property</li>
+
+<li>a blacklist of property names: what property to be excluded from the index via the <tt>excludePropertyNames</tt> property</li>
+
+<li>the <tt>reindex</tt> flag which when set to <tt>true</tt>, triggers a full content re-index.</li>
+</ul>
+<p>Example:</p>
+
+<div class="source">
+<pre>{
+ NodeBuilder index = root.child("oak:index");
+ index.child("lucene")
+ .setProperty("jcr:primaryType", "oak:queryIndexDefinition", Type.NAME)
+ .setProperty("type", "lucene")
+ .setProperty("async", "async")
+ .setProperty(PropertyStates.createProperty("includePropertyTypes", ImmutableSet.of(
+ PropertyType.TYPENAME_STRING, PropertyType.TYPENAME_BINARY), Type.STRINGS))
+ .setProperty(PropertyStates.createProperty("excludePropertyNames", ImmutableSet.of(
+ "jcr:createdBy", "jcr:lastModifiedBy"), Type.STRINGS))
+ .setProperty("reindex", true);
+}
+</pre></div></div>
+<div class="section">
+<h3>The Solr Full-Text Index<a name="The_Solr_Full-Text_Index"></a></h3>
+<p><tt>TODO</tt></p></div>
+<div class="section">
+<h3>The Node Type Index<a name="The_Node_Type_Index"></a></h3>
+<p>The <tt>NodeTypeIndex</tt> implements a <tt>QueryIndex</tt> using <tt>PropertyIndexLookup</tt>s on <tt>jcr:primaryType</tt> <tt>jcr:mixinTypes</tt> to evaluate a node type restriction on the filter. The cost for this index is the sum of the costs of the <tt>PropertyIndexLookup</tt> for queries on <tt>jcr:primaryType</tt> and <tt>jcr:mixinTypes</tt>.</p></div>
+<div class="section">
+<h3>Cost Calculation<a name="Cost_Calculation"></a></h3>
+<p>Each query index is expected to estimate the worst-case cost to query with the given filter. The returned value is between 1 (very fast; lookup of a unique node) and the estimated number of entries to traverse, if the cursor would be fully read, and if there could in theory be one network round-trip or disk read operation per node (this method may return a lower number if the data is known to be fully in memory).</p>
+<p>The returned value is supposed to be an estimate and doesn’t have to be very accurate. Please note this method is called on each index whenever a query is run, so the method should be reasonably fast (not read any data itself, or at least not read too much data).</p>
+<p>If an index implementation can not query the data, it has to return <tt>Double.POSITIVE_INFINITY</tt>.</p>
+<p>TODO Traversal warnings</p></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2012-2013
+ <a href="http://www.apache.org/">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+
+
+
+ <div id="ohloh" class="pull-right">
+ <script type="text/javascript" src="http://www.ohloh.net/p/jackrabbit-oak/widgets/project_users_logo.js"></script>
+ </div>
+ </div>
+ </footer>
+ </body>
</html>
\ No newline at end of file
Modified: jackrabbit/site/live/oak/docs/segmentmk.html
URL: http://svn.apache.org/viewvc/jackrabbit/site/live/oak/docs/segmentmk.html?rev=1535337&r1=1535336&r2=1535337&view=diff
==============================================================================
--- jackrabbit/site/live/oak/docs/segmentmk.html (original)
+++ jackrabbit/site/live/oak/docs/segmentmk.html Thu Oct 24 10:42:02 2013
@@ -1,417 +1,417 @@
-<!DOCTYPE html>
-<!--
- | Generated by Apache Maven Doxia at 2013-10-15
- | Rendered using Apache Maven Fluido Skin 1.3.0
--->
-<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
- <head>
- <meta charset="UTF-8" />
- <meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20131015" />
- <meta http-equiv="Content-Language" content="en" />
- <title>Jackrabbit Oak - </title>
- <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
- <link rel="stylesheet" href="./css/site.css" />
- <link rel="stylesheet" href="./css/print.css" media="print" />
-
-
- <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script>
-
-
- </head>
- <body class="topBarEnabled">
-
-
-
-
-
-
- <a href="http://github.com/apache/jackrabbit-oak">
- <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
- src="https://s3.amazonaws.com/github/ribbons/forkme_right_red_aa0000.png"
- alt="Fork me on GitHub">
- </a>
-
-
-
-
-
- <div id="topbar" class="navbar navbar-fixed-top ">
- <div class="navbar-inner">
- <div class="container-fluid">
- <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </a>
-
- <ul class="nav">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Overview <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
- <li> <a href="index.html" title="Jackrabbit Oak">Jackrabbit Oak</a>
-</li>
-
- <li> <a href="license.html" title="License">License</a>
-</li>
-
- <li> <a href="downloads.html" title="Downloads">Downloads</a>
-</li>
-
- <li> <a href="from_here.html" title="From here">From here</a>
-</li>
- </ul>
- </li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Concepts and architecture <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
- <li> <a href="overview.html" title="Overview">Overview</a>
-</li>
-
- <li> <a href="nodestate.html" title="Understanding the node state model">Understanding the node state model</a>
-</li>
-
- <li> <a href="microkernel.html" title="Microkernel">Microkernel</a>
-</li>
-
- <li> <a href="query.html" title="Query">Query</a>
-</li>
-
- <li> <a href="blobstore.html" title="BlobStore">BlobStore</a>
-</li>
- </ul>
- </li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Using Oak <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
- <li> <a href="use_getting_started.html" title="Getting Started">Getting Started</a>
-</li>
-
- <li> <a href="differences.html" title="Differences to Jackrabbit 2">Differences to Jackrabbit 2</a>
-</li>
-
- <li> <a href="dos_and_donts.html" title="Dos and don'ts">Dos and don'ts</a>
-</li>
-
- <li> <a href="when_things_go_wrong.html" title="When things go wrong">When things go wrong</a>
-</li>
- </ul>
- </li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developing Oak <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
- <li> <a href="dev_getting_started.html" title="Getting Started">Getting Started</a>
-</li>
-
- <li> <a href="participating.html" title="Participating">Participating</a>
-</li>
-
- <li> <a href="apidocs/index.html" title="API docs">API docs</a>
-</li>
- </ul>
- </li>
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown">Links <b class="caret"></b></a>
- <ul class="dropdown-menu">
-
- <li> <a href="http://jackrabbit.apache.org/oak" title="Apache Jackrabbit Oak">Apache Jackrabbit Oak</a>
-</li>
-
- <li> <a href="http://jackrabbit.apache.org/" title="Apache Jackrabbit">Apache Jackrabbit</a>
-</li>
- </ul>
- </li>
- </ul>
-
-
-
-
- </div>
-
- </div>
- </div>
- </div>
-
- <div class="container-fluid">
- <div id="banner">
- <div class="pull-left">
- <div id="bannerLeft">
- <h2>Oak Documentation</h2>
- </div>
- </div>
- <div class="pull-right"> </div>
- <div class="clear"><hr/></div>
- </div>
-
- <div id="breadcrumbs">
- <ul class="breadcrumb">
-
-
- <li id="publishDate">Last Published: 2013-10-15</li>
- <li class="divider">|</li> <li id="projectVersion">Version: 0.11-SNAPSHOT</li>
-
-
-
-
- </ul>
- </div>
-
-
- <div class="row-fluid">
- <div id="leftColumn" class="span3">
- <div class="well sidebar-nav">
-
-
- <ul class="nav nav-list">
- <li class="nav-header">Overview</li>
-
- <li>
-
- <a href="index.html" title="Jackrabbit Oak">
- <i class="none"></i>
- Jackrabbit Oak</a>
- </li>
-
- <li>
-
- <a href="license.html" title="License">
- <i class="none"></i>
- License</a>
- </li>
-
- <li>
-
- <a href="downloads.html" title="Downloads">
- <i class="none"></i>
- Downloads</a>
- </li>
-
- <li>
-
- <a href="from_here.html" title="From here">
- <i class="none"></i>
- From here</a>
- </li>
- <li class="nav-header">Concepts and architecture</li>
-
- <li>
-
- <a href="overview.html" title="Overview">
- <i class="none"></i>
- Overview</a>
- </li>
-
- <li>
-
- <a href="nodestate.html" title="Understanding the node state model">
- <i class="none"></i>
- Understanding the node state model</a>
- </li>
-
- <li>
-
- <a href="microkernel.html" title="Microkernel">
- <i class="none"></i>
- Microkernel</a>
- </li>
-
- <li>
-
- <a href="query.html" title="Query">
- <i class="none"></i>
- Query</a>
- </li>
-
- <li>
-
- <a href="blobstore.html" title="BlobStore">
- <i class="none"></i>
- BlobStore</a>
- </li>
- <li class="nav-header">Using Oak</li>
-
- <li>
-
- <a href="use_getting_started.html" title="Getting Started">
- <i class="none"></i>
- Getting Started</a>
- </li>
-
- <li>
-
- <a href="differences.html" title="Differences to Jackrabbit 2">
- <i class="none"></i>
- Differences to Jackrabbit 2</a>
- </li>
-
- <li>
-
- <a href="dos_and_donts.html" title="Dos and don'ts">
- <i class="none"></i>
- Dos and don'ts</a>
- </li>
-
- <li>
-
- <a href="when_things_go_wrong.html" title="When things go wrong">
- <i class="none"></i>
- When things go wrong</a>
- </li>
- <li class="nav-header">Developing Oak</li>
-
- <li>
-
- <a href="dev_getting_started.html" title="Getting Started">
- <i class="none"></i>
- Getting Started</a>
- </li>
-
- <li>
-
- <a href="participating.html" title="Participating">
- <i class="none"></i>
- Participating</a>
- </li>
-
- <li>
-
- <a href="apidocs/index.html" title="API docs">
- <i class="none"></i>
- API docs</a>
- </li>
- <li class="nav-header">Links</li>
-
- <li>
-
- <a href="http://jackrabbit.apache.org/oak" class="externalLink" title="Apache Jackrabbit Oak">
- <i class="none"></i>
- Apache Jackrabbit Oak</a>
- </li>
-
- <li>
-
- <a href="http://jackrabbit.apache.org/" class="externalLink" title="Apache Jackrabbit">
- <i class="none"></i>
- Apache Jackrabbit</a>
- </li>
- </ul>
-
-
-
- <hr class="divider" />
-
- <div id="poweredBy">
-
- <script type="text/javascript" src="https://apis.google.com/js/plusone.js"></script>
-
-
- <div class="g-plusone" data-href="http://jackrabbit.apache.org/oak-doc/" data-size="tall" ></div>
-
- <div class="clear"></div>
- <div class="clear"></div>
- <div class="clear"></div>
- <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
- <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" />
- </a>
- </div>
- </div>
- </div>
-
-
- <div id="bodyColumn" class="span9" >
-
- <!-- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. --><h1>SegmentMK design overview</h1>
-<h1>Segments</h1>
-<p>The content tree and all its revisions are stored in a collection of immutable <i>segments</i>. Each segment is identified by a UUID and typically contains a continuous subset of the content tree. Some segments might also be used to store commonly occurring property values or other shared data. Segments range from a few dozens of bytes to up to 256kB in size and are stored as documents in a MongoDB collection.</p>
-<p>Since segments are immutable, it’s easy for a client to keep a local in-memory cache of frequently accessed segments. Since segments also leverage locality of reference, i.e. nearby nodes are often stored in the same segment, it’s common for things like small child nodes to already exist in the cache by the time they get accessed. The intention is to avoid as many network or database roundtrips as possible by using local cache memory as efficiently as possible.</p>
-<p>Content within a segment can contain references to content within other segments. Each segment keeps a list of the UUIDs of all other segments it references. This list of segment references can be used to optimize both internal storage (as seen below) and garbage collection. Segments that are no longer referenced can be efficiently identified by traversing the graph of segment-level references without having to parse or even fetch the contents of each segment.</p>
-<p>The internal record structure of nodes is described in a moment once we first cover journal documents.</p>
-<h1>Journals</h1>
-<p>Journals are special, atomically updated documents that record the state of the repository as a sequence of references to successive root node records.</p>
-<p>A small system could consist of just a single journal and would serialize all repository updates through atomic updates of that journal. A larger system that needs more write throughput can have more journals, linked to each other in a tree hierarchy. Commits to journals in lower levels of the tree can proceed concurrently, but will need to be periodically merged back to the root journal. Potential conflicts and resulting data loss or inconsistency caused by such merges can be avoided by always committing against the root journal.</p>
-<p>Temporary branches used for large commits are also recorded as journals. A new private journal document is created for each branch and kept around until the branch gets merged or discarded. Branch journals contain an update timestamp that needs to be periodically refreshed by the client to prevent the branch from expiring and being reclaimed by the garbage collector.</p>
-<p>The root node references stored in journals are used as the starting point for garbage collection. It is assumed that all content currently visible to clients must be accessible through at least one of the journals. If a client wants to keep a reference to some old content revision that’s no longer referenced by one of the main journals, it should create an empty private branch based on that revision and keep refreshing the branch until that content is no longer needed.</p>
-<h1>Records</h1>
-<p>The content inside a segment is divided in records of different types: blocks, lists, maps, values, templates and nodes. These record types and their internal structures are described in subsections below.</p>
-<p>Each record is uniquely addressable by its location within the segment and the UUID of that segment. A single segment can contain up to 256kB of data and and references to up to 256 segments (including itself). Since all records are aligned at four-byte boundaries, 16 bits are needed to address all possible record locations within a segment. Thus only three bytes are needed to store a reference to any record in any segment (1 byte to identify the segment, 2 bytes for the record offset).</p>
-<div class="section">
-<h2>Block records<a name="Block_records"></a></h2>
-<p>Blocks are binary records of up to 4kB. They’re used as building blocks of large binary (or string) values and stored as-is with no extra metadata or structure. Blocks are the only record type that can’t contain references to other records. Block records are typically stored in <i>bulk segments</i> that consist only of block records and are thus easily identifiable as containing zero references to other segments.</p></div>
-<div class="section">
-<h2>List records<a name="List_records"></a></h2>
-<p>List records are used as components of more complex record types. Lists are used for storing arrays of values for multi-valued properties and sequences of blocks for large binary values.</p>
-<p>The list of references is split into pieces of up to 2^B references each (exact size TBD, B ~ 8) and those pieces are stored as records. If there are more than 2^B pieces like that, then a higher-level list is created of references to those pieces. This process is continued until the resulting list has less than 2^B entries. That top-level list is stored as a record prefixed with the total length of the list.</p>
-<p>The result is a hierarchically stored immutable list where each element can be accessed in O(log N) time and the size overhead of updating or appending list elements (and thus creating a new immutable list) is also O(log N).</p></div>
-<div class="section">
-<h2>Map records<a name="Map_records"></a></h2>
-<p>Like lists, maps are components of more complex record types. Maps store unordered sets of key-value pairs of record references and are used for nodes with a large number of properties or child nodes.</p>
-<p>Maps are stored using the hash array mapped trie (HAMT) data structure. The hash code of each key is split into pieces of B bits each (exact size TBD, B ~ 6) and the keys are sorted into 2^B packs based on the first B bits. If a pack contains less than 2^B entries, then it is stored directly as a list of key-value pairs. Otherwise the keys are split into sub-packs based on the next B bits of their hash codes. When all packs are stored, the list of top-level pack references gets stored along with the total number of entries in the map.</p>
-<p>The result is a hierarchically stored immutable map where each element can be accessed in O(log N) time and the size overhead of updating or inserting list elements is also O(log N).</p></div>
-<div class="section">
-<h2>Value records<a name="Value_records"></a></h2>
-<p>Value records are byte arrays used for storing all names and values of the content tree. Since item names can be thought of as name values and since all JCR and Oak values can be expressed in binary form, it is easiest to simply use that form for storing all values. The size overhead of such a form for small value types like booleans or dates is amortized by the facts that those types are used only for a minority of values in typical content trees and that repeating copies of a value can be stored just once.</p>
-<p>Small values, up to N kB (exact size TBD, N ~ 32), are stored inline in the record, prefixed by a byte or two to indicate the length of the value. Larger values are split into a list of fixed-size blocks and a possibly smaller tail block, and the value is stored as a list of block references.</p></div>
-<div class="section">
-<h2>Template records<a name="Template_records"></a></h2>
-<p>A template record describes the common structure of a family of related nodes. Since the structures of most nodes in a typical content tree fall into a small set of common templates, it makes sense to store such templates separately instead of repeating that information separately for each node. For example, the property names and types as well as child node names of all nt:file nodes are typically the same. The presence of mixins and different subtypes increases the number of different templates, but they’re typically still far fewer than nodes in the repository.</p>
-<p>A template record consists of a set of up to N (exact size TBD, N ~ 256) property name and type pairs. Additionally, since nodes that are empty or contain just a single child node are most common, a template record also contains information whether the node has zero, one or many child nodes. In case of a single child node, the template also contains the name of that node. For example, the template for typical mix:versionable nt:file nodes would be (using CND-like notation):</p>
-
-<div class="source">
-<pre>- jcr:primaryType (NAME)
-- jcr:mixinTypes (NAME) multiple
-- jcr:created (DATE)
-- jcr:uuid (STRING)
-- jcr:versionHistory (REFERENCE)
-- jcr:predecessors (REFERENCE) multiple
-- jcr:baseVersion (REFERENCE)
-+ jcr:content
-</pre></div>
-<p>The names used in a template are stored as separate value records and included by reference. This way multiple templates that for example all contain the “jcr:primaryType” property name don’t need to repeatedly store it.</p></div>
-<div class="section">
-<h2>Node records<a name="Node_records"></a></h2>
-<p>The overall structure of the content tree is stored in node records. Node records hold the actual content structure of the repository.</p>
-<p>A typical node record consists of a template reference followed by property value references (list references for multivalued properties) and zero, one or more child node entries as indicated by the template. If the node has more than one child nodes, then those entries are stored as an array of name-node pairs of references.</p>
-<p>A node that contains more than N properties or M child nodes (exact size TBD, M ~ 1k) is stored differently, using map records for the properties and child nodes. This way a node can become arbitrarily large and still remain reasonably efficient to access and modify. The main downside of this alternative storage layout is that the ordering of child nodes is lost.</p></div>
- </div>
- </div>
- </div>
-
- <hr/>
-
- <footer>
- <div class="container-fluid">
- <div class="row span12">Copyright © 2012-2013
- <a href="http://www.apache.org/">The Apache Software Foundation</a>.
- All Rights Reserved.
-
- </div>
-
-
-
-
-
-
- <div id="ohloh" class="pull-right">
- <script type="text/javascript" src="http://www.ohloh.net/p/jackrabbit-oak/widgets/project_users_logo.js"></script>
- </div>
- </div>
- </footer>
- </body>
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2013-10-24
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20131024" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Jackrabbit Oak - </title>
+ <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="./css/site.css" />
+ <link rel="stylesheet" href="./css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+ </head>
+ <body class="topBarEnabled">
+
+
+
+
+
+
+ <a href="http://github.com/apache/jackrabbit-oak">
+ <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
+ src="https://s3.amazonaws.com/github/ribbons/forkme_right_red_aa0000.png"
+ alt="Fork me on GitHub">
+ </a>
+
+
+
+
+
+ <div id="topbar" class="navbar navbar-fixed-top ">
+ <div class="navbar-inner">
+ <div class="container-fluid">
+ <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </a>
+
+ <ul class="nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Overview <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="index.html" title="Jackrabbit Oak">Jackrabbit Oak</a>
+</li>
+
+ <li> <a href="license.html" title="License">License</a>
+</li>
+
+ <li> <a href="downloads.html" title="Downloads">Downloads</a>
+</li>
+
+ <li> <a href="from_here.html" title="From here">From here</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Concepts and architecture <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="overview.html" title="Overview">Overview</a>
+</li>
+
+ <li> <a href="nodestate.html" title="Understanding the node state model">Understanding the node state model</a>
+</li>
+
+ <li> <a href="microkernel.html" title="Microkernel">Microkernel</a>
+</li>
+
+ <li> <a href="query.html" title="Query">Query</a>
+</li>
+
+ <li> <a href="blobstore.html" title="BlobStore">BlobStore</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Using Oak <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="use_getting_started.html" title="Getting Started">Getting Started</a>
+</li>
+
+ <li> <a href="differences.html" title="Differences to Jackrabbit 2">Differences to Jackrabbit 2</a>
+</li>
+
+ <li> <a href="dos_and_donts.html" title="Dos and don'ts">Dos and don'ts</a>
+</li>
+
+ <li> <a href="when_things_go_wrong.html" title="When things go wrong">When things go wrong</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developing Oak <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="dev_getting_started.html" title="Getting Started">Getting Started</a>
+</li>
+
+ <li> <a href="participating.html" title="Participating">Participating</a>
+</li>
+
+ <li> <a href="apidocs/index.html" title="API docs">API docs</a>
+</li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Links <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+
+ <li> <a href="http://jackrabbit.apache.org/oak" title="Apache Jackrabbit Oak">Apache Jackrabbit Oak</a>
+</li>
+
+ <li> <a href="http://jackrabbit.apache.org/" title="Apache Jackrabbit">Apache Jackrabbit</a>
+</li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <div id="bannerLeft">
+ <h2>Oak Documentation</h2>
+ </div>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li id="publishDate">Last Published: 2013-10-24</li>
+ <li class="divider">|</li> <li id="projectVersion">Version: 0.11-SNAPSHOT</li>
+
+
+
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">Overview</li>
+
+ <li>
+
+ <a href="index.html" title="Jackrabbit Oak">
+ <i class="none"></i>
+ Jackrabbit Oak</a>
+ </li>
+
+ <li>
+
+ <a href="license.html" title="License">
+ <i class="none"></i>
+ License</a>
+ </li>
+
+ <li>
+
+ <a href="downloads.html" title="Downloads">
+ <i class="none"></i>
+ Downloads</a>
+ </li>
+
+ <li>
+
+ <a href="from_here.html" title="From here">
+ <i class="none"></i>
+ From here</a>
+ </li>
+ <li class="nav-header">Concepts and architecture</li>
+
+ <li>
+
+ <a href="overview.html" title="Overview">
+ <i class="none"></i>
+ Overview</a>
+ </li>
+
+ <li>
+
+ <a href="nodestate.html" title="Understanding the node state model">
+ <i class="none"></i>
+ Understanding the node state model</a>
+ </li>
+
+ <li>
+
+ <a href="microkernel.html" title="Microkernel">
+ <i class="none"></i>
+ Microkernel</a>
+ </li>
+
+ <li>
+
+ <a href="query.html" title="Query">
+ <i class="none"></i>
+ Query</a>
+ </li>
+
+ <li>
+
+ <a href="blobstore.html" title="BlobStore">
+ <i class="none"></i>
+ BlobStore</a>
+ </li>
+ <li class="nav-header">Using Oak</li>
+
+ <li>
+
+ <a href="use_getting_started.html" title="Getting Started">
+ <i class="none"></i>
+ Getting Started</a>
+ </li>
+
+ <li>
+
+ <a href="differences.html" title="Differences to Jackrabbit 2">
+ <i class="none"></i>
+ Differences to Jackrabbit 2</a>
+ </li>
+
+ <li>
+
+ <a href="dos_and_donts.html" title="Dos and don'ts">
+ <i class="none"></i>
+ Dos and don'ts</a>
+ </li>
+
+ <li>
+
+ <a href="when_things_go_wrong.html" title="When things go wrong">
+ <i class="none"></i>
+ When things go wrong</a>
+ </li>
+ <li class="nav-header">Developing Oak</li>
+
+ <li>
+
+ <a href="dev_getting_started.html" title="Getting Started">
+ <i class="none"></i>
+ Getting Started</a>
+ </li>
+
+ <li>
+
+ <a href="participating.html" title="Participating">
+ <i class="none"></i>
+ Participating</a>
+ </li>
+
+ <li>
+
+ <a href="apidocs/index.html" title="API docs">
+ <i class="none"></i>
+ API docs</a>
+ </li>
+ <li class="nav-header">Links</li>
+
+ <li>
+
+ <a href="http://jackrabbit.apache.org/oak" class="externalLink" title="Apache Jackrabbit Oak">
+ <i class="none"></i>
+ Apache Jackrabbit Oak</a>
+ </li>
+
+ <li>
+
+ <a href="http://jackrabbit.apache.org/" class="externalLink" title="Apache Jackrabbit">
+ <i class="none"></i>
+ Apache Jackrabbit</a>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+
+ <script type="text/javascript" src="https://apis.google.com/js/plusone.js"></script>
+
+
+ <div class="g-plusone" data-href="http://jackrabbit.apache.org/oak-doc/" data-size="tall" ></div>
+
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <!-- Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. --><h1>SegmentMK design overview</h1>
+<h1>Segments</h1>
+<p>The content tree and all its revisions are stored in a collection of immutable <i>segments</i>. Each segment is identified by a UUID and typically contains a continuous subset of the content tree. Some segments might also be used to store commonly occurring property values or other shared data. Segments range from a few dozens of bytes to up to 256kB in size and are stored as documents in a MongoDB collection.</p>
+<p>Since segments are immutable, it’s easy for a client to keep a local in-memory cache of frequently accessed segments. Since segments also leverage locality of reference, i.e. nearby nodes are often stored in the same segment, it’s common for things like small child nodes to already exist in the cache by the time they get accessed. The intention is to avoid as many network or database roundtrips as possible by using local cache memory as efficiently as possible.</p>
+<p>Content within a segment can contain references to content within other segments. Each segment keeps a list of the UUIDs of all other segments it references. This list of segment references can be used to optimize both internal storage (as seen below) and garbage collection. Segments that are no longer referenced can be efficiently identified by traversing the graph of segment-level references without having to parse or even fetch the contents of each segment.</p>
+<p>The internal record structure of nodes is described in a moment once we first cover journal documents.</p>
+<h1>Journals</h1>
+<p>Journals are special, atomically updated documents that record the state of the repository as a sequence of references to successive root node records.</p>
+<p>A small system could consist of just a single journal and would serialize all repository updates through atomic updates of that journal. A larger system that needs more write throughput can have more journals, linked to each other in a tree hierarchy. Commits to journals in lower levels of the tree can proceed concurrently, but will need to be periodically merged back to the root journal. Potential conflicts and resulting data loss or inconsistency caused by such merges can be avoided by always committing against the root journal.</p>
+<p>Temporary branches used for large commits are also recorded as journals. A new private journal document is created for each branch and kept around until the branch gets merged or discarded. Branch journals contain an update timestamp that needs to be periodically refreshed by the client to prevent the branch from expiring and being reclaimed by the garbage collector.</p>
+<p>The root node references stored in journals are used as the starting point for garbage collection. It is assumed that all content currently visible to clients must be accessible through at least one of the journals. If a client wants to keep a reference to some old content revision that’s no longer referenced by one of the main journals, it should create an empty private branch based on that revision and keep refreshing the branch until that content is no longer needed.</p>
+<h1>Records</h1>
+<p>The content inside a segment is divided in records of different types: blocks, lists, maps, values, templates and nodes. These record types and their internal structures are described in subsections below.</p>
+<p>Each record is uniquely addressable by its location within the segment and the UUID of that segment. A single segment can contain up to 256kB of data and and references to up to 256 segments (including itself). Since all records are aligned at four-byte boundaries, 16 bits are needed to address all possible record locations within a segment. Thus only three bytes are needed to store a reference to any record in any segment (1 byte to identify the segment, 2 bytes for the record offset).</p>
+<div class="section">
+<h2>Block records<a name="Block_records"></a></h2>
+<p>Blocks are binary records of up to 4kB. They’re used as building blocks of large binary (or string) values and stored as-is with no extra metadata or structure. Blocks are the only record type that can’t contain references to other records. Block records are typically stored in <i>bulk segments</i> that consist only of block records and are thus easily identifiable as containing zero references to other segments.</p></div>
+<div class="section">
+<h2>List records<a name="List_records"></a></h2>
+<p>List records are used as components of more complex record types. Lists are used for storing arrays of values for multi-valued properties and sequences of blocks for large binary values.</p>
+<p>The list of references is split into pieces of up to 2^B references each (exact size TBD, B ~ 8) and those pieces are stored as records. If there are more than 2^B pieces like that, then a higher-level list is created of references to those pieces. This process is continued until the resulting list has less than 2^B entries. That top-level list is stored as a record prefixed with the total length of the list.</p>
+<p>The result is a hierarchically stored immutable list where each element can be accessed in O(log N) time and the size overhead of updating or appending list elements (and thus creating a new immutable list) is also O(log N).</p></div>
+<div class="section">
+<h2>Map records<a name="Map_records"></a></h2>
+<p>Like lists, maps are components of more complex record types. Maps store unordered sets of key-value pairs of record references and are used for nodes with a large number of properties or child nodes.</p>
+<p>Maps are stored using the hash array mapped trie (HAMT) data structure. The hash code of each key is split into pieces of B bits each (exact size TBD, B ~ 6) and the keys are sorted into 2^B packs based on the first B bits. If a pack contains less than 2^B entries, then it is stored directly as a list of key-value pairs. Otherwise the keys are split into sub-packs based on the next B bits of their hash codes. When all packs are stored, the list of top-level pack references gets stored along with the total number of entries in the map.</p>
+<p>The result is a hierarchically stored immutable map where each element can be accessed in O(log N) time and the size overhead of updating or inserting list elements is also O(log N).</p></div>
+<div class="section">
+<h2>Value records<a name="Value_records"></a></h2>
+<p>Value records are byte arrays used for storing all names and values of the content tree. Since item names can be thought of as name values and since all JCR and Oak values can be expressed in binary form, it is easiest to simply use that form for storing all values. The size overhead of such a form for small value types like booleans or dates is amortized by the facts that those types are used only for a minority of values in typical content trees and that repeating copies of a value can be stored just once.</p>
+<p>Small values, up to N kB (exact size TBD, N ~ 32), are stored inline in the record, prefixed by a byte or two to indicate the length of the value. Larger values are split into a list of fixed-size blocks and a possibly smaller tail block, and the value is stored as a list of block references.</p></div>
+<div class="section">
+<h2>Template records<a name="Template_records"></a></h2>
+<p>A template record describes the common structure of a family of related nodes. Since the structures of most nodes in a typical content tree fall into a small set of common templates, it makes sense to store such templates separately instead of repeating that information separately for each node. For example, the property names and types as well as child node names of all nt:file nodes are typically the same. The presence of mixins and different subtypes increases the number of different templates, but they’re typically still far fewer than nodes in the repository.</p>
+<p>A template record consists of a set of up to N (exact size TBD, N ~ 256) property name and type pairs. Additionally, since nodes that are empty or contain just a single child node are most common, a template record also contains information whether the node has zero, one or many child nodes. In case of a single child node, the template also contains the name of that node. For example, the template for typical mix:versionable nt:file nodes would be (using CND-like notation):</p>
+
+<div class="source">
+<pre>- jcr:primaryType (NAME)
+- jcr:mixinTypes (NAME) multiple
+- jcr:created (DATE)
+- jcr:uuid (STRING)
+- jcr:versionHistory (REFERENCE)
+- jcr:predecessors (REFERENCE) multiple
+- jcr:baseVersion (REFERENCE)
++ jcr:content
+</pre></div>
+<p>The names used in a template are stored as separate value records and included by reference. This way multiple templates that for example all contain the “jcr:primaryType” property name don’t need to repeatedly store it.</p></div>
+<div class="section">
+<h2>Node records<a name="Node_records"></a></h2>
+<p>The overall structure of the content tree is stored in node records. Node records hold the actual content structure of the repository.</p>
+<p>A typical node record consists of a template reference followed by property value references (list references for multivalued properties) and zero, one or more child node entries as indicated by the template. If the node has more than one child nodes, then those entries are stored as an array of name-node pairs of references.</p>
+<p>A node that contains more than N properties or M child nodes (exact size TBD, M ~ 1k) is stored differently, using map records for the properties and child nodes. This way a node can become arbitrarily large and still remain reasonably efficient to access and modify. The main downside of this alternative storage layout is that the ordering of child nodes is lost.</p></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2012-2013
+ <a href="http://www.apache.org/">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+
+
+
+ <div id="ohloh" class="pull-right">
+ <script type="text/javascript" src="http://www.ohloh.net/p/jackrabbit-oak/widgets/project_users_logo.js"></script>
+ </div>
+ </div>
+ </footer>
+ </body>
</html>
\ No newline at end of file