You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@jackrabbit.apache.org by mr...@apache.org on 2019/05/20 11:23:20 UTC
svn commit: r1859550 [7/26] - in /jackrabbit/site/live/archive: ./ wiki/
wiki/JCR/ wiki/JCR/attachments/ wiki/JCR/attachments/115513387/
wiki/JCR/attachments/115513390/ wiki/JCR/attachments/115513408/
wiki/JCR/attachments/115513413/ wiki/JCR/attachment...
Added: jackrabbit/site/live/archive/wiki/JCR/Clustering-the-Microkernel_115513378.html
URL: http://svn.apache.org/viewvc/jackrabbit/site/live/archive/wiki/JCR/Clustering-the-Microkernel_115513378.html?rev=1859550&view=auto
==============================================================================
--- jackrabbit/site/live/archive/wiki/JCR/Clustering-the-Microkernel_115513378.html (added)
+++ jackrabbit/site/live/archive/wiki/JCR/Clustering-the-Microkernel_115513378.html Mon May 20 11:23:18 2019
@@ -0,0 +1,115 @@
+<!DOCTYPE html>
+<html>
+ <head>
+ <title>Apache Jackrabbit : Clustering the Microkernel</title>
+ <link rel="stylesheet" href="styles/site.css" type="text/css" />
+ <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
+ </head>
+
+ <body class="theme-default aui-theme-default">
+ <div id="page">
+ <div id="main" class="aui-page-panel">
+ <div id="main-header">
+ <div id="breadcrumb-section">
+ <ol id="breadcrumbs">
+ <li class="first">
+ <span><a href="index.html">Apache Jackrabbit</a></span>
+ </li>
+ <li>
+ <span><a href="Home_70731.html">Home</a></span>
+ </li>
+ </ol>
+ </div>
+ <h1 id="title-heading" class="pagetitle">
+ <span id="title-text">
+ Apache Jackrabbit : Clustering the Microkernel
+ </span>
+ </h1>
+ </div>
+
+ <div id="content" class="view">
+ <div class="page-metadata">
+
+
+
+
+
+
+
+ Created by <span class='author'> ASF Infrabot</span> on May 20, 2019
+ </div>
+ <div id="main-content" class="wiki-content group">
+ <p>This page summarizes results from various online/offline and f2f discussions regarding clustering of the microkernel. It tries to fix some terms, sketches approaches which came up during the discussions and lists some of the open questions.</p>
+
+<style type='text/css'>/*<![CDATA[*/
+div.rbtoc1558350680177 {padding: 0px;}
+div.rbtoc1558350680177 ul {list-style: disc;margin-left: 0px;}
+div.rbtoc1558350680177 li {margin-left: 0px;padding-left: 0px;}
+
+/*]]>*/</style><div class='toc-macro rbtoc1558350680177'>
+<ul class='toc-indentation'>
+<li><a href='#ClusteringtheMicrokernel-Introduction'>Introduction</a></li>
+<li><a href='#ClusteringtheMicrokernel-Clustertopology'>Cluster topology</a></li>
+<li><a href='#ClusteringtheMicrokernel-Partitioning'>Partitioning</a></li>
+<li><a href='#ClusteringtheMicrokernel-Replication'>Replication</a></li>
+<li><a href='#ClusteringtheMicrokernel-DistributedB+Tree'>Distributed B+Tree</a></li>
+<li><a href='#ClusteringtheMicrokernel-Presentations'>Presentations</a></li>
+<li><a href='#ClusteringtheMicrokernel-References'>References</a></li>
+</ul>
+</div>
+
+<h3 id="ClusteringtheMicrokernel-Introduction">Introduction</h3>
+<p>Clustering is the common term for partitioning and replicating data. Partitioning means splitting data up among different storage locations such that each data item exists at a single location only. Partitioning enhances throughput through load balancing. Replication means a data item is kept in multiple storage locations at the same time. Replication enhances reliability through redundancy.</p>
+
+<p>The <a href="http://en.wikipedia.org/wiki/CAP_theorem" class="external-link" rel="nofollow">CAP</a> theorem states that it is impossible for a distributed computer system to provide consistency, availability and partition tolerance at the same time. (Note there is a slight clash of terminology: in the context of the CAP theorem partition tolerance means tolerance to the network becoming partitioned.)</p>
+
+<p>The clustering mechanism for the microkernel will trade consistency for availability in the face of network partition and for reduced latency under normal operation. The goal is to implement some form of <a href="http://en.wikipedia.org/wiki/Eventual_consistency" class="external-link" rel="nofollow">eventual consistency</a>.</p>
+
+<h3 id="ClusteringtheMicrokernel-Clustertopology">Cluster topology</h3>
+<p>On a first level, a cluster splits the JCR tree into various partitions. On a second level each partition may then be replicated to various cluster nodes.</p>
+
+<h3 id="ClusteringtheMicrokernel-Partitioning">Partitioning</h3>
+<p><p>A challenge for all partitioning schemes is enforcing atomicity of commits across different partitions. When atomicity is desired, some form of <a href="http://en.wikipedia.org/wiki/Atomic_commit" class="external-link" rel="nofollow">atomic commitment protocol</a> (ACP) has to be in place. The <a href="http://en.wikipedia.org/wiki/Two-phase_commit_protocol" class="external-link" rel="nofollow">two-phase commit protocol</a> seems to be standard here. Note however, that there is no ACP which is completely tolerant against network failures (See [1] proposition 7.1 and 7.2). That is, every ACP may cause cluster nodes to become blocked in the face of network failures. Furthermore no ACP can guarantee independent recovery of failed clusters.</p></p>
+
+<p>Move operations across partitions pose another challenge: with a naive implementation a whole subtree would have to be physically moved.</p>
+
+<p>One approach for partitioning the JCR tree is splitting it up by path: each cluster node contains the nodes pertaining to a direct child node of the root node. In this scenario the root node resembles a set of mount points onto which the various cluster nodes are mounted. Optimally the root node itself is stateless. An open question with this approach is how to aggregate the individual revision histories of the partitions into a consistent revision history of the root.</p>
+
+<p>Another approach for partitioning the JCR tree is splitting by subtree: in this scheme each cluster node contains some subtree of the entire JCR tree. The leave (JCR) nodes in each cluster node contain pointers to the cluster nodes which contain their actual child (JCR) nodes. One specific form of this scheme would dedicate a cluster node to the root (JCR) node and a couple of cluster nodes for the subtrees below. This scheme very much resembles the splitting by path scheme from above only that now the root is not state less. A disadvantage is that the root might become a bottleneck. However, since there is a dedicated cluster node for the root, it could be optimally tuned. On the other hand this scheme does not have the problem of aggregating the revision histories since these are effectively stored in the root node. Furthermore this scheme lends itself nicely to <a href="http://en.wikipedia.org/wiki/MapReduce" class="external-link" rel="nofollow">map reduce</a> approaches.</p>
+
+<h3 id="ClusteringtheMicrokernel-Replication">Replication</h3>
+<p>Replicating the same data items to various cluster nodes poses consistency concerns. A eventually consistent system guarantees, that after a finite time of quiescence, all cluster nodes in a replication set are in the same state. A possible approach to combine JCR with eventually consistent replication transparently, is to make cluster synchronization appear as normal session operations to clients. That is, when some items on a cluster node change due to a synchronization activity, the cluster node will generate JCR observation events for all sessions currently open on this cluster node.</p>
+
+<p><p>An open question is how to best implement cluster node synchronization: how (topological) should changes in cluster propagate to other clusters in order to ensure convergence. That is, in order not to arrive at an unstable system which propagates changes forth and back between cluster nodes. One approach which resembles branching and merging is proposed in [2]. However the branch operation doesn't seem viable. Another approach which might be more flexible is to use <a href="http://en.wikipedia.org/wiki/Vector_clock" class="external-link" rel="nofollow">vector clocks</a> for establishing a happens before relation on transactions from different cluster nodes.</p></p>
+
+<p>Another open question is how to synchronize conflicting transactions from different cluster nodes. Merging can be done to a certain degree. Other cases might have to rollback transactions on some cluster nodes. Consider for example the case where a node <em>x</em> is moved such that it becomes a child of node <em>y</em> on one cluster node and at them same time node <em>y</em> is moved such that it becomes a child of node <em>x</em> on another cluster node. The situation is inherently symmetric and neither move can follow the other. On cluster synchronization there seems to be no better way than to undo one (or both?) of the transactions which contained these move operations.</p>
+
+<h3 id="ClusteringtheMicrokernel-DistributedB+Tree">Distributed B+Tree</h3>
+<p>This approach was first mentioned on the <a href="http://markmail.org/message/jl66nqzct7s6ruju" class="external-link" rel="nofollow">dev list</a>. The basic idea is to store the MK nodes in a B+Tree and use the paths of the nodes as keys. To support efficient enumeration of child nodes, the order of the linearized MK tree is according to a breadth first search.</p>
+
+<p><p>The B+Tree could be implemented as described in the paper 'A practical scalable distributed B-tree' [3]. The B-Tree implementation is built on top of Sinfonia [4]. Sinfonia is a distributed block oriented storage mechanism, which supports simple transactions. Sinfonia works with optimistic concurrency control but still uses locks for a short amount of time to commit changes. Transactions that failed because of a conflict are simply retried until they succeed.</p></p>
+
+<h3 id="ClusteringtheMicrokernel-Presentations">Presentations</h3>
+<p><a href="https://docs.google.com/presentation/pub?id=131sVx5s58jAKE2FSVBfUZVQSl1W820_syyzLYRHGH6E&start=false&loop=false&delayms=3000" class="external-link" rel="nofollow">Clustering for Jackrabbit 3</a></p>
+
+<h3 id="ClusteringtheMicrokernel-References">References</h3>
+<p><p>[1] <a href="http://research.microsoft.com/en-us/people/philbe/ccontrol.aspx" class="external-link" rel="nofollow">Concurrency Control and Recovery in Database Systems</a>. Philip A. Bernstein, Vassos Hadzilacos, Nathan Goodman. 1987</p></p>
+
+<p><p>[2] <a href="http://research.microsoft.com/apps/pubs/default.aspx?id=155638" class="external-link" rel="nofollow">Eventually Consistent Transactions</a>. Sebastian Burckhardt, Manuel Fahndrich, Daan Leijen, and Mooly Sagiv. October 2011</p></p>
+
+<p><p>[3] <a href="http://research.microsoft.com/en-us/people/aguilera/distributed-btree-vldb2008.pdf" class="external-link" rel="nofollow">A practical scalable distributed B-tree</a>. Marcos K. Aguilera, Wojciech Golab, Mehul A. Shah. 2008</p></p>
+
+<p><p>[4] <a href="http://cs.ucla.edu/~kohler/class/08w-dsi/aguilera07sinfonia.pdf" class="external-link" rel="nofollow">Sinfonia: a new paradigm for building scalable distributed systems</a>. Marcos K. Aguilera, Arif Merchant, Mehul Shah, Alistair Veitch, Christos Karamanolis. 2007</p></p>
+ </div>
+
+
+
+ </div> </div>
+ <div id="footer" role="contentinfo">
+ <section class="footer-body">
+ <p>Document generated by Confluence on May 20, 2019 11:11</p>
+ <div id="footer-logo"><a href="http://www.atlassian.com/">Atlassian</a></div>
+ </section>
+ </div>
+ </div> </body>
+</html>
Propchange: jackrabbit/site/live/archive/wiki/JCR/Clustering-the-Microkernel_115513378.html
------------------------------------------------------------------------------
svn:eol-style = native
Propchange: jackrabbit/site/live/archive/wiki/JCR/Clustering-the-Microkernel_115513378.html
------------------------------------------------------------------------------
svn:executable = *
Added: jackrabbit/site/live/archive/wiki/JCR/Clustering_115513377.html
URL: http://svn.apache.org/viewvc/jackrabbit/site/live/archive/wiki/JCR/Clustering_115513377.html?rev=1859550&view=auto
==============================================================================
--- jackrabbit/site/live/archive/wiki/JCR/Clustering_115513377.html (added)
+++ jackrabbit/site/live/archive/wiki/JCR/Clustering_115513377.html Mon May 20 11:23:18 2019
@@ -0,0 +1,321 @@
+<!DOCTYPE html>
+<html>
+ <head>
+ <title>Apache Jackrabbit : Clustering</title>
+ <link rel="stylesheet" href="styles/site.css" type="text/css" />
+ <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
+ </head>
+
+ <body class="theme-default aui-theme-default">
+ <div id="page">
+ <div id="main" class="aui-page-panel">
+ <div id="main-header">
+ <div id="breadcrumb-section">
+ <ol id="breadcrumbs">
+ <li class="first">
+ <span><a href="index.html">Apache Jackrabbit</a></span>
+ </li>
+ <li>
+ <span><a href="Home_70731.html">Home</a></span>
+ </li>
+ </ol>
+ </div>
+ <h1 id="title-heading" class="pagetitle">
+ <span id="title-text">
+ Apache Jackrabbit : Clustering
+ </span>
+ </h1>
+ </div>
+
+ <div id="content" class="view">
+ <div class="page-metadata">
+
+
+
+
+
+
+
+ Created by <span class='author'> ASF Infrabot</span> on May 20, 2019
+ </div>
+ <div id="main-content" class="wiki-content group">
+ <h2 id="Clustering-Clustering">Clustering</h2>
+
+<style type='text/css'>/*<![CDATA[*/
+div.rbtoc1558350680147 {padding: 0px;}
+div.rbtoc1558350680147 ul {list-style: disc;margin-left: 0px;}
+div.rbtoc1558350680147 li {margin-left: 0px;padding-left: 0px;}
+
+/*]]>*/</style><div class='toc-macro rbtoc1558350680147'>
+<ul class='toc-indentation'>
+<li><a href='#Clustering-Clustering'>Clustering</a></li>
+<li><a href='#Clustering-Overview'>Overview</a></li>
+<li><a href='#Clustering-Limitations'>Limitations</a></li>
+<li><a href='#Clustering-Requirements'>Requirements</a></li>
+<li><a href='#Clustering-UniqueClusterNodeID'>Unique Cluster Node ID</a></li>
+<li><a href='#Clustering-SyncDelay'>Sync Delay</a></li>
+<li><a href='#Clustering-RemovingOldRevisions'>Removing Old Revisions</a></li>
+<li><a href='#Clustering-Easilyaddnewclusternodes'>Easily add new cluster nodes</a></li>
+<li><a href='#Clustering-JournalType'>Journal Type</a>
+<ul class='toc-indentation'>
+<li><a href='#Clustering-FileJournal'>File Journal</a></li>
+<li><a href='#Clustering-DatabaseJournal'>Database Journal</a></li>
+</ul>
+</li>
+<li><a href='#Clustering-SampleClusterConfigurations'>Sample Cluster Configurations</a>
+<ul class='toc-indentation'>
+<li><a href='#Clustering-Database(CompleteExample)'>Database (Complete Example)</a></li>
+<li><a href='#Clustering-FileJournal.1'>File Journal</a></li>
+<li><a href='#Clustering-OracleJournal'>Oracle Journal</a></li>
+<li><a href='#Clustering-PostgreSQLJournal'>PostgreSQL Journal</a></li>
+<li><a href='#Clustering-PersistenceManagerConfiguration'>Persistence Manager Configuration</a></li>
+<li><a href='#Clustering-DataStoreConfiguration'>Data Store Configuration</a></li>
+</ul>
+</li>
+<li><a href='#Clustering-ConcurrentWriteBehavior'>Concurrent Write Behavior</a></li>
+</ul>
+</div>
+
+<h2 id="Clustering-Overview">Overview</h2>
+
+<p>Clustering in Jackrabbit works as follows: content is shared between all cluster nodes. That means all Jackrabbit cluster nodes need access to the *same* persistent storage (persistence manager, data store, and repository file system).</p>
+
+<p>The persistence manager must be clusterable (eg. central database that allows for concurrent access, see <a href="PersistenceManagerFAQ_115513487.html">PersistenceManagerFAQ</a>); any <a href="DataStore_115513387.html">DataStore</a> (file or DB) is clusterable by its very nature, as they store content by unique hash ids. </p>
+
+<p>However, each cluster node needs its own (private) repository directory, including repository.xml file, workspace <a class="createlink" href="/confluence/pages/createpage.action?spaceKey=JCR&title=FileSystem&linkCreation=true&fromPageId=115513377">FileSystem</a> and <a href="Search_115513504.html">Search</a> index.</p>
+
+<p>Every change made by one cluster node is reported in a journal, which can be either file based or written to some database.</p>
+
+<h2 id="Clustering-Limitations">Limitations</h2>
+
+<p>Session scoped locks currently have no effect on other cluster nodes - see <a href="http://issues.apache.org/jira/browse/JCR-1173" class="external-link" rel="nofollow">http://issues.apache.org/jira/browse/JCR-1173</a></p>
+
+<h2 id="Clustering-Requirements">Requirements</h2>
+
+<p>In order to use clustering, the following prerequisites must be met:</p>
+
+<ul>
+ <li>Each cluster node must have its own repository configuration.</li>
+ <li>A <a href="DataStore_115513387.html">DataStore</a> must always be shared between nodes, if used.</li>
+ <li>The global <a class="createlink" href="/confluence/pages/createpage.action?spaceKey=JCR&title=FileSystem&linkCreation=true&fromPageId=115513377">FileSystem</a> on the repository level must be shared (only the one that is on the same level as the data store; only in the repository.xml file).</li>
+ <li>Each cluster node needs its own (private) workspace level and version <a class="createlink" href="/confluence/pages/createpage.action?spaceKey=JCR&title=FileSystem&linkCreation=true&fromPageId=115513377">FileSystem</a> (only those within the workspace and versioning configuration; the ones in the repository.xml and workspace.xml file).</li>
+ <li>Each cluster node needs its own (private) Search indexes.</li>
+ <li>Every cluster node must be assigned a unique ID.</li>
+ <li>A journal type must be chosen, either based on files or stored in a database.</li>
+ <li>Each cluster node must use the same (shared) journal.</li>
+ <li>The persistence managers must store their data in the same, globally accessible location (see <a href="PersistenceManagerFAQ_115513487.html">PersistenceManagerFAQ</a>).
+<br class="atl-forced-newline"/></li>
+</ul>
+
+
+<h2 id="Clustering-UniqueClusterNodeID">Unique Cluster Node ID</h2>
+
+<p>Every cluster node needs a unique ID. This ID can be either specified in the cluster configuration as <strong>id</strong> attribute or as value of the system property <strong>org.apache.jackrabbit.core.cluster.node_id</strong>. When copying repository configurations, do not forget to adapt the cluster node IDs if they are hardcoded. See below for some sample cluster configurations. A cluster id can be freely defined, the only requirement is that it has to be different on each cluster node.</p>
+
+<h2 id="Clustering-SyncDelay">Sync Delay</h2>
+
+<p>By default, cluster nodes read the journal and update their state every 5 seconds (5000 milliseconds). To use a different value, set the attribute <em>syncDelay</em> in the cluster configuration.</p>
+
+<h2 id="Clustering-RemovingOldRevisions">Removing Old Revisions</h2>
+
+<p>The journal in which cluster nodes write their changes can potentially become very large. By default, old revisions are not removed. This enables one to add a cluster node without much work: the new cluster node just replays the journal to get up to date (of course, if the journal contains data from two years of work then this might take a while...)</p>
+
+<p>As of Jackrabbit 1.5 there is the possibility to automatically clean the database-based journal. The local revision counter is automatically migrated to a new table in the database called LOCAL_REVISIONS. To support a proper migration the "revision" parameter must be present in the configuration. After the migration it can be removed.</p>
+
+<p>The clean-up task can be configured with three parameters:</p>
+<ul>
+ <li><strong>janitorEnabled</strong> specifies whether the clean-up task for the journal table is enabled (default = false)</li>
+ <li><strong>janitorSleep</strong> specifies the sleep time of the clean-up task in seconds (only useful when the clean-up task is enabled, default is 24 hours)</li>
+ <li><strong>janitorFirstRunHourOfDay</strong> specifies the hour at which the clean-up task initiates its first run (default = 3, which means 3:00 at night)
+<br class="atl-forced-newline"/></li>
+</ul>
+
+
+<p>The current solution has three known caveats:</p>
+<ul>
+ <li>If the janitor is enabled then you loose the possibility to easily add cluster nodes. (It is still possible but takes detailed knowledge of Jackrabbit.)</li>
+ <li>You must make sure that all cluster nodes have written their local revision to the database before the clean-up task runs for the first time because otherwise cluster nodes might miss updates (because they have been purged) and their local caches and search-indexes get out of sync.</li>
+ <li>If a cluster node is removed permanently from the cluster, then its entry in the LOCAL_REVISIONS table should be removed manually. Otherwise, the clean-up thread will not be effective.
+<br class="atl-forced-newline"/></li>
+</ul>
+
+
+<p>Related issue: <a href="http://issues.apache.org/jira/browse/JCR-1087" class="external-link" rel="nofollow">http://issues.apache.org/jira/browse/JCR-1087</a></p>
+
+<h2 id="Clustering-Easilyaddnewclusternodes">Easily add new cluster nodes</h2>
+
+<ul>
+ <li>Shutdown one of your instances</li>
+ <li>Get the current revision number that instance was from your database</li>
+ <li>Copy your whole Jackrabbit repository directory to another server/location</li>
+ <li>Start your original Jackrabbit again</li>
+ <li>Change the copied repository.xml with a new nodename in your clusterconfig</li>
+ <li>Add that nodename to your DB in JOURNAL_LOCAL_REVISIONS with the number from the original instance</li>
+ <li>Start your new Jackrabbit instance (or keep it for backup purposes)
+<br class="atl-forced-newline"/></li>
+</ul>
+
+
+<p>Related info: <a href="http://blog.liip.ch/archive/2011/05/10/add-new-instances-to-your-jackrabbit-cluster-the-non-time-consuming-way.html" class="external-link" rel="nofollow">http://blog.liip.ch/archive/2011/05/10/add-new-instances-to-your-jackrabbit-cluster-the-non-time-consuming-way.html</a> , <a href="https://github.com/chregu/Jackrabbit-clone-scripts/" class="external-link" rel="nofollow">https://github.com/chregu/Jackrabbit-clone-scripts/</a></p>
+
+<h2 id="Clustering-JournalType">Journal Type</h2>
+
+<p>The cluster nodes store information identifying items they modified in a journal. This journal must again be globally available to all nodes in the cluster. This can be either a folder in the file system or a database running standalone.</p>
+
+<h3 id="Clustering-FileJournal">File Journal</h3>
+
+<p>The file journal is configured through the following properties:</p>
+
+<ul>
+ <li><strong>revision</strong>: location of the cluster node's revision file</li>
+ <li><strong>directory</strong>: location of the journal folder
+<br class="atl-forced-newline"/></li>
+</ul>
+
+
+<h3 id="Clustering-DatabaseJournal">Database Journal</h3>
+
+<p>There are three Journal classes:</p>
+
+<ul>
+ <li>org.apache.jackrabbit.core.journal.DatabaseJournal</li>
+ <li>org.apache.jackrabbit.core.journal.OracleDatabaseJournal</li>
+ <li>org.apache.jackrabbit.core.journal.MSSqlDatabaseJournal
+<br class="atl-forced-newline"/></li>
+</ul>
+
+
+<p>If you use Oracle, you do need to use the OracleDatabaseJournal. The DatabaseJournal will not work.</p>
+
+<p>The database journal is configured through the following properties:</p>
+
+<ul>
+ <li><strong>revision</strong>: location of the cluster node's revision file</li>
+ <li><strong>driver</strong>: JDBC driver class name</li>
+ <li><strong>url</strong>: JDBC URL</li>
+ <li><strong>user</strong>: user name</li>
+ <li><strong>password</strong>: password
+<br class="atl-forced-newline"/></li>
+</ul>
+
+
+<h2 id="Clustering-SampleClusterConfigurations">Sample Cluster Configurations</h2>
+
+<h3 id="Clustering-Database(CompleteExample)">Database (Complete Example)</h3>
+
+<p>A sample repository.xml file that is using a clustered H2 database for all data (file system, data store, persistence managers, versioning, journal):</p>
+
+<p><a href="http://svn.apache.org/viewvc/jackrabbit/trunk/jackrabbit-core/src/test/resources/org/apache/jackrabbit/core/cluster/repository-h2.xml?view=co&content-type=text/plain" class="external-link" rel="nofollow">jackrabbit-core/src/test/resources/org/apache/jackrabbit/core/cluster/repository-h2.xml</a></p>
+
+<p>For details about H2 clustering see the <a href="http://h2database.com/html/advanced.html#clustering" class="external-link" rel="nofollow">H2 documentation</a>.</p>
+
+<h3 id="Clustering-FileJournal.1">File Journal</h3>
+
+<p>This section contains some sample cluster configurations. First, using a file based journal implementation, where the journal files are created in a share exported by NFS. Please note that for high availability, the NFS itself must be highly available (using a clustered file system).</p>
+
+<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
+<pre><Cluster id="node1">
+ <Journal class="org.apache.jackrabbit.core.journal.FileJournal">
+ <param name="revision" value="${rep.home}/revision.log" />
+ <param name="directory" value="/nfs/myserver/myjournal" />
+ </Journal>
+</Cluster>
+</pre>
+</div></div>
+
+<h3 id="Clustering-OracleJournal">Oracle Journal</h3>
+
+<p>In the next configuration, the journal is stored in an Oracle database, using a sync delay of 2 seconds (2000 milliseconds):</p>
+
+<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
+<pre><Cluster id="node1" syncDelay="2000">
+ <Journal class="org.apache.jackrabbit.core.journal.OracleDatabaseJournal">
+ <param name="revision" value="${rep.home}/revision.log" />
+ <param name="driver" value="oracle.jdbc.driver.OracleDriver" />
+ <param name="url" value="jdbc:oracle:thin:@myhost:1521:mydb" />
+ <param name="user" value="scott"/>
+ <param name="password" value="tiger"/>
+ </Journal>
+</Cluster>
+</pre>
+</div></div>
+
+<h3 id="Clustering-PostgreSQLJournal">PostgreSQL Journal</h3>
+
+<p>In the following configuration, the journal is stored in an PostgreSQL database, accessed via "JNDI" (See Also <a href="UsingJNDIDataSource_115513523.html">UsingJNDIDataSource</a>):</p>
+
+<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
+<pre><Cluster id="node1" syncDelay="2000">
+ <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
+ <param name="revision" value="${rep.home}/revision.log" />
+ <param name="driver" value="javax.naming.InitialContext"/>
+ <param name="url" value="java:jdbc/Workspace"/>
+ <param name="databaseType" value="postgresql"/>
+ </Journal>
+</Cluster>
+
+</pre>
+</div></div>
+
+<p><strong>Note</strong>: the journal implementation classes have been refactored in Jackrabbit 1.3. In earlier versions, journal implementations resided in the package <em>org.apache.jackrabbit.core.cluster</em>.</p>
+
+<h3 id="Clustering-PersistenceManagerConfiguration">Persistence Manager Configuration</h3>
+
+<p>All cluster nodes must point to the same persistence location. For performance reasons, only information identifying the modified items is stored in the journal. This implies, that all cluster nodes must configure the same persistence manager and persistence location, because they must have access to the items' actual content. The persistence manager needs to be transactional, and need to support concurrent access from multiple processes. When using Jackrabbit, one option is to use a database persistence manager, and use a database that does support concurrent access. The file system based persistence managers in Jackrabbit are not transactional and don't support concurrent access; Apache Derby doesn't support concurrent access in the embedded mode. The following sample shows a workspace's persistence manager configuration using an Oracle database:</p>
+
+<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
+<pre><PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.OraclePersistenceManager">
+ <param name="url" value="jdbc:oracle:thin:@myhost:1521:mydb" />
+ <param name="user" value="scott"/>
+ <param name="password" value="tiger"/>
+ <param name="schemaObjectPrefix" value="${wsp.name}_"/>
+</PersistenceManager>
+</pre>
+</div></div>
+
+<p>Previous versions of Jackrabbit that do not support this persistence manager may need to use org.apache.jackrabbit.core.persistence.db.OraclePersistenceManager.</p>
+
+
+<h3 id="Clustering-DataStoreConfiguration">Data Store Configuration</h3>
+
+<p>All cluster nodes must point to the same data store location. The data store should be used to store large binaries (all cluster nodes need to access the same data store). When not using the data store, one need to set the parameter <strong>externalBLOBs</strong> to <em>false</em> so that large binaries are stored in the persistence manager. The file system blob store does not support clustering, because it uses a local directory.</p>
+
+<h2 id="Clustering-ConcurrentWriteBehavior">Concurrent Write Behavior</h2>
+
+<p>When multiple cluster nodes write to the same nodes, those nodes must be locked first. If the nodes are not locked, then the operation may fail if the nodes were updated concurrently (or a little bit earlier). As an example, the following sequence may fail on session 2 (session 1 operates on cluster node 1, session 2 on cluster node 2), even the operations on session 2 are executed after session1.save():</p>
+
+<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
+<pre>// on cluster node 1
+session1.getRootNode().getNode("test").setProperty("x", "a");
+session1.save();
+// on cluster node 2
+session2.getRootNode().getNode("test").setProperty("x", "b");
+session2.save();
+</pre>
+</div></div>
+
+<p>The best solution is to use locking. If session2 operations are guaranteed to be executed after all session1 operations, another solution is to use session.refresh(), however only if cluster sync in refresh is enabled:</p>
+
+<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
+<pre>// on cluster node 1
+session1.getRootNode().getNode("test").setProperty("x", "a");
+session1.save();
+// on cluster node 2
+session2.refresh(false);
+session2.getRootNode().getNode("test").setProperty("x", "b");
+session2.save();
+</pre>
+</div></div>
+ </div>
+
+
+
+ </div> </div>
+ <div id="footer" role="contentinfo">
+ <section class="footer-body">
+ <p>Document generated by Confluence on May 20, 2019 11:11</p>
+ <div id="footer-logo"><a href="http://www.atlassian.com/">Atlassian</a></div>
+ </section>
+ </div>
+ </div> </body>
+</html>
Propchange: jackrabbit/site/live/archive/wiki/JCR/Clustering_115513377.html
------------------------------------------------------------------------------
svn:eol-style = native
Propchange: jackrabbit/site/live/archive/wiki/JCR/Clustering_115513377.html
------------------------------------------------------------------------------
svn:executable = *
Added: jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store-Cold-Storage-Delegates_115513380.html
URL: http://svn.apache.org/viewvc/jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store-Cold-Storage-Delegates_115513380.html?rev=1859550&view=auto
==============================================================================
--- jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store-Cold-Storage-Delegates_115513380.html (added)
+++ jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store-Cold-Storage-Delegates_115513380.html Mon May 20 11:23:18 2019
@@ -0,0 +1,83 @@
+<!DOCTYPE html>
+<html>
+ <head>
+ <title>Apache Jackrabbit : Composite Blob Store Cold Storage Delegates</title>
+ <link rel="stylesheet" href="styles/site.css" type="text/css" />
+ <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
+ </head>
+
+ <body class="theme-default aui-theme-default">
+ <div id="page">
+ <div id="main" class="aui-page-panel">
+ <div id="main-header">
+ <div id="breadcrumb-section">
+ <ol id="breadcrumbs">
+ <li class="first">
+ <span><a href="index.html">Apache Jackrabbit</a></span>
+ </li>
+ <li>
+ <span><a href="Home_70731.html">Home</a></span>
+ </li>
+ </ol>
+ </div>
+ <h1 id="title-heading" class="pagetitle">
+ <span id="title-text">
+ Apache Jackrabbit : Composite Blob Store Cold Storage Delegates
+ </span>
+ </h1>
+ </div>
+
+ <div id="content" class="view">
+ <div class="page-metadata">
+
+
+
+
+
+
+
+ Created by <span class='author'> ASF Infrabot</span> on May 20, 2019
+ </div>
+ <div id="main-content" class="wiki-content group">
+ <h1 id="CompositeBlobStoreColdStorageDelegates-CompositeBlobStoreColdStorageDelegates">Composite Blob Store Cold Storage Delegates</h1>
+
+<p>NOTE: The current status of this feature is a <strong>rejected feature</strong> for Oak 1.8 and a <strong>possible future feature</strong> for later Oak versions. It would depend on the implementation of some type of cold storage blob store, which currently does not exist.</p>
+
+<h2 id="CompositeBlobStoreColdStorageDelegates-Overview">Overview</h2>
+<p>A cold storage delegate is a <a href="Composite-Blob-Store_115513379.html">Composite Blob Store</a> delegate that stores data in a cold storage blob store. This is a blob store that stores data in some form of cold storage, like AWS Glacier. The primary feature of cold storage is that read access is not necessarily immediate and may take some amount of time. Cold storage services offer copious amounts of storage for low cost in exchange for very slow access time. The intended use is long-term archival.</p>
+
+<p>This is a possible future feature that can add value to the composite blob store concept in the area of auto archival use cases.</p>
+
+<h2 id="CompositeBlobStoreColdStorageDelegates-TechnicalDetails">Technical Details</h2>
+<h3 id="CompositeBlobStoreColdStorageDelegates-ImpactsonCompositeBlobStore">Impacts on Composite Blob Store</h3>
+<p>The existence of a cold storage delegate may have an impact on the delegate traversal strategy implementation. For example, an implementation may consider that a cold storage delegate should always be last resort for read or write requests. Or an implementation may wish to treat cold storage delegates as read-only delegates, or take some other approach.</p>
+
+<h4 id="CompositeBlobStoreColdStorageDelegates-ImpactonIntelligentDelegateTraversalStrategy">Impact on Intelligent Delegate Traversal Strategy</h4>
+<p>If we were to add support for cold storage delegates, it is expected that the following changes would be added to the Intelligent Delegate Traversal Strategy, which is the default strategy.</p>
+
+<h5 id="CompositeBlobStoreColdStorageDelegates-DelegateSearchOrder">Delegate Search Order</h5>
+<ul>
+ <li>Delegates for cold storage reads are always evaluated after every other option is exhausted.</li>
+ <li>Cold-storage delegates are treated as read-only; this strategy would not support writing to cold storage through the composite blob store. This may be done via an entirely different process.
+<br class="atl-forced-newline"/></li>
+</ul>
+
+
+<h5 id="CompositeBlobStoreColdStorageDelegates-Writes">Writes</h5>
+<p>Cold-storage delegates are treated as read-only delegates by this traversal strategy. Writes must be done via a different custom process.</p>
+
+<h5 id="CompositeBlobStoreColdStorageDelegates-Reads">Reads</h5>
+<p>After having tried to read the blob from every other delegate, the algorithm would attempt to fulfill read request from cold-storage delegates. Note that since it may take a long time (e.g. many minutes or hours) to fulfill these requests, a different mechanism for handling these responses would need to be implemented.</p>
+ </div>
+
+
+
+ </div> </div>
+ <div id="footer" role="contentinfo">
+ <section class="footer-body">
+ <p>Document generated by Confluence on May 20, 2019 11:11</p>
+ <div id="footer-logo"><a href="http://www.atlassian.com/">Atlassian</a></div>
+ </section>
+ </div>
+ </div> </body>
+</html>
Propchange: jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store-Cold-Storage-Delegates_115513380.html
------------------------------------------------------------------------------
svn:eol-style = native
Propchange: jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store-Cold-Storage-Delegates_115513380.html
------------------------------------------------------------------------------
svn:executable = *
Added: jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store-Storage-Filters_115513381.html
URL: http://svn.apache.org/viewvc/jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store-Storage-Filters_115513381.html?rev=1859550&view=auto
==============================================================================
--- jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store-Storage-Filters_115513381.html (added)
+++ jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store-Storage-Filters_115513381.html Mon May 20 11:23:18 2019
@@ -0,0 +1,133 @@
+<!DOCTYPE html>
+<html>
+ <head>
+ <title>Apache Jackrabbit : Composite Blob Store Storage Filters</title>
+ <link rel="stylesheet" href="styles/site.css" type="text/css" />
+ <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
+ </head>
+
+ <body class="theme-default aui-theme-default">
+ <div id="page">
+ <div id="main" class="aui-page-panel">
+ <div id="main-header">
+ <div id="breadcrumb-section">
+ <ol id="breadcrumbs">
+ <li class="first">
+ <span><a href="index.html">Apache Jackrabbit</a></span>
+ </li>
+ <li>
+ <span><a href="Home_70731.html">Home</a></span>
+ </li>
+ </ol>
+ </div>
+ <h1 id="title-heading" class="pagetitle">
+ <span id="title-text">
+ Apache Jackrabbit : Composite Blob Store Storage Filters
+ </span>
+ </h1>
+ </div>
+
+ <div id="content" class="view">
+ <div class="page-metadata">
+
+
+
+
+
+
+
+ Created by <span class='author'> ASF Infrabot</span> on May 20, 2019
+ </div>
+ <div id="main-content" class="wiki-content group">
+ <h1 id="CompositeBlobStoreStorageFilters-CompositeBlobStoreStorageFilters">Composite Blob Store Storage Filters</h1>
+
+<p>NOTE: The current status of this component is a <strong>rejected feature</strong> for Oak 1.8, and a <strong>possible future feature</strong> for later Oak versions.</p>
+
+<h2 id="CompositeBlobStoreStorageFilters-Overview">Overview</h2>
+<p>Storage filters add capability to <a href="Composite-Blob-Store_115513379.html">Composite Blob Store</a> by allowing configuration to define criteria that restrict which blobs can be stored in a composite blob store delegate. Storage filters would then be an optional element of delegate configuration. Delegates would not be required to have any storage filters at all.</p>
+
+<p>This is a possible future feature that can add a lot of value to the composite blob store concept. A major hurdle to this implementation is that JCR node information is not provided to data store implementations; the normal case is that a binary is created before the corresponding node. This problem has to be resolved in order to implement this feature. More discussion on this below.</p>
+
+<h2 id="CompositeBlobStoreStorageFilters-TechnicalDetails">Technical Details</h2>
+<h3 id="CompositeBlobStoreStorageFilters-StorageFilterTypes">Storage Filter Types</h3>
+<p>Storage filters operate on any combination of the following:</p>
+<ul>
+ <li>JCR path</li>
+ <li>JCR node type</li>
+ <li>Existence of JCR property</li>
+ <li>JCR property value
+<br class="atl-forced-newline"/></li>
+</ul>
+
+
+<h3 id="CompositeBlobStoreStorageFilters-ImpactsonCompositeBlobStore">Impacts on Composite Blob Store</h3>
+<p>The existence of a storage filter on a delegate blob store may have an impact on the delegate traversal strategy implementation. For example, an implementation may consider that a blob store with filters should take precedence over a blob store without filters in terms of choosing a write destination.</p>
+
+<h4 id="CompositeBlobStoreStorageFilters-ImpactonIntelligentDelegateTraversalStrategy">Impact on Intelligent Delegate Traversal Strategy</h4>
+<p>If storage filters were added to composite blob store, it is expected that the following changes would be added to the Intelligent Delegate Traversal Strategy, which is the default traversal strategy.</p>
+
+<h5 id="CompositeBlobStoreStorageFilters-DelegateSearchOrder">Delegate Search Order</h5>
+<p>Speaking generally, delegates with storage filters would take highest precedence, and would be tried first for reads and writes.</p>
+
+<h5 id="CompositeBlobStoreStorageFilters-Writes">Writes</h5>
+<p>Delegates that have storage filters always have write precedence over delegates that do not have storage filters - meaning writes always happen to delegates with filters if the filters match what is being written.</p>
+
+<p>The algorithm for writing a blob to the composite blob store would be changed:</p>
+<ul>
+ <li>When determining where to write the blob, start with delegates that have storage filters. Prefer writing to a delegate with a storage filter that matches the blob being written.</li>
+ <li>When searching for other versions of the same blob, <strong>also</strong> search using delegates that have filters that don't match the blob being written (after searching delegates without filters).
+ <ul>
+ <li>Configuration change can cause this situation - a blob may originally have been written to a blob store, and afterward configuration changes such that the blob would not have been written to that blob store originally. See "Using Non-matching Delegates" below for more.</li>
+ <li>If this case occurs, the code should also asynchronously remove the blob from the incorrect location once it has been written to the correct location.
+<br class="atl-forced-newline"/></li>
+ </ul>
+ </li>
+</ul>
+
+
+<h5 id="CompositeBlobStoreStorageFilters-Reads">Reads</h5>
+<p>Delegates that have storage filters always have read precedence over delegates that do not have storage filters - meaning we always attempt to read from them first, because matching filters is faster than checking a delegate to see if a blob exists.</p>
+
+<p>The algorithm for reading a blob from the composite blob store would be changed:</p>
+<ul>
+ <li>When searching for a blob, start with delegates that have storage filters that match the blob.</li>
+ <li>If no match, search delegates without storage filters.</li>
+ <li>If no match, search delegates that have storage filters that do not match the blob.
+ <ul>
+ <li>Configuration change can cause this situation - a blob may originally have been written to a blob store, and afterward configuration changes such that the blob would not have been written to that blob store originally. See "Using Non-matching Delegates" below for more.</li>
+ <li>If this case occurs, the code should also asynchronously move the blob from the incorrect location to the correct location.
+<br class="atl-forced-newline"/></li>
+ </ul>
+ </li>
+</ul>
+
+
+<h5 id="CompositeBlobStoreStorageFilters-UsingNon-matchingDelegates">Using Non-matching Delegates</h5>
+
+<p>This step is necessary to handle situations where blobs are temporarily located in the "wrong" blob store - in other words, when a blob is located in a delegate where it would not be written according to configuration. The most obvious case where this could occur is in the case of configuration change. A delegate D may be configured with certain storage filters, causing Blob B to be written there. Then the configuration is changed such that if B were being written now it would not have been written to D. This final step allows B to be found in D even though it doesn't match the storage filters.</p>
+
+<p>When this situation is encountered, the composite blob store should also initiate an asynchronous background job to move the blob from it's current location to the proper one - the location where it would be found if it were being created now - on read requests, or for write requests should write to the correct location and remove the blob asynchronously in the background from the current location after the write is done.</p>
+
+<h3 id="CompositeBlobStoreStorageFilters-TechnicalChallenges">Technical Challenges</h3>
+<p>The Data Store API doesn't have knowledge about any JCR node information for the binary being written, and doesn't have any provision to provide that information to any of the data store APIs today. The primary way a binary is created is to do so without any node information. Usually the node is created after the binary creation has taken place. This means that, while potentially useful, providing JCR information to the data store is problematic.</p>
+
+<h4 id="CompositeBlobStoreStorageFilters-PossibleSolutions">Possible Solutions</h4>
+<p>NOTE: (Matt Ryan): I don't fully understand the impacts of all suggestions here; additional suggestions and feedback encouraged.</p>
+<ul>
+ <li><strong>Rework the API to support providing JCR information in data store calls.</strong> While I'm not 100% sure of this, I assume the JCR node information is at least known at the time a binary is created; otherwise we wouldn't know whether it can be created, if there is already something with the same name in the node store, if the user has rights to create at this location, etc.</li>
+ <li><strong>Encode JCR information in the data identifier.</strong> This could be possible and might minimize public API contract changes. However, not all calls are based on a data identifier (e.g. DataStore::getRecordFromReference(), DataStore::addRecord()). And doing this would also probably imply upgrade challenges, as all existing blob IDs would have to be recreated to contain the new encoded information.</li>
+ <li><strong>Allow the creation of blobs in one location, and let curation move them later if needed.</strong> This would mean that blobs would usually (always?) be created in a non-filtered delegate first, and then (frequently or rarely, depending on filter configuration) would have to be moved to a more optimal location later, once node information was known. This could be done via a background curation process or on-demand when the blob is first requested.</li>
+</ul>
+ </div>
+
+
+
+ </div> </div>
+ <div id="footer" role="contentinfo">
+ <section class="footer-body">
+ <p>Document generated by Confluence on May 20, 2019 11:11</p>
+ <div id="footer-logo"><a href="http://www.atlassian.com/">Atlassian</a></div>
+ </section>
+ </div>
+ </div> </body>
+</html>
Propchange: jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store-Storage-Filters_115513381.html
------------------------------------------------------------------------------
svn:eol-style = native
Propchange: jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store-Storage-Filters_115513381.html
------------------------------------------------------------------------------
svn:executable = *
Added: jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store_115513379.html
URL: http://svn.apache.org/viewvc/jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store_115513379.html?rev=1859550&view=auto
==============================================================================
--- jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store_115513379.html (added)
+++ jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store_115513379.html Mon May 20 11:23:18 2019
@@ -0,0 +1,229 @@
+<!DOCTYPE html>
+<html>
+ <head>
+ <title>Apache Jackrabbit : Composite Blob Store</title>
+ <link rel="stylesheet" href="styles/site.css" type="text/css" />
+ <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
+ </head>
+
+ <body class="theme-default aui-theme-default">
+ <div id="page">
+ <div id="main" class="aui-page-panel">
+ <div id="main-header">
+ <div id="breadcrumb-section">
+ <ol id="breadcrumbs">
+ <li class="first">
+ <span><a href="index.html">Apache Jackrabbit</a></span>
+ </li>
+ <li>
+ <span><a href="Home_70731.html">Home</a></span>
+ </li>
+ </ol>
+ </div>
+ <h1 id="title-heading" class="pagetitle">
+ <span id="title-text">
+ Apache Jackrabbit : Composite Blob Store
+ </span>
+ </h1>
+ </div>
+
+ <div id="content" class="view">
+ <div class="page-metadata">
+
+
+
+
+
+
+
+ Created by <span class='author'> ASF Infrabot</span> on May 20, 2019
+ </div>
+ <div id="main-content" class="wiki-content group">
+ <h1 id="CompositeBlobStore-CompositeBlobStore">Composite Blob Store</h1>
+
+<p>NOTE: The current status of this component is a <strong>proposed feature</strong>.</p>
+
+<h2 id="CompositeBlobStore-Overview">Overview</h2>
+<p>The composite blob store is a multi-source blob store - a logical blob store consisting of at least two delegate blob stores. The union of all the data in all the delegate blob stores is presented to a user of the composite blob store as a single logical "view" of the data being stored.</p>
+
+<h2 id="CompositeBlobStore-Motivation">Motivation</h2>
+<p>A composite blob store offers flexibility in storage deployments that can address a number of user scenarios. Some of these user scenarios are already identified as existing customer use cases for binary management in Oak; see UC2, UC9, and UC14 in <a href="JCR-Binary-Usecase_115513420.html">JCR Binary Usecase</a>. These and other possible use cases are outlined in the "Use Cases" section below. Some of these may require additional features beyond those in the defined scope for composite blob store today.</p>
+
+<p>Speaking generally, some problems that might be addressable by a composite blob store include:</p>
+<ul>
+ <li>Enabling and/or simplifying development system / production system blob store sharing scenarios.</li>
+ <li>Storing blobs closer to the users that use them most, in geo-distributed environments.</li>
+ <li>Allowing the selection and use of different storage classes for different blobs based on frequency of access (including auto-archival).</li>
+ <li>Offering greater control within Oak to manage geo-redundancy and high availability, including on an object-by-object basis.</li>
+ <li>Choose different types of storage for different types of binary data in Oak based on some configurable criteria.
+<br class="atl-forced-newline"/></li>
+</ul>
+
+
+<h2 id="CompositeBlobStore-TechnicalDetails">Technical Details</h2>
+<p>To use the composite blob store, delegate blob stores are configured as data store factories. The configuration of each data store factory must specify the following:</p>
+<ul>
+ <li>Any standard configuration required to configure this data store</li>
+ <li>A role (any string value) identifying this delegate</li>
+ <li>Any configuration pertaining to this data store's role as a delegate</li>
+</ul>
+
+
+<p>For example, if configuring an S3DataStore as a delegate, a user would:</p>
+<ul>
+ <li>Configure standard S3DataStore values, like the access key, secret key, and bucket name</li>
+ <li>Define a role for this data store, e.g. "role=S3DS_1"</li>
+ <li>Add other configuration, if any, to configure this data store as a delegate (like whether it is a readOnly store)
+<br class="atl-forced-newline"/></li>
+</ul>
+
+
+<p>After configuring the delegates, the composite is configured using the PID "org.apache.jackrabbit.oak.plugins.blob.datastore.CompositeDataStore". A single configuration entry is required, which is a listing of the roles this composite manages. For example, suppose there are two delegate data stores. In the configuration of one delegate, it defines "role=S3DS_1". In the configuration of the second delegate, it defines "role=S3DS_2". To use these two delegates, the composite data store configuration would include the line "roles=S3DS_1,S3DS_2".</p>
+
+<h3 id="CompositeBlobStore-DelegateTraversal">Delegate Traversal</h3>
+<p>"Delegate Traversal" refers to the logic that is used to go through delegates on a read or write request to determine which delegate should be used for a request. The algorithm used to traverse delegates is extensible. The default implementation is called the Intelligent Delegate Traversal Strategy. It is "intelligent" because it attempts to interpret the provided configuration and apply a priority based on the interpretation of the configuration. Another possible default could be e.g. the Simple Delegate Traversal Strategy. This strategy simply attempts to use the delegates in the raw order they are specified in the configuration, with no logic applied. The Intelligent Delegate Traversal Strategy will probably provide the most unsurprising results for end users.</p>
+
+<h4 id="CompositeBlobStore-IntelligentDelegateTraversalStrategy">Intelligent Delegate Traversal Strategy</h4>
+
+<h5 id="CompositeBlobStore-Writes">Writes</h5>
+<p>====== Read-Only Delegates ======<br/>
+A delegate may be specified as a read-only delegate, in which case it will not accept any write requests. If it would otherwise have been chosen for a write request, the request will defer to the next delegate in the traversal that matches the request, if any.</p>
+
+<p>====== Delegate Write Preference ======<br/>
+The write algorithm is fairly simple: Excluding delegates that are read-only, iterate through delegates to select the first that can accept the write, and perform the write. Return the result of this write as the result of the composite blob store write. Priority is given to delegates that already have a matching blob ID.</p>
+
+<h5 id="CompositeBlobStore-Reads">Reads</h5>
+<p>The composite blob store fulfills read requests by deferring the read to delegate blob stores.</p>
+
+<p>====== Delegate Read Preference ======<br/>
+The read algorithm is also fairly simple:</p>
+<ul>
+ <li>Excluding delegates that are read-only, iterate through delegates and select the first that can satisfy the read request.</li>
+ <li>If no delegate can satisfy the read request, iterate through read-only delegates and select the first that can satisfy the read request.</li>
+ <li>Return the result of the delegate read, or an appropriate "not found" error message if no delegate has a match.
+<br class="atl-forced-newline"/></li>
+</ul>
+
+
+<p>The response to a read request is the result of the first successful read from a delegate. In this way the top priority result is always selected.</p>
+
+<p>Read-only delegates take lower precedence to writable delegates, as writable delegates may contain more up-to-date information which would be preferred.</p>
+
+<h3 id="CompositeBlobStore-Read-OnlyDelegates">Read-Only Delegates</h3>
+<p>The composite blob store supports the notion of a read-only delegate blob store. One or more of the delegate blob stores can be configured in read-only mode, meaning that it can be used to satisfy read requests but not write requests. An example use case for this scenario is where two content repositories are used, one for a production environment and one for a staging environment. The staging repository can be configured with a composite blob store that accesses to the production storage location in read-only mode, so tests can execute in staging using production data without modifying production data or the production store.</p>
+
+<p>Reads issued to a read-only delegate would be processed as normal. Read-only delegates are not considered for write requests, causing the composite blob store to move on to the next delegate to attempt to fulfill the write request.</p>
+
+<p>Note that configuring all delegates of a composite blob store as read-only delegates would make the blob store useless for storing blobs and thus should not be an allowed condition - at least one delegate blob store must not be a read-only delegate.</p>
+
+<h3 id="CompositeBlobStore-BlobID/DelegateMapping">Blob ID / Delegate Mapping</h3>
+<p>In order to avoid issuing read requests to delegates that do not contain the blob ID in question, the composite blob store must maintain a mapping of each blob ID to the delegate containing it. Bloom filters should be used for this purpose. This mapping must be created at startup and maintained as the system runs, and should be rebuilt every time data store garbage collection runs (among other things, the filter may need to be resized for the current number of blob IDs present).</p>
+
+<h2 id="CompositeBlobStore-UseCases">Use Cases</h2>
+<p>There are many possible use cases for the composite blob store. In order to manage the implementation of the capability, functionality is being added and supported one use case at a time. When this capability is released, the first supported use case will be the Staging Environment (listed below). Other use cases are listed here for reference but are not currently supported.</p>
+
+<h3 id="CompositeBlobStore-StagingEnvironment">Staging Environment</h3>
+<p>The composite blob store can be used to address a production/staging deployment use case, where one Oak repository is the production repository and another is the staging repository. The production repository accesses a single blob store. The staging repository uses a composite blob store to access a staging blob store as well as the production blob store in read-only mode. Thus staging can serve blobs out of either blob store but can only modify blobs on the staging blob store.</p>
+
+<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
+<pre>+-----------------+ +-----------------+
+| Production Env | | Staging Env |
+| +-------------+ | | +-------------+ |
+| | Oak | | +-----+ Oak | |
+| +------+------+ | | | +------+------+ |
+| | | Read- | | |
+| | | Only | | |
+| +------V------+ | | | +------V------+ |
+| | S3DataStore <------+ | | S3DataStore | |
+| +-------------+ | | +-------------+ |
+| | | |
++-----------------+ +-----------------+
+</pre>
+</div></div>
+
+<h3 id="CompositeBlobStore-HierarchicalBlobStore">Hierarchical Blob Store</h3>
+<p>The composite blob store directly addresses <a href="JCR-Binary-Usecase_115513420.html">JCR Binary Usecase</a> UC14 to store data in one of a number of blob stores based on a hierarchy.</p>
+
+<p>In the example below, blobs are initially stored in the FileDataStore and then once they are more than 30 days old are moved to !S3DataStore. They can be read from either location. Note that moving from one data store to the other fits under the category of curation, which is not in this scope.</p>
+
+<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
+<pre>+-------+
+| | <30 Days Old +---------------+
+| +----------------> FileDataStore |
+| | +---------------+
+| Oak |
+| |
+| | >=30 Days Old +-------------+
+| +-----------------> S3DataStore |
+| | +-------------+
++-------+
+</pre>
+</div></div>
+
+<h3 id="CompositeBlobStore-S3DataStoreClustering">S3DataStore Clustering</h3>
+<p>The composite blob store could be used to address <a href="JCR-Binary-Usecase_115513420.html">JCR Binary Usecase</a> UC9, where two Oak nodes in a cluster may both have a record of a blob in the node store but one node may temporarily not be able to access the blob in the case of async upload. This could be addressed by using a composite blob store where the first level blob store would be FileDataStore on an NFS mount and the second level blob store would be !S3DataStore without a cache. The composite blob store on each node will look for any asset in both the FileDataStore and the !S3DataStore, thus avoiding a split-brain scenario.</p>
+
+<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
+<pre>+-----------------------------+
+| Node 1 |
+| +-----+ |
+| | | |
+| | +-------------------------------+
+| | Oak | | |
+| | | +---------------+ | |
+| | +-->+ FileDataStore | | +------V------+
+| | | +-------^-------+ | | S3DataStore |
+| +-----+ | | +------+------+
++-------------------|---------+ |
+ | +------V------+
+ NFS | S3 Bucket |
+ | +------^------+
++-------------------|---------+ |
+| Node 2 | | +------+------+
+| +-----+ | | | S3DataStore |
+| | | +-------V-------+ | +------^------+
+| | +---> FileDataStore | | |
+| | Oak | +---------------+ | |
+| | | | |
+| | +-------------------------------+
+| | | |
+| +-----+ |
++-----------------------------+
+</pre>
+</div></div>
+
+<h3 id="CompositeBlobStore-ReplicationAcrossStorageRegions">Replication Across Storage Regions</h3>
+<p>The composite blob store can address <a href="JCR-Binary-Usecase_115513420.html">JCR Binary Usecase</a> UC2 by storing blobs close to users.</p>
+
+<p>(NOTE: This use case would rely upon <a href="Composite-Blob-Store-Storage-Filters_115513381.html">Composite Blob Store Storage Filters</a> which are not in scope for this release.)</p>
+
+<p>In this example, imagine a company with a main office in the United States and branch offices in Tokyo and Paris. Here there is a single Oak repository configured using a composite blob store with three delegates. The default delegate blob store is in the AWS "us-west-2" region, in Oregon in the United States, presumably near the main office. Two other delegate blob stores are configured, one in the AWS "ap-northeast-1" region (Tokyo), and one in the AWS "eu-west-2" region (London). When blobs are stored in Oak, if the "officeLocation" property is set on the stored blob, that will be used to determine where to store the blob. Any blobs stored with "officeLocation" set to any value besides "Tokyo" or "Paris", or blobs stored without this property set, will be stored in the default delegate blob store. </p>
+
+<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
+<pre> +-----+
+ | Oak |
+ +--+--+
+ |
+ +----------------+----------------+
+ | | |
+ officeLocation=Tokyo default officeLocation=Paris
+ | | |
+ +------V------+ +------V------+ +------V------+
+ | S3DataStore | | S3DataStore | | S3DataStore |
+ | (Tokyo) | | (Oregon) | | (London) |
+ +-------------+ +-------------+ +-------------+
+</pre>
+</div></div>
+ </div>
+
+
+
+ </div> </div>
+ <div id="footer" role="contentinfo">
+ <section class="footer-body">
+ <p>Document generated by Confluence on May 20, 2019 11:11</p>
+ <div id="footer-logo"><a href="http://www.atlassian.com/">Atlassian</a></div>
+ </section>
+ </div>
+ </div> </body>
+</html>
Propchange: jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store_115513379.html
------------------------------------------------------------------------------
svn:eol-style = native
Propchange: jackrabbit/site/live/archive/wiki/JCR/Composite-Blob-Store_115513379.html
------------------------------------------------------------------------------
svn:executable = *
Added: jackrabbit/site/live/archive/wiki/JCR/ConfigurationOverview_115513382.html
URL: http://svn.apache.org/viewvc/jackrabbit/site/live/archive/wiki/JCR/ConfigurationOverview_115513382.html?rev=1859550&view=auto
==============================================================================
--- jackrabbit/site/live/archive/wiki/JCR/ConfigurationOverview_115513382.html (added)
+++ jackrabbit/site/live/archive/wiki/JCR/ConfigurationOverview_115513382.html Mon May 20 11:23:18 2019
@@ -0,0 +1,68 @@
+<!DOCTYPE html>
+<html>
+ <head>
+ <title>Apache Jackrabbit : ConfigurationOverview</title>
+ <link rel="stylesheet" href="styles/site.css" type="text/css" />
+ <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
+ </head>
+
+ <body class="theme-default aui-theme-default">
+ <div id="page">
+ <div id="main" class="aui-page-panel">
+ <div id="main-header">
+ <div id="breadcrumb-section">
+ <ol id="breadcrumbs">
+ <li class="first">
+ <span><a href="index.html">Apache Jackrabbit</a></span>
+ </li>
+ <li>
+ <span><a href="Home_70731.html">Home</a></span>
+ </li>
+ </ol>
+ </div>
+ <h1 id="title-heading" class="pagetitle">
+ <span id="title-text">
+ Apache Jackrabbit : ConfigurationOverview
+ </span>
+ </h1>
+ </div>
+
+ <div id="content" class="view">
+ <div class="page-metadata">
+
+
+
+
+
+
+
+ Created by <span class='author'> ASF Infrabot</span> on May 20, 2019
+ </div>
+ <div id="main-content" class="wiki-content group">
+ <p>Jackrabbit's main configuration lies in the so-called <code>repository.xml</code> file. It contains global configuration such as Login and Access Management, Versioning or <a href="Clustering_115513377.html">Clustering</a>. Then it defines how the actual data for a certain workspace should be stored by choosing a <a class="createlink" href="/confluence/pages/createpage.action?spaceKey=JCR&title=PersistenceManager&linkCreation=true&fromPageId=115513382">PersistenceManager</a> and what search/query implementation to use by defining a <a href="SearchIndex_115513505.html">SearchIndex</a>.</p>
+
+<p>For each workspace that was created, there will also be a <code>workspace.xml</code> file created inside the workspace home directory that will be used for the workspace - these files have to be changed, too, because the workspace-specific configuration inside <code>repository.xml</code> is only used as a template for new workspaces, ie. if you use the createWorkspace of the Jackrabbit API, the workspace.xml is just a copy of the Workspace element inside repository.xml. You can also manually create the workspace folder with a workspace.xml file to create a new workspace yourself (Note that depending on the persistence manager you will also have to setup a database and configure the access to it).</p>
+
+<p>The default configuration of Jackrabbit can be used in many cases. To find out more about configuration options, see:</p>
+
+<ul>
+ <li><a href="http://jackrabbit.apache.org/jackrabbit-configuration.html" class="external-link" rel="nofollow">Jackrabbit Configuration</a>: Configuration files and variables</li>
+ <li><a href="PersistenceManagerFAQ_115513487.html">PersistenceManagerFAQ</a>: Persistence manager and file system configuration</li>
+ <li><a href="DataStore_115513387.html">DataStore</a>: How to configure and use the binary data store</li>
+ <li><a href="Clustering_115513377.html">Clustering</a>: How to set up and use clustering</li>
+ <li><a href="Search_115513504.html">Search</a>: How to set up and use search</li>
+ <li><a href="RepositoryLock_115513495.html">RepositoryLock</a>: How to configure repository locking</li>
+</ul>
+ </div>
+
+
+
+ </div> </div>
+ <div id="footer" role="contentinfo">
+ <section class="footer-body">
+ <p>Document generated by Confluence on May 20, 2019 11:11</p>
+ <div id="footer-logo"><a href="http://www.atlassian.com/">Atlassian</a></div>
+ </section>
+ </div>
+ </div> </body>
+</html>
Propchange: jackrabbit/site/live/archive/wiki/JCR/ConfigurationOverview_115513382.html
------------------------------------------------------------------------------
svn:eol-style = native
Propchange: jackrabbit/site/live/archive/wiki/JCR/ConfigurationOverview_115513382.html
------------------------------------------------------------------------------
svn:executable = *
Added: jackrabbit/site/live/archive/wiki/JCR/Conflict-handling-through-rebasing-branches_115513383.html
URL: http://svn.apache.org/viewvc/jackrabbit/site/live/archive/wiki/JCR/Conflict-handling-through-rebasing-branches_115513383.html?rev=1859550&view=auto
==============================================================================
--- jackrabbit/site/live/archive/wiki/JCR/Conflict-handling-through-rebasing-branches_115513383.html (added)
+++ jackrabbit/site/live/archive/wiki/JCR/Conflict-handling-through-rebasing-branches_115513383.html Mon May 20 11:23:18 2019
@@ -0,0 +1,128 @@
+<!DOCTYPE html>
+<html>
+ <head>
+ <title>Apache Jackrabbit : Conflict handling through rebasing branches</title>
+ <link rel="stylesheet" href="styles/site.css" type="text/css" />
+ <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
+ </head>
+
+ <body class="theme-default aui-theme-default">
+ <div id="page">
+ <div id="main" class="aui-page-panel">
+ <div id="main-header">
+ <div id="breadcrumb-section">
+ <ol id="breadcrumbs">
+ <li class="first">
+ <span><a href="index.html">Apache Jackrabbit</a></span>
+ </li>
+ <li>
+ <span><a href="Home_70731.html">Home</a></span>
+ </li>
+ </ol>
+ </div>
+ <h1 id="title-heading" class="pagetitle">
+ <span id="title-text">
+ Apache Jackrabbit : Conflict handling through rebasing branches
+ </span>
+ </h1>
+ </div>
+
+ <div id="content" class="view">
+ <div class="page-metadata">
+
+
+
+
+
+
+
+ Created by <span class='author'> ASF Infrabot</span> on May 20, 2019
+ </div>
+ <div id="main-content" class="wiki-content group">
+ <p><em>This is a slightly edited and somewhat amended version of my <a href="http://markmail.org/message/wtaarmdtgyf5lvjt" class="external-link" rel="nofollow">original proposal</a> posted to oak-dev@.</em></p>
+
+<p>There are various places in Oak where conflicting updates to the content tree can occur: committing changes against a non root revision, merging changes from a branch into trunk and synchronisation of cluster nodes.</p>
+
+<p>The Microkernel API currently doesn't specify a contract for merging conflicting changes but leaves it mostly to the underlying implementation. As <a href="http://markmail.org/message/4xwfwbax3kpoysbp" class="external-link" rel="nofollow">discussed before</a> this is not satisfactory and I'd like to tighten the contract.</p>
+
+<p>As announced earlier, I spent some time implementing various approaches for rebasing branches in Oak. Apart from the other problems this solves and which have been discussed at length, this also lends itself nicely for a central, clean and efficient way of handling conflicts.</p>
+
+<p>The core idea is to try to resolve all conflicts in a branch through rebasing it on top of the current trunk. After successfully rebasing a branch, merging it to trunk is as simple as fast forwarding the head of the trunk to the head of the branch.</p>
+
+<p>Here is a naïve sample implementation of the relevant methods in pseudo code. Note how commit is implemented in terms of branch and merge. It has not to be implemented that way but rather the observable behaviour should be like this.</p>
+
+<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
+<pre>/* Rebase branch on top of the head of the trunk. Returns false if a conflict occurs, true otherwise. */
+boolean rebase(branch) {
+ // See https://github.com/mduerig/jackrabbit-oak/commits/OAK-536
+ // for possible implementations
+}
+
+int merge(branch) {
+ atomic {
+ if (branch.baseRevision == trunk.headRevision) {
+ trunk.headRevision = branch.headRevision
+ return trunk.headRevision;
+ }
+ else {
+ throw new MicroKernelException("conflict");
+ }
+ }
+}
+
+int commit(baseRevision, jsop) {
+ branch = createBranchFrom(baseRevision)
+ branch.apply(jsop)
+ if (!rebase(branch)) {
+ throw new MicroKernelException("conflict");
+ }
+
+ // If merge fails at this point we could also retry the commit
+ // a certain number of times instead of failing right away.
+ return merge(branch)
+}
+</pre>
+</div></div>
+
+<p>Note the very simple <code>merge()</code> implementation: it only covers the fast forward case where the base of the branch to be merged is the head of the trunk and fails all other merges. <code>merge()</code> has a very narrow atomic section, which is basically a conditional update. Such operations are exposed by most operating systems / storage systems / platforms and should thus be easy to implement. </p>
+
+<p>The overall idea of implementing merge in this way is that callers need to take care of properly rebasing branches and resolving conflicts through the provided <code>rebase()</code> method. The <code>rebase()</code> method itself will detect a specific set of conflicts and annotates them with special conflict markers. This enables conflict resolution to be handled higher up the stack where we have a better understanding of the actual semantics of the conflict. Also note how all conflict handling takes place on private branches thus keeping the trunk free of contentions.</p>
+
+<p>In the <a href="https://github.com/mduerig/jackrabbit-oak/blob/OAK-536/oak-mk-api/src/main/java/org/apache/jackrabbit/mk/api/MicroKernel.java" class="external-link" rel="nofollow">current implementation</a> if <code>rebase()</code> results in a conflict, conflicting nodes are annotated with a conflict marker denoting the type of the conflict and the value(s) before the rebase operation. The conflict marker is an internal node with the name <code>:conflict</code> and is added to the node whose properties or child nodes are in conflict.</p>
+
+<p> <strong>addExistingProperty</strong></p>
+
+ <div class="indent20 inline" style="margin-left: 20px;"><style type="text/css">div.inline p:first-child { display: inline; }</style><p> A property has been added that has a different value than a property with the same name that has been added in trunk.</p></div>
+<p> <strong>removeRemovedProperty</strong></p>
+ <div class="indent20 inline" style="margin-left: 20px;"><style type="text/css">div.inline p:first-child { display: inline; }</style><p> A property has been removed while a property of the same name has been removed in trunk.</p></div>
+<p> <strong>removeChangedProperty</strong></p>
+ <div class="indent20 inline" style="margin-left: 20px;"><style type="text/css">div.inline p:first-child { display: inline; }</style><p> A property has been removed while a property of the same name has been changed in trunk.</p></div>
+<p> <strong>changeRemovedProperty</strong></p>
+ <div class="indent20 inline" style="margin-left: 20px;"><style type="text/css">div.inline p:first-child { display: inline; }</style><p> A property has been changed while a property of the same name has been removed in trunk. </p></div>
+<p> <strong>changeChangedProperty</strong></p>
+ <div class="indent20 inline" style="margin-left: 20px;"><style type="text/css">div.inline p:first-child { display: inline; }</style><p> A property has been changed while a property of the same name has been changed to a different value in trunk.</p></div>
+<p> <strong>addExistingNode</strong></p>
+ <div class="indent20 inline" style="margin-left: 20px;"><style type="text/css">div.inline p:first-child { display: inline; }</style><p> A node has been added that is different from a node of them same name that has been added to the trunk.</p></div>
+<p> <strong>removeRemovedNode</strong></p>
+ <div class="indent20 inline" style="margin-left: 20px;"><style type="text/css">div.inline p:first-child { display: inline; }</style><p> A node has been removed while a node of the same name has been removed in trunk.</p></div>
+<p> <strong>removeChangedNode</strong></p>
+ <div class="indent20 inline" style="margin-left: 20px;"><style type="text/css">div.inline p:first-child { display: inline; }</style><p> A node has been removed while a node of the same name has been changed in trunk.</p></div>
+<p> <strong>changeRemovedNode</strong></p>
+ <div class="indent20 inline" style="margin-left: 20px;"><style type="text/css">div.inline p:first-child { display: inline; }</style><p> A node has been changed while a node of the same name has been removed in trunk.</p></div>
+
+<p>In this context a node is regarded as changed if a property way added, a property was removed, a property was set to a different value, a child node was added, a child node was removed or a child node was changed.</p>
+
+<p>On conflict the conflict marker node carries the conflicting value of the branch while the rebased value in the branch itself will be set to the conflicting value of the trunk. In the case of conflicting properties, the conflicting value is the property value from the branch. In the case of conflicting node, the conflicting value is the node from the branch.</p>
+ </div>
+
+
+
+ </div> </div>
+ <div id="footer" role="contentinfo">
+ <section class="footer-body">
+ <p>Document generated by Confluence on May 20, 2019 11:11</p>
+ <div id="footer-logo"><a href="http://www.atlassian.com/">Atlassian</a></div>
+ </section>
+ </div>
+ </div> </body>
+</html>
Propchange: jackrabbit/site/live/archive/wiki/JCR/Conflict-handling-through-rebasing-branches_115513383.html
------------------------------------------------------------------------------
svn:eol-style = native
Propchange: jackrabbit/site/live/archive/wiki/JCR/Conflict-handling-through-rebasing-branches_115513383.html
------------------------------------------------------------------------------
svn:executable = *