You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-commits@jackrabbit.apache.org by am...@apache.org on 2017/05/03 05:20:46 UTC
svn commit: r1793609 - in /jackrabbit/oak/branches/1.6: ./
oak-blob-cloud-azure/ oak-blob-cloud-azure/pom.xml
oak-doc/src/site/markdown/osgi_config.md
oak-doc/src/site/markdown/plugins/blobstore.md oak-parent/pom.xml pom.xml
Author: amitj
Date: Wed May 3 05:20:45 2017
New Revision: 1793609
URL: http://svn.apache.org/viewvc?rev=1793609&view=rev
Log:
OAK-4933: Create a data store implementation that integrates with Microsoft Azure Blob Storage
Merged r1788387 from trunk
Added:
jackrabbit/oak/branches/1.6/oak-blob-cloud-azure/
- copied from r1788387, jackrabbit/oak/trunk/oak-blob-cloud-azure/
Modified:
jackrabbit/oak/branches/1.6/ (props changed)
jackrabbit/oak/branches/1.6/oak-blob-cloud-azure/pom.xml
jackrabbit/oak/branches/1.6/oak-doc/src/site/markdown/osgi_config.md
jackrabbit/oak/branches/1.6/oak-doc/src/site/markdown/plugins/blobstore.md
jackrabbit/oak/branches/1.6/oak-parent/pom.xml
jackrabbit/oak/branches/1.6/pom.xml
Propchange: jackrabbit/oak/branches/1.6/
------------------------------------------------------------------------------
--- svn:mergeinfo (original)
+++ svn:mergeinfo Wed May 3 05:20:45 2017
@@ -1,3 +1,3 @@
/jackrabbit/oak/branches/1.0:1665962
-/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784130,1784162,1784251,1784401,1784551,1784574,1785095,1785108,1785283,1785838,1785919,1785946,1787074,1787217,1788378,1790382,1792463,1792742,1793088
+/jackrabbit/oak/trunk:1781068,1781075,1781248,1781386,1781846,1781907,1782000,1782029,1782196,1782447,1782476,1782770,1782945,1782966,1782973,1782990,1783061,1783066,1783089,1783104-1783105,1783619,1783720,1783731,1783733,1783738,1783742,1783773,1783855,1783891,1784023,1784130,1784162,1784251,1784401,1784551,1784574,1785095,1785108,1785283,1785838,1785919,1785946,1787074,1787217,1788378,1788387,1790382,1792463,1792742,1793088
/jackrabbit/trunk:1345480
Modified: jackrabbit/oak/branches/1.6/oak-blob-cloud-azure/pom.xml
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.6/oak-blob-cloud-azure/pom.xml?rev=1793609&r1=1788387&r2=1793609&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.6/oak-blob-cloud-azure/pom.xml (original)
+++ jackrabbit/oak/branches/1.6/oak-blob-cloud-azure/pom.xml Wed May 3 05:20:45 2017
@@ -21,7 +21,7 @@
<parent>
<artifactId>oak-parent</artifactId>
<groupId>org.apache.jackrabbit</groupId>
- <version>1.8-SNAPSHOT</version>
+ <version>1.6.2-SNAPSHOT</version>
<relativePath>../oak-parent/pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Modified: jackrabbit/oak/branches/1.6/oak-doc/src/site/markdown/osgi_config.md
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.6/oak-doc/src/site/markdown/osgi_config.md?rev=1793609&r1=1793608&r2=1793609&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.6/oak-doc/src/site/markdown/osgi_config.md (original)
+++ jackrabbit/oak/branches/1.6/oak-doc/src/site/markdown/osgi_config.md Wed May 3 05:20:45 2017
@@ -433,6 +433,7 @@ All the above data stores enable local f
* _PID `org.apache.jackrabbit.oak.plugins.blob.datastore.SharedS3DataStore`_
* _PID `org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore`_
* _PID `org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore`_
+* _PID `org.apache.jackrabbit.oak.plugins.blob.datastore.AzureDataStore`_
cacheSize
: Default - 68719476736
Modified: jackrabbit/oak/branches/1.6/oak-doc/src/site/markdown/plugins/blobstore.md
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.6/oak-doc/src/site/markdown/plugins/blobstore.md?rev=1793609&r1=1793608&r2=1793609&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.6/oak-doc/src/site/markdown/plugins/blobstore.md (original)
+++ jackrabbit/oak/branches/1.6/oak-doc/src/site/markdown/plugins/blobstore.md Wed May 3 05:20:45 2017
@@ -78,6 +78,7 @@ Further Oak ships with multiple BlobStor
4. `S3DataStore` (with wrapper) - Stores the file in Amazon S3
5. `RDBBlobStore` - Store the file contents in chunks in a relational databases. Typically used with
`DocumentNodeStore`when using a relational DB persistence
+6. `AzureDataStore` (with wrapper) - Stores the file in Microsoft Azure Blob storage
In addition there are some more implementations which are considered **experimental**
@@ -98,6 +99,7 @@ can be used
* FileDataStore - This should be used if the blobs/binaries have to be shared between multiple
repositories. This would also be used when a JR2 repository is migrated to Oak
* S3DataStore - This should be used when binaries are stored in Amazon S3
+* AzureDataStore - This should be used when binaries are stored in Microsoft Azure Blob storage
#### DocumentNodeStore
@@ -112,7 +114,7 @@ one of the following can be used
#### Caching DataStore
-The DataStore implementations `S3DataStore` and `CachingFileDataStore` support local file system caching for the
+The DataStore implementations `S3DataStore`,`CachingFileDataStore` and `AzureDataStore` support local file system caching for the
files/blobs and extend the `AbstractSharedCachingDataStore` class which implements the caching functionality. The
`CachingFileDataStore` is useful when the DataStore is on nfs.
The cache has a size limit and is configured by the `cacheSize` parameter.
@@ -198,6 +200,7 @@ Blob Garbage Collection(GC) is applicabl
* FileDataStore
* S3DataStore
* SharedS3DataStore (since Oak 1.2.0)
+ * AzureDataStore
Oak implements a Mark and Sweep based Garbage Collection logic.
@@ -224,7 +227,7 @@ The garbage collection can be triggered
<a name="blobid-tracker"></a>
#### Caching of Blob ids locally (Oak 1.6.x)
-For the `FileDataStore` and `S3DataStore` the blob ids are cached locally on the disk when they are created which
+For the `FileDataStore`, `S3DataStore` and `AzureDataStore` the blob ids are cached locally on the disk when they are created which
speeds up the 'Mark BlobStore' phase. The locally tracked ids are synchronized with the data store periodically to enable
other cluster nodes or different repositories sharing the datastore to get a consolidated list of all blob ids. The
interval of synchronization is defined by the OSGi configuration parameter `blobTrackSnapshotIntervalInSecs` for the
@@ -249,13 +252,14 @@ following should be executed.
##### Registration
-On start of a repository configured to use a shared DataStore (same path or S3 bucket), a unique repository id is
+On start of a repository configured to use a shared DataStore (same path, S3 bucket or Azure container), a unique repository id is
generated and registered in the NodeStore as well as the DataStore.
In the DataStore this repository id is registered as an empty file with the format `repository-[repository-id]`
(e.g. repository-988373a0-3efb-451e-ab4c-f7e794189273). This empty file is created under:
* FileDataStore - Under the root directory configured for the datastore.
* S3DataStore - Under `META` folder in the S3 bucket configured.
+* AzureDataStore - Under `META` folder in the Azure container configured.
On start/configuration of all the repositories sharing the data store it should be confirmed that the unique
repositoryId per repository is registered in the DataStore. Refer the section below on [Checking Shared GC status](#check-shared-datastore-gc).
@@ -387,6 +391,7 @@ the steps:
* Remove the corresponding registered repository file (`repository-[repositoryId]`) from the DataStore
* FileDataStore - Remove the file from the data store root directory.
* S3DataStore - Remove the file from the `META` folder of the S3 bucket.
+ * AzureDataStore - Remove the file from the `META` folder of the Azure container.
* Remove other files corresponding to the particular repositoryId e.g. `markedTimestamp-[repositoryId]` or
`references-[repositoryId]`.
Modified: jackrabbit/oak/branches/1.6/oak-parent/pom.xml
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.6/oak-parent/pom.xml?rev=1793609&r1=1793608&r2=1793609&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.6/oak-parent/pom.xml (original)
+++ jackrabbit/oak/branches/1.6/oak-parent/pom.xml Wed May 3 05:20:45 2017
@@ -177,6 +177,7 @@
<filter>!org.apache.jackrabbit.oak.spi.blob.split</filter>
<filter>!org.apache.jackrabbit.oak.spi.blob.stats</filter>
<filter>!org.apache.jackrabbit.oak.blob.cloud.aws.s3</filter>
+ <filter>!org.apache.jackrabbit.oak.blob.cloud.azure.blobstorage</filter>
<filter>!org.apache.jackrabbit.oak</filter>
<filter>!org.apache.jackrabbit.oak.json</filter>
<filter>!org.apache.jackrabbit.oak.management</filter>
Modified: jackrabbit/oak/branches/1.6/pom.xml
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.6/pom.xml?rev=1793609&r1=1793608&r2=1793609&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.6/pom.xml (original)
+++ jackrabbit/oak/branches/1.6/pom.xml Wed May 3 05:20:45 2017
@@ -40,6 +40,7 @@
<module>oak-commons</module>
<module>oak-blob</module>
<module>oak-blob-cloud</module>
+ <module>oak-blob-cloud-azure</module>
<module>oak-core</module>
<module>oak-jcr</module>
<module>oak-upgrade</module>