You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@marmotta.apache.org by ss...@apache.org on 2013/11/08 21:28:00 UTC
svn commit: r1540172 - in /incubator/marmotta/site/trunk/content:
markdown/platform/cloud.md.vm resources/images/marmotta-cloud.dia
resources/images/marmotta-cloud.png site.xml
Author: sschaffert
Date: Fri Nov 8 20:28:00 2013
New Revision: 1540172
URL: http://svn.apache.org/r1540172
Log:
add documentation for Marmotta Cloud
Added:
incubator/marmotta/site/trunk/content/markdown/platform/cloud.md.vm
- copied, changed from r1540050, incubator/marmotta/site/trunk/content/markdown/platform/backends.md.vm
incubator/marmotta/site/trunk/content/resources/images/marmotta-cloud.dia (with props)
incubator/marmotta/site/trunk/content/resources/images/marmotta-cloud.png (with props)
Modified:
incubator/marmotta/site/trunk/content/site.xml
Copied: incubator/marmotta/site/trunk/content/markdown/platform/cloud.md.vm (from r1540050, incubator/marmotta/site/trunk/content/markdown/platform/backends.md.vm)
URL: http://svn.apache.org/viewvc/incubator/marmotta/site/trunk/content/markdown/platform/cloud.md.vm?p2=incubator/marmotta/site/trunk/content/markdown/platform/cloud.md.vm&p1=incubator/marmotta/site/trunk/content/markdown/platform/backends.md.vm&r1=1540050&r2=1540172&rev=1540172&view=diff
==============================================================================
--- incubator/marmotta/site/trunk/content/markdown/platform/backends.md.vm (original)
+++ incubator/marmotta/site/trunk/content/markdown/platform/cloud.md.vm Fri Nov 8 20:28:00 2013
@@ -1,145 +1,100 @@
-# Apache Marmotta Platform: Backends
+# Apache Marmotta Platform: Cloud Setup
Introduction
------------
-Starting with version 3.2, Apache Marmotta supports exchangeable triple store backends. This allows users of existing
-triple stores to simply connect to these and offer the data as Linked Data and it allows the use of special purpose
-triple stores like [BigData](http://www.bigdata.com). Currently, the following backends are available for Marmotta:
-
-* KiWi Triple Store: this is still the default backend for Apache Marmotta, as it is the most flexible and with most
- features; the KiWi triple store stores all data in a relational database
-* Sesame Native: allows to use a Sesame NativeStore as backend for Apache Marmotta; the Sesame NativeStore is a
- lightweight triple store that stores all data in a directory in the file system
-* BigData (experimental): allows to use a BigData triple store as backend; BigData is a high-performance triple store
- implementation for huge datasets; note that BigData is published under GPL license and thus not compatible with the
- Apache Software License; the BigData backend is therefore not built by default and needs to be compiled manually from
- the Marmotta source code;
-
-Note that not all triple stores support all Marmotta functionalities. At the moment, reasoning and versioning are only
-supported by the KiWi backend.
-
-Configuration
--------------
-
-Changing backends is currently only possible when assembling the web application .war file, because only one backend
-can be on the classpath for any installation. The easiest way to change the Marmotta backend is to copy over the
-configuration from the default webapp launcher in launchers/webapp and change the Maven build file (pom.xml) by adding
-dependencies to the appropriate backend.
-
-<h3>KiWi Backend</h3>
-
-For using the KiWi Backend, you should add the following dependencies to your pom.xml file:
-
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-backend-kiwi</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-ldcache-common</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-ldcache-kiwi</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-reasoner-kiwi</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-sparql</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-security</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-versioning-common</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-versioning-kiwi</artifactId>
- <version>${project.version}</version>
- </dependency>
-
-
-The administration interface will then also include an appropriate configuration screen for the KiWi backend (e.g. for
-configuring a different database backend than the default H2).
-
-<h3>Sesame Native Backend</h3>
-
-For using the Sesame Native backend, add the following dependencies to your pom.xml file (or replace the KiWi dependencies
-described above):
-
-
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-backend-native</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-ldcache-common</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-ldcache-file</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-sparql</artifactId>
- <version>${project.version}</version>
- </dependency>
-
-Note that the Sesame Native backend currently does not support versioning and reasoning, therefore these modules
-are not configured here. The administration interface will then again include an appropriate configuration screen
-for configuring the backend.
-
-<h3>BigData Backend</h3>
-
-The BigData library is unfortunately only available with GPL license, which is not fully compatible with the more liberal
-Apache License used by Apache Marmotta. For this reason, we do not distribute binary versions of the BigData backend.
-Instead, you need to manually check out the source code of Marmotta from the GIT repository and build the whole source
-tree with the following command:
-
- mvn clean install -Pnon-free
-
-After this, for using the Sesame Native backend, add the following dependencies to your pom.xml file (or replace the
-KiWi dependencies described above):
-
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-backend-bigdata</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-ldcache-common</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-ldcache-file</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.marmotta</groupId>
- <artifactId>marmotta-sparql</artifactId>
- <version>${project.version}</version>
- </dependency>
-
-Note that the BigData library still uses the Sesame 2.6 API while Marmotta uses Sesame 2.7. Also, BigData bypasses the
-normal SPARQL query evaluation of Sesame, so we cannot easily integrate our transaction functionalities. For this reason,
-the BigData integration is still experimental and we do not recommend it for production use yet.
\ No newline at end of file
+Starting with version 3.2, Apache Marmotta can be configured to run in a clustered our cloud environment. Instances
+in the cluster are managed using Apache Zookeeper, which needs to be installed separately. Also, clustered setup
+only works when using the KiWi triple store backend, and using a common database (or database cluster) for all servers
+in a cluster.
+
+The following diagram gives an overview over a typical cluster setup of Apache Marmotta:
+
+<img src="../images/marmotta-cloud.png"/>
+
+
+Zookeeper Setup
+---------------
+
+Setting up Zookeeper for Apache Marmotta is straightforward, as the Zookeeper module does most of the initialisation
+for you. To connect to a running Zookeeper server, the following configuration options need to be passed over to
+Marmotta, either as system properties or as servlet context parameters:
+
+* `zookeeper.server` (required): comma-separated list of Zookeeper server names with optional port numbers for all
+ Zookeeper instances in your setup
+* `zookeeper.timeout` (optional, default 60000): timeout for connecting to the Zookeeper servers; if the server does not respond
+ within this time in milliseconds, the connection is ignored
+* `zookeeper.cluster` (optional, default "default"): name of the cluster to which this instance belongs; if not given,
+ the cluster name will be "default"
+* `zookeeper.instance` (optional, default random UUID): name of the instance inside the cluster; if not given, will
+ be created as a random UUID
+
+The main feature of the Zookeeper module is that Marmotta instances can be automatically configured through Zookeeper.
+Marmotta instances will react to configuration changes in Zookeeper and update their local configuration accordingly.
+The Apache Marmotta configuration stored in Zookeeper follows the following structure:
+
+ /marmotta
+ |
+ /config - global configuration options
+ |
+ /<config_key> <config_value>
+ /...
+ /clusters
+ |
+ /default
+ |
+ /config - cluster-level configuration options
+ |
+ /<config_key> <config_value>
+ /...
+ /snowflake - used for generating unique IDs
+ /instances
+ |
+ /<instance_name>
+ |
+ /config - instance-level configuration options
+ |
+ /<config_key> <config_value>
+ /...
+ /...
+ /...
+
+Configuration values can be stored on either the global level, in which case the configuration applies to all Marmotta
+instances, the cluster level, in which case the configuration applies to only those Marmotta instances in a single
+cluster, or the instance level, in which case the configuration only applies to a specific Marmotta instance. More specific
+configurations take precedence over more generic configurations.
+
+
+Database Setup
+--------------
+
+All servers in a cluster need to be configured to access the same database. Therefore, the following configuration
+properties should be defined on the cluster level in Zookeeper:
+
+* `database.type`: type of the database to use; PostgreSQL or MySQL are preferrable
+* `database.url`: JDBC URL for accessing the database
+* `database.user`: user name for accessing the database
+* `database.password`: password for accessing the database
+
+When initialising the cluster for the first time, it is advisable to first start up only a single Marmotta instance to
+let it allow setting up the necessary database tables. When the database initialisation is complete, all other instances
+cam be started up in any order. When starting up a new instance, the Zookeeper module will automatically create a proper
+datacenter ID for generating database IDs that are unique over the cluster.
+
+When running in a high-load environment, it is also useful to startup the database in a database cluster. This is e.g.
+supported by PostgreSQL.
+
+
+Additional Configuration
+------------------------
+
+For properly running Apache Marmotta in a cluster, the following additional configurations need to be considered:
+
+* `kiwi.context` should be set on the cluster level so that all URIs are constructed using the same URI prefix;
+ normally, this option is automatically set to the host name of the server, but for a cluster it should be set to the
+ host name of the load balancer
+* `kiwi.host` should also be set on the cluster level so that webpage links in the admin interface are properly
+ configured; note however that the Marmotta admin user interface should not be used for changing the configuration
+ - this should be done through Zookeeper
+
+
Added: incubator/marmotta/site/trunk/content/resources/images/marmotta-cloud.dia
URL: http://svn.apache.org/viewvc/incubator/marmotta/site/trunk/content/resources/images/marmotta-cloud.dia?rev=1540172&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/marmotta/site/trunk/content/resources/images/marmotta-cloud.dia
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/marmotta/site/trunk/content/resources/images/marmotta-cloud.png
URL: http://svn.apache.org/viewvc/incubator/marmotta/site/trunk/content/resources/images/marmotta-cloud.png?rev=1540172&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/marmotta/site/trunk/content/resources/images/marmotta-cloud.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Modified: incubator/marmotta/site/trunk/content/site.xml
URL: http://svn.apache.org/viewvc/incubator/marmotta/site/trunk/content/site.xml?rev=1540172&r1=1540171&r2=1540172&view=diff
==============================================================================
--- incubator/marmotta/site/trunk/content/site.xml (original)
+++ incubator/marmotta/site/trunk/content/site.xml Fri Nov 8 20:28:00 2013
@@ -86,6 +86,7 @@
<item name="SPARQL Module" href="platform/sparql-module.html" />
<item name="User Module" href="platform/user-module.html" />
<item name="Versioning Module" href="platform/versioning-module.html" />
+ <item name="Cloud Setup" href="platform/cloud.html" />
<item name="Client Library" href="client-library.html" />
</menu>
<menu name="KiWi Triple Store">