You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-commits@jackrabbit.apache.org by re...@apache.org on 2019/02/28 13:27:37 UTC
svn commit: r1854524 - in /jackrabbit/oak/branches/1.10: ./
oak-doc/src/site/markdown/nodestore/document/rdb-document-store.md
oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/rdb/RDBDocumentStore.java
Author: reschke
Date: Thu Feb 28 13:27:37 2019
New Revision: 1854524
URL: http://svn.apache.org/viewvc?rev=1854524&view=rev
Log:
OAK-8080: RDB*Store: move DB-specific config hints from Javadoc into oak-doc (ported to 1.10)
Modified:
jackrabbit/oak/branches/1.10/ (props changed)
jackrabbit/oak/branches/1.10/oak-doc/src/site/markdown/nodestore/document/rdb-document-store.md
jackrabbit/oak/branches/1.10/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/rdb/RDBDocumentStore.java
Propchange: jackrabbit/oak/branches/1.10/
------------------------------------------------------------------------------
--- svn:mergeinfo (original)
+++ svn:mergeinfo Thu Feb 28 13:27:37 2019
@@ -1,3 +1,3 @@
/jackrabbit/oak/branches/1.0:1665962
-/jackrabbit/oak/trunk:1850874,1850882,1851236,1851253,1851451,1852052,1852084,1852120,1852451,1852492-1852493,1852528,1852582,1852584,1852601,1852920,1853141,1853229,1853393,1853429,1853433,1853441,1853866,1853868,1853870,1853893,1853969,1853997,1854034,1854044,1854058,1854113,1854462,1854466,1854468
+/jackrabbit/oak/trunk:1850874,1850882,1851236,1851253,1851451,1852052,1852084,1852120,1852451,1852492-1852493,1852528,1852582,1852584,1852601,1852920,1853141,1853229,1853393,1853429,1853433,1853441,1853866,1853868,1853870,1853893,1853969,1853997,1854034,1854044,1854058,1854113,1854373,1854377,1854380,1854385,1854401,1854403,1854462,1854466,1854468
/jackrabbit/trunk:1345480
Modified: jackrabbit/oak/branches/1.10/oak-doc/src/site/markdown/nodestore/document/rdb-document-store.md
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.10/oak-doc/src/site/markdown/nodestore/document/rdb-document-store.md?rev=1854524&r1=1854523&r2=1854524&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.10/oak-doc/src/site/markdown/nodestore/document/rdb-document-store.md (original)
+++ jackrabbit/oak/branches/1.10/oak-doc/src/site/markdown/nodestore/document/rdb-document-store.md Thu Feb 28 13:27:37 2019
@@ -57,7 +57,7 @@ will attempt to start anyway):
~~~
-## Database Creation
+## <a name="database-creation"></a> Database Creation
`RDBDocumentStore` relies on JDBC, and thus, in general, can not create
database instances (that said, certain DBs such as Apache Derby or H2DB can create the
@@ -70,8 +70,62 @@ the correct operation:
- character fields must be able to store any Unicode code point - UTF-8 encoding is recommended
- the collation for character fields needs to sort by Unicode code points
+- BLOBs need to support sizes of ~16MB
-## Table Creation
+The subsections below give examples that have been found to work during the
+development of `RDBDocumentStore`.
+
+
+### <a name="database-creation-db2"></a> DB2
+
+Creating a database called `OAK`:
+~~~
+create database oak USING CODESET UTF-8 TERRITORY DEFAULT COLLATE USING IDENTITY;
+~~~
+
+To verify, check that INFO level log message written by `RDBDocumentStore`
+upon startup. For example:
+
+~~~
+14:47:20.332 INFO [main] RDBDocumentStore.java:1065 RDBDocumentStore (SNAPSHOT) instantiated for database DB2/NT64 SQL11014 (11.1), using driver: IBM Data Server Driver for JDBC and SQLJ 4.19.77 (4.19), connecting to: jdbc:db2://localhost:50276/OAK, properties: {DB2ADMIN.CODEPAGE=1208, DB2ADMIN.COLLATIONSCHEMA=SYSIBM, DB2ADMIN.COLLATIONNAME=IDENTITY}, transaction isolation level: TRANSACTION_READ_COMMITTED (2), DB2ADMIN.NODES: ID VARCHAR(512), MODIFIED BIGINT, HASBINARY SMALLINT, DELETEDONCE SMALLINT, MODCOUNT BIGINT, CMODCOUNT BIGINT, DSIZE BIGINT, VERSION SMALLINT, SDTYPE SMALLINT, SDMAXREVTIME BIGINT, DATA VARCHAR(16384), BDATA BLOB(1073741824) /* {BIGINT=-5, BLOB=2004, SMALLINT=5, VARCHAR=12} */ /* index DB2ADMIN.NODES_MOD on DB2ADMIN.NODES (MODIFIED ASC) other (#0, p0), unique index DB2ADMIN.NODES_PK on DB2ADMIN.NODES (ID ASC) clustered (#0, p0), index DB2ADMIN.NODES_SDM on DB2ADMIN.NODES (SDMAXREVTIME ASC) other (#0, p0), index DB2ADMIN.NODES_SDT on DB2ADMIN.NODES (SDT
YPE ASC) other (#0, p0), index DB2ADMIN.NODES_VSN on DB2ADMIN.NODES (VERSION ASC) other (#0, p0) */
+~~~
+
+
+### <a name="database-creation-mysql"></a> MySQL
+
+Creating a database called `OAK`:
+~~~
+create database oak DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+~~~
+
+Also make sure to configure the `max_allowed_packet` parameter
+for the server (mysqld) to a value greater than 4M (such as 8388608).
+
+To verify, check that INFO level log message written by `RDBDocumentStore`
+upon startup. For example:
+
+~~~
+13:40:46.637 INFO [main] RDBDocumentStore.java:1065 RDBDocumentStore (SNAPSHOT) instantiated for database MySQL 8.0.15 (8.0), using driver: MySQL Connector/J mysql-connector-java-8.0.15 (Revision: 79a4336f140499bd22dd07f02b708e163844e3d5) (8.0), connecting to: jdbc:mysql://localhost:3306/oak?serverTimezone=UTC, properties: {character_set_database=utf8mb4, character_set_client=utf8mb4, character_set_connection=utf8mb4, character_set_results=, max_allowed_packet=8388608, collation_database=utf8mb4_unicode_ci, character_set_system=utf8, collation_server=utf8mb4_0900_ai_ci, collation=utf8mb4_unicode_ci, character_set_filesystem=binary, character_set_server=utf8mb4, collation_connection=utf8mb4_0900_ai_ci}, transaction isolation level: TRANSACTION_REPEATABLE_READ (4), .nodes: ID VARBINARY(512), MODIFIED BIGINT(20), HASBINARY SMALLINT(6), DELETEDONCE SMALLINT(6), MODCOUNT BIGINT(20), CMODCOUNT BIGINT(20), DSIZE BIGINT(20), VERSION SMALLINT(6), SDTYPE SMALLINT(6), SDMAXREVTIME BIGI
NT(20), DATA VARCHAR(16000), BDATA LONGBLOB(2147483647) /* {BIGINT=-5, LONGBLOB=-4, SMALLINT=5, VARBINARY=-3, VARCHAR=12} */ /* unique index oak.PRIMARY on nodes (ID ASC) other (#0, p0), index oak.NODES_MOD on nodes (MODIFIED ASC) other (#0, p0), index oak.NODES_SDM on nodes (SDMAXREVTIME ASC) other (#0, p0), index oak.NODES_SDT on nodes (SDTYPE ASC) other (#0, p0), index oak.NODES_VSN on nodes (VERSION ASC) other (#0, p0) */
+~~~
+
+
+### <a name="database-creation-postgresql"></a> PostgreSQL
+
+Creating a database called `OAK`:
+~~~
+CREATE DATABASE "oak" TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'C' LC_CTYPE = 'C';
+~~~
+
+To verify, check that INFO level log message written by `RDBDocumentStore`
+upon startup. For example:
+
+~~~
+16:26:28.172 INFO [main] RDBDocumentStore.java:1065 RDBDocumentStore (SNAPSHOT) instantiated for database PostgreSQL 10.6 (10.6), using driver: PostgreSQL JDBC Driver 42.2.5 (42.2), connecting to: jdbc:postgresql:oak, properties: {datcollate=C, pg_encoding_to_char(encoding)=UTF8}, transaction isolation level: TRANSACTION_READ_COMMITTED (2), .nodes: id varchar(512), modified int8, hasbinary int2, deletedonce int2, modcount int8, cmodcount int8, dsize int8, version int2, sdtype int2, sdmaxrevtime int8, data varchar(16384), bdata bytea(2147483647) /* {bytea=-2, int2=5, int8=-5, varchar=12} */ /* index nodes_mod on public.nodes (modified ASC) other (#0, p1), unique index nodes_pkey on public.nodes (id ASC) other (#0, p1), index nodes_sdm on public.nodes (sdmaxrevtime ASC) other (#0, p1), index nodes_sdt on public.nodes (sdtype ASC) other (#0, p1), index nodes_vsn on public.nodes (version ASC) other (#0, p1) */
+~~~
+
+
+
+## <a name="table-creation"></a> Table Creation
The implementation will try to create all tables and indices when they are not present
yet. Of course this requires that the configured database user actually has
@@ -86,10 +140,11 @@ If it does not, the system will not star
diagnostics in the log file.
Administrators who want to create tables upfront can do so. The DDL statements
-for the supported databases can be dumped using [RDBHelper](/oak/docs/apidocs/org/apache/jackrabbit/oak/plugins/document/rdb/RDBHelper.html).
+for the supported databases can be dumped using [RDBHelper](/oak/docs/apidocs/org/apache/jackrabbit/oak/plugins/document/rdb/RDBHelper.html)
+or, more recently, using `oak-run rdbddldump` (see [below](#rdbddldump)).
-## Upgrade from earlier versions
+## <a name="upgrade"></a> Upgrade from earlier versions
As of Oak 1.8, the database layout has been slightly extended (see
[API docs for RDBDocumentStore](/oak/docs/apidocs/org/apache/jackrabbit/oak/plugins/document/rdb/RDBDocumentStore.html#apidocs.versioning)
@@ -120,8 +175,106 @@ and log diagnostics about the failed upg
~~~
The upgrade can then be done
-at a later point of time by executing the required DDL statements.
+at a later point of time by executing the required DDL statements.
+## <a name="rdbddldump"></a> oak-run rdbddldump
+`@since Oak 1.8.12` `@since Oak 1.10.1` `@since Oak 1.12`
+The `rdbddldump` prints out the DDL statements that Oak would use to create or
+update a database. It can be used to create the tables upfront, or to obtain
+the DDL statements needed to upgrade to a newer schema version.
+
+By default, it will print out the DDL statements for all supported databases,
+with a target of the latest schema version.
+
+The `--db` switch can be used to specify the database type (note that precise
+spelling is needed, otherwise the code will fall back to a generic database
+type).
+
+The `--initial` switch selects the initial database schema (and defaults to
+the most recent one).
+
+The `--upgrade` switch selects the target database schema (and defaults to
+the most recent one).
+
+Selecting a higher "upgrade" version then the "initial" version causes the
+tool to create separate DDL statements for the initial table schema (which may
+already be there), and then to add individual statements for the upgrade to
+the target schema.
+
+For instance:
+
+~~~
+java -jar oak-run-*.jar rdbddldump --db DB2 --initial 0 --upgrade 2
+~~~
+
+will dump statements for DB2, initially creating schema version 0 tables,
+and then include DDL statements to upgrade to version 2 (the latter would
+be applicable if an installation needed to be upgraded from an Oak version
+older than 1.8 to 1.8 or newer).
+
+~~~
+-- DB2
+
+ -- creating table CLUSTERNODES for schema version 0
+ create table CLUSTERNODES (ID varchar(512) not null, MODIFIED bigint, HASBINARY smallint, DELETEDONCE smallint, MODCOUNT bigint, CMODCOUNT bigint, DSIZE bigint, DATA varchar(16384), BDATA blob(1073741824))
+ create unique index CLUSTERNODES_pk on CLUSTERNODES ( ID ) cluster
+ alter table CLUSTERNODES add constraint CLUSTERNODES_pk primary key ( ID )
+ create index CLUSTERNODES_MOD on CLUSTERNODES (MODIFIED)
+ -- upgrading table CLUSTERNODES to schema version 1
+ alter table CLUSTERNODES add VERSION smallint
+ -- upgrading table CLUSTERNODES to schema version 2
+ alter table CLUSTERNODES add SDTYPE smallint
+ alter table CLUSTERNODES add SDMAXREVTIME bigint
+ create index CLUSTERNODES_VSN on CLUSTERNODES (VERSION)
+ create index CLUSTERNODES_SDT on CLUSTERNODES (SDTYPE) exclude null keys
+ create index CLUSTERNODES_SDM on CLUSTERNODES (SDMAXREVTIME) exclude null keys
+
+ -- creating table JOURNAL for schema version 0
+ create table JOURNAL (ID varchar(512) not null, MODIFIED bigint, HASBINARY smallint, DELETEDONCE smallint, MODCOUNT bigint, CMODCOUNT bigint, DSIZE bigint, DATA varchar(16384), BDATA blob(1073741824))
+ create unique index JOURNAL_pk on JOURNAL ( ID ) cluster
+ alter table JOURNAL add constraint JOURNAL_pk primary key ( ID )
+ create index JOURNAL_MOD on JOURNAL (MODIFIED)
+ -- upgrading table JOURNAL to schema version 1
+ alter table JOURNAL add VERSION smallint
+ -- upgrading table JOURNAL to schema version 2
+ alter table JOURNAL add SDTYPE smallint
+ alter table JOURNAL add SDMAXREVTIME bigint
+ create index JOURNAL_VSN on JOURNAL (VERSION)
+ create index JOURNAL_SDT on JOURNAL (SDTYPE) exclude null keys
+ create index JOURNAL_SDM on JOURNAL (SDMAXREVTIME) exclude null keys
+
+ -- creating table NODES for schema version 0
+ create table NODES (ID varchar(512) not null, MODIFIED bigint, HASBINARY smallint, DELETEDONCE smallint, MODCOUNT bigint, CMODCOUNT bigint, DSIZE bigint, DATA varchar(16384), BDATA blob(1073741824))
+ create unique index NODES_pk on NODES ( ID ) cluster
+ alter table NODES add constraint NODES_pk primary key ( ID )
+ create index NODES_MOD on NODES (MODIFIED)
+ -- upgrading table NODES to schema version 1
+ alter table NODES add VERSION smallint
+ -- upgrading table NODES to schema version 2
+ alter table NODES add SDTYPE smallint
+ alter table NODES add SDMAXREVTIME bigint
+ create index NODES_VSN on NODES (VERSION)
+ create index NODES_SDT on NODES (SDTYPE) exclude null keys
+ create index NODES_SDM on NODES (SDMAXREVTIME) exclude null keys
+
+ -- creating table SETTINGS for schema version 0
+ create table SETTINGS (ID varchar(512) not null, MODIFIED bigint, HASBINARY smallint, DELETEDONCE smallint, MODCOUNT bigint, CMODCOUNT bigint, DSIZE bigint, DATA varchar(16384), BDATA blob(1073741824))
+ create unique index SETTINGS_pk on SETTINGS ( ID ) cluster
+ alter table SETTINGS add constraint SETTINGS_pk primary key ( ID )
+ create index SETTINGS_MOD on SETTINGS (MODIFIED)
+ -- upgrading table SETTINGS to schema version 1
+ alter table SETTINGS add VERSION smallint
+ -- upgrading table SETTINGS to schema version 2
+ alter table SETTINGS add SDTYPE smallint
+ alter table SETTINGS add SDMAXREVTIME bigint
+ create index SETTINGS_VSN on SETTINGS (VERSION)
+ create index SETTINGS_SDT on SETTINGS (SDTYPE) exclude null keys
+ create index SETTINGS_SDM on SETTINGS (SDMAXREVTIME) exclude null keys
+
+ -- creating blob store tables
+ create table DATASTORE_META (ID varchar(64) not null primary key, LVL int, LASTMOD bigint)
+ create table DATASTORE_DATA (ID varchar(64) not null primary key, DATA blob(2097152))
+~~~
Modified: jackrabbit/oak/branches/1.10/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/rdb/RDBDocumentStore.java
URL: http://svn.apache.org/viewvc/jackrabbit/oak/branches/1.10/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/rdb/RDBDocumentStore.java?rev=1854524&r1=1854523&r2=1854524&view=diff
==============================================================================
--- jackrabbit/oak/branches/1.10/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/rdb/RDBDocumentStore.java (original)
+++ jackrabbit/oak/branches/1.10/oak-store-document/src/main/java/org/apache/jackrabbit/oak/plugins/document/rdb/RDBDocumentStore.java Thu Feb 28 13:27:37 2019
@@ -218,21 +218,23 @@ import com.google.common.collect.Sets;
* existing version 0 and 1 tables to version 2.
* <h4>DB-specific information</h4>
* <p>
- * <em>Note that the database needs to be created/configured to support all
- * Unicode characters in text fields, and to collate by Unicode code point (in
- * DB2: "collate using identity", in PostgreSQL: "C"). THIS IS NOT THE
- * DEFAULT!</em>
+ * Databases need to be configured so that:
+ * <ul>
+ * <li>Text fields support all Unicode code points,</li>
+ * <li>Collation of text fields happens by Unicode code point,</li>
+ * <li>and BLOBs need to support at least 16 MB.</li>
+ * </ul>
* <p>
- * <em>For MySQL, the database parameter "max_allowed_packet" needs to be
- * increased to support ~16M blobs.</em>
- *
+ * See the
+ * <a href="https://jackrabbit.apache.org/oak/docs/nodestore/document/rdb-document-store.html#database-creation">RDBDocumentStore documentation</a>
+ * for more information.
* <h3 id="apidocs.table-creation">Table Creation</h3>
* <p>
* The code tries to create the tables when they are not present. Likewise, it
* tries to upgrade to a newer schema when needed.
* <p>
* Users/Administrators who prefer to stay in control over table generation can
- * create them "manually". The oak-run "<a href="https://jackrabbit.apache.org/oak/docs/nodestore/document/rdb-document-store.html#oak-run_rdbddldump"><code>rdbddldump</code></a>"
+ * create them "manually". The oak-run "<a href="https://jackrabbit.apache.org/oak/docs/nodestore/document/rdb-document-store.html#rdbddldump"><code>rdbddldump</code></a>"
* command can be used to print out the DDL statements that would have been used for auto-creation
* and/or automatic schema updates.
*