You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by bbende <gi...@git.apache.org> on 2018/06/01 18:33:07 UTC
[GitHub] nifi-registry pull request #121: NIFIREG-173 Refactor metadata DB to be inde...
GitHub user bbende opened a pull request:
https://github.com/apache/nifi-registry/pull/121
NIFIREG-173 Refactor metadata DB to be independent of H2
The approach here is to create a new DB with a schema and DataSource that is not specific to H2 and migrate existing data.
During start up if the previous DB properties are populated and the new DB has not been setup yet, then it initiate a migration of the data from the old DB to new DB. For anyone starting up for the first time without the legacy DB then it will just start with the new schema and DB.
I tested taking an existing 0.1.0 registry and dropping the old H2 DB into a build of this PR and it migrated over to the new H2 DB and was able to continue using the app as normal. Future restarts after that don't trigger the migration since the new DB already exists at that point.
I also tested the same scenario as above, but using Postgres as the target DB by copying the Postgres driver jar into the lib directory and configuring the appropriate properties in nifi-registry.properties.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bbende/nifi-registry db-refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi-registry/pull/121.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #121
----
commit b8ac082b0c0449a038aa09a38d67789e8b1559db
Author: Bryan Bende <bb...@...>
Date: 2018-05-30T18:31:26Z
NIFIREG-173 Refactor metadata DB to be independent of H2
----
---
[GitHub] nifi-registry pull request #121: NIFIREG-173 Refactor metadata DB to be inde...
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/121#discussion_r192790706
--- Diff: nifi-registry-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -867,12 +867,32 @@ content of the flows saved to the registry. For further details on persistence p
These properties define the settings for the Registry database, which keeps track of metadata about buckets and all items stored in buckets.
+The 0.1.0 release leveraged an embedded H2 database that was configured via the following properties:
+
|====
|*Property*|*Description*
|nifi.registry.db.directory|The location of the Registry database directory. The default value is `./database`.
|nifi.registry.db.url.append|This property specifies additional arguments to add to the connection string for the Registry database. The default value should be used and should not be changed. It is: `;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE`.
|====
+The 0.2.0 release introduced a more flexible approach which allows leveraging an external database. This new approach
+is configured via the following properties:
--- End diff --
Good writeup. Clear and concise.
---
[GitHub] nifi-registry pull request #121: NIFIREG-173 Refactor metadata DB to be inde...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi-registry/pull/121
---
[GitHub] nifi-registry issue #121: NIFIREG-173 Refactor metadata DB to be independent...
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on the issue:
https://github.com/apache/nifi-registry/pull/121
Will review...
---
[GitHub] nifi-registry pull request #121: NIFIREG-173 Refactor metadata DB to be inde...
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/121#discussion_r192781546
--- Diff: nifi-registry-framework/src/main/java/org/apache/nifi/registry/db/CustomFlywayMigrationStrategy.java ---
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.db;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.registry.db.migration.BucketEntityV1;
+import org.apache.nifi.registry.db.migration.FlowEntityV1;
+import org.apache.nifi.registry.db.migration.FlowSnapshotEntityV1;
+import org.apache.nifi.registry.db.migration.LegacyDataSourceFactory;
+import org.apache.nifi.registry.db.migration.LegacyDatabaseService;
+import org.apache.nifi.registry.db.migration.LegacyEntityMapper;
+import org.apache.nifi.registry.properties.NiFiRegistryProperties;
+import org.apache.nifi.registry.service.MetadataService;
+import org.flywaydb.core.Flyway;
+import org.flywaydb.core.api.FlywayException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.boot.autoconfigure.flyway.FlywayMigrationStrategy;
+import org.springframework.jdbc.core.JdbcTemplate;
+import org.springframework.stereotype.Component;
+
+import javax.sql.DataSource;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+
+/**
+ * Custom Flyway migration strategy that lets us perform data migration from the original database used in the
+ * 0.1.0 release, to the new database. The data migration will be triggered when it is determined that new database
+ * is brand new AND the legacy DB properties are specified. If the primary database already contains the 'BUCKET' table,
+ * or if the legacy database properties are not specified, then no data migration is performed.
+ */
+@Component
+public class CustomFlywayMigrationStrategy implements FlywayMigrationStrategy {
--- End diff --
This is a nice solution to determining when to migrate databases. Nice work!
---
[GitHub] nifi-registry pull request #121: NIFIREG-173 Refactor metadata DB to be inde...
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-registry/pull/121#discussion_r192790400
--- Diff: nifi-registry-framework/src/main/resources/db/migration/V2__Initial.sql ---
@@ -0,0 +1,58 @@
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
--- End diff --
I tried this schema in H2, Postgres and MySQL. For the most part looks good. I did get this error in MySQL:
```
Column length too big for column 'DESCRIPTION' (max = 21845); use BLOB or TEXT instead
```
I did a bit of research and this was the best explanation I found:
https://dev.mysql.com/doc/refman/5.7/en/column-count-limit.html
Seems like 65535 should be fine so long as the total row size stays below that limit. Maybe the 21845 value in the error message is due to a text encoding type (did not dig into it past that). In any case, to maintain mysql compatibility, you may want to lower those max length sizes or change the description fields to TEXT as I don't anticipate we will need to search on those.
Aside from that, this schema looks good to me as a clean starting point for the new database. Nice work.
---