You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@unomi.apache.org by jk...@apache.org on 2022/07/28 10:13:30 UTC
[unomi] branch migrationTests updated (19040201d -> 5006b77eb)
This is an automated email from the ASF dual-hosted git repository.
jkevan pushed a change to branch migrationTests
in repository https://gitbox.apache.org/repos/asf/unomi.git
from 19040201d Oops
new 198f827eb UNOMI-632: add a bit of documentation on migration testing, for Unomi developers
new 5006b77eb UNOMI-633: update documentation regarding migration command
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
itests/README.md | 178 +++++++++++++++++++--
manual/src/main/asciidoc/shell-commands.adoc | 11 +-
.../unomi/shell/migration/actions/Migrate.java | 6 +-
3 files changed, 182 insertions(+), 13 deletions(-)
[unomi] 01/02: UNOMI-632: add a bit of documentation on migration testing, for Unomi developers
Posted by jk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jkevan pushed a commit to branch migrationTests
in repository https://gitbox.apache.org/repos/asf/unomi.git
commit 198f827eb7fe2014a25ee41deb6d904eed3629a4
Author: Kevan <ke...@jahia.com>
AuthorDate: Thu Jul 28 11:19:14 2022 +0200
UNOMI-632: add a bit of documentation on migration testing, for Unomi developers
---
itests/README.md | 178 ++++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 169 insertions(+), 9 deletions(-)
diff --git a/itests/README.md b/itests/README.md
index 845ea669c..ae2eee662 100644
--- a/itests/README.md
+++ b/itests/README.md
@@ -38,16 +38,16 @@ have what you expect and it could work fine on your machine but not on others.
If possible clean what your test created at the end of its execution or at the very least make sure to use unique IDs
-When you need a service from Unomi to execute your test inject it with a filer:
+When you need a service from Unomi to execute your test, you can add it to the BaseIT code:
```java
-@Inject @Filter(timeout = 60000)
-protected ProfileService profileService;
+@Before
+public void waitForStartup() throws InterruptedException {
+ // ...
+ // init unomi services that are available once unomi:start have been called
+ persistenceService=getOsgiService(PersistenceService.class, 600000);
+}
```
-This will ensure the service is available before starting the test and if you need a resource like an URL you can do something like this:
-```java
-@Inject @Filter(value="(configDiscriminator=IMPORT)", timeout = 60000)
-protected ImportExportConfigurationService<ImportConfiguration> importConfigurationService;
-```
+This will ensure the service is available before starting the test.
## Running integration tests
You can run the integration tests along with the build by doing:
@@ -82,4 +82,164 @@ https://maven.apache.org/surefire/maven-failsafe-plugin/examples/single-test.htm
Here's an example:
- mvn clean install -Dit.karaf.debug=hold:true -Dit.test=org.apache.unomi.itests.graphql.GraphQLEventIT
\ No newline at end of file
+ mvn clean install -Dit.karaf.debug=hold:true -Dit.test=org.apache.unomi.itests.graphql.GraphQLEventIT
+
+## Migration tests
+
+Migration can now be tested, by reusing an ElasticSearch snapshot.
+The snapshot should be from a Unomi version where you want to start the migration from.
+
+The snapshot is copied to the /target folder using a maven ant plugin:
+
+ <plugin>
+ <artifactId>maven-antrun-plugin</artifactId>
+ <version>1.8</version>
+ <executions>
+ <execution>
+ <phase>generate-resources</phase>
+ <configuration>
+ <tasks>
+ <unzip src="${project.basedir}/src/test/resources/migration/snapshots_repository.zip" dest="${project.build.directory}" />
+ </tasks>
+ </configuration>
+ <goals>
+ <goal>run</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+
+Also the ElasticSearch maven plugin is configured to allow this snapshot repository using conf:
+
+ <path.repo>${project.build.directory}/snapshots_repository</path.repo>
+
+Now that migration accept configuration file we can provide it, this allows to benefit from silent migration (in BaseIT configuration):
+
+ replaceConfigurationFile("etc/org.apache.unomi.migration.cfg", new File("src/test/resources/migration/org.apache.unomi.migration.cfg")),
+
+The config should contain all the required prop for the migration you want to do, example:
+
+ esAddress = http://localhost:9400
+ httpClient.trustAllCertificates = true
+ indexPrefix = context
+
+Then in the first Test of the suite you can restore the Snapshot and run the migration cmd, like this:
+
+```java
+public class Migrate16xTo200IT extends BaseIT {
+
+ @Override
+ @Before
+ public void waitForStartup() throws InterruptedException {
+
+ // Restore snapshot from 1.6.x
+ try (CloseableHttpClient httpClient = HttpUtils.initHttpClient(true)) {
+ // Create snapshot repo
+ HttpUtils.executePutRequest(httpClient, "http://localhost:9400/_snapshot/snapshots_repository/", resourceAsString("migration/create_snapshots_repository.json"), null);
+ // Get snapshot, insure it exists
+ String snapshot = HttpUtils.executeGetRequest(httpClient, "http://localhost:9400/_snapshot/snapshots_repository/snapshot_1.6.x", null);
+ if (snapshot == null || !snapshot.contains("snapshot_1.6.x")) {
+ throw new RuntimeException("Unable to retrieve 1.6.x snapshot for ES restore");
+ }
+ // Restore the snapshot
+ HttpUtils.executePostRequest(httpClient, "http://localhost:9400/_snapshot/snapshots_repository/snapshot_1.6.x/_restore?wait_for_completion=true", "{}", null);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ // Do migrate the data set
+ executeCommand("unomi:migrate 1.6.0 true");
+ // Call super for starting Unomi and wait for the complete startup
+ super.waitForStartup();
+ }
+
+ @After
+ public void cleanup() throws InterruptedException {
+ // Do some cleanup for next tests
+ }
+
+ @Test
+ public void checkMigratedData() throws Exception {
+ // call Unomi services to check the migrated data is correct.
+ }
+}
+```
+
+### How to update a migration test ElasticSearch Snapshot ?
+
+In the following example we want to modify the snapshot: `snapshot_1.6.x`.
+This snapshot has been done on Unomi 1.6.x using ElasticSearch 7.11.0.
+So we will set up locally those servers in the exact same versions.
+(For now just download them and do not start them yet.)
+
+First we need to extract the zip of the snapshot repository from the test resources:
+
+ /src/test/resources/migration/snapshots_repository.zip
+
+In my case I unzip it to:
+
+ /servers/elasticsearch-7.11.0/
+
+So I have the following folders structure:
+
+ /servers/elasticsearch-7.11.0/snapshots_repository/snapshots
+
+Now we need to configure our ElasticSearch server to allow this path as repo, edit the `elasticsearch.yml` to add this:
+
+ path:
+ repo:
+ - /servers/elasticsearch-7.11.0/snapshots_repository
+
+Start ElasticSearch server.
+Now we have to add the snapshot repository, do the following request on your ElasticSearch instance:
+
+ PUT /_snapshot/snapshots_repository/
+ {
+ "type": "fs",
+ "settings": {
+ "location": "snapshots"
+ }
+ }
+
+Now we need to restore the snapshot we want to modify,
+but first let's try to see if the snapshot with the id `snapshot_1.6.x` correctly exists:
+
+ GET /_snapshot/snapshots_repository/snapshot_1.6.x
+
+If the snapshot exists we can restore it:
+
+ POST /_snapshot/snapshots_repository/snapshot_1.6.x/_restore?wait_for_completion=true
+ {}
+
+At the end of the previous request ElasticSearch should be ready and our Unomi snapshot is restored to version `1.6.x`.
+Now make sure your Unomi server is correctly configured to connect to your running ElasticSearch, then start the Unomi server.
+In my case it's Unomi version 1.6.0.
+
+Once Unomi started you can perform all the operations you want to be able to add the required data to the next snapshot, like:
+- creating new events
+- creating new profiles with new data to be migrated
+- create rules/segments etc ...
+- anything you want to be part of the new snapshot.
+
+(NOTE: that it is important to add new data to the existing snapshot, but try to not removing things,
+they are probably used by the actual migration tests already.)
+
+Once you data updated we need to recreate the snapshot, first we delete the old snapshot:
+
+ DELETE /_snapshot/snapshots_repository/snapshot_1.6.x
+
+Then we recreate it:
+
+ PUT /_snapshot/snapshots_repository/snapshot_1.6.x
+
+Once the process finished (check the ElasticSearch logs to see that the snapshot is correctly created),
+we need to remove the snapshot repository from our local ElasticSearch
+
+ DELETE /_snapshot/snapshots_repository
+
+And the final step is, zipping the new version of the snapshot repository and replace it in the test resources:
+
+ zip -r snapshots_repository.zip /servers/elasticsearch-7.11.0/snapshots_repository
+ cp /servers/elasticsearch-7.11.0/snapshots_repository.zip src/test/resources/migration/snapshots_repository.zip
+
+Now you can modify the migration test class to test that your added data in 1.6.x is correctly migrated in 2.0.0
[unomi] 02/02: UNOMI-633: update documentation regarding migration command
Posted by jk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jkevan pushed a commit to branch migrationTests
in repository https://gitbox.apache.org/repos/asf/unomi.git
commit 5006b77ebd25d0b907962a1e96f943cc7884dfff
Author: Kevan <ke...@jahia.com>
AuthorDate: Thu Jul 28 12:13:18 2022 +0200
UNOMI-633: update documentation regarding migration command
---
manual/src/main/asciidoc/shell-commands.adoc | 11 +++++++++--
.../org/apache/unomi/shell/migration/actions/Migrate.java | 6 ++++--
2 files changed, 13 insertions(+), 4 deletions(-)
diff --git a/manual/src/main/asciidoc/shell-commands.adoc b/manual/src/main/asciidoc/shell-commands.adoc
index e951a3071..a4ffb4836 100644
--- a/manual/src/main/asciidoc/shell-commands.adoc
+++ b/manual/src/main/asciidoc/shell-commands.adoc
@@ -41,15 +41,22 @@ karaf@root()> help unomi:migrate
DESCRIPTION
unomi:migrate
- This will Migrate your date in ES to be compliant with current version
+ This will Migrate your date in ES to be compliant with current version.
+ It's possible to configure the migration using OSGI configuration file: org.apache.unomi.migration.cfg,
+ if no configuration is provided then questions will be prompted during the migration process.
SYNTAX
- unomi:migrate [fromVersionWithoutSuffix]
+ unomi:migrate [fromVersionWithoutSuffix] [silent]
ARGUMENTS
fromVersionWithoutSuffix
Origin version without suffix/qualifier (e.g: 1.2.0)
(defaults to 1.2.0)
+ silent
+ Should the migration process be silent ?. (No questions will be prompted in the console)
+ The configuration org.apache.unomi.migration.cfg is required for this option to work properly
+ (defaults to false)
+
```
==== Lifecycle commands
diff --git a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/actions/Migrate.java b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/actions/Migrate.java
index 8a3f69bf8..af57a6ae4 100644
--- a/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/actions/Migrate.java
+++ b/tools/shell-commands/src/main/java/org/apache/unomi/shell/migration/actions/Migrate.java
@@ -44,7 +44,8 @@ import java.util.stream.Stream;
import static org.apache.unomi.shell.migration.MigrationConfig.CONFIG_TRUST_ALL_CERTIFICATES;
-@Command(scope = "unomi", name = "migrate", description = "This will Migrate your data in ES to be compliant with current version")
+@Command(scope = "unomi", name = "migrate", description = "This will Migrate your data in ES to be compliant with current version. " +
+ "It's possible to configure the migration using OSGI configuration file: org.apache.unomi.migration.cfg, if no configuration is provided then questions will be prompted during the migration process.")
@Service
public class Migrate implements Action {
@@ -61,7 +62,8 @@ public class Migrate implements Action {
@Argument(name = "originVersion", description = "Origin version without suffix/qualifier (e.g: 1.2.0)", valueToShowInHelp = "1.2.0")
private String originVersion;
- @Argument(index = 1, name = "silent", description = "Should the migration process be silent ? (default: false)", valueToShowInHelp = "true")
+ @Argument(index = 1, name = "silent", description = "Should the migration process be silent ?. " +
+ "The configuration org.apache.unomi.migration.cfg is required for this option to work properly", valueToShowInHelp = "false")
private boolean silent = false;
public Object execute() throws Exception {