You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@unomi.apache.org by am...@apache.org on 2017/06/06 15:36:42 UTC
[15/17] incubator-unomi git commit: DMF-1343 : Improvement and
documentation
DMF-1343 : Improvement and documentation
Project: http://git-wip-us.apache.org/repos/asf/incubator-unomi/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-unomi/commit/3c602041
Tree: http://git-wip-us.apache.org/repos/asf/incubator-unomi/tree/3c602041
Diff: http://git-wip-us.apache.org/repos/asf/incubator-unomi/diff/3c602041
Branch: refs/heads/feature-DMF-1343
Commit: 3c602041a7402dddb4ab119a7c62984149db65c1
Parents: 2f975d3
Author: Abdelkader Midani <am...@apache.org>
Authored: Wed May 24 17:26:14 2017 +0200
Committer: Abdelkader Midani <am...@apache.org>
Committed: Tue Jun 6 17:35:46 2017 +0200
----------------------------------------------------------------------
extensions/router/README.md | 113 +++++++++++++++++++
.../ProfileImportConfigUpdateRouteBuilder.java | 2 +-
.../resources/OSGI-INF/blueprint/blueprint.xml | 6 +-
.../main/resources/org.apache.unomi.router.cfg | 4 +-
.../ImportConfigurationServiceEndPoint.java | 37 +++---
.../resources/OSGI-INF/blueprint/blueprint.xml | 14 ++-
6 files changed, 154 insertions(+), 22 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-unomi/blob/3c602041/extensions/router/README.md
----------------------------------------------------------------------
diff --git a/extensions/router/README.md b/extensions/router/README.md
new file mode 100644
index 0000000..8fa1994
--- /dev/null
+++ b/extensions/router/README.md
@@ -0,0 +1,113 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+Unomi Router
+==========================
+
+## Getting started
+Unomi Router Extension a Karaf Feature that provide an Enterprise Application Integration tool.
+It is optional so you must configure it and install it in Karaf, and can be used for Machine - Machine or Human - Machine integration with Unomi.
+Mainly Unomi Router Extension aim to make it easy to import third party applications/platforms profiles into Unomi.
+This extension is implemented using Apache Camel routes and is using Apache Kafka to buffer import process and make it failsafe.
+
+## Getting started
+1. Configure your Unomi Router:
+ In the `etc/org.apache.unomi.sfdc.cfg` file, you might want to update the following settings:
+ Kafka settings
+ >`#Kafka settings`
+
+ >`kafka.host=localhost`
+
+ >`kafka.port=9092`
+
+ >`kafka.import.topic=camel-deposit`
+
+ >`kafka.import.groupId=unomi-import-group`
+
+ Kafka host and port with the topic name and the groupId ti which the topic is assigned
+
+ >`#Import One Shot upload directory`
+
+ >`import.oneshot.uploadDir=/tmp/unomi_oneshot_import_configs/`
+
+ Path to the folder where unomi should stock file imported for a oneshot processing
+
+
+2. Deploy into Apache Unomi using the following commands from the Apache Karaf shell:
+ ```sh
+ $ feature:repo-add mvn:org.apache.unomi/unomi-router-karaf-feature/${version}/xml/features
+ $ feature:install unomi-router-karaf-feature
+ ```
+
+3. Send your import configuration:
+
+ An import configuration is nothing else than a simple JSON to describe how you want to import your data (Profiles).
+ To create/update an import configuration
+
+ `POST /cxs/importConfiguration`
+ ```json
+ {
+ "itemId": "f57f1f86-97bf-4ba0-b4e4-7d5e77e7c0bd",
+ "itemType": "importConfig",
+ "scope": "integration",
+ "name": "Test Recurrent",
+ "description": "Just test recurrent import",
+ "configType": "recurrent",
+ "properties": {
+ "source": "{file/ftp}://{path}?fileName={file-name}.csv&move=.done&consumer.delay=20000",
+ "mapping": {
+ "firstName": 0,
+ "lastName": 1,
+ ...
+ }
+ },
+ "mergingProperty": "email",
+ "overwriteExistingProfiles": true,
+ "propertiesToOverwrite": ["firstName", "lastName"],
+ "active": true
+ }
+ ```
+
+ Omit the `itemId` when creating new entry, `configType` can be '**recurrent**' for file/ftp/network path polling or '**oneshot**' for one time import.
+
+ The `properties.source` attribute is an Apache Camel endpoint uri (See http://camel.apache.org/uris.html for more details). Unomi Router is designed to use **File** and **FTP** Camel components.
+
+ The attribute `properties.mapping` is a Map of:
+ * Key: Profile property id in Unomi
+ * Value: Index of the column in the imported file to copy the in the previous property.
+
+ The attribute `mergingProperty` is the profile property id in Unomi to use to check for duplication.
+
+ The attribute `propertiesToOverwrite` is a list of profile properties ids to overwrite, if **null** all properties
+ will be overwritten.
+
+ The attribute `active` is the flag to activate or deactivate the import configuration.
+
+ Concerning oneshot import configuration using the previously described service will only create the import configuration, to send the file to process
+ you need to call :
+
+ `POST /cxs/importConfiguration/oneshot`
+
+ `Content-Type : multipart/form-data`
+
+ First multipart with the name '**importConfigId**' is the importConfiguration to use to import the file, second one with the name '**file**' is the file to import.
+
+
+
+
+
+
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-unomi/blob/3c602041/extensions/router/router-core/src/main/java/org/apache/unomi/router/core/route/ProfileImportConfigUpdateRouteBuilder.java
----------------------------------------------------------------------
diff --git a/extensions/router/router-core/src/main/java/org/apache/unomi/router/core/route/ProfileImportConfigUpdateRouteBuilder.java b/extensions/router/router-core/src/main/java/org/apache/unomi/router/core/route/ProfileImportConfigUpdateRouteBuilder.java
index ac71798..40575d5 100644
--- a/extensions/router/router-core/src/main/java/org/apache/unomi/router/core/route/ProfileImportConfigUpdateRouteBuilder.java
+++ b/extensions/router/router-core/src/main/java/org/apache/unomi/router/core/route/ProfileImportConfigUpdateRouteBuilder.java
@@ -38,7 +38,7 @@ public class ProfileImportConfigUpdateRouteBuilder extends RouteBuilder {
logger.info("Preparing REST Configuration for servlet with context path [/importConfigAdmin]");
restConfiguration().component("servlet")
.contextPath("/importConfigAdmin")
- .enableCORS(true)
+ .enableCORS(false)
.bindingMode(RestBindingMode.json)
.dataFormatProperty("prettyPrint", "true");
http://git-wip-us.apache.org/repos/asf/incubator-unomi/blob/3c602041/extensions/router/router-core/src/main/resources/OSGI-INF/blueprint/blueprint.xml
----------------------------------------------------------------------
diff --git a/extensions/router/router-core/src/main/resources/OSGI-INF/blueprint/blueprint.xml b/extensions/router/router-core/src/main/resources/OSGI-INF/blueprint/blueprint.xml
index ff29755..4c36b9e 100644
--- a/extensions/router/router-core/src/main/resources/OSGI-INF/blueprint/blueprint.xml
+++ b/extensions/router/router-core/src/main/resources/OSGI-INF/blueprint/blueprint.xml
@@ -79,9 +79,9 @@
<property name="bundleContext" ref="blueprintBundleContext"/>
</bean>
- <camelContext id="httpEndpoint" xmlns="http://camel.apache.org/schema/blueprint">
- <routeBuilder ref="profileImportConfigUpdateRouteBuilder" />
- </camelContext>
+ <camel:camelContext id="httpEndpoint" xmlns="http://camel.apache.org/schema/blueprint">
+ <camel:routeBuilder ref="profileImportConfigUpdateRouteBuilder" />
+ </camel:camelContext>
<bean id="profileImportConfigUpdateRouteBuilder" class="org.apache.unomi.router.core.route.ProfileImportConfigUpdateRouteBuilder">
<property name="profileImportCamelContext" ref="camelContext"/>
http://git-wip-us.apache.org/repos/asf/incubator-unomi/blob/3c602041/extensions/router/router-core/src/main/resources/org.apache.unomi.router.cfg
----------------------------------------------------------------------
diff --git a/extensions/router/router-core/src/main/resources/org.apache.unomi.router.cfg b/extensions/router/router-core/src/main/resources/org.apache.unomi.router.cfg
index b11d983..ff2c8ef 100644
--- a/extensions/router/router-core/src/main/resources/org.apache.unomi.router.cfg
+++ b/extensions/router/router-core/src/main/resources/org.apache.unomi.router.cfg
@@ -14,7 +14,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-kafka.host=localhost
+
+#Kafka
+ settingskafka.host=localhost
kafka.port=9092
kafka.import.topic=camel-deposit
kafka.import.groupId=unomi-import-group
http://git-wip-us.apache.org/repos/asf/incubator-unomi/blob/3c602041/extensions/router/router-rest/src/main/java/org/apache/unomi/router/rest/ImportConfigurationServiceEndPoint.java
----------------------------------------------------------------------
diff --git a/extensions/router/router-rest/src/main/java/org/apache/unomi/router/rest/ImportConfigurationServiceEndPoint.java b/extensions/router/router-rest/src/main/java/org/apache/unomi/router/rest/ImportConfigurationServiceEndPoint.java
index e81930a..42670b2 100644
--- a/extensions/router/router-rest/src/main/java/org/apache/unomi/router/rest/ImportConfigurationServiceEndPoint.java
+++ b/extensions/router/router-rest/src/main/java/org/apache/unomi/router/rest/ImportConfigurationServiceEndPoint.java
@@ -52,9 +52,11 @@ import java.util.List;
public class ImportConfigurationServiceEndPoint {
private static final Logger logger = LoggerFactory.getLogger(ImportConfigurationServiceEndPoint.class.getName());
+ private final String CONFIG_TYPE_RECURRENT = "recurrent";
private ImportConfigurationService importConfigurationService;
private String uploadDir;
+ private String serverPort;
public ImportConfigurationServiceEndPoint () {
logger.info("Initializing import configuration service endpoint...");
@@ -70,6 +72,11 @@ public class ImportConfigurationServiceEndPoint {
this.uploadDir = uploadDir;
}
+ @WebMethod(exclude = true)
+ public void setServerPort(String serverPort) {
+ this.serverPort = serverPort;
+ }
+
/**
* Retrieves all the import configurations.
*
@@ -122,21 +129,23 @@ public class ImportConfigurationServiceEndPoint {
@Consumes(MediaType.APPLICATION_JSON)
public ImportConfiguration saveImportConfiguration(ImportConfiguration importConfiguration) {
ImportConfiguration importConfigSaved = importConfigurationService.save(importConfiguration);
- CloseableHttpClient httpClient = HttpClients.createDefault();
- try {
- HttpPut httpPut = new HttpPut("http://localhost:8181/importConfigAdmin/");
- StringEntity input = new StringEntity(new ObjectMapper().writeValueAsString(importConfigSaved));
- input.setContentType(MediaType.APPLICATION_JSON);
- httpPut.setEntity(input);
-
- HttpResponse response = httpClient.execute(httpPut);
-
- if (response.getStatusLine().getStatusCode() != 200) {
- throw new RuntimeException("Failed : HTTP error code : "
- + response.getStatusLine().getStatusCode());
+ if(CONFIG_TYPE_RECURRENT.equals(importConfigSaved.getConfigType())) {
+ CloseableHttpClient httpClient = HttpClients.createDefault();
+ try {
+ HttpPut httpPut = new HttpPut("http://localhost:"+serverPort+"/importConfigAdmin/");
+ StringEntity input = new StringEntity(new ObjectMapper().writeValueAsString(importConfigSaved));
+ input.setContentType(MediaType.APPLICATION_JSON);
+ httpPut.setEntity(input);
+
+ HttpResponse response = httpClient.execute(httpPut);
+
+ if (response.getStatusLine().getStatusCode() != 200) {
+ throw new RuntimeException("Failed : HTTP error code : "
+ + response.getStatusLine().getStatusCode());
+ }
+ } catch (IOException e) {
+ logger.warn("Unable to update Camel route [{}]", importConfiguration.getItemId());
}
- } catch (IOException e) {
- logger.warn("Unable to update Camel route [{}]", importConfiguration.getItemId());
}
return importConfigSaved;
}
http://git-wip-us.apache.org/repos/asf/incubator-unomi/blob/3c602041/extensions/router/router-rest/src/main/resources/OSGI-INF/blueprint/blueprint.xml
----------------------------------------------------------------------
diff --git a/extensions/router/router-rest/src/main/resources/OSGI-INF/blueprint/blueprint.xml b/extensions/router/router-rest/src/main/resources/OSGI-INF/blueprint/blueprint.xml
index a5cf1be..7e3b5d1 100644
--- a/extensions/router/router-rest/src/main/resources/OSGI-INF/blueprint/blueprint.xml
+++ b/extensions/router/router-rest/src/main/resources/OSGI-INF/blueprint/blueprint.xml
@@ -21,15 +21,22 @@
xmlns:cxf="http://cxf.apache.org/blueprint/core" xmlns:jaxrs="http://cxf.apache.org/blueprint/jaxrs"
xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
- http://cxf.apache.org/blueprint/jaxrs http://cxf.apache.org/schemas/blueprint/jaxrs.xsd
- http://cxf.apache.org/blueprint/core http://cxf.apache.org/schemas/blueprint/core.xsd
- http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0 http://aries.apache.org/schemas/blueprint-cm/blueprint-cm-1.1.0.xsd">
+ http://cxf.apache.org/blueprint/jaxrs http://cxf.apache.org/schemas/blueprint/jaxrs.xsd
+ http://cxf.apache.org/blueprint/core http://cxf.apache.org/schemas/blueprint/core.xsd
+ http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0 http://aries.apache.org/schemas/blueprint-cm/blueprint-cm-1.1.0.xsd">
<cm:property-placeholder persistent-id="org.apache.unomi.router" update-strategy="reload">
<cm:default-properties>
<cm:property name="import.oneshot.uploadDir" value="/tmp/oneshot_import_configs/"/>
</cm:default-properties>
</cm:property-placeholder>
+ <cm:property-placeholder persistent-id="org.apache.unomi.cluster"
+ update-strategy="reload" placeholder-prefix="${cluster.">
+ <cm:default-properties>
+ <cm:property name="contextserver.port" value="8181"/>
+ </cm:default-properties>
+ </cm:property-placeholder>
+
<cxf:bus id="cxsServiceBus">
<cxf:features>
<cxf:logging/>
@@ -67,6 +74,7 @@
<bean id="importConfigurationServiceEndPoint" class="org.apache.unomi.router.rest.ImportConfigurationServiceEndPoint">
<property name="importConfigurationService" ref="importConfigurationService"/>
<property name="uploadDir" value="${import.oneshot.uploadDir}"/>
+ <property name="serverPort" value="${contextserver.port}"/>
</bean>
</blueprint>