You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@unomi.apache.org by sh...@apache.org on 2019/07/09 13:22:37 UTC
[unomi] 01/01: UNOMI-208 Improve Unomi documentation - Add
configuration to generate Swagger documentation for Privacy service and
Salesforce REST service documentation using the JAX RS Analyzer Maven
plugin - Improvements to documentation with lots of samples. - Corrections
in the request examples
This is an automated email from the ASF dual-hosted git repository.
shuber pushed a commit to branch UNOMI-208-improve-documentation
in repository https://gitbox.apache.org/repos/asf/unomi.git
commit b367c08850ef15eb2c3ebdd0ffabb975f6fd9a80
Author: sergehuber <sh...@jahia.com>
AuthorDate: Tue Jul 9 15:19:13 2019 +0200
UNOMI-208 Improve Unomi documentation
- Add configuration to generate Swagger documentation for Privacy service and Salesforce REST service documentation using the JAX RS Analyzer Maven plugin
- Improvements to documentation with lots of samples.
- Corrections in the request examples
---
extensions/privacy-extension/rest/pom.xml | 21 ++
.../unomi/privacy/rest/PrivacyServiceEndPoint.java | 1 +
extensions/salesforce-connector/rest/pom.xml | 16 +
.../org/apache/unomi/sfdc/rest/SFDCEndPoint.java | 1 +
manual/src/main/asciidoc/actions.adoc | 69 +++++
manual/src/main/asciidoc/concepts.adoc | 208 ++++++++++++-
manual/src/main/asciidoc/conditions.adoc | 103 +++++++
.../main/asciidoc/how-profile-tracking-works.adoc | 34 ++
.../src/main/asciidoc/images/unomi-rule-engine.png | Bin 0 -> 134094 bytes
manual/src/main/asciidoc/index.adoc | 14 +
manual/src/main/asciidoc/privacy.adoc | 80 +++++
manual/src/main/asciidoc/recipes.adoc | 343 +++++++++++++++++++++
manual/src/main/asciidoc/request-examples.adoc | 3 +-
13 files changed, 878 insertions(+), 15 deletions(-)
diff --git a/extensions/privacy-extension/rest/pom.xml b/extensions/privacy-extension/rest/pom.xml
index eb5779f..ec4eb87 100644
--- a/extensions/privacy-extension/rest/pom.xml
+++ b/extensions/privacy-extension/rest/pom.xml
@@ -81,4 +81,25 @@
</dependency>
</dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.sebastian-daschner</groupId>
+ <artifactId>jaxrs-analyzer-maven-plugin</artifactId>
+ <version>0.17</version>
+ <executions>
+ <execution>
+ <goals>
+ <goal>analyze-jaxrs</goal>
+ </goals>
+ <configuration>
+ <backend>swagger</backend>
+ <deployedDomain>unomi.apache.org</deployedDomain>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
</project>
\ No newline at end of file
diff --git a/extensions/privacy-extension/rest/src/main/java/org/apache/unomi/privacy/rest/PrivacyServiceEndPoint.java b/extensions/privacy-extension/rest/src/main/java/org/apache/unomi/privacy/rest/PrivacyServiceEndPoint.java
index 3fe1855..5e9e1b6 100644
--- a/extensions/privacy-extension/rest/src/main/java/org/apache/unomi/privacy/rest/PrivacyServiceEndPoint.java
+++ b/extensions/privacy-extension/rest/src/main/java/org/apache/unomi/privacy/rest/PrivacyServiceEndPoint.java
@@ -37,6 +37,7 @@ import java.util.List;
allowAllOrigins = true,
allowCredentials = true
)
+@Path("/")
public class PrivacyServiceEndPoint {
private PrivacyService privacyService;
diff --git a/extensions/salesforce-connector/rest/pom.xml b/extensions/salesforce-connector/rest/pom.xml
index 68c50fa..3e023fe 100644
--- a/extensions/salesforce-connector/rest/pom.xml
+++ b/extensions/salesforce-connector/rest/pom.xml
@@ -126,6 +126,22 @@
</instructions>
</configuration>
</plugin>
+ <plugin>
+ <groupId>com.sebastian-daschner</groupId>
+ <artifactId>jaxrs-analyzer-maven-plugin</artifactId>
+ <version>0.17</version>
+ <executions>
+ <execution>
+ <goals>
+ <goal>analyze-jaxrs</goal>
+ </goals>
+ <configuration>
+ <backend>swagger</backend>
+ <deployedDomain>unomi.apache.org</deployedDomain>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
</plugins>
</build>
diff --git a/extensions/salesforce-connector/rest/src/main/java/org/apache/unomi/sfdc/rest/SFDCEndPoint.java b/extensions/salesforce-connector/rest/src/main/java/org/apache/unomi/sfdc/rest/SFDCEndPoint.java
index c7a5c3f..e10c9f4 100644
--- a/extensions/salesforce-connector/rest/src/main/java/org/apache/unomi/sfdc/rest/SFDCEndPoint.java
+++ b/extensions/salesforce-connector/rest/src/main/java/org/apache/unomi/sfdc/rest/SFDCEndPoint.java
@@ -38,6 +38,7 @@ import java.util.Map;
allowAllOrigins = true,
allowCredentials = true
)
+@Path("/")
public class SFDCEndPoint {
private SFDCService sfdcService;
diff --git a/manual/src/main/asciidoc/actions.adoc b/manual/src/main/asciidoc/actions.adoc
new file mode 100644
index 0000000..3cbe239
--- /dev/null
+++ b/manual/src/main/asciidoc/actions.adoc
@@ -0,0 +1,69 @@
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+=== Built-in actions
+
+Unomi comes with quite a lot of built-in actions. Instead of detailing them one by one you will find here an overview of
+what an action descriptor looks like:
+
+[source,json]
+----
+{
+ "metadata": {
+ "id": "UNIQUE_IDENTIFIER_STRING",
+ "name": "DISPLAYABLE_ACTION_NAME",
+ "description": "DISPLAYABLE_ACTION_DESCRIPTION",
+ "systemTags": [
+ "profileTags",
+ "event",
+ "availableToEndUser",
+ "allowMultipleInstances"
+ ],
+ "readOnly": true
+ },
+ "actionExecutor": "ACTION_EXECUTOR_ID",
+ "parameters": [
+ ... parameters specific to each action ...
+ ]
+}
+----
+
+The ACTION_EXECUTOR_ID points to a OSGi Blueprint parameter that is defined when implementing the action in a plugin.
+Here's an example of such a registration:
+
+From https://github.com/apache/unomi/blob/master/plugins/mail/src/main/resources/OSGI-INF/blueprint/blueprint.xml
+
+[source,xml]
+----
+ <bean id="sendMailActionImpl" class="org.apache.unomi.plugins.mail.actions.SendMailAction">
+ <!-- ... bean properties ... -->
+ </bean>
+ <service id="sendMailAction" ref="sendMailActionImpl" interface="org.apache.unomi.api.actions.ActionExecutor">
+ <service-properties>
+ <entry key="actionExecutorId" value="sendMail"/>
+ </service-properties>
+ </service>
+----
+
+In the above example the ACTION_EXECUTOR_ID is `sendMail`
+
+==== Existing actions descriptors
+
+Here is a non-exhaustive list of actions built into Apache Unomi. Feel free to browse the source code if you want to
+discover more. But the list below should get you started with the most useful actions:
+
+- https://github.com/apache/unomi/tree/master/plugins/baseplugin/src/main/resources/META-INF/cxs/actions
+- https://github.com/apache/unomi/tree/master/plugins/request/src/main/resources/META-INF/cxs/actions
+- https://github.com/apache/unomi/tree/master/plugins/mail/src/main/resources/META-INF/cxs/actions
+
+Of course it is also possible to build your own custom actions by developing custom Unomi plugins/extensions.
\ No newline at end of file
diff --git a/manual/src/main/asciidoc/concepts.adoc b/manual/src/main/asciidoc/concepts.adoc
index 6fc8525..1c0af11 100644
--- a/manual/src/main/asciidoc/concepts.adoc
+++ b/manual/src/main/asciidoc/concepts.adoc
@@ -11,17 +11,29 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//
-Apache Unomi gathers information about users actions, information that is processed and stored by Unomi services. The collected information can then be used to personalize content, derive insights on user behavior, categorize the user profiles into segments along user-definable dimensions or acted upon by algorithms.
+Apache Unomi gathers information about users actions, information that is processed and stored by Unomi services. The
+collected information can then be used to personalize content, derive insights on user behavior, categorize the user
+profiles into segments along user-definable dimensions or acted upon by algorithms.
=== Items and types
-Unomi structures the information it collects using the concept of `Item` which provides the base information (an identifier and a type) the context server needs to process and store the data. Items are persisted according to their type (structure) and identifier (identity). This base structure can be extended, if needed, using properties in the form of key-value pairs.
+Unomi structures the information it collects using the concept of `Item` which provides the base information (an
+identifier and a type) the context server needs to process and store the data. Items are persisted according to their
+type (structure) and identifier (identity). This base structure can be extended, if needed, using properties in the
+form of key-value pairs.
-These properties are further defined by the `Item`’s type definition which explicits the `Item`’s structure and semantics. By defining new types, users specify which properties (including the type of values they accept) are available to items of that specific type.
+These properties are further defined by the `Item`’s type definition which explicits the `Item`’s structure and
+semantics. By defining new types, users specify which properties (including the type of values they accept) are
+available to items of that specific type.
-Unomi defines default value types: `date`, `email`, `integer` and `string`, all pretty self-explanatory. While you can think of these value types as "primitive" types, it is possible to extend Unomi by providing additional value types.
+Unomi defines default value types: `date`, `email`, `integer` and `string`, all pretty self-explanatory. While you can
+think of these value types as "primitive" types, it is possible to extend Unomi by providing additional value types.
-Additionally, most items are also associated to a scope, which is a concept that Unomi uses to group together related items. A given scope is represented in Unomi by a simple string identifier and usually represents an application or set of applications from which Unomi gathers data, depending on the desired analysis granularity. In the context of web sites, a scope could, for example, represent a site or family of related sites being analyzed. Scopes allow clients accessing the context [...]
+Additionally, most items are also associated to a scope, which is a concept that Unomi uses to group together related
+items. A given scope is represented in Unomi by a simple string identifier and usually represents an application or set
+of applications from which Unomi gathers data, depending on the desired analysis granularity. In the context of web
+sites, a scope could, for example, represent a site or family of related sites being analyzed. Scopes allow clients
+accessing the context server to filter data to only see relevant data.
_Base `Item` structure:_
@@ -35,7 +47,11 @@ _Base `Item` structure:_
}
----
-Some types can be dynamically defined at runtime by calling to the REST API while other extensions are done via Unomi plugins. Part of extending Unomi, therefore, is a matter of defining new types and specifying which kind of Unomi entity (e.g. profiles) they can be affected to. For example, the following JSON document can be passed to Unomi to declare a new property type identified (and named) `tweetNb`, tagged with the `social` tag, targeting profiles and using the `integer` value type.
+Some types can be dynamically defined at runtime by calling to the REST API while other extensions are done via Unomi
+plugins. Part of extending Unomi, therefore, is a matter of defining new types and specifying which kind of Unomi
+entity (e.g. profiles) they can be affected to. For example, the following JSON document can be passed to Unomi to
+declare a new property type identified (and named) `tweetNb`, tagged with the `social` tag, targeting profiles and
+using the `integer` value type.
_Example JSON type definition:_
@@ -61,7 +77,10 @@ ____
=== Events
-Users' actions are conveyed from clients to the context server using events. Of course, the required information depends on what is collected and users' interactions with the observed systems but events minimally provide a type, a scope and source and target items. Additionally, events are timestamped. Conceptually, an event can be seen as a sentence, the event's type being the verb, the source the subject and the target the object.
+Users' actions are conveyed from clients to the context server using events. Of course, the required information
+depends on what is collected and users' interactions with the observed systems but events minimally provide a type, a
+scope and source and target items. Additionally, events are timestamped. Conceptually, an event can be seen as a
+sentence, the event's type being the verb, the source the subject and the target the object.
_Event structure:_
@@ -76,9 +95,13 @@ _Event structure:_
}
----
-Source and target can be any Unomi item but are not limited to them. In particular, as long as they can be described using properties and Unomi’s type mechanism and can be processed either natively or via extension plugins, source and target can represent just about anything. Events can also be triggered as part of Unomi’s internal processes for example when a rule is triggered.
+Source and target can be any Unomi item but are not limited to them. In particular, as long as they can be described
+using properties and Unomi’s type mechanism and can be processed either natively or via extension plugins, source and
+target can represent just about anything. Events can also be triggered as part of Unomi’s internal processes for example
+when a rule is triggered.
-Events are sent to Unomi from client applications using the JSON format and a typical page view event from a web site could look something like the following:
+Events are sent to Unomi from client applications using the JSON format and a typical page view event from a web site
+could look something like the following:
_Example page view event:_
@@ -114,18 +137,175 @@ _Example page view event:_
=== Profiles
-By processing events, Unomi progressively builds a picture of who the user is and how they behave. This knowledge is embedded in `Profile` object. A profile is an `Item` with any number of properties and optional segments and scores. Unomi provides default properties to cover common data (name, last name, age, email, etc.) as well as default segments to categorize users. Unomi users are, however, free and even encouraged to create additional properties and segments to better suit their needs.
+By processing events, Unomi progressively builds a picture of who the user is and how they behave. This knowledge is
+embedded in `Profile` object. A profile is an `Item` with any number of properties and optional segments and scores.
+Unomi provides default properties to cover common data (name, last name, age, email, etc.) as well as default segments
+to categorize users. Unomi users are, however, free and even encouraged to create additional properties and segments to
+better suit their needs.
-Contrary to other Unomi items, profiles are not part of a scope since we want to be able to track the associated user across applications. For this reason, data collected for a given profile in a specific scope is still available to any scoped item that accesses the profile information.
+Contrary to other Unomi items, profiles are not part of a scope since we want to be able to track the associated user
+across applications. For this reason, data collected for a given profile in a specific scope is still available to any
+scoped item that accesses the profile information.
-It is interesting to note that there is not necessarily a one to one mapping between users and profiles as users can be captured across applications and different observation contexts. As identifying information might not be available in all contexts in which data is collected, resolving profiles to a single physical user can become complex because physical users are not observed directly. Rather, their portrait is progressively patched together and made clearer as Unomi captures more an [...]
+It is interesting to note that there is not necessarily a one to one mapping between users and profiles as users can be
+captured across applications and different observation contexts. As identifying information might not be available in
+all contexts in which data is collected, resolving profiles to a single physical user can become complex because
+physical users are not observed directly. Rather, their portrait is progressively patched together and made clearer as
+Unomi captures more and more traces of their actions. Unomi will merge related profiles as soon as collected data
+permits positive association between distinct profiles, usually as a result of the user performing some identifying
+action in a context where the user hadn’t already been positively identified.
=== Sessions
-A session represents a time-bounded interaction between a user (via their associated profile) and a Unomi-enabled application. A session represents the sequence of actions the user performed during its duration. For this reason, events are associated with the session during which they occurred. In the context of web applications, sessions are usually linked to HTTP sessions.
+A session represents a time-bounded interaction between a user (via their associated profile) and a Unomi-enabled
+application. A session represents the sequence of actions the user performed during its duration. For this reason,
+events are associated with the session during which they occurred. In the context of web applications, sessions are
+usually linked to HTTP sessions.
+
+=== Segments
+
+Segments are used to group profiles together, and are based on conditions that are executed on profiles to determine if
+they are part of a segment or not. This also means that a profile may enter or leave a segment based on changes in their
+properties, making segments a highly dynamic concept.
+
+Here is an example of a simple segment definition registered using the REST API:
+
+[source]
+----
+curl -X POST http://localhost:8181/cxs/segments \
+--user karaf:karaf \
+-H "Content-Type: application/json" \
+-d @- <<'EOF'
+{
+ "metadata": {
+ "id": "leads",
+ "name": "Leads",
+ "scope": "systemscope",
+ "description": "You can customize the list below by editing the leads segment.",
+ "readOnly":true
+ },
+ "condition": {
+ "type": "booleanCondition",
+ "parameterValues": {
+ "operator" : "and",
+ "subConditions": [
+ {
+ "type": "profilePropertyCondition",
+ "parameterValues": {
+ "propertyName": "properties.leadAssignedTo",
+ "comparisonOperator": "exists"
+ }
+ }
+ ]
+ }
+ }
+}
+EOF
+----
+
+For more details on the conditions and how they are structured using conditions, see the next section.
+
+=== Conditions
+
+Conditions are a very useful notion inside of Apache Unomi, as they are used as the basis for multiple other objects.
+Conditions may be used as parts of:
+
+- Segments
+- Rules
+- Queries
+- Campaigns
+- Goals
+- Profile filters
+
+A condition is composed of two basic elements:
+
+- a condition type identifier
+- a list of parameter values for the condition, that can be of any type, and in some cases may include sub-conditions
+
+A condition type identifier is a string that contains a unique identifier for a condition type. Example condition types
+may include `booleanCondition`, `eventTypeCondition`, `eventPropertyCondition`, and so on. Plugins may implement new
+condition types that may implement any logic that may be needed. The parameter values are simply lists of objects that
+may be used to configure the condition. In the case of a `booleanCondition` for example one of the parameter values will
+be an `operator` that will contain values such as `and` or `or` and a second parameter value called `subConditions`
+that contains a list of conditions to evaluate with that operator. The result of a condition is always a boolean
+value of true or false.
+
+Apache Unomi provides quite a lot of built-in condition types, including boolean types that make it possible to
+compose conditions using operators such as `and`, `or` or `not`. Composition is an essential element of building more
+complex conditions.
+
+Here is an example of a complex condition:
+
+[source,json]
+----
+{
+ "condition": {
+ "type": "booleanCondition",
+ "parameterValues": {
+ "operator":"or",
+ "subConditions":[
+ {
+ "type": "eventTypeCondition",
+ "parameterValues": {
+ "eventTypeId": "sessionCreated"
+ }
+ },
+ {
+ "type": "eventTypeCondition",
+ "parameterValues": {
+ "eventTypeId": "sessionReassigned"
+ }
+ }
+ ]
+ }
+ }
+}
+----
+
+As we can see in the above example we use the boolean `or` condition to check if the event type is of type `sessionCreated`
+or `sessionReassigned`.
+
+For a more complete list of available conditions, see the <<Built-in conditions>> reference section.
+
+=== Rules
+
+image::unomi-rule-engine.png[Unomi Rule Engine]
+
+Apache Unomi has a built-in rule engine that is one of the most important components of its architecture. Every time
+an event is received by the server, it is evaluated against all the rules and the ones matching the incoming event will
+be executed. You can think of a rule as a structure that looks like this:
+
+ when
+ conditions
+ then
+ actions
+
+Basically when a rule is evaluated, all the conditions in the `when` part are evaluated and if the result matches
+(meaning it evaluates to `true`) then the actions will be executed in sequence.
+
+The real power of Apache Unomi comes from the fact that `conditions` and `actions` are fully pluggeable and that plugins
+may implement new conditions and/or actions to perform any task. You can imagine conditions checking incoming event data
+against third-party systems or even against authentication systesm, and actions actually pulling or pushing data to third-party
+systems.
+
+For example the Salesforce CRM connector is simply a set of actions that pull and push data into the CRM. It is then
+just a matter of setting up the proper rules with the proper conditions to determine when and how the data will be
+pulled or pushed into the third-party system.
+
+==== Actions
+
+Actions are executed by rules in a sequence, and an action is only executed once the previous action has finished
+executing. If an action generates an exception, it will be logged and the execution sequence will continue unless in the
+case of a Runtime exception (such as a NullPointerException).
+
+Actions are implemented as Java classes, and as such may perform any kind of tasks that may include calling web hooks,
+setting profile properties, extracting data from the incoming request (such as resolving location from an IP address),
+or even pulling and/or pushing data to third-party systems such as a CRM server.
+
+Apache Unomi also comes with built-in actions. You may find the list of built-in actions in the <<Built-in actions>> section.
=== Request flow
Here is an overview of how Unomi processes incoming requests to the `ContextServlet`.
-image::unomi-request.png[Unomi request overview]
\ No newline at end of file
+image::unomi-request.png[Unomi request overview]
diff --git a/manual/src/main/asciidoc/conditions.adoc b/manual/src/main/asciidoc/conditions.adoc
new file mode 100644
index 0000000..52a01b8
--- /dev/null
+++ b/manual/src/main/asciidoc/conditions.adoc
@@ -0,0 +1,103 @@
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+=== Built-in conditions
+
+Apache Unomi comes with an extensive collection of built-in conditions. Instead of detailing them one by one you will
+find here an overview of what a JSON condition descriptor looks like:
+
+[source,json]
+----
+{
+ "metadata": {
+ "id": "booleanCondition",
+ "name": "booleanCondition",
+ "description": "",
+ "systemTags": [
+ "profileTags",
+ "logical",
+ "condition",
+ "profileCondition",
+ "eventCondition",
+ "sessionCondition",
+ "sourceEventCondition"
+ ],
+ "readOnly": true
+ },
+ "conditionEvaluator": "booleanConditionEvaluator",
+ "queryBuilder": "booleanConditionESQueryBuilder",
+ "parameters": [
+ {
+ "id": "operator",
+ "type": "String",
+ "multivalued": false,
+ "defaultValue": "and"
+ },
+ {
+ "id": "subConditions",
+ "type": "Condition",
+ "multivalued": true
+ }
+ ]
+}
+----
+
+Note that conditions have two important identifiers:
+
+- conditionEvaluator
+- queryBuilder
+
+This is because conditions can either be used to build queries or to evaluate a condition in real time. When implementing
+a new condition type, both implementations much be provided. Here's an example an OSGi Blueprint registration for the
+above condition descriptor:
+
+From https://github.com/apache/unomi/blob/master/plugins/baseplugin/src/main/resources/OSGI-INF/blueprint/blueprint.xml
+
+[source,xml]
+----
+...
+ <service
+ interface="org.apache.unomi.persistence.elasticsearch.conditions.ConditionESQueryBuilder">
+ <service-properties>
+ <entry key="queryBuilderId" value="booleanConditionESQueryBuilder"/>
+ </service-properties>
+ <bean class="org.apache.unomi.plugins.baseplugin.conditions.BooleanConditionESQueryBuilder"/>
+ </service>
+...
+ <!-- Condition evaluators -->
+ <service interface="org.apache.unomi.persistence.elasticsearch.conditions.ConditionEvaluator">
+ <service-properties>
+ <entry key="conditionEvaluatorId" value="booleanConditionEvaluator"/>
+ </service-properties>
+ <bean class="org.apache.unomi.plugins.baseplugin.conditions.BooleanConditionEvaluator"/>
+ </service>
+...
+----
+
+As you can see two Java classes are used to build a single condition. You don't need to understand all these details in
+order to use conditions, but this might be interesting to know if you're interested in building your own condition
+implementations. For more details on building your own custom plugins/extensions, please refer to the corresponding
+sections.
+
+==== Existing condition descriptors
+
+Here is a non-exhaustive list of conditions built into Apache Unomi. Feel free to browse the source code if you want to
+discover more. But the list below should get you started with the most useful conditions:
+
+- https://github.com/apache/unomi/tree/master/plugins/baseplugin/src/main/resources/META-INF/cxs/conditions
+
+Of course it is also possible to build your own custom conditions by developing custom Unomi plugins/extensions.
+
+You will also note that some conditions can re-use a `parentCondition`. This is a way to inherit from another condition
+type to make them more specific.
\ No newline at end of file
diff --git a/manual/src/main/asciidoc/how-profile-tracking-works.adoc b/manual/src/main/asciidoc/how-profile-tracking-works.adoc
new file mode 100644
index 0000000..56847df
--- /dev/null
+++ b/manual/src/main/asciidoc/how-profile-tracking-works.adoc
@@ -0,0 +1,34 @@
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+=== How profile tracking works
+
+In this section you will learn how Apache Unomi keeps track of visitors.
+
+Steps:
+
+1. A visitor comes to a website
+2. The web server resolves a previous request session ID if it exists, or if it doesn't it create a new sessionID
+3. A request to Apache Unomi's /context.json servlet is made passing the web server session ID as a query parameter
+4. Unomi uses the sessionID and tries to load an existing session, if none is found a new session is created with the
+ID passed by the web server
+5. If a session was found, the profile ID is extracted from the session and if it not found, Unomi looks for a cookie
+called `context-profile-id` to read the profileID. If no profileID is found or if the session didn't exist, a new
+profile ID is created by Apache Unomi
+6. If the profile ID existed, the corresponding profile is loaded by Apache Unomi, otherwise a new profile is created
+7. If events were passed along with the request to the context.json endpoint, they are processed against the profile
+8. The updated profile is sent back as a response to the context.json request. Along with the response
+
+It is important to note that the profileID is always server-generated. Injecting a custom cookie with a non-valid
+profile ID will result in failure to load the profile. Profile ID are UUIDs, which make them (pretty) safe from brute-
+forcing.
diff --git a/manual/src/main/asciidoc/images/unomi-rule-engine.png b/manual/src/main/asciidoc/images/unomi-rule-engine.png
new file mode 100644
index 0000000..c65dbd8
Binary files /dev/null and b/manual/src/main/asciidoc/images/unomi-rule-engine.png differ
diff --git a/manual/src/main/asciidoc/index.adoc b/manual/src/main/asciidoc/index.adoc
index c3eb2ba..f611a7a 100644
--- a/manual/src/main/asciidoc/index.adoc
+++ b/manual/src/main/asciidoc/index.adoc
@@ -36,6 +36,8 @@ include::concepts.adoc[]
include::getting-started.adoc[]
+include::recipes.adoc[]
+
include::request-examples.adoc[]
include::web-tracker.adoc[]
@@ -44,14 +46,26 @@ include::configuration.adoc[]
include::useful-unomi-urls.adoc[]
+include::how-profile-tracking-works.adoc[]
+
== Consent management
include::consent-api.adoc[]
+== Privacy management
+
+include::privacy.adoc[]
+
== Cluster setup
include::clustering.adoc[]
+== Reference
+
+include::conditions.adoc[]
+
+include::actions.adoc[]
+
== Integration samples
include::samples/samples.adoc[]
diff --git a/manual/src/main/asciidoc/privacy.adoc b/manual/src/main/asciidoc/privacy.adoc
new file mode 100644
index 0000000..93eb353
--- /dev/null
+++ b/manual/src/main/asciidoc/privacy.adoc
@@ -0,0 +1,80 @@
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+Apache Unomi provides an endpoint to manage visitor privacy. You will find in this section information about what it
+includes as well as how to use it.
+
+==== Setting up access to the privacy endpoint
+
+The privacy endpoint is a bit special, because despite being protected by basic authentication as the rest of the REST
+API is is actually designed to be available to end-users.
+
+So in effect it should usually be proxied so that public internet users can access the endpoint but the proxy should
+also check if the profile ID wasn't manipulated in some way.
+
+Apache Unomi doesn't provide (for the moment) such a proxy, but basically it should do the following:
+
+1. check for potential attack activity (could be based on IDS policies or even rate detection), and at the minimum check
+that the profile ID cookie seems authentic (for example by checking that it is often coming from the same IP or the same
+geographic location)
+2. proxy to /cxs/privacy
+
+==== Anonymizing a profile
+
+It is possible to anonymize a profile, meaning it will remove all "identifying" property values from the profile.
+Basically all properties with the tag `personalIdentifierProperties` will be purged from the profile.
+
+Here's an example of a request to anonymize a profile:
+
+[source]
+----
+curl -X POST http://localhost:8181/cxs/profiles/{profileID}/anonymize?scope=ASCOPE
+----
+
+where `{profileID}` must be replaced by the actual identifier of a profile
+and `ASCOPE` must be replaced by a scope identifier.
+
+==== Downloading profile data
+
+It is possible to download the profile data of a user. This will only download the profile for a user using the
+specified ID as a cookie value.
+
+Warning: this operation can also be sensitive so it would be better to protected with a proxy that can perform some
+validation on the requests to make sure no one is trying to download a profile using some kind of "guessing" of profile
+IDs.
+
+[source]
+----
+curl -X GET http://localhost:8181/client/myprofile.[json,csv,yaml,text] \
+--cookie "context-profile-id=PROFILE-ID"
+----
+
+where `PROFILE-ID` is the profile identifier for which to download the profile.
+
+==== Deleting a profile
+
+It is possible to delete a profile, but this works a little differently than you might expect. In all cases the data
+contained in the profile will be completely erased. If the `withData` optional flag is set to true, all past event and
+session data will also be detached from the current profile and anonymized.
+
+[source]
+----
+curl -X DELETE http://localhost:8181/cxs/profiles/{profileID}?withData=false --user karaf:karaf
+----
+
+where `{profileID}` must be replaced by the actual identifier of a profile
+and the `withData` specifies whether the data associated with the profile must be anonymized or not
+
+==== Related
+
+You might also be interested in the <<Consent API>> section that describe how to manage profile consents.
diff --git a/manual/src/main/asciidoc/recipes.adoc b/manual/src/main/asciidoc/recipes.adoc
new file mode 100644
index 0000000..7a9e6af
--- /dev/null
+++ b/manual/src/main/asciidoc/recipes.adoc
@@ -0,0 +1,343 @@
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+=== Recipes
+
+==== Introduction
+
+In this section of the documentation we provide quick recipes focused on helping you achieve a specific result with
+Apache Unomi.
+
+==== How to read a profile
+
+The simplest way to retrieve profile data for the current profile is to simply send a request to the /context.json
+endpoint. However you will need to send a body along with that request. Here's an example:
+
+Here is an example that will retrieve all the session and profile properties.
+
+[source]
+----
+curl -X POST http://localhost:8181/context.json?sessionId=1234 \
+-H "Content-Type: application/json" \
+-d @- <<'EOF'
+{
+ "source": {
+ "itemId":"homepage",
+ "itemType":"page",
+ "scope":"example"
+ },
+ "requiredProfileProperties":["*"],
+ "requiredSessionProperties":["*"],
+ "requireSegments":true
+}
+EOF
+----
+
+The `requiredProfileProperties` and `requiredSessionProperties` are properties that take an array of property names
+that should be retrieved. In this case we use the wildcard character '*' to say we want to retrieve all the available
+properties. The structure of the JSON object that you should send is a JSON-serialized version of the http://unomi.incubator.apache.org/unomi-api/apidocs/org/apache/unomi/api/ContextRequest.html[ContextRequest]
+Java class.
+
+Note that it is also possible to access a profile's data through the /cxs/profiles/ endpoint but that really should be
+reserved to administrative purposes. All public accesses should always use the /context.json endpoint for consistency
+and security.
+
+==== How to update a profile from the public internet
+
+Before we get into how to update a profile directly from a request coming from the public internet, we'll quickly talk
+first about how NOT to do it, because we often see users using the following anti-patterns.
+
+===== How NOT to update a profile from the public internet
+
+Please avoid using the /cxs/profile endpoint. This endpoint was initially the only way to update a profile but it has
+multiple issues:
+
+- it requires authenticated access. The temptation can be great to use this endpoint because it is simple to access
+but the risk is that developers might include the credentials to access it in non-secure parts of code such as
+client-side code. Since there is no difference between this endpoint and any other administration-focused endpoints,
+attackers could easily re-use stolen credentials to wreak havock on the whole platform.
+- No history of profile modifications is kept: this can be a problem for multiple reasons: you might want to keep an
+trail of profile modifications, or even a history of profile values in case you want to understand how a profile
+property was modified.
+- Even when protected using some kind of proxy, potentially the whole profile properties might be modified, including
+ones that you might not want to be overriden.
+
+===== Recommended ways to update a profile
+
+Instead you can use the following solutions to update profiles:
+
+- (Preferred) Use you own custom event(s) to send data you want to be inserted in a profile, and use rules to map the
+event data to the profile. This is simpler than it sounds, as usually all it requires is setting up a simple rule and
+you're ready to update profiles using events. This is also the safest way to update a profile because if you design your
+events to be as specific as possible to your needs, only the data that you specified will be copied to the profile,
+making sure that even in the case an attacker tries to send more data using your custom event it will simply be ignored.
+
+- Use the protected built-in "updateProperties" event. This event is designed to be used for administrative purposes
+only. Again, prefer the custom events solution because as this is a protected event it will require sending the Unomi
+key as a request header, and as Unomi only supports a single key for the moment it could be problematic if the key is
+intercepted. But at least by using an event you will get the benefits of auditing and historical property modification
+tracing.
+
+Let's go into more detail about the preferred way to update a profile. Let's consider the following example of a rule:
+
+[source]
+----
+curl -X POST http://localhost:8181/cxs/rules \
+--user karaf:karaf \
+-H "Content-Type: application/json" \
+-d @- <<'EOF'
+{
+ "metadata": {
+ "id": "setContactInfo",
+ "name": "Copy the received contact info to the current profile",
+ "description": "Copies the contact info received in a custom event called 'contactInfoSubmitted' to the current profile"
+ },
+ "raiseEventOnlyOnceForSession": false,
+ "condition": {
+ "type": "eventTypeCondition",
+ "parameterValues": {
+ "eventTypeId": "contactInfoSubmitted"
+ }
+ },
+ "actions": [
+ {
+ "type": "setPropertyAction",
+ "parameterValues": {
+ "setPropertyName": "properties(firstName)",
+ "setPropertyValue": "eventProperty::properties(firstName)",
+ "setPropertyStrategy": "alwaysSet"
+ }
+ },
+ {
+ "type": "setPropertyAction",
+ "parameterValues": {
+ "setPropertyName": "properties(lastName)",
+ "setPropertyValue": "eventProperty::properties(lastName)",
+ "setPropertyStrategy": "alwaysSet"
+ }
+ },
+ {
+ "type": "setPropertyAction",
+ "parameterValues": {
+ "setPropertyName": "properties(email)",
+ "setPropertyValue": "eventProperty::properties(email)",
+ "setPropertyStrategy": "alwaysSet"
+ }
+ }
+ ]
+}
+EOF
+----
+
+What this rule does is that it listen for a custom event (events don't need any registration, you can simply start
+sending them to Apache Unomi whenever you like) of type 'contactInfoSubmitted' and it will search for properties called
+'firstName', 'lastName' and 'email' and copy them over to the profile with corresponding property names. You could of
+course change any of the property names to find your needs. For example you might want to prefix the profile properties
+with the source of the event, such as 'mobileApp:firstName'.
+
+You could then simply send the `contactInfoSubmitted` event using a request similar to this one:
+
+[source]
+----
+curl -X POST http://localhost:8181/eventcollector \
+-H "Content-Type: application/json" \
+-d @- <<'EOF'
+{
+ "sessionId" : "1234",
+ "events":[
+ {
+ "eventType":"contactInfoSubmitted",
+ "scope": "example",
+ "source":{
+ "itemType": "site",
+ "scope":"example",
+ "itemId": "mysite"
+ },
+ "target":{
+ "itemType":"form",
+ "scope":"example",
+ "itemId":"contactForm",
+ },
+ "properties" : {
+ "firstName" : "John",
+ "lastName" : "Doe",
+ "email" : "john.doe@acme.com"
+ }
+ }
+ ]
+}
+EOF
+----
+
+
+==== How to search for profile events
+
+Sometimes you want to retrieve events for a known profile. You will need to provide a query in the body of the request
+that looks something like this (and https://unomi.incubator.apache.org/rest-api-doc/#1768188821[documentation is available in the REST API]) :
+
+[source]
+----
+curl -X POST http://localhost:8181/cxs/event/search \
+--user karaf:karaf \
+-H "Content-Type: application/json" \
+-d @- <<'EOF'
+{ "offset" : 0,
+ "limit" : 20,
+ "condition" : {
+ "type": "eventPropertyCondition",
+ "parameterValues" : {
+ "propertyName" : "profileId",
+ "comparisonOperator" : "equals",
+ "propertyValue" : "PROFILE_ID"
+ }
+ }
+}
+EOF
+----
+
+where PROFILE_ID is a profile identifier. This will indeed retrieve all the events for a given profile.
+
+==== How to create a new rule
+
+There are basically two ways to create a new rule :
+
+- Using the REST API
+- Packaging it as a predefined rule in a plugin
+
+In both cases the JSON structure for the rule will be exactly the same, and in most scenarios it will be more
+interesting to use the REST API to create and manipulate rules, as they don't require any development or deployments
+on the Apache Unomi server.
+
+[source]
+----
+curl -X POST http://localhost:8181/cxs/rules \
+--user karaf:karaf \
+-H "Content-Type: application/json" \
+-d @- <<'EOF'
+{
+ "metadata": {
+ "id": "exampleEventCopy",
+ "name": "Example Copy Event to Profile",
+ "description": "Copy event properties to profile properties"
+ },
+ "condition": {
+ "type": "eventTypeCondition",
+ "parameterValues": {
+ "eventTypeId" : "myEvent"
+ }
+ },
+ "actions": [
+ {
+ "parameterValues": {
+ },
+ "type": "allEventToProfilePropertiesAction"
+ }
+ ]
+}
+EOF
+----
+
+The above rule will be executed if the incoming event is of type `myEvent` and will simply copy all the properties
+contained in the event to the current profile.
+
+==== How to search for profiles
+
+In order to search for profiles you will have to use the /cxs/profiles/search endpoint that requires a Query JSON
+structure. Here's an example of a profile search with a Query object:
+
+[source]
+----
+curl -X POST http://localhost:8181/cxs/rules \
+--user karaf:karaf \
+-H "Content-Type: application/json" \
+-d @- <<'EOF'
+{
+ "text" : "unomi",
+ "offset" : 0,
+ "limit" : 10,
+ "sortby" : "properties.lastName:asc,properties.firstName:desc",
+ "condition" : {
+ "type" : "booleanCondition",
+ "parameterValues" : {
+ "operator" : "and",
+ "subConditions" : [
+ {
+ "type": "profilePropertyCondition",
+ "parameterValues": {
+ "propertyName": "properties.leadAssignedTo",
+ "comparisonOperator": "exists"
+ }
+ },
+ {
+ "type": "profilePropertyCondition",
+ "parameterValues": {
+ "propertyName": "properties.lastName",
+ "comparisonOperator": "exists"
+ }
+ },
+ ]
+ }
+ }
+}
+EOF
+----
+
+In the above example, you search for all the profiles that have the `leadAssignedTo` and `lastName` properties and that
+have the `unomi` value anywhere in their profile property values. You are also specifying that you only want 10 results
+beginning at offset 0. The results will be also sorted in alphabetical order for the `lastName` property value, and then
+by reverse alphabetical order for the `firstName` property value.
+
+As you can see, queries can be quite complex. Please remember that the more complex the more resources it will consume
+on the server and potentially this could affect performance.
+
+==== Getting / updating consents
+
+You can find information on how to retrieve or create/update consents in the <<Consent API>> section.
+
+==== How to send a login event to Unomi
+
+Tracking logins must be done carefully with Unomi. A login event is considered a "privileged" event and therefore for
+not be initiated from the public internet. Ideally user authentication should always be validated by a trusted third-
+party even if it is a well-known social platform such as Facebook or Twitter. Basically what should NEVER be done:
+
+1. Login to a social platform
+2. Call back to the originating page
+3. Send a login event to Unomi from the page originating the login in step 1
+
+The problem with this, is that any attacker could simply directly call step 3 without any kind of security. Instead the
+flow should look something like this:
+
+1. Login to a social platform
+2. Call back to a special secured system that performs an server-to-server call to send the login event to Apache
+Unomi using the Unomi key.
+
+For simplicity reasons, in our login example, the first method is used, but it really should never be done like this
+in production because of the aforementioned security issues. The second method, although a little more involved, is
+much preferred.
+
+When sending a login event, you can setup a rule that can check a profile property to see if profiles can be merged on an
+universal identifier such as an email address.
+
+In our login sample we provide an example of such a rule. You can find it here:
+
+https://github.com/apache/unomi/blob/master/samples/login-integration/src/main/resources/META-INF/cxs/rules/exampleLogin.json
+
+As you can see in this rule, we call an action called :
+
+ mergeProfilesOnPropertyAction
+
+with as a parameter value the name of the property on which to perform the merge (the email). What this means is that
+upon successful login using an email, Unomi will look for other profiles that have the same email and merge them into
+a single profile. Because of the merge, this should only be done for authenticated profiles, otherwise this could be a
+security issue since it could be a way to load data from other profiles by merging their data !
+
diff --git a/manual/src/main/asciidoc/request-examples.adoc b/manual/src/main/asciidoc/request-examples.adoc
index 123ff02..a534acc 100644
--- a/manual/src/main/asciidoc/request-examples.adoc
+++ b/manual/src/main/asciidoc/request-examples.adoc
@@ -119,10 +119,11 @@ respond quickly and minimize network traffic. Here is an example of using this s
[source]
----
-curl -X POST http://localhost:8181/context.json?sessionId=1234 \
+curl -X POST http://localhost:8181/eventcollector \
-H "Content-Type: application/json" \
-d @- <<'EOF'
{
+ "sessionId" : "1234",
"events":[
{
"eventType":"view",