You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@unomi.apache.org by sh...@apache.org on 2020/11/24 15:24:48 UTC
svn commit: r1883788 - in /unomi/website/manual: 1_1_x/index.html
1_2_x/index.html 1_3_x/index.html 1_4_x/index.html
1_5_x/images/expression-filtering-layers.png 1_5_x/index.html
latest/images/expression-filtering-layers.png latest/index.html
Author: shuber
Date: Tue Nov 24 15:24:48 2020
New Revision: 1883788
URL: http://svn.apache.org/viewvc?rev=1883788&view=rev
Log:
Site checkin for project Apache Unomi :: Root Project
Added:
unomi/website/manual/1_5_x/images/expression-filtering-layers.png (with props)
unomi/website/manual/latest/images/expression-filtering-layers.png (with props)
Modified:
unomi/website/manual/1_1_x/index.html
unomi/website/manual/1_2_x/index.html
unomi/website/manual/1_3_x/index.html
unomi/website/manual/1_4_x/index.html
unomi/website/manual/1_5_x/index.html
unomi/website/manual/latest/index.html
Modified: unomi/website/manual/1_1_x/index.html
URL: http://svn.apache.org/viewvc/unomi/website/manual/1_1_x/index.html?rev=1883788&r1=1883787&r2=1883788&view=diff
==============================================================================
--- unomi/website/manual/1_1_x/index.html (original)
+++ unomi/website/manual/1_1_x/index.html Tue Nov 24 15:24:48 2020
@@ -1722,7 +1722,7 @@ done by disabling the elasticsearch auto
</div>
<div id="footer">
<div id="footer-text">
-Last updated 2020-05-14 16:06:48 +0200
+Last updated 2020-11-04 10:32:53 +0100
</div>
</div>
</body>
Modified: unomi/website/manual/1_2_x/index.html
URL: http://svn.apache.org/viewvc/unomi/website/manual/1_2_x/index.html?rev=1883788&r1=1883787&r2=1883788&view=diff
==============================================================================
--- unomi/website/manual/1_2_x/index.html (original)
+++ unomi/website/manual/1_2_x/index.html Tue Nov 24 15:24:48 2020
@@ -2821,7 +2821,7 @@ definition file.</p>
</div>
<div id="footer">
<div id="footer-text">
-Last updated 2020-05-14 16:06:48 +0200
+Last updated 2020-11-04 10:32:53 +0100
</div>
</div>
</body>
Modified: unomi/website/manual/1_3_x/index.html
URL: http://svn.apache.org/viewvc/unomi/website/manual/1_3_x/index.html?rev=1883788&r1=1883787&r2=1883788&view=diff
==============================================================================
--- unomi/website/manual/1_3_x/index.html (original)
+++ unomi/website/manual/1_3_x/index.html Tue Nov 24 15:24:48 2020
@@ -3401,7 +3401,7 @@ definition file.</p>
</div>
<div id="footer">
<div id="footer-text">
-Last updated 2020-05-14 16:06:48 +0200
+Last updated 2020-11-04 10:32:53 +0100
</div>
</div>
</body>
Modified: unomi/website/manual/1_4_x/index.html
URL: http://svn.apache.org/viewvc/unomi/website/manual/1_4_x/index.html?rev=1883788&r1=1883787&r2=1883788&view=diff
==============================================================================
--- unomi/website/manual/1_4_x/index.html (original)
+++ unomi/website/manual/1_4_x/index.html Tue Nov 24 15:24:48 2020
@@ -6145,7 +6145,7 @@ They allow to modify an item, that would
</div>
<div id="footer">
<div id="footer-text">
-Last updated 2020-05-14 16:06:48 +0200
+Last updated 2020-11-24 14:42:54 +0100
</div>
</div>
</body>
Added: unomi/website/manual/1_5_x/images/expression-filtering-layers.png
URL: http://svn.apache.org/viewvc/unomi/website/manual/1_5_x/images/expression-filtering-layers.png?rev=1883788&view=auto
==============================================================================
Binary file - no diff available.
Propchange: unomi/website/manual/1_5_x/images/expression-filtering-layers.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Modified: unomi/website/manual/1_5_x/index.html
URL: http://svn.apache.org/viewvc/unomi/website/manual/1_5_x/index.html?rev=1883788&r1=1883787&r2=1883788&view=diff
==============================================================================
--- unomi/website/manual/1_5_x/index.html (original)
+++ unomi/website/manual/1_5_x/index.html Tue Nov 24 15:24:48 2020
@@ -72,7 +72,14 @@
<li><a href="#_installing_the_maxmind_geoiplite2_ip_lookup_database">3.5. Installing the MaxMind GeoIPLite2 IP lookup database</a></li>
<li><a href="#_installing_geonames_database">3.6. Installing Geonames database</a></li>
<li><a href="#_rest_api_security">3.7. REST API Security</a></li>
-<li><a href="#_scripting_security">3.8. Scripting security</a></li>
+<li><a href="#_scripting_security">3.8. Scripting security</a>
+<ul class="sectlevel3">
+<li><a href="#_multi_layer_scripting_filtering_system">3.8.1. Multi-layer scripting filtering system</a></li>
+<li><a href="#_scripts_and_expressions">3.8.2. Scripts and expressions</a></li>
+<li><a href="#_scripting_expression_filtering_configuration_parameters">3.8.3. Scripting expression filtering configuration parameters</a></li>
+<li><a href="#_scripting_roadmap">3.8.4. Scripting roadmap</a></li>
+</ul>
+</li>
<li><a href="#_automatic_profile_merging">3.9. Automatic profile merging</a></li>
<li><a href="#_securing_a_production_environment">3.10. Securing a production environment</a></li>
<li><a href="#_integrating_with_an_apache_http_web_server">3.11. Integrating with an Apache HTTP web server</a></li>
@@ -1825,24 +1832,243 @@ org.ops4j.pax.web.ssl.keypassword=${env:
</div>
<div class="sect2">
<h3 id="_scripting_security">3.8. Scripting security</h3>
+<div class="sect3">
+<h4 id="_multi_layer_scripting_filtering_system">3.8.1. Multi-layer scripting filtering system</h4>
+<div class="paragraph">
+<p>The scripting security system is multi-layered.</p>
+</div>
+<div class="paragraph">
+<p>For requests coming in through the /context.json endpoint, the following flow is used to secure incoming requests:</p>
+</div>
+<div class="imageblock">
+<div class="content">
+<img src="images/expression-filtering-layers.png" alt="Expression filtering layers">
+</div>
+</div>
+<div class="paragraph">
+<p>Conditions submitted through the context.json public endpoint are first sanitized, meaning that any scripting directly
+injected is removed. However, as conditions can use sub conditions that include scripting, only the first directly
+injected layer of scripts are removed.</p>
+</div>
+<div class="paragraph">
+<p>The second layer is the expression filtering system, that uses an allow-listing mechanism to only accept pre-vetted
+expressions (through configuration and deployment on the server side). Any unrecognized expression will not be accepted.</p>
+</div>
+<div class="paragraph">
+<p>Finally, once the script starts executing in the scripting engine, a filtering class loader will only let the script
+access classes that have been allowed.</p>
+</div>
+<div class="paragraph">
+<p>This multi-layered approach makes it possible to retain a high level of security even if one layer is poorly
+configured or abused.</p>
+</div>
+<div class="paragraph">
+<p>For requests coming in through the secure APIs such as rules, only the condition sanitizing step is skipped,
+otherwise the rest of the filtering system is the same.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_scripts_and_expressions">3.8.2. Scripts and expressions</h4>
+<div class="paragraph">
+<p>Apache Unomi allows using different types of expressions in the following subsystems:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>context.json filters and personalization queries</p>
+</li>
+<li>
+<p>rule conditions and actions parameters</p>
+</li>
+</ul>
+</div>
<div class="paragraph">
-<p>By default, scripting (using in conditions, segments and rules) is controlled by a custom classloader that is quite
-restrictive and using a white-list/black list system. It is controlled through the following property in the
-<code>unomi.custom.system.properties</code> file:</p>
+<p>Apache Unomi uses two integrated scripting languages to provide this functionality: OGNL and MVEL
+OGNL is deprecated and is now disabled by default in 1.5.2 as it is little used (and replaced by better performing
+hardcoded property lookups). MVEL is more commonly used in rule actions as in the following example:</p>
+</div>
+<div class="paragraph">
+<p>From <a href="https://github.com/apache/unomi/blob/unomi-1.5.x/plugins/baseplugin/src/main/resources/META-INF/cxs/rules/sessionAssigned.json">https://github.com/apache/unomi/blob/unomi-1.5.x/plugins/baseplugin/src/main/resources/META-INF/cxs/rules/sessionAssigned.json</a>:</p>
</div>
<div class="listingblock">
<div class="content">
-<pre class="highlight"><code>org.apache.unomi.scripting.allow=${env:UNOMI_ALLOW_SCRIPTING_CLASSES:-org.apache.unomi.api.Event,org.apache.unomi.api.Profile,org.apache.unomi.api.Session,org.apache.unomi.api.Item,org.apache.unomi.api.CustomItem,ognl.*,java.lang.Object,java.util.Map,java.lang.Integer,org.mvel2.*}
-org.apache.unomi.scripting.forbid=${env:UNOMI_FORBID_SCRIPTING_CLASSES:-}</code></pre>
+<pre class="highlight"><code class="language-json" data-lang="json">{
+ "metadata": {
+ "id": "_ajhg9u2s5_sessionAssigned",
+ "name": "Session assigned to a profile",
+ "description": "Update profile visit information",
+ "readOnly":true
+ },
+
+ "condition": {
+ "type": "booleanCondition",
+ "parameterValues": {
+ "subConditions":[
+ {
+ "type": "eventTypeCondition",
+ "parameterValues": {
+ "eventTypeId": "sessionCreated"
+ }
+ },
+ {
+ "type": "eventTypeCondition",
+ "parameterValues": {
+ "eventTypeId": "sessionReassigned"
+ }
+ }
+
+ ],
+ "operator":"or"
+
+ }
+ },
+
+ "actions": [
+ {
+ "parameterValues": {
+ "setPropertyName": "properties.previousVisit",
+ "setPropertyValue": "profileProperty::lastVisit",
+ "storeInSession": false
+ },
+ "type": "setPropertyAction"
+ },
+ {
+ "parameterValues": {
+ "setPropertyName": "properties.lastVisit",
+ "setPropertyValue": "now",
+ "storeInSession": false
+ },
+ "type": "setPropertyAction"
+ },
+ {
+ "parameterValues": {
+ "setPropertyName": "properties.nbOfVisits",
+ "setPropertyValue": "script::profile.properties.?nbOfVisits != null ? (profile.properties.nbOfVisits + 1) : 1",
+ "storeInSession": false
+ },
+ "type": "setPropertyAction"
+ }
+ ]
+
+}</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>As we see in the above example, we use an MVEL script with the setPropertyAction to set a property value.
+Starting with version 1.5.2, any expression use in rules MUST be allow-listed.</p>
+</div>
+<div class="paragraph">
+<p>OGNL was previously used wherever a parameter could be used, but MVEL could only be used with a âscript::â prefix.
+Starting with version 1.5.2 OGNL will no longer be allowed and is replaced by a compatible âhardcodedâ property
+lookup system, while MVEL requires allow-listing the scripts that are to be used.</p>
+</div>
+<div class="paragraph">
+<p>By default, Apache Unomi comes with some built-in allowed expressions that cover all the internal uses cases.</p>
+</div>
+<div class="paragraph">
+<p>Default allowed MVEL expressions (from <a href="https://github.com/apache/unomi/blob/unomi-1.5.x/plugins/baseplugin/src/main/resources/META-INF/cxs/expressions/mvel.json">https://github.com/apache/unomi/blob/unomi-1.5.x/plugins/baseplugin/src/main/resources/META-INF/cxs/expressions/mvel.json</a>) :</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlight"><code class="language-json" data-lang="json">[
+ "\\Q'systemProperties.goals.'+goalId+'TargetReached'\\E",
+ "\\Q'now-'+since+'d'\\E",
+ "\\Q'scores.'+scoringPlanId\\E",
+ "\\QminimumDuration*1000\\E",
+ "\\QmaximumDuration*1000\\E",
+ "\\Qprofile.properties.?nbOfVisits != null ? (profile.properties.nbOfVisits + 1) : 1\\E",
+ "\\Qsession != null ? session.size + 1 : 0\\E",
+ "\\Q'properties.optimizationTest_'+event.target.itemId\\E",
+ "\\Qevent.target.properties.variantId\\E",
+ "\\Qprofile.properties.?systemProperties.goals.\\E[\\w\\_]*\\QReached != null ? (profile.properties.systemProperties.goals.\\E[\\w\\_]*\\QReached) : 'now'\\E",
+ "\\Qprofile.properties.?systemProperties.campaigns.\\E[\\w\\_]*\\QEngaged != null ? (profile.properties.systemProperties.campaigns.\\E[\\w\\_]*\\QEngaged) : 'now'\\E"
+]</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>If you require or are already using custom expressions, you should add a plugin to Apache Unomi to allow for this.
+The choice of a plugin was to make sure only system administrators and solution developers could provide such a
+list, avoiding the possibility to provide it through an API call or another security sensitive deployment mechanism.</p>
</div>
+<div class="paragraph">
+<p>There is another way of allow-listing expressions through configuration, see the âscripting configuration parametersâ section below.</p>
</div>
<div class="paragraph">
-<p>If you encounter any errors while trying to access a class in a condition or an action it might be due to this
-restrictive configuration.</p>
+<p>Procedure to add allowed expressions:</p>
</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>Create a new Apache Unomi plugin project.</p>
+</li>
+<li>
+<p>Create a JSON file in src/main/resources/META-INF/cxs/expressions/mvel.json with an array of regular expressions that will contain the allowed expressions.</p>
+</li>
+<li>
+<p>Build the project and deploy it to Apache Unomi</p>
+</li>
+</ol>
+</div>
+<div class="paragraph">
+<p>Warning: Do not make regular expressions too general. They should actually be as specific as possible to avoid potential injection of malicious code.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_scripting_expression_filtering_configuration_parameters">3.8.3. Scripting expression filtering configuration parameters</h4>
<div class="paragraph">
-<p>If you need, for example when adding a custom item type, to adjust these, please be careful as scripts may be called
-directly from the context.json personalization conditions and therefore should be kept minimal.</p>
+<p>Alongside with the allow-listing technology, there are new configuration parameters to control the security of the scripting engines:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlight"><code># These parameters control the list of classes that are allowed or forbidden when executing expressions.
+org.apache.unomi.scripting.allow=${env:UNOMI_ALLOW_SCRIPTING_CLASSES:-org.apache.unomi.api.Event,org.apache.unomi.api.Profile,org.apache.unomi.api.Session,org.apache.unomi.api.Item,org.apache.unomi.api.CustomItem,ognl.*,java.lang.Object,java.util.Map,java.util.HashMap,java.lang.Integer,org.mvel2.*}
+org.apache.unomi.scripting.forbid=${env:UNOMI_FORBID_SCRIPTING_CLASSES:-}
+
+# This parameter controls the whole expression filtering system. It is not recommended to turn it off. The main reason to turn it off would be to check if it is interfering with something, but it should always be active in production.
+org.apache.unomi.scripting.filter.activated=${env:UNOMI_SCRIPTING_FILTER_ACTIVATED:-true}
+
+# The following parameters control the filtering using regular expressions for each scripting sub-system.
+# The "collections" parameter tells the expression filtering system which configurations to expect. By default only MVEL and/or OGNL are accepted values, but in the future these might be replaced by new scripting sub-systems.
+org.apache.unomi.scripting.filter.collections=${env:UNOMI_SCRIPTING_FILTER_COLLECTIONS:-mvel,ognl}
+
+# For each scripting sub-system, there is an allow and a forbid property that reference a .json files,
+# you can either edit this files or reference your own file directly in the following config.
+# Note: You can add new expressions to the "allow" file, although it is better to add them inside any plugins you may be adding.
+# This configuration is only designed to compensate for the cases where something was not properly designed or to deal with compatibility issues.
+# Just be VERY careful to make your patterns AS SPECIFIC AS POSSIBLE in order to avoid introducing a way to abuse the expression filtering.
+# Note: It is NOT recommended to change the built-in "forbid" value unless you are having issues with its value.
+# Note: mvel-allow.json contains an empty array: [], this mean nothing is allowed, so far.
+# If you want to allow all expression, just remove the property org.apache.unomi.scripting.filter.mvel.allow, but this is not recommended
+# It's better to list your expressions, and provide them in the mvel-allow.json file
+# example: ["\\Qsession.size + 1\\E"]
+org.apache.unomi.scripting.filter.mvel.allow=${env:UNOMI_SCRIPTING_FILTER_MVEL_ALLOW:-${karaf.etc}/mvel-allow.json}
+org.apache.unomi.scripting.filter.mvel.forbid=${env:UNOMI_SCRIPTING_FILTER_MVEL_FORBID:-${karaf.etc}/mvel-forbid.json}
+org.apache.unomi.scripting.filter.ognl.allow=${env:UNOMI_SCRIPTING_FILTER_OGNL_ALLOW:-${karaf.etc}/ognl-allow.json}
+org.apache.unomi.scripting.filter.ognl.forbid=${env:UNOMI_SCRIPTING_FILTER_OGNL_FORBID:-${karaf.etc}/ognl-forbid.json}
+
+# This parameter controls whether OGNL scripting is allowed in expressions. Because of security reasons it is deactivated by default. If you run into compatibility issues you could reactivate it but it is at your own risk.
+org.apache.unomi.security.properties.useOGNLScripting=${env:UNOMI_SCRIPTING_USE_OGNL:-false}
+
+# This parameter controls the condition sanitizing done on the ContextServlet (/context.json). If will remove any expressions that start with "script::". It is not recommended to change this value, unless you run into compatibility issues.
+org.apache.unomi.security.personalization.sanitizeConditions=${env:UNOMI_SECURITY_SANITIZEPERSONALIZATIONCONDITIONS:-true}</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_scripting_roadmap">3.8.4. Scripting roadmap</h4>
+<div class="paragraph">
+<p>Scripting will probably undergo major changes in future versions of Apache Unomi, possibly replacing MVEL with a
+more secure implementation of a scripting language, or possibly even removed completely (see Groovy actions below).</p>
+</div>
+<div class="paragraph">
+<p>It is recommended that scripting be avoided as in most cases it could be replaced by custom action implementations,
+which are also possible using the Groovy Scripting language. The main difference is in the deployment mechanism,
+Groovy actions or Java actions must be deployed as plugins, which require system administrator access and permissions.</p>
+</div>
+<div class="paragraph">
+<p>These changes will not happen on maintenance versions of Apache Unomi, only in the next major version. Maintenance
+versions will of course maintain compatibility with existing scripting solutions.</p>
+</div>
</div>
</div>
<div class="sect2">
@@ -6396,7 +6622,7 @@ only be sent if the user has authenticat
</div>
<div class="listingblock">
<div class="content">
-<pre class="highlight"><code>bundle:install mvn:org.apache.unomi/login-integration-samples/${project.version}</code></pre>
+<pre class="highlight"><code>bundle:install mvn:org.apache.unomi/login-integration-sample/${project.version}</code></pre>
</div>
</div>
<div class="paragraph">
@@ -8747,7 +8973,7 @@ They allow to modify an item, that would
</div>
<div id="footer">
<div id="footer-text">
-Last updated 2020-06-05 17:34:24 +0200
+Last updated 2020-11-24 14:42:54 +0100
</div>
</div>
</body>
Added: unomi/website/manual/latest/images/expression-filtering-layers.png
URL: http://svn.apache.org/viewvc/unomi/website/manual/latest/images/expression-filtering-layers.png?rev=1883788&view=auto
==============================================================================
Binary file - no diff available.
Propchange: unomi/website/manual/latest/images/expression-filtering-layers.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Modified: unomi/website/manual/latest/index.html
URL: http://svn.apache.org/viewvc/unomi/website/manual/latest/index.html?rev=1883788&r1=1883787&r2=1883788&view=diff
==============================================================================
--- unomi/website/manual/latest/index.html (original)
+++ unomi/website/manual/latest/index.html Tue Nov 24 15:24:48 2020
@@ -72,7 +72,14 @@
<li><a href="#_installing_the_maxmind_geoiplite2_ip_lookup_database">3.5. Installing the MaxMind GeoIPLite2 IP lookup database</a></li>
<li><a href="#_installing_geonames_database">3.6. Installing Geonames database</a></li>
<li><a href="#_rest_api_security">3.7. REST API Security</a></li>
-<li><a href="#_scripting_security">3.8. Scripting security</a></li>
+<li><a href="#_scripting_security">3.8. Scripting security</a>
+<ul class="sectlevel3">
+<li><a href="#_multi_layer_scripting_filtering_system">3.8.1. Multi-layer scripting filtering system</a></li>
+<li><a href="#_scripts_and_expressions">3.8.2. Scripts and expressions</a></li>
+<li><a href="#_scripting_expression_filtering_configuration_parameters">3.8.3. Scripting expression filtering configuration parameters</a></li>
+<li><a href="#_scripting_roadmap">3.8.4. Scripting roadmap</a></li>
+</ul>
+</li>
<li><a href="#_automatic_profile_merging">3.9. Automatic profile merging</a></li>
<li><a href="#_securing_a_production_environment">3.10. Securing a production environment</a></li>
<li><a href="#_integrating_with_an_apache_http_web_server">3.11. Integrating with an Apache HTTP web server</a></li>
@@ -1825,24 +1832,243 @@ org.ops4j.pax.web.ssl.keypassword=${env:
</div>
<div class="sect2">
<h3 id="_scripting_security">3.8. Scripting security</h3>
+<div class="sect3">
+<h4 id="_multi_layer_scripting_filtering_system">3.8.1. Multi-layer scripting filtering system</h4>
+<div class="paragraph">
+<p>The scripting security system is multi-layered.</p>
+</div>
+<div class="paragraph">
+<p>For requests coming in through the /context.json endpoint, the following flow is used to secure incoming requests:</p>
+</div>
+<div class="imageblock">
+<div class="content">
+<img src="images/expression-filtering-layers.png" alt="Expression filtering layers">
+</div>
+</div>
+<div class="paragraph">
+<p>Conditions submitted through the context.json public endpoint are first sanitized, meaning that any scripting directly
+injected is removed. However, as conditions can use sub conditions that include scripting, only the first directly
+injected layer of scripts are removed.</p>
+</div>
+<div class="paragraph">
+<p>The second layer is the expression filtering system, that uses an allow-listing mechanism to only accept pre-vetted
+expressions (through configuration and deployment on the server side). Any unrecognized expression will not be accepted.</p>
+</div>
+<div class="paragraph">
+<p>Finally, once the script starts executing in the scripting engine, a filtering class loader will only let the script
+access classes that have been allowed.</p>
+</div>
+<div class="paragraph">
+<p>This multi-layered approach makes it possible to retain a high level of security even if one layer is poorly
+configured or abused.</p>
+</div>
+<div class="paragraph">
+<p>For requests coming in through the secure APIs such as rules, only the condition sanitizing step is skipped,
+otherwise the rest of the filtering system is the same.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_scripts_and_expressions">3.8.2. Scripts and expressions</h4>
+<div class="paragraph">
+<p>Apache Unomi allows using different types of expressions in the following subsystems:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>context.json filters and personalization queries</p>
+</li>
+<li>
+<p>rule conditions and actions parameters</p>
+</li>
+</ul>
+</div>
<div class="paragraph">
-<p>By default, scripting (using in conditions, segments and rules) is controlled by a custom classloader that is quite
-restrictive and using a white-list/black list system. It is controlled through the following property in the
-<code>unomi.custom.system.properties</code> file:</p>
+<p>Apache Unomi uses two integrated scripting languages to provide this functionality: OGNL and MVEL
+OGNL is deprecated and is now disabled by default in 1.5.2 as it is little used (and replaced by better performing
+hardcoded property lookups). MVEL is more commonly used in rule actions as in the following example:</p>
+</div>
+<div class="paragraph">
+<p>From <a href="https://github.com/apache/unomi/blob/unomi-1.5.x/plugins/baseplugin/src/main/resources/META-INF/cxs/rules/sessionAssigned.json">https://github.com/apache/unomi/blob/unomi-1.5.x/plugins/baseplugin/src/main/resources/META-INF/cxs/rules/sessionAssigned.json</a>:</p>
</div>
<div class="listingblock">
<div class="content">
-<pre class="highlight"><code>org.apache.unomi.scripting.allow=${env:UNOMI_ALLOW_SCRIPTING_CLASSES:-org.apache.unomi.api.Event,org.apache.unomi.api.Profile,org.apache.unomi.api.Session,org.apache.unomi.api.Item,org.apache.unomi.api.CustomItem,ognl.*,java.lang.Object,java.util.Map,java.lang.Integer,org.mvel2.*}
-org.apache.unomi.scripting.forbid=${env:UNOMI_FORBID_SCRIPTING_CLASSES:-}</code></pre>
+<pre class="highlight"><code class="language-json" data-lang="json">{
+ "metadata": {
+ "id": "_ajhg9u2s5_sessionAssigned",
+ "name": "Session assigned to a profile",
+ "description": "Update profile visit information",
+ "readOnly":true
+ },
+
+ "condition": {
+ "type": "booleanCondition",
+ "parameterValues": {
+ "subConditions":[
+ {
+ "type": "eventTypeCondition",
+ "parameterValues": {
+ "eventTypeId": "sessionCreated"
+ }
+ },
+ {
+ "type": "eventTypeCondition",
+ "parameterValues": {
+ "eventTypeId": "sessionReassigned"
+ }
+ }
+
+ ],
+ "operator":"or"
+
+ }
+ },
+
+ "actions": [
+ {
+ "parameterValues": {
+ "setPropertyName": "properties.previousVisit",
+ "setPropertyValue": "profileProperty::lastVisit",
+ "storeInSession": false
+ },
+ "type": "setPropertyAction"
+ },
+ {
+ "parameterValues": {
+ "setPropertyName": "properties.lastVisit",
+ "setPropertyValue": "now",
+ "storeInSession": false
+ },
+ "type": "setPropertyAction"
+ },
+ {
+ "parameterValues": {
+ "setPropertyName": "properties.nbOfVisits",
+ "setPropertyValue": "script::profile.properties.?nbOfVisits != null ? (profile.properties.nbOfVisits + 1) : 1",
+ "storeInSession": false
+ },
+ "type": "setPropertyAction"
+ }
+ ]
+
+}</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>As we see in the above example, we use an MVEL script with the setPropertyAction to set a property value.
+Starting with version 1.5.2, any expression use in rules MUST be allow-listed.</p>
+</div>
+<div class="paragraph">
+<p>OGNL was previously used wherever a parameter could be used, but MVEL could only be used with a âscript::â prefix.
+Starting with version 1.5.2 OGNL will no longer be allowed and is replaced by a compatible âhardcodedâ property
+lookup system, while MVEL requires allow-listing the scripts that are to be used.</p>
+</div>
+<div class="paragraph">
+<p>By default, Apache Unomi comes with some built-in allowed expressions that cover all the internal uses cases.</p>
+</div>
+<div class="paragraph">
+<p>Default allowed MVEL expressions (from <a href="https://github.com/apache/unomi/blob/unomi-1.5.x/plugins/baseplugin/src/main/resources/META-INF/cxs/expressions/mvel.json">https://github.com/apache/unomi/blob/unomi-1.5.x/plugins/baseplugin/src/main/resources/META-INF/cxs/expressions/mvel.json</a>) :</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlight"><code class="language-json" data-lang="json">[
+ "\\Q'systemProperties.goals.'+goalId+'TargetReached'\\E",
+ "\\Q'now-'+since+'d'\\E",
+ "\\Q'scores.'+scoringPlanId\\E",
+ "\\QminimumDuration*1000\\E",
+ "\\QmaximumDuration*1000\\E",
+ "\\Qprofile.properties.?nbOfVisits != null ? (profile.properties.nbOfVisits + 1) : 1\\E",
+ "\\Qsession != null ? session.size + 1 : 0\\E",
+ "\\Q'properties.optimizationTest_'+event.target.itemId\\E",
+ "\\Qevent.target.properties.variantId\\E",
+ "\\Qprofile.properties.?systemProperties.goals.\\E[\\w\\_]*\\QReached != null ? (profile.properties.systemProperties.goals.\\E[\\w\\_]*\\QReached) : 'now'\\E",
+ "\\Qprofile.properties.?systemProperties.campaigns.\\E[\\w\\_]*\\QEngaged != null ? (profile.properties.systemProperties.campaigns.\\E[\\w\\_]*\\QEngaged) : 'now'\\E"
+]</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>If you require or are already using custom expressions, you should add a plugin to Apache Unomi to allow for this.
+The choice of a plugin was to make sure only system administrators and solution developers could provide such a
+list, avoiding the possibility to provide it through an API call or another security sensitive deployment mechanism.</p>
</div>
+<div class="paragraph">
+<p>There is another way of allow-listing expressions through configuration, see the âscripting configuration parametersâ section below.</p>
</div>
<div class="paragraph">
-<p>If you encounter any errors while trying to access a class in a condition or an action it might be due to this
-restrictive configuration.</p>
+<p>Procedure to add allowed expressions:</p>
</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>Create a new Apache Unomi plugin project.</p>
+</li>
+<li>
+<p>Create a JSON file in src/main/resources/META-INF/cxs/expressions/mvel.json with an array of regular expressions that will contain the allowed expressions.</p>
+</li>
+<li>
+<p>Build the project and deploy it to Apache Unomi</p>
+</li>
+</ol>
+</div>
+<div class="paragraph">
+<p>Warning: Do not make regular expressions too general. They should actually be as specific as possible to avoid potential injection of malicious code.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_scripting_expression_filtering_configuration_parameters">3.8.3. Scripting expression filtering configuration parameters</h4>
<div class="paragraph">
-<p>If you need, for example when adding a custom item type, to adjust these, please be careful as scripts may be called
-directly from the context.json personalization conditions and therefore should be kept minimal.</p>
+<p>Alongside with the allow-listing technology, there are new configuration parameters to control the security of the scripting engines:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlight"><code># These parameters control the list of classes that are allowed or forbidden when executing expressions.
+org.apache.unomi.scripting.allow=${env:UNOMI_ALLOW_SCRIPTING_CLASSES:-org.apache.unomi.api.Event,org.apache.unomi.api.Profile,org.apache.unomi.api.Session,org.apache.unomi.api.Item,org.apache.unomi.api.CustomItem,ognl.*,java.lang.Object,java.util.Map,java.util.HashMap,java.lang.Integer,org.mvel2.*}
+org.apache.unomi.scripting.forbid=${env:UNOMI_FORBID_SCRIPTING_CLASSES:-}
+
+# This parameter controls the whole expression filtering system. It is not recommended to turn it off. The main reason to turn it off would be to check if it is interfering with something, but it should always be active in production.
+org.apache.unomi.scripting.filter.activated=${env:UNOMI_SCRIPTING_FILTER_ACTIVATED:-true}
+
+# The following parameters control the filtering using regular expressions for each scripting sub-system.
+# The "collections" parameter tells the expression filtering system which configurations to expect. By default only MVEL and/or OGNL are accepted values, but in the future these might be replaced by new scripting sub-systems.
+org.apache.unomi.scripting.filter.collections=${env:UNOMI_SCRIPTING_FILTER_COLLECTIONS:-mvel,ognl}
+
+# For each scripting sub-system, there is an allow and a forbid property that reference a .json files,
+# you can either edit this files or reference your own file directly in the following config.
+# Note: You can add new expressions to the "allow" file, although it is better to add them inside any plugins you may be adding.
+# This configuration is only designed to compensate for the cases where something was not properly designed or to deal with compatibility issues.
+# Just be VERY careful to make your patterns AS SPECIFIC AS POSSIBLE in order to avoid introducing a way to abuse the expression filtering.
+# Note: It is NOT recommended to change the built-in "forbid" value unless you are having issues with its value.
+# Note: mvel-allow.json contains an empty array: [], this mean nothing is allowed, so far.
+# If you want to allow all expression, just remove the property org.apache.unomi.scripting.filter.mvel.allow, but this is not recommended
+# It's better to list your expressions, and provide them in the mvel-allow.json file
+# example: ["\\Qsession.size + 1\\E"]
+org.apache.unomi.scripting.filter.mvel.allow=${env:UNOMI_SCRIPTING_FILTER_MVEL_ALLOW:-${karaf.etc}/mvel-allow.json}
+org.apache.unomi.scripting.filter.mvel.forbid=${env:UNOMI_SCRIPTING_FILTER_MVEL_FORBID:-${karaf.etc}/mvel-forbid.json}
+org.apache.unomi.scripting.filter.ognl.allow=${env:UNOMI_SCRIPTING_FILTER_OGNL_ALLOW:-${karaf.etc}/ognl-allow.json}
+org.apache.unomi.scripting.filter.ognl.forbid=${env:UNOMI_SCRIPTING_FILTER_OGNL_FORBID:-${karaf.etc}/ognl-forbid.json}
+
+# This parameter controls whether OGNL scripting is allowed in expressions. Because of security reasons it is deactivated by default. If you run into compatibility issues you could reactivate it but it is at your own risk.
+org.apache.unomi.security.properties.useOGNLScripting=${env:UNOMI_SCRIPTING_USE_OGNL:-false}
+
+# This parameter controls the condition sanitizing done on the ContextServlet (/context.json). If will remove any expressions that start with "script::". It is not recommended to change this value, unless you run into compatibility issues.
+org.apache.unomi.security.personalization.sanitizeConditions=${env:UNOMI_SECURITY_SANITIZEPERSONALIZATIONCONDITIONS:-true}</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_scripting_roadmap">3.8.4. Scripting roadmap</h4>
+<div class="paragraph">
+<p>Scripting will probably undergo major changes in future versions of Apache Unomi, possibly replacing MVEL with a
+more secure implementation of a scripting language, or possibly even removed completely (see Groovy actions below).</p>
+</div>
+<div class="paragraph">
+<p>It is recommended that scripting be avoided as in most cases it could be replaced by custom action implementations,
+which are also possible using the Groovy Scripting language. The main difference is in the deployment mechanism,
+Groovy actions or Java actions must be deployed as plugins, which require system administrator access and permissions.</p>
+</div>
+<div class="paragraph">
+<p>These changes will not happen on maintenance versions of Apache Unomi, only in the next major version. Maintenance
+versions will of course maintain compatibility with existing scripting solutions.</p>
+</div>
</div>
</div>
<div class="sect2">
@@ -6396,7 +6622,7 @@ only be sent if the user has authenticat
</div>
<div class="listingblock">
<div class="content">
-<pre class="highlight"><code>bundle:install mvn:org.apache.unomi/login-integration-samples/${project.version}</code></pre>
+<pre class="highlight"><code>bundle:install mvn:org.apache.unomi/login-integration-sample/${project.version}</code></pre>
</div>
</div>
<div class="paragraph">
@@ -8931,7 +9157,7 @@ They allow to modify an item, that would
</div>
<div id="footer">
<div id="footer-text">
-Last updated 2020-06-05 17:34:24 +0200
+Last updated 2020-11-24 14:42:54 +0100
</div>
</div>
</body>