You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@unomi.apache.org by js...@apache.org on 2022/07/07 12:09:04 UTC
[unomi] 01/01: UNOMI-625 : add documentation on json schema
This is an automated email from the ASF dual-hosted git repository.
jsinovassinnaik pushed a commit to branch UNOMI-625-add-json-schema-documentation
in repository https://gitbox.apache.org/repos/asf/unomi.git
commit 7cb3cd1e86ea950b2604721ee3fc11f0119f1338
Author: jsinovassin <js...@jahia.com>
AuthorDate: Thu Jul 7 14:08:35 2022 +0200
UNOMI-625 : add documentation on json schema
---
.../asciidoc/images/process-creation-extension.png | Bin 0 -> 145044 bytes
.../asciidoc/images/process-creation-schema.png | Bin 0 -> 136021 bytes
manual/src/main/asciidoc/index.adoc | 2 +
.../jsonSchema/extend-an-existing-schema.adoc | 87 +++++++
.../src/main/asciidoc/jsonSchema/introduction.adoc | 280 +++++++++++++++++++++
.../main/asciidoc/jsonSchema/json-schema-api.adoc | 138 ++++++++++
.../src/main/asciidoc/jsonSchema/json-schema.adoc | 21 ++
7 files changed, 528 insertions(+)
diff --git a/manual/src/main/asciidoc/images/process-creation-extension.png b/manual/src/main/asciidoc/images/process-creation-extension.png
new file mode 100644
index 000000000..0c7357a85
Binary files /dev/null and b/manual/src/main/asciidoc/images/process-creation-extension.png differ
diff --git a/manual/src/main/asciidoc/images/process-creation-schema.png b/manual/src/main/asciidoc/images/process-creation-schema.png
new file mode 100644
index 000000000..4158eb50b
Binary files /dev/null and b/manual/src/main/asciidoc/images/process-creation-schema.png differ
diff --git a/manual/src/main/asciidoc/index.adoc b/manual/src/main/asciidoc/index.adoc
index 2976cc0fa..111ba170e 100644
--- a/manual/src/main/asciidoc/index.adoc
+++ b/manual/src/main/asciidoc/index.adoc
@@ -38,6 +38,8 @@ include::request-examples.adoc[]
include::configuration.adoc[]
+include::jsonSchema/json-schema.adoc[]
+
include::migrations/migrations.adoc[]
== Queries and aggregations
diff --git a/manual/src/main/asciidoc/jsonSchema/extend-an-existing-schema.adoc b/manual/src/main/asciidoc/jsonSchema/extend-an-existing-schema.adoc
new file mode 100644
index 000000000..d82501f97
--- /dev/null
+++ b/manual/src/main/asciidoc/jsonSchema/extend-an-existing-schema.adoc
@@ -0,0 +1,87 @@
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+=== Extend an existing schema
+
+==== When a extension is needed?
+
+Apache Unomi provides predefined schemas to validate some known events such as a view event.
+
+The Apache Unomi JSON schemas are designed to consider invalid any properties which are not defined in the JSON schema.
+So if an unknown property is part of the event, the event will be considered as invalid.
+
+This means that if your events include additional properties, you will need extensions to describe these.
+
+==== Understanding how extensions are merged in unomi
+
+An extension schema is a JSON schema whose id will be overridden and be defined by a keyword named *extends* in the *self* part of the extension.
+
+When sending an extension through the API, it will be persisted in Elasticsearch then will be merged to the targeted schema.
+
+What does “merge a schema” mean?
+The merge will simply add in the *allOf* keyword of the targeted schema a reference to the extensions.
+It means that to be valid, an event should be valid against the base schema and against the ones added in the *allOf*.
+
+Example of an extension to allow to add a new property in the view event properties:
+
+[source]
+----
+{
+ "$id": "https://vendor.test.com/schemas/json/events/dummy/extension/1-0-0",
+ "$schema": "https://json-schema.org/draft/2019-09/schema",
+ "self":{
+ "vendor":"com.vendor.test",
+ "name":"dummyExtension",
+ "format":"jsonschema",
+ "extends": "https://unomi.apache.org/schemas/json/events/view/properties/1-0-0",
+ "version":"1-0-0"
+ },
+ "title": "DummyEventExtension",
+ "type": "object",
+ "properties": {
+ "myNewProp": {
+ "type": "string"
+ }
+ }
+}
+----
+
+When validating the events of type view, the extension will be added to the schema with the id *\https://unomi.apache.org/schemas/json/events/view/properties/1-0-0* like the following:
+
+[source]
+----
+"allOf": [{
+ "$ref": "https://vendor.test.com/schemas/json/events/dummy/extension/1-0-0"
+}]
+----
+
+With this extension the property *myNewProp* can now be added to the event.
+
+[source]
+----
+…
+"properties": {
+ "myNewProp" : "newValue"
+},
+…
+----
+
+Process when adding extension:
+
+image::process-creation-extension.png[pdfwidth=35%,align=center]
+
+==== How to add an extension through the API
+
+Since an extension is also a JSON schema, it is possible to add extensions by calling the endpoint to add a JSON schema.
+By calling `POST {{url}}/cxs/jsonSchema` with the JSON schema in the payload of the request, the extension will be persisted and will be merged to the targeted schema.
diff --git a/manual/src/main/asciidoc/jsonSchema/introduction.adoc b/manual/src/main/asciidoc/jsonSchema/introduction.adoc
new file mode 100644
index 000000000..4a233cfdf
--- /dev/null
+++ b/manual/src/main/asciidoc/jsonSchema/introduction.adoc
@@ -0,0 +1,280 @@
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+=== Introduction
+
+==== What is a JSON Schema
+
+https://json-schema.org/specification.html[JSON Schema] is a powerful standard for validating the structure of JSON data.
+Described as a JSON object, a JSON schema file contains format, types, patterns, and more.
+Used against JSON data, a JSON schema validates that the data is compatible with the specified schema.
+
+Example of a basic JSON schema that validate that the path property is a string property:
+
+[source]
+----
+{
+ "$id":"https://unomi.apache.org/schemas/json/example/1-0-0",
+ "$schema":"https://json-schema.org/draft/2019-09/schema",
+ "title":"Example of a basic schema",
+ "type":"object",
+ "properties":{
+ "path":{
+ "type":"string",
+ "$comment":"Example of a property."
+ }
+ }
+}
+----
+
+[source]
+----
+{
+ "path": "example/of/path" //Is valid
+}
+----
+
+[source]
+----
+{
+ "path": 100 // Is not valid
+}
+----
+
+Apache Unomi is using json-schema-validator to integrate JSON schema.
+The library and its source code is available at: https://github.com/networknt/json-schema-validator[https://github.com/networknt/json-schema-validator], you can refer to the feature’s pom.xml available at https://github.com/apache/unomi/blob/master/extensions/json-schema/services/pom.xml#L35[json-schema/service/pom.xml] to identify which version of the library is currently integrated.
+
+You can discover and play with JSON schema using online tools such as https://www.jsonschemavalidator.net/[JSON Schema Validator].
+Such tools allow you to validate a schema against JSON data (such as the example above), and can point to particular errors.
+More details about JSON schema are available on the official specification’s website: https://json-schema.org/specification.html[https://json-schema.org/specification.html]
+
+==== Key concepts
+
+This section details concepts that are important to understand in order to use JSON schema validation with Apache Unomi.
+
+===== $id keyword
+
+The *$id* keyword:
+
+Each schema in Apache Unomi should have a *$id*, the *$id* value is an URI which will be used to retrieve the schema and must be unique.
+
+Example:
+
+[source]
+----
+{
+ "$id":"https://unomi.apache.org/schemas/json/example/1-0-0"
+}
+----
+
+===== $ref keyword
+
+The *$ref* keyword allows you to reference another JSON schema by its *$id* keyword.
+It’s possible to separate complex structures or repetitive parts of schema into other small files and use *$ref* to include them into several json schemas.
+
+Example with a person and an address:
+
+[source]
+----
+{
+ "$id": "https://example.com/schemas/address",
+ "type": "object",
+ "properties": {
+ "street_address": { "type": "string" },
+ "city": { "type": "string" },
+ "state": { "type": "string" }
+ }
+}
+----
+
+[source]
+----
+{
+ "type": "object",
+ "properties": {
+ "first_name":{ "type": "string" },
+ "last_name": { "type": "string" },
+ "shipping_address": {
+ "$ref": "https://example.com/schemas/address"
+ },
+ "billing_address": {
+ "$ref": "https://example.com/schemas/address"
+ }
+ }
+}
+----
+
+More details about *$ref* can be found in the specifications: https://json-schema.org/understanding-json-schema/structuring.html#ref[https://json-schema.org/understanding-json-schema/structuring.html#ref]
+
+===== allOf keyword
+
+The allOf keyword is an array of fields which allows schema composition.
+The data will be valid against a schema if the data are valid against all of the given subschemas in the allOf part and are valid against the properties defined in the schema.
+
+[source]
+----
+{
+ "$id": "https://unomi.apache.org/schemas/json/example/1-0-0",
+ "$schema": "https://json-schema.org/draft/2019-09/schema",
+ "type": "object",
+ "allOf": [
+ {
+ "type": "object",
+ "properties": {
+ "fromAllOf": {
+ "type": "integer",
+ "$comment": "Example of allOf."
+ }
+ }
+ }
+ ],
+ "properties": {
+ "myProperty": {
+ "type": "string",
+ "$comment": "Example of a property."
+ }
+ }
+}
+----
+
+Valid JSON:
+
+[source]
+----
+{
+ "myProperty": "My property",
+ "fromAllOf" : 10
+}
+----
+
+Invalid JSON:
+
+[source]
+----
+{
+ "myProperty": "My property",
+ "fromAllOf" : "My value"
+}
+----
+
+It’s also possible to use a reference *$ref* in the *allOf* keyword to reference another schema.
+
+In UNOMI, there is an example of using *$ref* in the *allOf* keyword to validate the properties which are defined in the event schema.
+This schema contains properties common to all events.
+It’s done in the the view event schema.
+The file can be found on github: https://github.com/apache/unomi/blob/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas/events/view/view.json#L13[view.json]
+More details about allOf can be found in the specifications: https://json-schema.org/understanding-json-schema/reference/combining.html#allof[https://json-schema.org/understanding-json-schema/reference/combining.html#allof]
+
+===== unevaluatedProperties keyword
+
+The *unevaluatedProperties* keyword is useful for schema composition.
+This keyword is similar to *additionalProperties* except that it can recognize properties declared in sub schemas.
+When setting the *unevaluatedProperties* value to *false*, the properties which are not present in the properties part and are not present in the sub schemas will be considered as invalid.
+
+Example with the following schema:
+
+[source]
+----
+{
+ "$id": "https://unomi.apache.org/schemas/json/example/1-0-0",
+ "$schema": "https://json-schema.org/draft/2019-09/schema",
+ "type": "object",
+ "allOf": [
+ {
+ "$ref": "https://unomi.apache.org/schemas/json/subschema/1-0-0"
+ }
+ ],
+ "properties": {
+ "myProperty": {
+ "type": "string",
+ "$comment": "Example of a property."
+ }
+ },
+ "unevaluatedProperties": false
+}
+----
+
+Sub schema:
+
+[source]
+----
+{
+ "$id": "https://unomi.apache.org/schemas/json/subschema/1-0-0",
+ "$schema": "https://json-schema.org/draft/2019-09/schema",
+ "type": "object",
+ "properties": {
+ "fromAllOf": {
+ "type": "string",
+ "$comment": "Example of allOf."
+ }
+ }
+}
+----
+
+With the following data, the validation will fail because the property *myNewProperty* is not defined neither the *properties* part nor the *allOf* part.
+
+[source]
+----
+{
+ "myProperty": "My property",
+ "fromAllOf" : 10,
+ "myNewProperty": "another one" //Not valid
+}
+----
+
+==== How are JSON Schema used in Unomi
+
+JSON Schema is used in UNOMI to validate the data coming from the two public endpoints */contextRequest* and */eventCollector*.
+Both endpoints have a custom deserializer which will begin by validating the payload of the request, then will filter invalid events present in this payload.
+If an event is not valid it will not be processed by the system.
+The internal events are not validated by JSON schema as they are not sent through the public endpoints.
+
+In UNOMI, each event type must have an associated JSON schema.
+To validate an event, UNOMI will search for a schema in which the target of the schema is *events*, and with the name of the schema matching the event type.
+
+A custom keyword named *self* has to be present in the JSON schemas to store the information related to each schema.
+The following example is the *self* part of the view event JSON schema.
+Having the target set to *events* and the name set to *view*, this schema will be used to validate the events of type *view*.
+
+[source]
+----
+…
+"self":{
+ "vendor":"org.apache.unomi",
+ "target" : "events",
+ "name": "view",
+ "format":"jsonschema",
+ "version":"1-0-0"
+},
+…
+----
+
+Link to the schema on github: https://github.com/apache/unomi/blob/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas/events/view/view.json[view.json].
+
+A set of predefined schema are present in UNOMI, these schemas can be found under the folder : https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas[extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas].
+
+These schemas will be loaded in memory at startup.
+Each schema where the *target* value is set to *events*, will be used to validate events.
+The others are simply used as part of JSON schema or can be used in additional JSON schemas.
+
+It’s possible to add JSON schema to validate your own event by using the API, the explanations to manage JSON schema through the API are
+in the <<Create / update a JSON schema to validate an event, Create / update a JSON schema to validate an event>> section.
+
+Contrary to the predefined schemas, the schemas added through the API will be persisted in elasticsearch in the jsonSchema index.
+Schemas persisted in Elasticsearch do not need a restart of the platform to reflect changes.
+
+Process of creation of schemas:
+
+image::process-creation-schema.png[pdfwidth=35%,align=center]
+
diff --git a/manual/src/main/asciidoc/jsonSchema/json-schema-api.adoc b/manual/src/main/asciidoc/jsonSchema/json-schema-api.adoc
new file mode 100644
index 000000000..f9fcb1cb9
--- /dev/null
+++ b/manual/src/main/asciidoc/jsonSchema/json-schema-api.adoc
@@ -0,0 +1,138 @@
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+=== JSON schema API
+
+The JSON schema endpoints are private, so the user has to be authenticated to manage the JSON schema in UNOMI.
+
+==== List existing schemas
+
+The REST endpoint GET `{{url}}/cxs/jsonSchema` allows to get all ids of available schemas and subschemas.
+
+List of predefined schemas:
+
+[source]
+----
+[
+ "https://unomi.apache.org/schemas/json/events/modifyConsent/properties/1-0-0",
+ "https://unomi.apache.org/schemas/json/item/1-0-0",
+ "https://unomi.apache.org/schemas/json/events/login/1-0-0",
+ "https://unomi.apache.org/schemas/json/events/modifyConsent/1-0-0",
+ "https://unomi.apache.org/schemas/json/consentType/1-0-0",
+ "https://unomi.apache.org/schemas/json/items/page/properties/1-0-0",
+ "https://unomi.apache.org/schemas/json/items/page/properties/attributes/1-0-0",
+ "https://unomi.apache.org/schemas/json/events/incrementInterest/1-0-0",
+ "https://unomi.apache.org/schemas/json/events/view/flattenProperties/1-0-0",
+ "https://unomi.apache.org/schemas/json/interests/1-0-0",
+ "https://unomi.apache.org/schemas/json/items/site/1-0-0",
+ "https://unomi.apache.org/schemas/json/items/page/properties/pageInfo/1-0-0",
+ "https://unomi.apache.org/schemas/json/rest/requestIds/1-0-0",
+ "https://unomi.apache.org/schemas/json/rest/eventscollectorrequest/1-0-0",
+ "https://unomi.apache.org/schemas/json/events/view/properties/1-0-0",
+ "https://unomi.apache.org/schemas/json/items/page/1-0-0",
+ "https://unomi.apache.org/schemas/json/URLParameters/1-0-0",
+ "https://unomi.apache.org/schemas/json/event/1-0-0",
+ "https://unomi.apache.org/schemas/json/timestampeditem/1-0-0",
+ "https://unomi.apache.org/schemas/json/events/updateProperties/1-0-0",
+ "https://unomi.apache.org/schemas/json/consent/1-0-0",
+ "https://unomi.apache.org/schemas/json/events/incrementInterest/flattenProperties/1-0-0",
+ "https://unomi.apache.org/schemas/json/events/view/1-0-0"
+]
+----
+
+Custom schemas will also be present in this list once added.
+
+==== Read a schema
+
+It’s possible to get a schema by its id by calling the endpoint `POST {{url}}/cxs/jsonSchema/query` with the id of the schema in the payload of the query.
+
+Example:
+
+[source]
+----
+curl --location --request POST 'http://localhost:8181/cxs/jsonSchema/query' \
+-u 'karaf:karaf'
+--header 'Content-Type: text/plain' \
+--header 'Cookie: context-profile-id=0f2fbca8-c242-4e6d-a439-d65fcbf0f0a8' \
+--data-raw 'https://unomi.apache.org/schemas/json/event/1-0-0'
+----
+
+==== Create / update a JSON schema to validate an event
+
+It’s possible to add or update JSON schema by calling the endpoint `POST {{url}}/cxs/jsonSchema` with the JSON schema in the payload of the request.
+If the JSON schema exists it will be updated with the new one.
+
+Example of creation:
+
+[source]
+----
+curl --location --request POST 'http://localhost:8181/cxs/jsonSchema' \
+-u 'karaf:karaf' \
+--header 'Content-Type: application/json' \
+--header 'Cookie: context-profile-id=0f2fbca8-c242-4e6d-a439-d65fcbf0f0a8' \
+--data-raw '{
+ "$id": "https://vendor.test.com/schemas/json/events/dummy/1-0-0",
+ "$schema": "https://json-schema.org/draft/2019-09/schema",
+ "self": {
+ "vendor": "com.vendor.test",
+ "name": "dummy",
+ "format": "jsonschema",
+ "target": "events",
+ "version": "1-0-0"
+ },
+ "title": "DummyEvent",
+ "type": "object",
+ "allOf": [
+ {
+ "$ref": "https://unomi.apache.org/schemas/json/event/1-0-0"
+ }
+ ],
+ "properties": {
+ "properties": {
+ "$ref": "https://vendor.test.com/schemas/json/events/dummy/properties/1-0-0"
+ }
+ },
+ "unevaluatedProperties": false
+}'
+----
+
+==== Deleting a schema
+
+To delete a schema, call the endpoint `POST {{url}}/cxs/jsonSchema/delete` with the id of the schema into the payload of the request
+
+Example:
+
+[source]
+----
+curl --location --request POST 'http://localhost:8181/cxs/jsonSchema/delete' \
+-u 'karaf:karaf' \
+--header 'Content-Type: text/plain' \
+--header 'Cookie: context-profile-id=0f2fbca8-c242-4e6d-a439-d65fcbf0f0a8' \
+--data-raw 'https://vendor.test.com/schemas/json/events/dummy/1-0-0'
+----
+
+==== Error Management
+
+When calling an endpoint with invalid data, such as an invalid value for the *sessionId* property in the contextRequest object or eventCollectorRequest object, the server would respond with a 400 error code and the message *Request rejected by the server because: Invalid received data*.
+
+==== Details on invalid events
+
+If it’s an event which is incorrect the server will continue to process the request but will exclude the invalid events.
+Running Apache Unomi with the Debug flag will add to the logs the reason why events are rejected.
+You can set the log level of the class validating the events to debug by using the following karaf command:
+
+[source]
+----
+log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl
+----
diff --git a/manual/src/main/asciidoc/jsonSchema/json-schema.adoc b/manual/src/main/asciidoc/jsonSchema/json-schema.adoc
new file mode 100644
index 000000000..a4586a6ae
--- /dev/null
+++ b/manual/src/main/asciidoc/jsonSchema/json-schema.adoc
@@ -0,0 +1,21 @@
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+== JSON schemas
+
+include::introduction.adoc[]
+
+include::json-schema-api.adoc[]
+
+include::extend-an-existing-schema.adoc[]