You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@unomi.apache.org by "Kevan Jahanshahi (Jira)" <ji...@apache.org> on 2022/05/23 09:44:00 UTC
[jira] [Resolved] (UNOMI-571) JSON Schema extension system
[ https://issues.apache.org/jira/browse/UNOMI-571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevan Jahanshahi resolved UNOMI-571.
------------------------------------
Resolution: Fixed
Finally here is the PR for the JSON Schema extension implementation, available for review: [https://github.com/apache/unomi/pull/426]
It’s well covered by tests, validation, extensions, endpoints, save, delete, update, etc and a lot of effort have been done to stabilize the overall json-schema implem ({*}tests are green on the PR{*})
A newly discovered bug have been identified: https://issues.apache.org/jira/browse/UNOMI-572
Dont be surprise to find a TODO in the code of the PR that is referencing this ticket.
> JSON Schema extension system
> ----------------------------
>
> Key: UNOMI-571
> URL: https://issues.apache.org/jira/browse/UNOMI-571
> Project: Apache Unomi
> Issue Type: New Feature
> Reporter: Kevan Jahanshahi
> Assignee: Kevan Jahanshahi
> Priority: Major
> Attachments: form.contact.json, form.json, form.properties.json
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h4. The goal here is to re-introduce the extension system we originally planned to implement to have a better integration with JSON Schema using the power of _[schema composition|https://json-schema.org/understanding-json-schema/reference/combining.html]_ _(allOf{-}, anyOf, oneOf{-})_ keyword.
> The idea is:
> * to store extensions as full JSON schemas
> * update the original extended JSON schemas in RAM with *allOf* -OR *anyOf* OR *oneOf*- referencing the extensions (only one default cardinality will be handle at first: {*}allOf{*})
> h1. Why do we need extensions system for json schema ?
> because Unomi is going to close coimpletely the structure of events and incoming data for security purpose, any unknow property not mapped in a json schema will end up by rejecting the event.
> Due to this we are loosing the power of the events data structure that was free until Unomi 2.
> To counter this limitation we want the basic Object and schema provided by Unomi by default to be extensible so new properties and custom properties could still be used in events that would use Unomi based object.
> h1. Necessary changes, code and implem:
> h3. Create the extension it self:
> The extension that was a piece of JSON schema in first implem should now be a full JSON Schema definition that will only contains the new props to be validated on the same instance (object, event).
> impacts:
> * -remove the endpoints and custom storage of the JSON Schema extensions- *(Already done)*
> * use existing endpoints of JSON Schema to create them
> ** (we will probably have to introduce a new param or schema property to identify those schemas as extensions)
> h4. Before (first implem that have been reverted during the refacto):
> {code:java}
> {
> "id": "extension-test-event-1",
> "schemaId": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0",
> "metadata": {
> "description": "Description of the extension",
> "id": "extension-test-event-1",
> "name": "The name of the extension"
> },
> "priority": "10",
> "extension": {
> "allOf": [
> {
> "$ref": "https://unomi.apache.org/schemas/json/customitem/1-0-0"
> }
> ],
> "properties": {
> "properties": {
> "properties": {
> "floatProperty": {
> "type": "number",
> "maximum": 100,
> "description": "Extension of float property"
> },
> "stringProperty": {
> "type": "string",
> "maxLength": 100,
> "description": "Updated description"
> },
> "newProperty": {
> "type": "string",
> "description": "A new element"
> }
> }
> }
> }
> }
> }
> {code}
> h4. After (expected implem):
> {code:java}
> {
> "$id": "https://unomi.apache.org/schemas/json/events/test-event-extension-1/1-0-0",
> "$schema": "https://json-schema.org/draft/2019-09/schema",
> "self": {
> "vendor": "org.apache.unomi",
> "name": "testEventExtension",
> "format": "jsonschema",
> "target": "events",
> "version": "1-0-0"
> },
> "title": "TestEventExtension1",
> "type": "object",
> "allOf": [
> {
> "$ref": "https://unomi.apache.org/schemas/json/customitem/1-0-0"
> }
> ],
> "properties": {
> "properties": {
> "properties": {
> "floatProperty": {
> "type": "number",
> "maximum": 100,
> "description": "Extension of float property"
> },
> "stringProperty": {
> "type": "string",
> "maxLength": 100,
> "description": "Updated description"
> },
> "newProperty": {
> "type": "string",
> "description": "A new element"
> }
> }
> }
> }
> }
> {code}
> * Some additional informations may be required on JSON Schema that are extensions, like:
> ** {*}-priority-{*}: NOT necessary anymore, there is no notion of priority in the allOf, anyOf, oneOf
> ** {-}*cardinality*{-}: NOT necessary in current implem, only "{*}allOf{*}" will be used and by default. (we will see if it's needed in next steps)
> ** {*}originalSchemaID{*}: the id of the original schema to extend.
> ** This infos may be store in the self part of the JSON Schema that will be extension (final structure could be discussed with the other dev of the team), here is an example:
>
> {code:java}
> "self": {
> "vendor": "org.apache.unomi",
> "name": "testEventExtension",
> "format": "jsonschema",
> "target": "events",
> "version": "1-0-0",
> "unomiExtends": {
> "schema": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0"
> }
> }
> {code}
>
> h3. The merge:
> The merge was originally design to merge both the original JSON shema and the extension schema in a final JSON schema merged and stored in RAM.
> This was a bit complex to implement and could be handle in a more easy and fashioned way using the allOf, anyOf capabilities.
> Now that that the extensions are. real schemas we can just reference them using this notation (reusing above exemple)
> h4. original JSON Schema to be extended:
> {code:java}
> {
> "$id": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0",
> "$schema": "https://json-schema.org/draft/2019-09/schema",
> "self": {
> "vendor": "org.apache.unomi",
> "name": "testEventForExtension",
> "format": "jsonschema",
> "target": "events",
> "version": "1-0-0"
> },
> "title": "TestEvent",
> "type": "object",
> "allOf": [
> {
> "$ref": "https://unomi.apache.org/schemas/json/event/1-0-0"
> }
> ],
> "properties": {
> "properties": {
> "type": "object",
> "properties": {
> "floatProperty": {
> "type": "number"
> },
> "stringProperty": {
> "type": "string",
> "maxLength": 50,
> "description": "Initial description"
> }
> }
> }
> }
> }{code}
> h4. Original JSON Schema extended:
> {code:java}
> {
> "$id": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0",
> "$schema": "https://json-schema.org/draft/2019-09/schema",
> "self": {
> "vendor": "org.apache.unomi",
> "name": "testEventForExtension",
> "format": "jsonschema",
> "target": "events",
> "version": "1-0-0"
> },
> "title": "TestEvent",
> "type": "object",
> "allOf": [
> {
> "$ref": "https://unomi.apache.org/schemas/json/event/1-0-0"
> },
> {
> "$ref": "https://unomi.apache.org/schemas/json/events/test-event-extension-1/1-0-0"
> }
> ],
> "properties": {
> "properties": {
> "type": "object",
> "properties": {
> "floatProperty": {
> "type": "number"
> },
> "stringProperty": {
> "type": "string",
> "maxLength": 50,
> "description": "Initial description"
> }
> }
> }
> }
> }
> {code}
> As you can see we still need a merge operation that will store the results in RAM, but the only part to update in the original schema is the "{*}allOf{*}" property, by adding the extension schema URIs.
> h3. Changes on the original schema:
> In first implem we wanted to use the {*}additionalProperties: false{*}. but this is not possible because this keyword doesn't work with allOf, anyOf or other sub schemas combination.
> So we will have to use: *unevaluatedProperties: false* insteads.
> Read:
> * [https://json-schema.org/understanding-json-schema/reference/object.html#unevaluated-properties]
> * [https://json-schema.org/understanding-json-schema/reference/object.html#additional-properties]
> NOTE that *unevaluatedProperties: false* should only be set on all our original schemas.
> Also we have to consider refactoring all our original schemas to avoid having nested objects in schemas.
> I attached 3 schemas to the ticket as an example of schema and extension for the form event.
> [^form.json] is the Original JSON Schema for the form event it self
> [^form.properties.json] is the Original JSON Schema for the form properties
> [^form.contact.json] is the extension for a given form, providing fields for a contact form and extending the [^form.properties.json] schema.
> This does represent the final structure goal of schemas and extensions.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)