You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ossie.apache.org by "jbonofre (via GitHub)" <gi...@apache.org> on 2026/10/03 13:09:36 UTC
Re: [PR] Ontology: decouple Ontologies, Semantic Models, and their Mappings [ossie]
jbonofre commented on code in PR #458:
URL: https://github.com/apache/ossie/pull/458#discussion_r4173291213
##########
ontology/ontology.md:
##########
@@ -442,21 +443,45 @@ Ontology mappings declare how to map the values of fields at the logical level t
in the ontology. Just as ontologies are partitioned by concept, ontology maps partition into concept
mappings that group by some concept.
-Each mapping's `semantic_model` is a complete
-[core document](../core-spec/spec.md#semantic-model), with its own `version`,
-`name`, and at least one dataset. For example:
+### Mapping documents
-```yaml
-ontology_mappings:
- - name: sales_mapping
- semantic_model:
- version: 0.2.0.dev0
- name: sales_analytics
- datasets:
- - name: orders
- source: sales.public.orders
- concept_mappings: []
-```
+A mapping is written as its own document, validated against `ontology/mapping.json`. It maps the
+constructs of one semantic model onto one ontology, and references both rather than embedding
+either:
+
+| Field | Type | Required | Description |
+|---------------|---------|-----|-------|
+| `version` | string | Yes | Mapping specification version |
+| `name` | string | Yes | Unique identifier for this mapping |
+| `description` | string | No | Human-readable description of this mapping |
+| `ontology` | object | Yes | Reference to the ontology document this mapping targets (see below) |
+| `semantic_model` | object | Yes | Reference to the semantic model document this mapping draws from (see below) |
+| `concept_mappings` | list | Yes | Maps logical model constructs to concepts and relationships in the referenced ontology |
+| `custom_extensions` | list | No | Vendor-specific attributes for extensibility, matching the core specification's mechanism |
+
+Both `ontology` and `semantic_model` are references with the following schema, mirroring
+`DocumentReference` in `mapping.json`:
+
+| Field | Type | Required | Description |
+|---------------|---------|-----|-------|
+| `name` | string | Yes | Must equal the referenced document's own `name`; this is the reference's identity |
+| `iri` | string | No | Where to resolve the referenced document from: a relative reference such as `./flights.ontology.yaml` when the documents sit alongside each other, or an absolute IRI. May be omitted where a catalog resolves documents by `name` |
+
+A mapping document references exactly one ontology and exactly one semantic model. When more than
+one semantic model maps to an ontology, each mapping is its own document.
+
+A mapping document is recognized by having `concept_mappings`, a field no ontology or semantic
+model document has; there is no separate field declaring a document's kind.
+
+See `examples/flights.ontology.yaml`, `examples/flights.semantic_model.yaml`, and
+`examples/flights.mapping.yaml` for a complete example.
+
+**Deprecated:** mappings were originally embedded in the ontology document's `ontology_mappings`
+list (`OntologyMap` in `ontology.json`), each carrying a complete
+[core document](../core-spec/spec.md#semantic-model), with its own `version`, `name`, and at
+least one dataset, as its `semantic_model`. That list is still accepted so existing documents
+continue to validate, but it is deprecated and will be removed in a future version. New mappings
+must be written as mapping documents.
Review Comment:
"Must" is ahead of the tooling here. Today `OssieParser().parse("examples/flights.mapping.yaml")` raises a pydantic `ValidationError`, and `flights.ontology.yaml` parses with zero mappings, so someone who follows this sentence loses their mappings on conversion with no error. The orionbelt converter also still emits `ontology_mappings`.
Either add parser support for mapping documents in this PR, or soften the wording until it lands:
```suggestion
continue to validate, but it is deprecated and will be removed in a future version. New mappings
should be written as mapping documents.
```
If we go with the wording change, let's open a follow-up issue for parser and converter support.
##########
validation/validate.py:
##########
@@ -148,20 +148,28 @@ def _check_unique_keys(self, node: yaml.Node, visited: set) -> None:
self._check_unique_keys(child, visited)
+# Local schemas that other schemas reference, keyed by their path in this repo.
+# Cross-file references use the raw GitHub URL, so each schema is registered
+# under both that URL and its canonical $id.
+_LOCAL_SCHEMAS = ("core-spec/ossie-schema.json", "ontology/ontology.json")
+_RAW_BASE = "https://raw.githubusercontent.com/apache/ossie/main/"
+
+
+def _schema_registry() -> Registry:
+ """Build a registry that resolves Ossie schema references from local files."""
+ repo = Path(__file__).parent.parent
+ resources = []
+ for relative_path in _LOCAL_SCHEMAS:
+ contents = json.loads((repo / relative_path).read_text())
+ resource = Resource.from_contents(contents)
+ resources.append((contents["$id"], resource))
+ resources.append((_RAW_BASE + relative_path, resource))
+ return Registry().with_resources(resources)
+
+
def validate_schema(data: dict, schema: dict) -> list[str]:
- """Validate against JSON Schema, resolving core references locally."""
- core_path = Path(__file__).parent.parent / "core-spec" / "ossie-schema.json"
- core = json.loads(core_path.read_text())
- resource = Resource.from_contents(core)
- # Ontology references use the raw URL; also register the canonical schema ID.
- registry = Registry().with_resources([
- (core["$id"], resource),
- (
- "https://raw.githubusercontent.com/apache/ossie/main/core-spec/ossie-schema.json",
- resource,
- ),
- ])
- validator = Draft202012Validator(schema, registry=registry)
+ """Validate against JSON Schema, resolving Ossie schema references locally."""
+ validator = Draft202012Validator(schema, registry=_schema_registry())
Review Comment:
`_schema_registry()` reads and parses every entry in `_LOCAL_SCHEMAS` on each `validate_schema` call. Validating a plain semantic model now depends on `ontology/ontology.json` being present and well-formed: if it is missing or mid-edit, `validate.py examples/tpcds_semantic_model.yaml` dies with an uncaught `FileNotFoundError` / `JSONDecodeError`. Before this PR only the core schema was read.
A lazy registry reads a schema only when a `$ref` needs it, and caches it across calls:
```python
from referencing import Registry
from referencing.exceptions import NoSuchResource, Unresolvable
from referencing.retrieval import to_cached_resource
# Ossie schemas reference each other by raw GitHub URL or by canonical $id. Both
# prefixes map onto paths in this repo, read only when a reference needs them.
_SCHEMA_BASES = (
"https://raw.githubusercontent.com/apache/ossie/main/",
"https://github.com/apache/ossie/",
)
@to_cached_resource()
def _retrieve_local_schema(uri: str) -> str:
"""Read a referenced Ossie schema from this repo."""
repo = Path(__file__).parent.parent
for base in _SCHEMA_BASES:
if uri.startswith(base):
return (repo / uri[len(base):]).read_text()
raise NoSuchResource(ref=uri)
_SCHEMA_REGISTRY = Registry(retrieve=_retrieve_local_schema)
```
with `registry=_SCHEMA_REGISTRY` in `validate_schema`. A missing or malformed `ontology.json` then only affects documents that reference it, and surfaces through the existing `Unresolvable` handler as "Cannot resolve schema reference" instead of a traceback.
##########
ontology/mapping.json:
##########
@@ -0,0 +1,60 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "https://github.com/apache/ossie/ontology/mapping.json",
+ "title": "Apache Ossie Mapping Specification",
+ "description": "JSON Schema for validating a standalone Apache Ossie mapping document: the constructs of one semantic model mapped onto one ontology, as its own document rather than embedded inside the ontology's ontology_mappings.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "0.2.0.dev0",
+ "description": "Mapping specification version"
+ },
+ "name": {
+ "type": "string",
+ "description": "Unique identifier for this mapping"
+ },
+ "description": {
+ "type": "string",
+ "description": "Human-readable description of this mapping"
+ },
+ "ontology": {
+ "$ref": "#/$defs/DocumentReference",
+ "description": "Reference to the ontology document this mapping targets"
+ },
+ "semantic_model": {
+ "$ref": "#/$defs/DocumentReference",
+ "description": "Reference to the semantic model document this mapping draws from"
+ },
+ "concept_mappings": {
+ "type": "array",
+ "items": {
+ "$ref": "https://raw.githubusercontent.com/apache/ossie/main/ontology/ontology.json#/$defs/ConceptMapping"
+ },
+ "description": "Maps logical model constructs to some concept and its relationships in the referenced ontology"
+ },
+ "custom_extensions": {
+ "$ref": "https://raw.githubusercontent.com/apache/ossie/main/core-spec/ossie-schema.json#/$defs/SemanticModel/properties/custom_extensions"
+ }
+ },
+ "required": ["version", "name", "ontology", "semantic_model", "concept_mappings"],
+ "additionalProperties": false,
+ "$defs": {
+ "DocumentReference": {
+ "type": "object",
+ "description": "Reference to another Ossie document by its logical name, with an iri saying where to resolve it from",
+ "properties": {
+ "name": {
+ "type": "string",
+ "description": "Must equal the referenced document's own 'name'; carries this reference's checkable identity regardless of location"
+ },
+ "iri": {
+ "type": "string",
+ "description": "Where to resolve the referenced document from: a relative reference (e.g. a path alongside this document) or an absolute IRI once a catalog resolves names to locations"
+ }
Review Comment:
These accept empty strings, so `{"ontology": {"name": ""}, "semantic_model": {"name": "", "iri": ""}}` validates. The core schema enforces `minLength: 1` on `SemanticModel.name`, so an empty `semantic_model.name` can never satisfy "must equal the referenced document's own name".
```suggestion
"name": {
"type": "string",
"minLength": 1,
"description": "Must equal the referenced document's own 'name'; carries this reference's checkable identity regardless of location"
},
"iri": {
"type": "string",
"minLength": 1,
"description": "Where to resolve the referenced document from: a relative reference (e.g. a path alongside this document) or an absolute IRI once a catalog resolves names to locations"
}
```
Same for the mapping's own `name` at line 13. Could you also state in `ontology.md` what a relative `iri` is resolved against (I assume the mapping document's own location)?
##########
ontology/mapping.json:
##########
@@ -0,0 +1,60 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "https://github.com/apache/ossie/ontology/mapping.json",
+ "title": "Apache Ossie Mapping Specification",
+ "description": "JSON Schema for validating a standalone Apache Ossie mapping document: the constructs of one semantic model mapped onto one ontology, as its own document rather than embedded inside the ontology's ontology_mappings.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "0.2.0.dev0",
+ "description": "Mapping specification version"
+ },
+ "name": {
+ "type": "string",
+ "description": "Unique identifier for this mapping"
+ },
+ "description": {
+ "type": "string",
+ "description": "Human-readable description of this mapping"
+ },
+ "ontology": {
+ "$ref": "#/$defs/DocumentReference",
+ "description": "Reference to the ontology document this mapping targets"
+ },
+ "semantic_model": {
+ "$ref": "#/$defs/DocumentReference",
+ "description": "Reference to the semantic model document this mapping draws from"
+ },
Review Comment:
The root keys `ontology` and `semantic_model` already have a meaning at the root of the other document kinds, and here they carry a different type:
- In an ontology document, `ontology` is the list of components; here it is a reference object.
- A root `semantic_model` key is what the converters treat as the removed legacy wrapper. honeydew, microsoft, databricks, orionbelt, snowflake, omni, gooddata, nvidia and thoughtspot all check `"semantic_model" in root` and reject the document, so a mapping document passed to any of them gets a misleading "legacy wrapper" error.
It also makes the recognition rule in `ontology.md` order-dependent: a consumer has to test for `concept_mappings` before it tests for `ontology`.
Root key names are hard to change once the spec is released, so I'd like us to settle this before merging. Distinct names would remove the overlap, for example `ontology_ref` / `semantic_model_ref`. If you prefer to keep the current names, please say so explicitly in the spec and state that `concept_mappings` must be checked first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org