You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ossie.apache.org by jb...@apache.org on 2026/10/17 04:43:06 UTC
(ossie) branch main updated: feat(nvidia): export identifier names, field-named keys and an embedded mapping (#599)
This is an automated email from the ASF dual-hosted git repository.
jbonofre pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/ossie.git
The following commit(s) were added to refs/heads/main by this push:
new ef91cb61 feat(nvidia): export identifier names, field-named keys and an embedded mapping (#599)
ef91cb61 is described below
commit ef91cb61092c60f4b918016a81cfdbe483e113bc
Author: liofnv <lf...@nvidia.com>
AuthorDate: Fri Oct 16 23:43:00 2026 -0500
feat(nvidia): export identifier names, field-named keys and an embedded mapping (#599)
Ontology tools could not read the ontology export. RelationalAI's Ossie
converter (apache/ossie#412), run on a TPC-H model exported from Auto
Ontology, failed on three things this changes:
- Names were Auto Ontology's display text, such as `Order Line Item`, so
mapping expressions quoted them (`"Order Line Item"."Phone Number"`), which
that converter's formula grammar does not parse. Datasets, fields, metrics
and relationships are now lowercase identifiers (`order_line_item`), with a
trailing underscore for words SQL reserves, a prefix for a leading digit and
a number for a collision. Where the identifier drops more than case and
word separators, the original is kept in `ai_context` synonyms. Expressions
lifted out of Auto Ontology SQL are requalified with the dataset names, and
mapping expressions quote only names SQL cannot read bare.
- Keys and relationships named physical columns, while that converter looks
them up among the dataset's fields, so a key column exposed as `part_key`
but named `P_PARTKEY` was not found. They now name the field that reads the
column. On import, a key or relationship column that names a field stands
for its column; any other is the physical column, as earlier exports and
other tools write it.
- It reads only mappings embedded in the ontology, which the specification
deprecates but still accepts for tools that require them.
--embedded-ontology-output (convert_auto_ontology_to_embedded_ossie_ontology)
writes that form, the mapping carrying the semantic model whole.
Auto Ontology names still round-trip: one function names every item for both
directions, and the native snapshot is matched under an item's own name, the
display name earlier exports used and the identifier exported now, so files
written before this change import as before. Two terms sharing a name are
still rejected, since the snapshot could not tell them apart.
---
converters/nvidia/README.md | 47 ++-
.../src/ossie_nvidia_auto_ontology/__init__.py | 2 +
.../src/ossie_nvidia_auto_ontology/converter.py | 6 +-
.../ossie_nvidia_auto_ontology/native_converter.py | 460 ++++++++++++++++++---
.../src/ossie_nvidia_auto_ontology/ontology.py | 73 +++-
converters/nvidia/tests/test_converter.py | 254 +++++++++++-
converters/nvidia/tests/test_ontology.py | 124 +++++-
7 files changed, 888 insertions(+), 78 deletions(-)
diff --git a/converters/nvidia/README.md b/converters/nvidia/README.md
index af275187..c517a0b0 100644
--- a/converters/nvidia/README.md
+++ b/converters/nvidia/README.md
@@ -108,6 +108,18 @@ SQL is preserved, so it returns to Auto Ontology unchanged. Custom analyses
remain global by becoming model-level Ossie metrics. Relationships are
recovered from joins, then physical foreign keys, then semantic foreign keys.
+Auto Ontology names are display text, such as `Order Line Item`, while Ossie
+names are identifiers that tools write unquoted, as an ontology mapping does in
+`dataset.field`. Each dataset, field, metric and relationship is therefore
+named in lowercase words joined by underscores, such as `order_line_item`. A
+name that SQL reserves gets a trailing underscore (`from_`), one that would
+start with a digit is prefixed with what it names (`field_2024_total`), and one
+that would collide with another in the same namespace gets a number
+(`customer_id_2`). Where the identifier does not spell out the original name,
+because it dropped punctuation or had to be told apart, the original is kept as
+a synonym in `ai_context`. Expressions lifted out of Auto Ontology SQL refer to
+datasets by these names too. Every item keeps its own name on the way back.
+
Auto Ontology term names are unique across every database it catalogs, so it
renames a term whose name another database already uses to
`Name (database.schema.table)`. An Ossie document is its own namespace, so the
@@ -118,6 +130,12 @@ instead; each key and relationship column that no field exposes is exported as
a field named after the column, marked so that it does not become an attribute
on the way back.
+Primary keys, unique keys and relationships name the fields that read their
+columns, which is how ontology tools look them up. On the way back, a key or
+relationship column that names a field of its dataset stands for the column
+that field reads; any other is the physical column itself, as earlier exports
+and other tools write it.
+
### Ontology and mapping
Auto Ontology's terms are business concepts, its attributes are properties of
@@ -144,8 +162,30 @@ ontology_yaml, mapping_yaml = convert_auto_ontology_to_ossie_ontology(
```
The two outputs are written together. The mapping references both other
-documents by name and, from the CLI, by a path relative to itself. They are
-read off the semantic model as follows:
+documents by name and, from the CLI, by a path relative to itself.
+
+Tools that do not read mapping documents yet read the mapping embedded in the
+ontology instead, a form the specification deprecates but still accepts for
+them. `--embedded-ontology-output` writes the ontology that way, the mapping
+carrying the semantic model whole, on its own or alongside the other outputs:
+
+```bash
+uv run ossie-nvidia-auto-ontology import \
+ --input tpcds.auto_ontology.yaml \
+ --embedded-ontology-output tpcds.ontology.embedded.yaml
+```
+
+```python
+from ossie_nvidia_auto_ontology import (
+ convert_auto_ontology_to_embedded_ossie_ontology,
+)
+
+ontology_yaml = convert_auto_ontology_to_embedded_ossie_ontology(
+ auto_ontology_yaml, model_name="tpcds"
+)
+```
+
+The ontology and mapping are read off the semantic model as follows:
| Semantic model | Ontology | Mapping |
|---|---|---|
@@ -157,7 +197,8 @@ read off the semantic model as follows:
A foreign-key column is the link to another concept, so it does not also become
a value of its own concept. Auto Ontology records no verbalizations, so each
relationship is verbalized from its field name, such as
-`{Customer} has Account Balance {Decimal}`. A field whose type is unknown, or
+`{Customer} has account balance {Decimal}`. Mapping expressions quote a name
+only where SQL could not read it bare, so `customer.account_balance`. A field whose type is unknown, or
has no built-in value type (`Time`, `Opaque`), is left out rather than guessed.
A concept whose dataset has no key, or whose key columns are not all
identifying fields or links, has nothing to name its objects by: it is declared
diff --git a/converters/nvidia/src/ossie_nvidia_auto_ontology/__init__.py b/converters/nvidia/src/ossie_nvidia_auto_ontology/__init__.py
index f27c37e6..a5618d6f 100644
--- a/converters/nvidia/src/ossie_nvidia_auto_ontology/__init__.py
+++ b/converters/nvidia/src/ossie_nvidia_auto_ontology/__init__.py
@@ -19,6 +19,7 @@
from .converter import (
AutoOntologyConversionError,
+ convert_auto_ontology_to_embedded_ossie_ontology,
convert_auto_ontology_to_ossie,
convert_auto_ontology_to_ossie_ontology,
convert_ossie_to_auto_ontology,
@@ -26,6 +27,7 @@ from .converter import (
__all__ = [
"AutoOntologyConversionError",
+ "convert_auto_ontology_to_embedded_ossie_ontology",
"convert_auto_ontology_to_ossie",
"convert_auto_ontology_to_ossie_ontology",
"convert_ossie_to_auto_ontology",
diff --git a/converters/nvidia/src/ossie_nvidia_auto_ontology/converter.py b/converters/nvidia/src/ossie_nvidia_auto_ontology/converter.py
index 34e782a6..2bd7fec0 100644
--- a/converters/nvidia/src/ossie_nvidia_auto_ontology/converter.py
+++ b/converters/nvidia/src/ossie_nvidia_auto_ontology/converter.py
@@ -23,10 +23,14 @@ from .native_converter import (
convert_ossie_to_auto_ontology,
main,
)
-from .ontology import convert_auto_ontology_to_ossie_ontology
+from .ontology import (
+ convert_auto_ontology_to_embedded_ossie_ontology,
+ convert_auto_ontology_to_ossie_ontology,
+)
__all__ = [
"AutoOntologyConversionError",
+ "convert_auto_ontology_to_embedded_ossie_ontology",
"convert_auto_ontology_to_ossie",
"convert_auto_ontology_to_ossie_ontology",
"convert_ossie_to_auto_ontology",
diff --git a/converters/nvidia/src/ossie_nvidia_auto_ontology/native_converter.py b/converters/nvidia/src/ossie_nvidia_auto_ontology/native_converter.py
index adc300bc..77f53715 100644
--- a/converters/nvidia/src/ossie_nvidia_auto_ontology/native_converter.py
+++ b/converters/nvidia/src/ossie_nvidia_auto_ontology/native_converter.py
@@ -50,6 +50,55 @@ _SIMPLE_COLUMN = re.compile(
r"^(?:(?P<qualifier>[A-Za-z_][A-Za-z0-9_]*)\.)?"
r"(?P<column>[A-Za-z_][A-Za-z0-9_]*)$"
)
+# A name tools can write unquoted, as in an ontology mapping's ``dataset.field``.
+_PLAIN_NAME = re.compile(r"[^\W\d]\w*")
+_CAMEL_BOUNDARY = re.compile(r"(?<=[^\W_])(?=[A-Z][a-z])|(?<=[a-z0-9])(?=[A-Z])")
+_NAME_SEPARATORS = re.compile(r"[\W_]+")
+_WORD_SEPARATORS = re.compile(r"[\s_-]+")
+# Words SQL reserves so firmly that a parser cannot read them as a table or
+# column name without quotes. A name that would be one gets a trailing
+# underscore instead.
+_SQL_RESERVED_NAMES = frozenset(
+ {
+ "and",
+ "any",
+ "as",
+ "between",
+ "case",
+ "create",
+ "cross",
+ "desc",
+ "distinct",
+ "drop",
+ "else",
+ "except",
+ "fetch",
+ "for",
+ "from",
+ "grant",
+ "having",
+ "in",
+ "inner",
+ "insert",
+ "intersect",
+ "into",
+ "join",
+ "like",
+ "not",
+ "on",
+ "or",
+ "outer",
+ "select",
+ "set",
+ "then",
+ "union",
+ "update",
+ "using",
+ "when",
+ "where",
+ "with",
+ }
+)
# sqlglot's default parser first, since it is closest to ANSI, then the
# dialects Auto Ontology connections commonly report.
_SQL_DIALECTS = (
@@ -257,6 +306,7 @@ def convert_ossie_to_auto_ontology(
native = _native_snapshot(model)
preserved = _index_native_document(native)
datasets: dict[str, dict[str, Any]] = {}
+ field_columns: dict[str, dict[str, str]] = {}
for item in source_datasets:
if not isinstance(item, dict) or not item.get("name"):
raise AutoOntologyConversionError(
@@ -282,9 +332,11 @@ def convert_ossie_to_auto_ontology(
for column in preserved_table.get("columns") or []:
if isinstance(column, dict) and column.get("name"):
_add_catalog_column(datasets[name], str(column["name"]))
+ field_columns[name] = _field_columns(item, name, str(source["table"]))
+ datasets[name]["original"] = _keys_as_columns(item, field_columns[name])
relationships = [
- _validate_relationship(item, datasets)
+ _relationship_as_columns(_validate_relationship(item, datasets), field_columns)
for item in model.get("relationships") or []
]
for relationship in relationships:
@@ -386,7 +438,7 @@ def convert_ossie_to_auto_ontology(
term["columns_attributes"].append(
{
"id": attr_id,
- "name": field_name,
+ "name": str(preserved_attr.get("name") or field_name),
"description": str(field.get("description") or ""),
"column_id": column_ids[(name, source_column)],
}
@@ -494,7 +546,7 @@ def convert_ossie_to_auto_ontology(
or preserved_attr.get("id")
or _stable_id("sql-attribute", name, field_name)
),
- "name": field_name,
+ "name": str(preserved_attr.get("name") or field_name),
"description": str(field.get("description") or ""),
"sql": full_sql,
"sql_column_is": resolved_column_ids,
@@ -541,7 +593,7 @@ def convert_ossie_to_auto_ontology(
or preserved_analysis.get("id")
or _stable_id("custom-analysis", str(model["name"]), name)
),
- "name": name,
+ "name": str(preserved_analysis.get("name") or name),
"description": str(metric.get("description") or ""),
"sql": full_sql,
"sql_column_is": referenced_column_ids,
@@ -601,7 +653,14 @@ def convert_auto_ontology_to_ossie(
term_columns: dict[str, set[str]] = defaultdict(set)
sampled_unique_columns: dict[str, list[str]] = {}
attr_owner: dict[str, tuple[str, dict[str, Any]]] = {}
- dataset_names = _dataset_names(terms, catalog)
+ names = _ossie_names(root, catalog)
+ term_names: set[str] = set()
+ # Auto Ontology SQL aliases each table by its term's name.
+ dataset_by_alias = {
+ str(term["name"]): names["datasets"][str(term.get("id") or "")]
+ for term in terms
+ if isinstance(term, dict) and term.get("name")
+ }
for term in terms:
term_id = _required_id(term, "Auto Ontology term")
@@ -626,11 +685,14 @@ def convert_auto_ontology_to_ossie(
raise AutoOntologyConversionError(
"Every Auto Ontology term needs a non-empty name"
)
- name = dataset_names[term_id]
- if name in term_id_by_name:
+ # Every Ossie name is told apart, but two terms that share theirs could
+ # not be told apart when the native snapshot is matched back to them.
+ if str(term["name"]) in term_names:
raise AutoOntologyConversionError(
- f"Duplicate Auto Ontology term name {name!r}"
+ f"Duplicate Auto Ontology term name {term['name']!r}"
)
+ term_names.add(str(term["name"]))
+ name = names["datasets"][term_id]
term_id_by_name[name] = term_id
datasets_by_table[table_id].append(name)
term_by_id[term_id] = term
@@ -642,6 +704,7 @@ def convert_auto_ontology_to_ossie(
dataset["description"] = str(term["description"])
if table["item"].get("pk"):
dataset["primary_key"] = list(table["item"]["pk"])
+ _with_original_name(dataset, names["spelled_from"][term_id])
sampled_unique_columns[name] = [
str(column.get("name"))
for column in table["item"].get("columns") or []
@@ -660,16 +723,16 @@ def convert_auto_ontology_to_ossie(
f"Column attribute {attribute.get('name')!r} references "
f"unknown column {column_id!r}"
)
- field_name = str(attribute.get("name") or column["name"])
- if field_name in field_names_by_term[term_id]:
+ attribute_name = str(attribute.get("name") or column["name"])
+ if attribute_name in field_names_by_term[term_id]:
raise AutoOntologyConversionError(
- f"Duplicate field name {field_name!r} in Auto Ontology "
+ f"Duplicate field name {attribute_name!r} in Auto Ontology "
f"term {name!r}"
)
- field_names_by_term[term_id].add(field_name)
+ field_names_by_term[term_id].add(attribute_name)
term_columns[name].add(str(column["name"]))
field: dict[str, Any] = {
- "name": field_name,
+ "name": names["fields"][str(attribute["id"])],
"expression": _ossie_expression(str(column["name"])),
}
datatype = _ossie_datatype(column["item"].get("type"))
@@ -677,6 +740,7 @@ def convert_auto_ontology_to_ossie(
field["datatype"] = datatype
if attribute.get("description"):
field["description"] = str(attribute["description"])
+ _with_original_name(field, attribute_name)
fields_by_term[term_id].append(field)
attr_owner[str(attribute["id"])] = (term_id, attribute)
@@ -712,9 +776,11 @@ def convert_auto_ontology_to_ossie(
attached_table_id=represented_table_id,
)
condition = _filter_condition_from_sql(sql)
- ossie_expression = condition or _expression_from_sql(sql)
+ ossie_expression = _requalified(
+ condition or _expression_from_sql(sql), dataset_by_alias
+ )
field: dict[str, Any] = {
- "name": name,
+ "name": names["fields"][attribute_id],
"expression": _ossie_expression(
ossie_expression,
_ossie_dialect(dialects, sql_databases),
@@ -737,6 +803,7 @@ def convert_auto_ontology_to_ossie(
]
if attribute.get("description"):
field["description"] = str(attribute["description"])
+ _with_original_name(field, name)
fields_by_term[term_id].append(field)
relationships = _relationships_from_auto_ontology(
@@ -763,6 +830,7 @@ def convert_auto_ontology_to_ossie(
)
if fields_by_term[term_id]:
dataset["fields"] = fields_by_term[term_id]
+ _name_key_columns_by_field(datasets, relationships)
metrics: list[dict[str, Any]] = []
for analysis in semantic.get("custom_analyses") or []:
@@ -779,9 +847,9 @@ def convert_auto_ontology_to_ossie(
analysis.get("sql_column_is") or [],
catalog,
)
- ossie_expression = _expression_from_sql(sql)
+ ossie_expression = _requalified(_expression_from_sql(sql), dataset_by_alias)
metric: dict[str, Any] = {
- "name": name,
+ "name": names["metrics"][analysis_id],
"expression": _ossie_expression(
ossie_expression,
_ossie_dialect(dialects, sql_databases),
@@ -800,6 +868,7 @@ def convert_auto_ontology_to_ossie(
}
if analysis.get("description"):
metric["description"] = str(analysis["description"])
+ _with_original_name(metric, name)
metrics.append(metric)
database_names = {
@@ -1057,47 +1126,78 @@ def _index_native_document(root: dict[str, Any] | None) -> dict[str, Any]:
for table_id, table in catalog["tables"].items()
}
terms = root["semantic_layer"].get("terms") or []
- dataset_names = _dataset_names(terms, catalog)
- # A term is found under its own name and under the dataset name it was
- # exported as, so a document from either side keeps its identifiers.
+ names = _ossie_names(root, catalog)
+
+ # Every item is found under its own name, under the display name earlier
+ # exports gave it, and under the name it is exported as now, so a document
+ # from any of them keeps its identifiers. The name exported now wins over
+ # another item's older name, should the two meet.
+ def known_as(
+ item: Mapping[str, Any], exported: Mapping[str, str]
+ ) -> tuple[list[str], str]:
+ item_id = str(item.get("id") or "")
+ own = str(item.get("name") or "")
+ older = list(dict.fromkeys((own, names["spelled_from"].get(item_id, own))))
+ return older, exported.get(item_id, own)
+
+ def index(
+ kind: str,
+ item: Any,
+ older_keys: Iterable[tuple[str, ...] | str],
+ exported_key: tuple[str, ...] | str,
+ ) -> None:
+ for key in older_keys:
+ result[kind].setdefault(key, item)
+ result[kind][exported_key] = item
+
term_names = {
- str(term.get("id")): list(
- dict.fromkeys(
- (str(term.get("name") or ""), dataset_names[str(term.get("id") or "")])
- )
- )
- for term in terms
+ str(term.get("id")): known_as(term, names["datasets"]) for term in terms
}
for term in terms:
represents = term.get("represents") or []
if len(represents) != 1 or str(represents[0]) not in table_source:
continue
source = table_source[str(represents[0])]
- for term_name in term_names[str(term.get("id"))]:
- result["terms"][(term_name, source)] = term
- for attribute in term.get("columns_attributes") or []:
- column = catalog["columns"].get(str(attribute.get("column_id") or ""))
- if column:
- result["column_attributes"][
- (
- term_name,
- str(attribute.get("name") or ""),
- source,
- column["name"],
- )
- ] = attribute
+ older_terms, exported_term = term_names[str(term.get("id"))]
+ index(
+ "terms",
+ term,
+ [(name, source) for name in older_terms],
+ (exported_term, source),
+ )
+ for attribute in term.get("columns_attributes") or []:
+ column = catalog["columns"].get(str(attribute.get("column_id") or ""))
+ if not column:
+ continue
+ older_fields, exported_field = known_as(attribute, names["fields"])
+ index(
+ "column_attributes",
+ attribute,
+ [
+ (term_name, field_name, source, column["name"])
+ for term_name in older_terms
+ for field_name in older_fields
+ ],
+ (exported_term, exported_field, source, column["name"]),
+ )
for group in _SQL_GROUPS:
for attribute in (root["semantic_layer"].get("sql_attributes") or {}).get(
group, []
):
item = dict(attribute)
item["source_group"] = group
- for term_name in term_names.get(str(attribute.get("term_id") or ""), [""]):
- result["sql_attributes"][
- (term_name, str(attribute.get("name") or ""))
- ] = item
+ older_terms, exported_term = term_names.get(
+ str(attribute.get("term_id") or ""), ([""], "")
+ )
+ older_fields, exported_field = known_as(attribute, names["fields"])
+ index(
+ "sql_attributes",
+ item,
+ [(term, field) for term in older_terms for field in older_fields],
+ (exported_term, exported_field),
+ )
for analysis in root["semantic_layer"].get("custom_analyses") or []:
- result["custom_analyses"][str(analysis.get("name") or "")] = analysis
+ index("custom_analyses", analysis, *known_as(analysis, names["metrics"]))
return result
@@ -1209,17 +1309,31 @@ def _validate_auto_ontology_sql_databases(
return databases
-def _dataset_names(
- terms: Iterable[Mapping[str, Any]],
+def _ossie_names(
+ root: Mapping[str, Any],
catalog: Mapping[str, Any],
-) -> dict[str, str]:
- """Name the dataset each Auto Ontology term becomes, keyed by term id.
+) -> dict[str, dict[str, str]]:
+ """Name what each Auto Ontology item becomes in Ossie, keyed by item id.
+
+ ``datasets`` names a term's dataset, ``fields`` the field of a column or
+ SQL attribute, and ``metrics`` the metric of a custom analysis;
+ ``spelled_from`` holds the text each of those names was spelled from.
+ Both directions go through here, so an export and the native snapshot it
+ carries agree on every name.
+
+ Auto Ontology names are display text, such as ``Order Line Item``, while
+ Ossie names are identifiers that tools write unquoted, as an ontology
+ mapping does in ``dataset.field``. Each name is spelled as one, and told
+ apart from any other it would then collide with in the same namespace.
Auto Ontology term names are unique across every database it catalogs, so
a term whose name another database already uses is renamed
``Name (database.schema.table)``. An Ossie document is its own namespace,
so that suffix is dropped again wherever the plain name is free in it.
"""
+ semantic = root.get("semantic_layer") or {}
+ terms = [term for term in semantic.get("terms") or [] if isinstance(term, dict)]
+ spelled_from: dict[str, str] = {}
raw: dict[str, str] = {}
plain: dict[str, str] = {}
for term in terms:
@@ -1236,13 +1350,99 @@ def _dataset_names(
if suffix and name.endswith(suffix) and name != suffix
else name
)
- taken = Counter(plain.values())
+ free = Counter(_ossie_identifier(name, "dataset") for name in plain.values())
+ taken: set[str] = set()
+ datasets: dict[str, str] = {}
+ for term_id, name in plain.items():
+ if free[_ossie_identifier(name, "dataset")] > 1:
+ name = raw[term_id]
+ spelled_from[term_id] = name
+ datasets[term_id] = _unique_name(_ossie_identifier(name, "dataset"), taken)
+
+ # Fields in the order the export lists them: column attributes first.
+ fields: dict[str, str] = {}
+ taken_by_term: dict[str, set[str]] = defaultdict(set)
+
+ def add_field(item_id: str, name: str, term_id: str) -> None:
+ spelled_from[item_id] = name
+ fields[item_id] = _unique_name(
+ _ossie_identifier(name, "field"), taken_by_term[term_id]
+ )
+
+ for term in terms:
+ for attribute in term.get("columns_attributes") or []:
+ if isinstance(attribute, dict):
+ column = catalog["columns"].get(str(attribute.get("column_id") or ""))
+ add_field(
+ str(attribute.get("id") or ""),
+ str(attribute.get("name") or (column or {}).get("name") or ""),
+ str(term.get("id") or ""),
+ )
+ for group in _SQL_GROUPS:
+ for attribute in (semantic.get("sql_attributes") or {}).get(group) or []:
+ if isinstance(attribute, dict):
+ add_field(
+ str(attribute.get("id") or ""),
+ str(attribute.get("name") or ""),
+ str(attribute.get("term_id") or ""),
+ )
+
+ taken_metrics: set[str] = set()
+ metrics: dict[str, str] = {}
+ for analysis in semantic.get("custom_analyses") or []:
+ if isinstance(analysis, dict):
+ analysis_id = str(analysis.get("id") or "")
+ spelled_from[analysis_id] = str(analysis.get("name") or "")
+ metrics[analysis_id] = _unique_name(
+ _ossie_identifier(spelled_from[analysis_id], "metric"), taken_metrics
+ )
return {
- term_id: plain[term_id] if taken[plain[term_id]] == 1 else raw[term_id]
- for term_id in raw
+ "datasets": datasets,
+ "fields": fields,
+ "metrics": metrics,
+ "spelled_from": spelled_from,
}
+def _ossie_identifier(text: str, kind: str) -> str:
+ """Spell *text* as a lowercase identifier, its words joined by underscores.
+
+ *kind* names what is being named, and stands in for a name that has no
+ word in it or prefixes one that would start with a digit.
+ """
+ name = _NAME_SEPARATORS.sub("_", _CAMEL_BOUNDARY.sub("_", text)).strip("_")
+ name = name.lower() or kind
+ if name[0].isdigit():
+ name = f"{kind}_{name}"
+ return f"{name}_" if name in _SQL_RESERVED_NAMES else name
+
+
+def _is_plain_name(name: str) -> bool:
+ """Report whether *name* can be written in SQL without quoting it."""
+ return bool(_PLAIN_NAME.fullmatch(name)) and name.lower() not in _SQL_RESERVED_NAMES
+
+
+def _unique_name(name: str, taken: set[str]) -> str:
+ candidate, count = name, 1
+ while candidate in taken:
+ count += 1
+ candidate = f"{name}_{count}"
+ taken.add(candidate)
+ return candidate
+
+
+def _with_original_name(item: dict[str, Any], original: str) -> None:
+ """Keep *original* as a synonym of *item* where its name does not spell it out.
+
+ A name that differs only in case and in how words are separated reads the
+ same, so it needs no synonym; one that dropped punctuation, or had to be
+ told apart from another, does.
+ """
+ words = _WORD_SEPARATORS.sub("_", _CAMEL_BOUNDARY.sub("_", original))
+ if words.strip("_").lower() != item["name"]:
+ item["ai_context"] = {"synonyms": [original]}
+
+
def _referenced_unique_keys(
dataset: Mapping[str, Any],
sampled_unique_columns: list[str],
@@ -1308,13 +1508,8 @@ def _key_column_fields(
for column in dict.fromkeys(key_columns):
if column in exposed:
continue
- field_name, count = column, 1
- while field_name in taken:
- count += 1
- field_name = f"{column}_{count}"
- taken.add(field_name)
field: dict[str, Any] = {
- "name": field_name,
+ "name": _unique_name(_ossie_identifier(column, "field"), taken),
"expression": _ossie_expression(column),
}
datatype = _ossie_datatype(catalog_columns.get(column, {}).get("type"))
@@ -1327,6 +1522,50 @@ def _key_column_fields(
return key_fields
+def _name_key_columns_by_field(
+ datasets: list[dict[str, Any]],
+ relationships: list[dict[str, Any]],
+) -> None:
+ """Name each key and relationship column by the field that reads it.
+
+ Auto Ontology knows keys and links by physical column, but ontology tools
+ look the columns of an Ossie key or relationship up among the dataset's
+ fields. By now every such column is a field, so each is named by it.
+ """
+ by_column = {
+ str(dataset["name"]): _field_names_by_column(dataset) for dataset in datasets
+ }
+ for dataset in datasets:
+ names = by_column[str(dataset["name"])]
+ if dataset.get("primary_key"):
+ dataset["primary_key"] = [
+ names.get(column, column) for column in dataset["primary_key"]
+ ]
+ if dataset.get("unique_keys"):
+ dataset["unique_keys"] = [
+ [names.get(column, column) for column in key]
+ for key in dataset["unique_keys"]
+ ]
+ for relationship in relationships:
+ for side, key in (("from", "from_columns"), ("to", "to_columns")):
+ names = by_column[str(relationship[side])]
+ relationship[key] = [
+ names.get(column, column) for column in relationship[key]
+ ]
+
+
+def _field_names_by_column(dataset: Mapping[str, Any]) -> dict[str, str]:
+ """Name the first field of *dataset* that passes each source column through."""
+ names: dict[str, str] = {}
+ for field in dataset.get("fields") or []:
+ column = _simple_source_column(
+ field["expression"]["dialects"][0]["expression"], str(dataset["name"]), ""
+ )
+ if column:
+ names.setdefault(column, str(field["name"]))
+ return names
+
+
def _relationships_from_auto_ontology(
root: dict[str, Any],
catalog: Mapping[str, Any],
@@ -1477,7 +1716,9 @@ def _relationships_from_auto_ontology(
# Named from its own columns only, so adding another link between the
# same datasets never renames this one.
source_columns = "_".join(source for source, _ in columns)
- base_name = f"{from_name}_{source_columns}_to_{to_name}"
+ base_name = _ossie_identifier(
+ f"{from_name}_{source_columns}_to_{to_name}", "relationship"
+ )
name, count = base_name, 1
while name in used_names:
count += 1
@@ -1704,6 +1945,73 @@ def _validate_relationship(
return relationship
+def _field_columns(
+ dataset: Mapping[str, Any],
+ dataset_name: str,
+ table_name: str,
+) -> dict[str, str]:
+ """Return the source column each pass-through field of *dataset* reads.
+
+ A field whose expression cannot be read is left out here; it fails with
+ its own error once the dataset's fields are converted.
+ """
+ columns: dict[str, str] = {}
+ for field in dataset.get("fields") or []:
+ if not isinstance(field, dict) or not field.get("name"):
+ continue
+ name = str(field["name"])
+ try:
+ expression = _pick_expression(
+ _normalize_expressions(field.get("expression"), name), name
+ )
+ except AutoOntologyConversionError:
+ continue
+ column = _simple_source_column(expression, dataset_name, table_name)
+ if column:
+ columns.setdefault(name, column)
+ return columns
+
+
+def _keys_as_columns(
+ dataset: dict[str, Any],
+ field_columns: Mapping[str, str],
+) -> dict[str, Any]:
+ """Return *dataset* with its keys naming source columns.
+
+ A key column names a field when the dataset has one by that name, as
+ ontology tools read it and this converter exports it; otherwise it is the
+ physical column itself, as earlier exports and other tools write it.
+ """
+ result = dict(dataset)
+ if isinstance(dataset.get("primary_key"), list):
+ result["primary_key"] = [
+ field_columns.get(str(column), column) for column in dataset["primary_key"]
+ ]
+ if isinstance(dataset.get("unique_keys"), list):
+ result["unique_keys"] = [
+ [field_columns.get(str(column), column) for column in key]
+ if isinstance(key, list)
+ else key
+ for key in dataset["unique_keys"]
+ ]
+ return result
+
+
+def _relationship_as_columns(
+ relationship: dict[str, Any],
+ field_columns: Mapping[str, Mapping[str, str]],
+) -> dict[str, Any]:
+ """Return *relationship* with its columns naming source columns, as keys do.
+
+ *field_columns* holds what ``_field_columns`` returns, by dataset name.
+ """
+ result = dict(relationship)
+ for side, key in (("from", "from_columns"), ("to", "to_columns")):
+ columns = field_columns.get(str(relationship[side]), {})
+ result[key] = [columns.get(str(column), column) for column in relationship[key]]
+ return result
+
+
def _normalize_expressions(value: Any, name: str) -> list[dict[str, str]]:
if not isinstance(value, dict):
raise AutoOntologyConversionError(f"{name!r} has no valid expression")
@@ -2118,6 +2426,25 @@ def _expression_from_sql(sql: str) -> str:
return expression.sql()
+def _requalified(expression: str, dataset_by_alias: Mapping[str, str]) -> str:
+ """Return *expression* with columns qualified by their dataset's name.
+
+ An expression lifted out of Auto Ontology SQL keeps the table aliases
+ that SQL gave it, which are term names; in Ossie the same table is the
+ dataset of that term, and goes by the dataset's name.
+ """
+ parsed = _parse_sql(expression)
+ if parsed is None:
+ return expression
+ changed = False
+ for column in parsed.find_all(exp.Column):
+ dataset = dataset_by_alias.get(column.table) if column.table else None
+ if dataset and dataset != column.table:
+ column.set("table", exp.to_identifier(dataset))
+ changed = True
+ return parsed.sql() if changed else expression
+
+
def _filter_condition_from_sql(sql: str) -> str | None:
"""Return the condition of a SQL attribute that selects rows, not a value.
@@ -2599,6 +2926,14 @@ def _build_parser() -> argparse.ArgumentParser:
type=Path,
help="Also write the mapping from that ontology onto the semantic model",
)
+ import_parser.add_argument(
+ "--embedded-ontology-output",
+ type=Path,
+ help=(
+ "Also write the ontology with the mapping and semantic model embedded "
+ "in it, for tools that do not read mapping documents yet"
+ ),
+ )
return parser
@@ -2643,6 +2978,15 @@ def main(argv: list[str] | None = None) -> None:
)
args.ontology_output.write_text(ontology, encoding="utf-8")
args.mapping_output.write_text(mapping, encoding="utf-8")
+ if args.command == "import" and args.embedded_ontology_output is not None:
+ from .ontology import convert_auto_ontology_to_embedded_ossie_ontology
+
+ args.embedded_ontology_output.write_text(
+ convert_auto_ontology_to_embedded_ossie_ontology(
+ source, model_name=args.name
+ ),
+ encoding="utf-8",
+ )
except (AutoOntologyConversionError, OSError, UnicodeError) as exc:
print(f"Error: {exc}", file=sys.stderr)
raise SystemExit(1) from exc
diff --git a/converters/nvidia/src/ossie_nvidia_auto_ontology/ontology.py b/converters/nvidia/src/ossie_nvidia_auto_ontology/ontology.py
index a9f44ac8..d8bc1bc3 100644
--- a/converters/nvidia/src/ossie_nvidia_auto_ontology/ontology.py
+++ b/converters/nvidia/src/ossie_nvidia_auto_ontology/ontology.py
@@ -36,6 +36,10 @@ import yaml
from .native_converter import (
OSSIE_VERSION,
_dump_yaml,
+ _field_columns,
+ _is_plain_name,
+ _keys_as_columns,
+ _relationship_as_columns,
_simple_source_column,
convert_auto_ontology_to_ossie,
)
@@ -97,6 +101,44 @@ def convert_auto_ontology_to_ossie_ontology(
return _dump_yaml(ontology), _dump_yaml(mapping)
+def convert_auto_ontology_to_embedded_ossie_ontology(
+ auto_ontology_yaml: str,
+ *,
+ model_name: str | None = None,
+) -> str:
+ """Return the Ossie ontology for a native document, its mapping embedded.
+
+ The mapping carries the semantic model ``convert_auto_ontology_to_ossie``
+ returns for the same document and name, so the one document is complete.
+ See ``embed_mapping`` for when this form is wanted.
+ """
+ semantic_model = yaml.safe_load(
+ convert_auto_ontology_to_ossie(auto_ontology_yaml, model_name=model_name)
+ )
+ ontology, mapping = build_ontology_documents(semantic_model)
+ return _dump_yaml(embed_mapping(ontology, mapping, semantic_model))
+
+
+def embed_mapping(
+ ontology: dict[str, Any],
+ mapping: dict[str, Any],
+ semantic_model: dict[str, Any],
+) -> dict[str, Any]:
+ """Return *ontology* with *mapping* embedded in it, as ``ontology_mappings``.
+
+ The specification has replaced embedded mappings with mapping documents
+ but still accepts them, and asks for them where a tool requires them:
+ until tools read mapping documents, some read only this form. An embedded
+ mapping carries its semantic model whole rather than referencing it.
+ """
+ embedded: dict[str, Any] = {"name": mapping["name"]}
+ if mapping.get("description"):
+ embedded["description"] = mapping["description"]
+ embedded["semantic_model"] = semantic_model
+ embedded["concept_mappings"] = mapping["concept_mappings"]
+ return {**ontology, "ontology_mappings": [embedded]}
+
+
def build_ontology_documents(
semantic_model: dict[str, Any],
*,
@@ -105,8 +147,20 @@ def build_ontology_documents(
) -> tuple[dict[str, Any], dict[str, Any]]:
"""Return the ontology and the mapping documents for *semantic_model*."""
model_name = str(semantic_model["name"])
- datasets = semantic_model.get("datasets") or []
- relationships = semantic_model.get("relationships") or []
+ # Keys and relationships are read by source column, whether the model
+ # names their columns by field, as this converter exports them, or not.
+ field_columns = {
+ str(dataset["name"]): _field_columns(dataset, str(dataset["name"]), "")
+ for dataset in semantic_model.get("datasets") or []
+ }
+ datasets = [
+ _keys_as_columns(dataset, field_columns[str(dataset["name"])])
+ for dataset in semantic_model.get("datasets") or []
+ ]
+ relationships = [
+ _relationship_as_columns(relationship, field_columns)
+ for relationship in semantic_model.get("relationships") or []
+ ]
concepts = _read_concepts(datasets, relationships)
ontology: dict[str, Any] = {
@@ -236,10 +290,11 @@ def _relationships(
if item.get("description"):
relationship["description"] = str(item["description"])
verb = "is identified by" if identifies else "has"
+ reading = field_name.replace("_", " ")
relationship |= {
"roles": [{"concept": value_type}],
"multiplicity": "OneToOne" if identifies else "ManyToOne",
- "verbalizes": [f"{{{concept.name}}} {verb} {field_name} {{{value_type}}}"],
+ "verbalizes": [f"{{{concept.name}}} {verb} {reading} {{{value_type}}}"],
}
relationships.append(relationship)
for name, link in concept.links:
@@ -414,11 +469,17 @@ def _field_column(dataset: dict[str, Any], item: dict[str, Any]) -> str | None:
def _field_expression(dataset: dict[str, Any], item: dict[str, Any]) -> str:
- return f"{_quote(str(dataset['name']))}.{_quote(str(item['name']))}"
+ return f"{_sql_name(str(dataset['name']))}.{_sql_name(str(item['name']))}"
-def _quote(identifier: str) -> str:
- return '"' + identifier.replace('"', '""') + '"'
+def _sql_name(name: str) -> str:
+ """Write *name* as SQL reads it, quoted only where it has to be.
+
+ Not every tool that reads a mapping expression parses quoted names, while
+ every one reads an identifier, which is what this converter names
+ datasets and fields with.
+ """
+ return name if _is_plain_name(name) else '"' + name.replace('"', '""') + '"'
def _pascal_case(text: str) -> str:
diff --git a/converters/nvidia/tests/test_converter.py b/converters/nvidia/tests/test_converter.py
index 72846407..600acabe 100644
--- a/converters/nvidia/tests/test_converter.py
+++ b/converters/nvidia/tests/test_converter.py
@@ -647,9 +647,12 @@ def test_split_links_pick_their_dataset_from_all_links_of_the_table_pair() -> No
ossie = yaml.safe_load(convert_auto_ontology_to_ossie(yaml.safe_dump(native)))
+ # The added attribute exposes referrer_id, so the relationship names its field.
assert _relationship_columns(ossie) == {
"orders_customer_id_to_customers": [("customer_id", "customer_id")],
- "orders_referrer_id_to_customers": [("referrer_id", "customer_id")],
+ "orders_referrer_id_to_customers": [
+ ("orders_orders_referrer_id", "customer_id")
+ ],
}
@@ -684,10 +687,11 @@ def test_a_composite_key_known_only_from_semantic_fks_stays_one_relationship() -
ossie = convert_auto_ontology_to_ossie(yaml.safe_dump(native))
+ # The added attribute exposes the target's region, so its field is named.
assert _relationship_columns(yaml.safe_load(ossie)) == {
"orders_customer_id_region_to_customers": [
("customer_id", "customer_id"),
- ("region", "region"),
+ ("region", "customers_customers_region"),
],
}
restored = yaml.safe_load(convert_ossie_to_auto_ontology(ossie))
@@ -1165,12 +1169,256 @@ def test_a_qualified_name_stays_qualified_where_the_plain_name_is_taken() -> Non
)
orders["name"] = "customers (analytics.public.orders)"
- assert set(_datasets(native)) == {
+ ossie = convert_auto_ontology_to_ossie(yaml.safe_dump(native))
+ datasets = {item["name"]: item for item in yaml.safe_load(ossie)["datasets"]}
+
+ assert set(datasets) == {"customers", "customers_analytics_public_orders"}
+ assert datasets["customers_analytics_public_orders"]["ai_context"] == {
+ "synonyms": ["customers (analytics.public.orders)"]
+ }
+ restored = yaml.safe_load(convert_ossie_to_auto_ontology(ossie))
+ assert {term["name"] for term in restored["semantic_layer"]["terms"]} == {
"customers",
"customers (analytics.public.orders)",
}
+def _term(native: dict[str, Any], name: str) -> dict[str, Any]:
+ return next(
+ term for term in native["semantic_layer"]["terms"] if term["name"] == name
+ )
+
+
+def _attribute(term: dict[str, Any], name: str) -> dict[str, Any]:
+ return next(item for item in term["columns_attributes"] if item["name"] == name)
+
+
+def _display_named_native() -> dict[str, Any]:
+ """The sales fixture as Auto Ontology names things, for people."""
+ native = yaml.safe_load(_auto_ontology_yaml())
+ orders = _term(native, "orders")
+ orders["name"] = "Customer Orders"
+ _attribute(orders, "order_date")["name"] = "Order Date"
+ _attribute(_term(native, "customers"), "customer_id")["name"] = "Customer Number"
+ native["semantic_layer"]["sql_attributes"]["manual"][0]["name"] = "Net Total %"
+ analysis = native["semantic_layer"]["custom_analyses"][0]
+ analysis["name"] = "Revenue per Customer"
+ # Auto Ontology aliases each table by its term's name.
+ analysis["sql"] = (
+ 'SELECT SUM("Customer Orders".subtotal) / COUNT(DISTINCT '
+ 'customers.customer_id) AS "Revenue per Customer" FROM '
+ '"analytics"."public"."orders" AS "Customer Orders" JOIN '
+ '"analytics"."public"."customers" AS "customers" ON '
+ '"Customer Orders"."customer_id" = "customers"."customer_id"'
+ )
+ return native
+
+
+def test_display_names_become_identifiers_and_come_back() -> None:
+ native = _display_named_native()
+
+ ossie = convert_auto_ontology_to_ossie(yaml.safe_dump(native))
+ model = yaml.safe_load(ossie)
+ datasets = {dataset["name"]: dataset for dataset in model["datasets"]}
+ order_fields = {
+ item["name"]: item for item in datasets["customer_orders"]["fields"]
+ }
+
+ assert set(datasets) == {"customer_orders", "customers"}
+ assert set(order_fields) == {"order_id", "customer_id", "order_date", "net_total"}
+ assert [metric["name"] for metric in model["metrics"]] == ["revenue_per_customer"]
+ # Only a name the identifier does not spell out is kept as a synonym.
+ assert order_fields["net_total"]["ai_context"] == {"synonyms": ["Net Total %"]}
+ assert "ai_context" not in order_fields["order_date"]
+ assert "ai_context" not in datasets["customer_orders"]
+ assert (
+ model["metrics"][0]["expression"]["dialects"][0]["expression"]
+ == "SUM(customer_orders.subtotal) / COUNT(DISTINCT customers.customer_id)"
+ )
+ restored = yaml.safe_load(convert_ossie_to_auto_ontology(ossie))
+ assert restored["semantic_layer"] == native["semantic_layer"]
+ assert restored["data_layer"] == native["data_layer"]
+
+
+def test_keys_and_relationships_name_the_fields_that_read_them() -> None:
+ model = yaml.safe_load(
+ convert_auto_ontology_to_ossie(yaml.safe_dump(_display_named_native()))
+ )
+ datasets = {dataset["name"]: dataset for dataset in model["datasets"]}
+
+ assert datasets["customers"]["primary_key"] == ["customer_number"]
+ assert datasets["customer_orders"]["primary_key"] == ["order_id"]
+ assert model["relationships"] == [
+ {
+ "name": "customer_orders_customer_id_to_customers",
+ "from": "customer_orders",
+ "to": "customers",
+ "from_columns": ["customer_id"],
+ "to_columns": ["customer_number"],
+ }
+ ]
+
+
+def test_keys_and_relationships_may_name_columns_or_fields() -> None:
+ """Earlier exports and other tools name physical columns; both import alike."""
+ by_field = yaml.safe_load(
+ convert_auto_ontology_to_ossie(yaml.safe_dump(_display_named_native()))
+ )
+ by_column = deepcopy(by_field)
+ customers = next(
+ dataset for dataset in by_column["datasets"] if dataset["name"] == "customers"
+ )
+ customers["primary_key"] = ["customer_id"]
+ by_column["relationships"][0]["to_columns"] = ["customer_id"]
+
+ restored = [
+ yaml.safe_load(convert_ossie_to_auto_ontology(yaml.safe_dump(document)))
+ for document in (by_field, by_column)
+ ]
+
+ assert restored[0] == restored[1]
+ tables = {
+ table["name"]: table
+ for table in restored[0]["data_layer"]["databases"][0]["schemas"][0]["tables"]
+ }
+ assert tables["customers"]["pk"] == ["customer_id"]
+ assert restored[0]["data_layer"]["joins"][0]["join_columns"] == [
+ {"source": "customer_id", "target": "customer_id"}
+ ]
+
+
+def test_a_plain_ossie_model_resolves_field_named_keys_to_columns() -> None:
+ model = yaml.safe_load(_ossie_yaml())
+ customers = next(
+ dataset for dataset in model["datasets"] if dataset["name"] == "customers"
+ )
+ key_field = next(
+ item for item in customers["fields"] if item["name"] == "customer_id"
+ )
+ key_field["name"] = "customer_number"
+ customers["primary_key"] = ["customer_number"]
+ model["relationships"][0]["to_columns"] = ["customer_number"]
+
+ native = yaml.safe_load(convert_ossie_to_auto_ontology(yaml.safe_dump(model)))
+
+ tables = {
+ table["name"]: table
+ for table in native["data_layer"]["databases"][0]["schemas"][0]["tables"]
+ }
+ assert tables["customers"]["pk"] == ["customer_id"]
+ # The key names a field, so no column of that name reaches the catalog.
+ assert [column["name"] for column in tables["customers"]["columns"]] == [
+ "customer_id",
+ "name",
+ ]
+ assert native["data_layer"]["joins"][0]["join_columns"] == [
+ {"source": "customer_id", "target": "customer_id"}
+ ]
+
+
+def test_an_export_from_before_identifier_names_keeps_its_identifiers() -> None:
+ native = _display_named_native()
+ _term(native, "Customer Orders")["name"] = (
+ "Customer Orders (analytics.public.orders)"
+ )
+ model = yaml.safe_load(convert_auto_ontology_to_ossie(yaml.safe_dump(native)))
+ # Earlier exports used Auto Ontology's names, a term's without its suffix,
+ # and named the physical columns of keys and relationships.
+ renamed = {
+ "customer_orders": "Customer Orders",
+ "order_date": "Order Date",
+ "customer_number": "Customer Number",
+ "net_total": "Net Total %",
+ "revenue_per_customer": "Revenue per Customer",
+ }
+ for item in [
+ *model["datasets"],
+ *(item for dataset in model["datasets"] for item in dataset["fields"]),
+ *model["metrics"],
+ ]:
+ item["name"] = renamed.get(item["name"], item["name"])
+ item.pop("ai_context", None)
+ for dataset in model["datasets"]:
+ dataset["primary_key"] = [
+ "customer_id" if column == "customer_number" else column
+ for column in dataset["primary_key"]
+ ]
+ relationship = model["relationships"][0]
+ relationship |= {"from": "Customer Orders", "to_columns": ["customer_id"]}
+
+ restored = yaml.safe_load(convert_ossie_to_auto_ontology(yaml.safe_dump(model)))
+
+ assert restored["semantic_layer"] == native["semantic_layer"]
+ assert restored["data_layer"] == native["data_layer"]
+
+
+def test_a_field_renamed_in_ossie_renames_its_attribute() -> None:
+ model = yaml.safe_load(
+ convert_auto_ontology_to_ossie(yaml.safe_dump(_display_named_native()))
+ )
+ orders = next(
+ dataset for dataset in model["datasets"] if dataset["name"] == "customer_orders"
+ )
+ order_date = next(item for item in orders["fields"] if item["name"] == "order_date")
+ order_date["name"] = "placed_on"
+
+ restored = yaml.safe_load(convert_ossie_to_auto_ontology(yaml.safe_dump(model)))
+
+ names = {
+ attribute["name"]
+ for attribute in _term(restored, "Customer Orders")["columns_attributes"]
+ }
+ assert names == {"order_id", "customer_id", "placed_on"}
+
+
+def test_names_that_meet_as_identifiers_are_told_apart() -> None:
+ native = yaml.safe_load(_auto_ontology_yaml())
+ customers = _term(native, "customers")
+ _attribute(customers, "customer_name")["name"] = "Customer-ID"
+ _attribute(customers, "customer_id")["name"] = "Customer ID"
+
+ ossie = convert_auto_ontology_to_ossie(yaml.safe_dump(native))
+ fields = next(
+ dataset["fields"]
+ for dataset in yaml.safe_load(ossie)["datasets"]
+ if dataset["name"] == "customers"
+ )
+
+ assert [(item["name"], item.get("ai_context")) for item in fields] == [
+ ("customer_id", None),
+ ("customer_id_2", {"synonyms": ["Customer-ID"]}),
+ ]
+ restored = yaml.safe_load(convert_ossie_to_auto_ontology(ossie))
+ assert restored["semantic_layer"] == native["semantic_layer"]
+
+
+@pytest.mark.parametrize(
+ ("display_name", "identifier"),
+ [
+ ("Order Line Item", "order_line_item"),
+ ("OrderLineItem", "order_line_item"),
+ ("C_NATIONKEY", "c_nationkey"),
+ ("Revenue %", "revenue"),
+ ("From", "from_"),
+ ("select", "select_"),
+ ("2024 Total", "field_2024_total"),
+ ("%", "field"),
+ ("Café Owner", "café_owner"),
+ ],
+)
+def test_names_are_identifiers_sql_reads_unquoted(
+ display_name: str, identifier: str
+) -> None:
+ from sqlglot import exp, parse_one
+
+ from ossie_nvidia_auto_ontology.native_converter import _ossie_identifier
+
+ assert _ossie_identifier(display_name, "field") == identifier
+ column = parse_one(f"dataset.{identifier}")
+ assert isinstance(column, exp.Column)
+ assert column.name == identifier
+
+
def test_auto_ontology_requires_one_represented_table_per_term() -> None:
native = yaml.safe_load(_auto_ontology_yaml())
term = native["semantic_layer"]["terms"][0]
diff --git a/converters/nvidia/tests/test_ontology.py b/converters/nvidia/tests/test_ontology.py
index da9e1b37..0424e0de 100644
--- a/converters/nvidia/tests/test_ontology.py
+++ b/converters/nvidia/tests/test_ontology.py
@@ -27,8 +27,13 @@ from typing import Any
import pytest
import yaml
-from ossie_nvidia_auto_ontology import convert_auto_ontology_to_ossie_ontology
+from ossie_nvidia_auto_ontology import (
+ convert_auto_ontology_to_embedded_ossie_ontology,
+ convert_auto_ontology_to_ossie,
+ convert_auto_ontology_to_ossie_ontology,
+)
from ossie_nvidia_auto_ontology.converter import main
+from ossie_nvidia_auto_ontology.ontology import build_ontology_documents
FIXTURES = Path(__file__).parent / "fixtures"
REPOSITORY = Path(__file__).resolve().parents[3]
@@ -86,7 +91,7 @@ def test_terms_become_concepts_identified_by_their_key() -> None:
assert relationships["customer_id"]["roles"] == [{"concept": "Integer"}]
assert relationships["customer_name"]["multiplicity"] == "ManyToOne"
assert relationships["customer_name"]["verbalizes"] == [
- "{Customers} has customer_name {String}"
+ "{Customers} has customer name {String}"
]
@@ -107,7 +112,7 @@ def test_a_foreign_key_links_two_concepts_instead_of_holding_a_value() -> None:
assert {
"object_mapping": {
"concept": "Customers",
- "expression": '"orders"."customer_id"',
+ "expression": "orders.customer_id",
},
"relationship": "customers",
} in children
@@ -118,9 +123,9 @@ def test_a_simple_identifier_maps_objects_with_one_expression() -> None:
orders = _concept_mapping(mapping, "Orders")
- assert orders["object_mappings"] == [{"expression": '"orders"."order_id"'}]
+ assert orders["object_mappings"] == [{"expression": "orders.order_id"}]
assert orders["link_mappings"][0]["object_mapping"] == {
- "expression": '"orders"."order_id"'
+ "expression": "orders.order_id"
}
@@ -135,11 +140,11 @@ def test_a_key_through_a_link_maps_objects_through_the_linked_concept() -> None:
"referent_mappings": [
{
"relationship": "customer_id",
- "expression": '"orders"."customer_id"',
+ "expression": "orders.customer_id",
}
],
},
- {"relationship": "order_id", "expression": '"orders"."order_id"'},
+ {"relationship": "order_id", "expression": "orders.order_id"},
]
}
]
@@ -237,3 +242,108 @@ def test_cli_requires_the_ontology_and_mapping_outputs_together(
)
assert raised.value.code == 2
+
+
+def test_mapping_expressions_quote_only_names_sql_cannot_read_bare() -> None:
+ semantic_model = {
+ "version": "0.2.0.dev0",
+ "name": "shop",
+ "datasets": [
+ {
+ "name": "order lines",
+ "source": "shop.public.order_lines",
+ "primary_key": ["select"],
+ "fields": [
+ {
+ "name": "select",
+ "expression": {
+ "dialects": [{"dialect": "ANSI_SQL", "expression": "ID"}]
+ },
+ "datatype": "Integer",
+ }
+ ],
+ }
+ ],
+ }
+
+ _, mapping = build_ontology_documents(semantic_model)
+
+ assert mapping["concept_mappings"][0]["object_mappings"] == [
+ {"expression": '"order lines"."select"'}
+ ]
+
+
+def test_keys_naming_fields_or_columns_read_as_the_same_ontology() -> None:
+ native = _native()
+ customers = next(
+ term
+ for term in native["semantic_layer"]["terms"]
+ if term["name"] == "customers"
+ )
+ customers["columns_attributes"][0]["name"] = "Customer Number"
+ by_field = yaml.safe_load(convert_auto_ontology_to_ossie(yaml.safe_dump(native)))
+ by_column = yaml.safe_load(yaml.safe_dump(by_field))
+ for dataset in by_column["datasets"]:
+ if dataset["name"] == "customers":
+ dataset["primary_key"] = ["customer_id"]
+ by_column["relationships"][0]["to_columns"] = ["customer_id"]
+
+ assert build_ontology_documents(by_field) == build_ontology_documents(by_column)
+
+
+def test_the_embedded_form_carries_the_mapping_and_its_semantic_model() -> None:
+ native = yaml.safe_dump(_native())
+ ontology, mapping = (
+ yaml.safe_load(document)
+ for document in convert_auto_ontology_to_ossie_ontology(native)
+ )
+
+ embedded = yaml.safe_load(convert_auto_ontology_to_embedded_ossie_ontology(native))
+
+ assert embedded.pop("ontology_mappings") == [
+ {
+ "name": mapping["name"],
+ "description": mapping["description"],
+ "semantic_model": yaml.safe_load(convert_auto_ontology_to_ossie(native)),
+ "concept_mappings": mapping["concept_mappings"],
+ }
+ ]
+ assert embedded == ontology
+
+
+def test_cli_writes_an_embedded_ontology_that_passes_official_validation(
+ tmp_path: Path,
+) -> None:
+ source = tmp_path / "sales.auto_ontology.yaml"
+ source.write_text(yaml.safe_dump(_native()), encoding="utf-8")
+ embedded = tmp_path / "sales.ontology.embedded.yaml"
+
+ main(
+ [
+ "import",
+ "--input",
+ str(source),
+ "--output",
+ str(tmp_path / "sales.semantic_model.yaml"),
+ "--embedded-ontology-output",
+ str(embedded),
+ ]
+ )
+ result = subprocess.run(
+ [
+ sys.executable,
+ str(VALIDATOR),
+ "--schema",
+ str(REPOSITORY / "ontology" / "ontology.json"),
+ str(embedded),
+ ],
+ check=False,
+ capture_output=True,
+ text=True,
+ )
+
+ assert result.returncode == 0, result.stdout + result.stderr
+ text = embedded.read_text(encoding="utf-8")
+ # Repeated mappings are written out, not as YAML aliases a reader may refuse.
+ assert "&id" not in text
+ assert yaml.safe_load(text)["ontology_mappings"][0]["concept_mappings"]