You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ossie.apache.org by "georgea-tk (via GitHub)" <gi...@apache.org> on 2026/08/04 15:24:07 UTC
[I] msi_to_osi silently discards config.meta instead of mapping it to custom_extensions [ossie]
georgea-tk opened a new issue, #303:
URL: https://github.com/apache/ossie/issues/303
`MSIToOSIConverter` silently discards `config.meta` from metrics, dimensions and semantic models. MSI carries the metadata, Ossie has a place to put it (`custom_extensions`), but the converter never reads one or writes the other — and unlike other lossy paths in this converter, no `ConverterIssue` is emitted.
**Root cause**
`converters/dbt/src/ossie_dbt/msi_to_osi.py`. Each OSI object is built from a fixed handful of attributes:
```python
OSIMetric(
name=metric.name,
expression=self._make_expression(expr),
description=metric.description,
)
```
```python
return OSIField(
name=dim.name,
expression=self._make_expression(expr),
dimension=OSIDimension(is_time=is_time),
label=dim.label,
description=dim.description,
)
```
`grep -n '\.config\|meta' converters/dbt/src/ossie_dbt/msi_to_osi.py` returns nothing — `.config` is never accessed anywhere in the converter.
Both sides of the mapping already exist:
- **Source:** `PydanticMetric`, `PydanticSemanticModel` and `PydanticDimension` all expose a `config` field of type `PydanticSemanticLayerElementConfig`, whose `meta` is an arbitrary dict. dbt writes user-supplied `config.meta` into `semantic_manifest.json`, so it is present in the object the converter is handed.
- **Destination:** `custom_extensions` is defined on `Field`, `Metric`, `Dataset`, `Relationship` and `SemanticModel` in `core-spec/osi-schema.json`, and `Vendor` is documented as "Any string value is accepted", with `DBT` given as an example.
**Repro** (HEAD `88e0011148283302c9a04cd0287e00e0b9d87354`), using the repo's own test helpers:
```python
import sys
sys.path.insert(0, "src"); sys.path.insert(0, "."); sys.path.insert(0, "../../python/src")
from ossie_dbt.msi_to_osi import MSIToOSIConverter
from tests.helpers import _entity, _manifest, _simple_metric
from metricflow_semantic_interfaces.implementations.element_config import PydanticSemanticLayerElementConfig
from metricflow_semantic_interfaces.implementations.elements.dimension import PydanticDimension
from metricflow_semantic_interfaces.implementations.elements.measure import PydanticMeasure
from metricflow_semantic_interfaces.test_utils import semantic_model_with_guaranteed_meta
from metricflow_semantic_interfaces.type_enums import AggregationType, DimensionType, EntityType
meta = {"value_format": '[>=1000000]$#,##0.00,,"M";$#,##0.00', "group_label": "Booking Costs"}
sm = semantic_model_with_guaranteed_meta(
name="bookings",
entities=[_entity("booking", EntityType.PRIMARY, "booking_id")],
measures=[PydanticMeasure(name="amount", agg=AggregationType.SUM, expr="amount_eur")],
dimensions=[PydanticDimension(name="status", type=DimensionType.CATEGORICAL,
config=PydanticSemanticLayerElementConfig(meta=meta))],
)
metric = _simple_metric("total_amount", "amount")
metric.config = PydanticSemanticLayerElementConfig(meta=meta)
res = MSIToOSIConverter().convert(_manifest(semantic_models=[sm], metrics=[metric]))
osi_metric = res.output.semantic_model[0].metrics[0]
osi_field = [f for f in res.output.semantic_model[0].datasets[0].fields if f.name == "status"][0]
print("metric.custom_extensions:", osi_metric.custom_extensions)
print("field.custom_extensions :", osi_field.custom_extensions)
print("ISSUES:", res.issues)
```
Run from `converters/dbt/`.
Observed:
```
metric.custom_extensions: None
field.custom_extensions : None
ISSUES: []
```
Expected: the `meta` dict carried through as a `custom_extensions` entry — or, if that is deliberately out of scope, a `ConverterIssue` so the loss is visible.
**Why this matters**
`config.meta` is the only place dbt users can attach metadata that the MSI spec does not model — units, display formatting, grouping, ownership, links back to a source system. For anyone migrating an existing BI semantic layer into dbt it tends to be where all the presentation metadata ends up, precisely because the spec has nowhere else for it. Discarding it at the Ossie boundary means that content cannot reach any downstream consumer, even ones that would understand it.
It is also inconsistent with how this converter handles its other lossy paths: `CUMULATIVE` metrics emit `CUMULATIVE_SEMANTICS_LOSS`, conversion metrics emit `CONVERSION_METRIC_DROPPED`, natural entities emit `NATURAL_ENTITY_DROPPED`. This loss is silent.
**Suggested fix**
Map `config.meta` onto a single `custom_extensions` entry when non-empty, e.g.
```python
def _meta_extensions(config) -> Optional[List[OSICustomExtension]]:
meta = getattr(config, "meta", None) if config else None
if not meta:
return None
return [OSICustomExtension(vendor_name="DBT", data=json.dumps(meta))]
```
applied in `_convert_metrics`, `_convert_dimension`, `_convert_entity`, `_convert_measure` and `_convert_dataset`. `CustomExtension.data` is a JSON string, so the dict serialises directly.
Happy to send a PR if the approach sounds right. Two things worth deciding first:
1. **Vendor name.** `DBT` matches the schema's own example, but `COMMON` may be more appropriate if the intent is that any consumer may read it rather than dbt-specific tooling.
2. **Round-tripping.** `osi_to_msi` would need the inverse to make this lossless in both directions. Happy to include that in the same PR.
**Note:** the same `msi_to_osi` implementation is vendored in `dbt-labs/metricflow` as `metricflow/converters/msi_to_osi.py` (byte-identical apart from one word in a docstring) and ships inside dbt-core 1.12, where it generates `target/osi_document.json`. The same gap exists there.
Related: #111 covers `custom_extensions` being dropped again on the Snowflake side, so both would need addressing for metadata to survive a full dbt → Ossie → Snowflake conversion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org