You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ossie.apache.org by "christianeu-db (via GitHub)" <gi...@apache.org> on 2026/10/12 22:33:24 UTC

[I] We do not measure model portability across converters [ossie]

christianeu-db opened a new issue, #576:
URL: https://github.com/apache/ossie/issues/576

   Following the `dev@` thread [[DISCUSS][PROPOSAL] Measuring Semantic Interoperability in Ossie](https://lists.apache.org/thread.html/dnl13g1nlh7l2o6gnw9qjp7wqmrh0tds). Opening this for the technical discussion:
   
   ### The gap
   
   Ossie's goal is to "standardize how we exchange semantic metadata across analytics, AI and BI platforms." We test each converter, validate against the schema, and run CI per converter. None of that measures how much of a model survives a round trip through the portable core.
   
   A converter can pass its own tests, read and write valid Ossie documents, and still return a model no other implementation can consume. We currently have no way to tell that apart from success, or to see whether it is improving.
   
   ### Evidence
   
   A model with one dataset, one field and one `SUM` metric, run `Ossie -> vendor -> Ossie` at `0f04d8c` in both `ANSI_SQL` and `OSSIE_SQL_2026`. 14 converters measured. Three were excluded: one is not a vendor converter, one targets a table catalog with no metric concept, and one is export-only and newly merged.
   
   | Result | Count |
   |---|---|
   | Metric returns in any dialect | 12 of 14 |
   | Metric returns tagged `ANSI_SQL` | **8 of 14** |
   | Metric returns tagged `OSSIE_SQL_2026` | **1 of 14** |
   | Metric returns in a vendor dialect only | 4 of 14 |
   | Metric dropped with no diagnostic | 1 of 14 |
   | No vendor -> Ossie direction | 1 of 14 |
   
   One `SUM` over one column is close to the smallest possible test, so treat these as a floor.
   
   The metric was lost in three independent ways:
   
   1. **The dialect tag.** Three converters accept the fixture tagged `ANSI_SQL` and drop the same expression text tagged `OSSIE_SQL_2026`. The two fixtures differ only in the tag. 
   2. **The returned dialect.** Four return the metric only in their own dialect: valid, correct, unusable by the next tool.
   3. **The reference form.** `SUM(amount)` is spec-legal (`FieldExpr: Field | Field '.' Field`, and `validate.py` accepts it), and 2 of the 11 converters measured on it refused the document. One reported a cross-dataset metric on a model with exactly one dataset.
   
   Note: this is a changing landscape. #547 fixes one row above, and two converters have landed since the measurement, taking `converters/` from 17 directories to 19. The table is a snapshot of `0f04d8c`, not of current `main`.
   
   <details>
   <summary>The model used (ANSI version)</summary>
   
   ```yaml
   version: 0.2.0.dev0
   name: hello_world
   description: Minimal single-table model used to measure converter round-trip fidelity.
   datasets:
     - name: sales
       source: hello.public.sales
       description: Single fact table holding one monetary column.
       fields:
         - name: amount
           datatype: Decimal
           description: Transaction amount.
           expression:
             dialects:
               - dialect: ANSI_SQL
                 expression: amount
   metrics:
     - name: total_amount
       datatype: Decimal
       description: Sum of the amount column.
       expression:
         dialects:
           - dialect: ANSI_SQL
             expression: SUM(sales.amount)
   ```
   
   Two variants made the result diagnostic: the same document with both `dialect:` tags changed to `OSSIE_SQL_2026` and nothing else, which attributes a failure to tag handling rather than parsing; and the same document with the metric expression changed to `SUM(amount)`. The `datatype` is required in practice — one converter refuses to export without it, and another silently emits a monetary column as a string.
   
   </details>
   
   ### Looking for collaborators
   
   A single-converter round trip is one path. `vendor A -> Ossie -> vendor B` is the one users care about, and nothing here measures it yet. I would like help thinking through:
   
   - **Test cases.** Which reference models, and in what order of complexity.
   - **What portability means, and across which actions.** Export, import, a round trip through one converter, a hand-off between two, or all of these scored separately. I'd initially propose focusing on `Ossie -> vendor -> Ossie`
   - **What counts as preserved.** The run above scored survival (the measure returns in any dialect) separately from portability (it returns as `ANSI_SQL` or `OSSIE_SQL_2026`), because a vendor-dialect return is a working measure the next tool cannot read. Is that the right split, and what else belongs?
   - **A fair denominator.** An export-only converter and a non-semantic target have no round trip to lose. Scoring them as failures penalizes everyone else; excluding them case by case makes the figure unstable.
   - **Standardizing measurement across converters.** Entry points differ, one has no CLI, and one targets a live service, so comparable results need an agreed invocation and grading rule rather than per-converter judgement.
   - Anything else people think is worth measuring.
   
   Replies here or on the `dev@` thread are equally welcome.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add measurements for model portability across converters [ossie]

Posted by "MrBlack1995 (via GitHub)" <gi...@apache.org>.
MrBlack1995 commented on issue #576:
URL: https://github.com/apache/ossie/issues/576#issuecomment-6185497123

   Thanks @kayemkim, this is exactly the grounding the strawman needed. Running all three hello-world variants through the nine drivable converters on eabe562 and putting the 25 returned docs through `validate.py` turns the rubric discussion into something we can calibrate against instead of argue in the abstract.
   
   Agreed on the core point: conformance is a floor here, not a gate. If all 25 come back passing, and `validate.py` skips DAX, THOUGHTSPOT and SIGMA by design so it cannot even speak to the vendor-only returns, then output validity separates nothing at this level. The grader has to diff the returned document against the input element by element. That is a harness job, not a validator job, and we should build it as one.
   
   On the two outcomes the rubric has no row for, both need to be first class:
   
   - ADDED (orionbelt materializing `_sum_amount` alongside `total_amount` on the SUM(amount) variant): correct and round-trippable, and the next tool now carries an extra element. The grader should record it, not penalize it.
   - RELABELED (dbt, wisdom, honeydew, and orionbelt re-tagging OSSIE_SQL_2026 as ANSI_SQL with the text unchanged, only cube keeping the tag): this is the important one. PORTABLE as currently defined scores all five the same and hides the 1-of-14 figure from the first run. +1 to keeping PORTABLE as the top-level score and tracking "tag preserved" as a sub-metric under it, so that signal survives.
   
   On the reference form, agreed we should not bake an answer into the ladder while #486 (qualified `dataset.field` only) and #343 (bare name as a source column inside a dataset-scoped metric) are open. Let us carry the qualified form as the primary fixture at every rung and keep the bare form as a sibling variant, the way the third run already does. That keeps the suite honest regardless of how those two land.
   
   On mechanics, building on `converters/cube/tools/interop_matrix.py` and the interop/ harness in #348 (adapters plus a manifest) rather than beside them makes sense to me. Where it finally lives follows #523, so I do not think we need to settle placement to start.
   
   Also noted and reproduced from your run: sigma dropping SUM(amount) as a cross-dataset metric on a one-dataset model, and thoughtspot exiting non-zero and leaving it out of the TML.
   
   So concretely, I would like to take you up on the offer: ladder fixtures as `validate.py`-passing YAML, plus a stdlib-only grader that diffs input against returned output into the per-element outcomes above, including ADDED and RELABELED.
   
   To unblock that, I will post the M0 fixture set here within the next week: the three variants (ANSI_SQL qualified as primary, OSSIE_SQL_2026 qualified for the tag-sensitivity case, and the bare SUM(amount) variant), each validate.py-passing, with a small manifest row per fixture naming the element under test and the reference form. Once we are happy with the M0 shape you can post the M0 table for all three variants against the nine converters, and we extend the same shape up the ladder. If you have a preferred manifest layout, say so and I will build the proposal around it.
   
   And no problem @ekut, understood on the executor focus. We will keep the validation thread in #573 and not pull you into the suite shaping.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add measurements for model portability across converters [ossie]

Posted by "MrBlack1995 (via GitHub)" <gi...@apache.org>.
MrBlack1995 commented on issue #576:
URL: https://github.com/apache/ossie/issues/576#issuecomment-6161201333

   This resonates. I have been working the same gap from the Microsoft converter side, and I would like to help on the measurement itself.
   
   I ran your fixture through `converters/microsoft` at current `main` (`7496811`) to put a number on it. Ossie to vendor to Ossie, across all three variants (the ANSI_SQL fixture, the `OSSIE_SQL_2026`-tagged variant, and the `SUM(amount)` variant): the `total_amount` metric comes back tagged **DAX only**, as `SUM('sales'[amount])`. So Microsoft is a "returns in a vendor dialect only" row. The portable input is rewritten to DAX on import and never translated back, so even the `OSSIE_SQL_2026`-tagged input does not survive the round trip.
   
   I am building a deterministic DAX to `OSSIE_SQL_2026` translation in that converter (refuse by default, DAX kept as a fallback). That should move the Microsoft row from vendor-only to portable, and I will post the before/after here as evidence that the metric is actionable.
   
   Two opinions from the converter side on your open questions:
   
   - The survival vs portability split is the right one. A vendor-dialect return is a green test and a dead hand off, so scoring them separately is what makes "passing" distinguishable from "portable".
   - Denominator: score export-only and non-semantic targets on a separate axis rather than as round-trip failures. Otherwise one newly merged export-only converter destabilizes everyone's number, as your `0f04d8c` snapshot already shows.
   
   Happy to standardize the Microsoft invocation into whatever harness shape you and Justin settle on. I will follow up on the `dev@` thread too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] We do not measure model portability across converters [ossie]

Posted by "christianeu-db (via GitHub)" <gi...@apache.org>.
christianeu-db commented on issue #576:
URL: https://github.com/apache/ossie/issues/576#issuecomment-6136903507

   @ekut and @kayemkim curious if you'd be interested in collaborating on this item. Tagging you both since you've been critically evaluating the core spec as a building block


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add measurements for model portability across converters [ossie]

Posted by "kayemkim (via GitHub)" <gi...@apache.org>.
kayemkim commented on issue #576:
URL: https://github.com/apache/ossie/issues/576#issuecomment-6182120554

   Before answering the shape question I ran the hello world model, all three variants, through the nine Python converters I could drive both ways from the command line on current main (eabe562), and put every returned document through validate.py.
   
   All 25 documents that came back pass. That includes the three where the metric is gone (omni and sigma on the OSSIE_SQL_2026 tag, sigma on `SUM(amount)`), the four where it returns in DAX or THOUGHTSPOT only, and one where a metric appeared that was not in the input. validate.py skips DAX, THOUGHTSPOT and SIGMA expressions by design, so it cannot even say whether a vendor-only return is well formed. Conformance of the output is a floor the scoring can assume, not a gate that separates anything at this level. The grader has to diff the returned document against the input, element by element, and that is a harness job rather than a validator job.
   
   Two outcomes the strawman rubric has no row for, both visible on M0:
   
   - ADDED. orionbelt returns two metrics for one on the `SUM(amount)` variant. OBML materializes a `_sum_amount` measure that the `total_amount` definition references, and both come back as Ossie metrics. Correct and round-trippable, and a second element the next tool now carries.
   - RELABELED. dbt, wisdom and honeydew return the OSSIE_SQL_2026 variant tagged ANSI_SQL with the text unchanged (honeydew says so: "no ANSI_SQL dialect found; using first available"), and orionbelt does the same while quoting the identifiers. Only cube keeps the tag. PORTABLE as defined scores all five the same, which hides the 1-of-14 figure from Chris's first run. I would keep PORTABLE as the top-level score and track "tag preserved" under it.
   
   Also confirming two rows from the first run on this tree: sigma drops `SUM(amount)` as a cross-dataset metric on a one-dataset model, and thoughtspot exits non-zero and leaves it out of the TML.
   
   On the ladder, every rung can be written against the spec as it stands. The axis to hold is the reference form. Whether `SUM(amount)` is a valid model-level metric is exactly what #486 (qualified `dataset.field` only) and #343 (bare name means a source column, inside a dataset-scoped metric) are settling, so I would carry the qualified form as the primary fixture at every rung and keep the bare form as a variant, the way Chris's third run already does, rather than bake one answer into the ladder.
   
   On mechanics, two things already in the tree are worth building on rather than beside. `converters/cube/tools/interop_matrix.py` runs a Cube model through Ossie to every other spoke with a per-converter `uv` invocation and a validate.py step, and #348 proposes an `interop/` harness with adapters and a manifest. Where any of this lives presumably follows #523.
   
   What I can take: the ladder fixtures as validate.py-passing YAML, and a grader that diffs input against returned output into the per-element outcomes above, stdlib only. Once the fixture set is agreed I will post the M0 table for all three variants against the nine converters here.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add measurements for model portability across converters [ossie]

Posted by "MrBlack1995 (via GitHub)" <gi...@apache.org>.
MrBlack1995 commented on issue #576:
URL: https://github.com/apache/ossie/issues/576#issuecomment-6162041941

   @ekut @kayemkim, Chris pulled you in here and I am glad he did. I would rather we shape this together than have me hand down a rubric, so treat the below as a strawman to argue with, not a proposal.
   
   The reason your angles fit: a fidelity measure is only as good as the edges it pins down. Alex, your #498 (where NULLs sort in a window `ORDER BY`) is exactly the kind of expression-language edge a round-trip ladder has to make explicit. Kyoung Min, your #488 and #393 are the validation and conformance half of the same question: a document can round-trip and still not actually conform. I would love both of those perspectives on the shape.
   
   A starting point to react to:
   
   ### A graduated ladder of reference models
   
   Each model adds exactly one capability on top of the previous, so a failure localizes to a single construct rather than a whole model.
   
   | Model | Adds | Element it exercises |
   |---|---|---|
   | M0 | one dataset, one field, one `SUM` metric | the existing baseline |
   | M1 | a scalar field expression (calculated column), e.g. `amount * 1.1` or a `CASE` | field expression, no aggregation |
   | M2 | the core aggregate set: `SUM`, `COUNT`, `COUNT(DISTINCT)`, `AVG`, `MIN`, `MAX` | aggregates |
   | M3 | a filtered aggregate in the portable form `SUM(CASE WHEN status = 'completed' THEN amount ELSE 0 END)` | conditional aggregation |
   | M4 | two datasets and a relationship, with a metric that aggregates across the join | relationships plus cross-dataset metric |
   | M5 | a metric that references another metric | metric composition |
   | M6 | date/time scalars: `YEAR`, `DATE_TRUNC`, `DATEDIFF` | date/time core |
   | M7 | one construct deliberately outside the portable core (e.g. a window or time-intelligence measure) | the refusal/stash path, to confirm it is diagnosed rather than silently dropped |
   
   ### A per-element fidelity rubric (strawman)
   
   Extending the survival-vs-portability split into a per-element outcome, scored for each metric and field a model contains:
   
   | Outcome | Meaning |
   |---|---|
   | PORTABLE | returns tagged `ANSI_SQL` or `OSSIE_SQL_2026` |
   | SURVIVED (vendor only) | returns in a vendor dialect only: a working element the next tool cannot read |
   | DROPPED (diagnosed) | removed, with a warning or notice |
   | DROPPED (silent) | removed with no diagnostic |
   | REJECTED | the converter refuses the whole document |
   
   Portability score is PORTABLE over in-scope elements; SURVIVED and DROPPED (silent) tracked separately rather than folded into pass/fail. I would treat diagnostic quality as its own axis: a diagnosed drop beats a silent one, and M7 should score well if the unsupported construct comes back with a clear reason. Conformance (does the returned document actually validate, per Kyoung Min's #488/#393) is arguably a gate before any of this scoring even applies.
   
   Open questions I do not think we should answer alone:
   
   - Denominator: how to score export-only and non-semantic targets without destabilizing everyone's number.
   - Direction: start with `Ossie -> vendor -> Ossie`, and when to add `vendor A -> Ossie -> vendor B`.
   - Mechanics: entry points differ (one has no CLI, one is a live service), so a uniform invocation matters. If Julian's `.ost` (#501) and the view expander (#502) are the intended home, the ladder should probably be expressed there.
   
   What I can bring concretely: I will run this whole ladder through the Microsoft converter and post the actual numbers, so we are arguing over real results rather than a hypothetical. Where do the two of you want to take the shape from here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add measurements for model portability across converters [ossie]

Posted by "MrBlack1995 (via GitHub)" <gi...@apache.org>.
MrBlack1995 commented on issue #576:
URL: https://github.com/apache/ossie/issues/576#issuecomment-6211984316

   @kayemkim your eabe562 run is the right way to start, so I turned exactly that into a committed baseline rung and the first three rungs above it, one capability added per step, so a difference localizes to a single construct. Everything below is a strawman to push on, not a fixed suite.
   
   **M0, baseline (what you already ran).** One dataset, one field, one `SUM` metric. The three variants are the ones you drove: `ANSI_SQL` qualified (`SUM(sales.amount)`), the same model tagged `OSSIE_SQL_2026` to isolate tag survival, and the bare-reference form (`SUM(amount)`). This just pins your hello-world-times-three run as a named artifact so we calibrate on identical files rather than on wording.
   
   **M1, scalar field, no aggregation.** Adds one calculated-column field expression (`amount * 1.1`) next to the base column. Isolates whether a plain scalar field survives, separate from any aggregate.
   
   **M2, the core aggregate set.** `SUM`, `COUNT`, `COUNT(DISTINCT ...)`, `AVG`, `MIN`, `MAX` as six metrics over one dataset. This is the set the expression spec marks REQUIRED, so a converter dropping or relabelling any one of them is the first real signal.
   
   **M3, conditional aggregation.** One filtered aggregate in the spec's portable form, `SUM(CASE WHEN sales.status = 'completed' THEN sales.amount ELSE 0 END)`. This is the portable shape of the filtered-measure pattern, so it tells us which converters keep a `CASE`-guarded aggregate intact versus flatten it.
   
   Each rung carries the qualified `dataset.field` form as primary and, where it applies (M0, M2, M3), the bare form as a sibling variant, matching the open #486 / #343 reference-form question rather than pre-deciding it. All of them are authored in portable dialect on purpose: a vendor-tagged input only exercises one converter, and the point here is the cross-converter diff.
   
   I am deliberately stopping at M3. M4 would add relationships and cross-dataset metrics, which overlaps Justin's query-interface work in #354, and metric-to-metric composition depends on dataset-scoped metrics that are still in flight in #591. The window and offset rungs (prior period, running total, ratio-to-total) depend on #587's NULL-placement default landing first. I would rather calibrate the grader on M0-M3, where the semantics are settled, before we climb into those.
   
   On outcome vocabulary, I kept it to what your run can already produce per element against the returned document: PORTABLE (with a tag-preserved sub-flag), SURVIVED-vendor-only, ADDED, RELABELED, DROPPED-diagnosed, DROPPED-silent, REJECTED. Conformance stays a floor (`validate.py` passing), scoring stays the grader's job, consistent with #348.
   
   I put the fixtures and the manifest up as #596 so they are reviewable in-tree rather than pasted here; all eight validate on current main (9531429). I placed them under `compliance/portability/` as a starting point, but where they should live (next to the interop harness in #348, or under `validation/`) is your call since you have driven the converters. Tell me the path you would expect and I will move them to match.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add measurements for model portability across converters [ossie]

Posted by "ekut (via GitHub)" <gi...@apache.org>.
ekut commented on issue #576:
URL: https://github.com/apache/ossie/issues/576#issuecomment-6180827333

   Thanks for thinking of me, Chris, and @MrBlack1995 for the detailed strawman. Measuring this across converters is useful work. I can't commit time to shaping the suite, though: my focus is on the executor side. The part that touches validation I've answered in #573.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org