You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ossie.apache.org by jb...@apache.org on 2026/09/24 14:06:16 UTC
(ossie) branch main updated: feat(thoughtspot): bidirectional ThoughtSpot TML <-> Ossie converter (#364)
This is an automated email from the ASF dual-hosted git repository.
jbonofre pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/ossie.git
The following commit(s) were added to refs/heads/main by this push:
new d628f683 feat(thoughtspot): bidirectional ThoughtSpot TML <-> Ossie converter (#364)
d628f683 is described below
commit d628f6830d27493943d10e0e84069e8c5c27cb51
Author: nordlaw78 <dj...@users.noreply.github.com>
AuthorDate: Fri Sep 25 00:05:42 2026 +1000
feat(thoughtspot): bidirectional ThoughtSpot TML <-> Ossie converter (#364)
* feat(thoughtspot): package skeleton, CI workflow, and ASF header gate
* fix(thoughtspot): add ASF header to pyproject.toml, pin CI actions to SHAs
* feat(thoughtspot): vocabulary constants with dialect registration flag
Adds ossie_thoughtspot.constants (VENDOR_KEY, DIALECT, FALLBACK_DIALECT,
STASH_VERSION, SPEC_SERIES, DIALECT_IS_REGISTERED) per task-2-brief.md.
VENDOR_KEY and DIALECT are kept as separate names despite sharing a value
today (P6) since they are governed by different upstream processes.
Also adds converters/thoughtspot/.gitignore, copied verbatim from
converters/honeydew/.gitignore, so a local uv build's egg-info directory
can no longer land in a commit (Task 1 had to delete it by hand).
* feat(thoughtspot): YAML 1.2 codec so on/off/yes/no survive a round trip
* test(thoughtspot): pin loader/dumper fixes independently with plain-PyYAML controls
Complete YAML11_BOOL_TOKENS (add Y, N, YES, NO, OFF) and add two differential
tests so the dumper's quoting fix and the loader's 1.2-resolver fix each have
a test that fails if that half is removed -- the prior tests only exercised
the loader, and 9/11 dumper-quote assertions passed against plain SafeDumper.
* feat(thoughtspot): structured never-silent issue reporting
* feat(thoughtspot): custom_extensions stash with staleness-guarded restore
* feat(thoughtspot): identifier normalisation with case-folded collision resolution
* fix(thoughtspot): reject ambiguous column refs; document normalise ASCII-only limitation
Review findings on 9ad8fe6:
- split_column_ref now raises on a reference containing more than one '::'
instead of silently mis-splitting (e.g. a table name formatted with '::'
in it corrupted the table/column boundary). Delimiter/escaping redesign
is left to Plan C; loud failure is the interim behaviour.
- normalise's ASCII-only behaviour (non-ASCII characters dropped, not
transliterated) is now documented as a known limitation rather than left
implicit, with tests pinning current behaviour for accented Latin and a
CJK-only name so it can't silently regress.
* feat(thoughtspot): key derivation that refuses to manufacture false keys
* fix(thoughtspot): report empty-to_columns relationships, document KD3, add duplicate-key test
Review findings on task 7:
- KD1: a to-one, non-residual relationship with empty to_columns previously
vanished silently (no key, no issue) because it still "qualified" for the
seen-building loop but was excluded by its own `if cols` guard, while the
KD2 loop was gated on seen being non-empty. _qualifies() now requires
non-empty to_columns, and the KD2 loop reports an empty-to_columns
relationship unconditionally rather than only when a key was derived.
- Documented KD3 (orientation re-checked downstream by
converters/databricks) in the module docstring, resolving the dangling
KD1-KD3 reference.
- Added a test for two qualifying relationships agreeing on the same
columns (e.g. a dimension joined from two fact tables), which must
collapse into one unique key and still yield a primary key rather than
being mistaken for disagreement.
* docs(thoughtspot): README with direction statement and coverage matrix
Documents the two conversion directions, the 1+N TML document shape, and the
Ossie #351 dialect caveat. The coverage matrix enumerates what the converter
does not carry (L1-L6); L2 (row-level security) is called out as error
severity, naming every affected table, since rls_rules is the primary
mechanism ThoughtSpot customers are actively migrating onto. The Status
section lists the foundations Tasks 1-7 actually shipped (YAML 1.2 codec,
issue reporting, custom_extensions stash, identifier and key derivation).
test_readme.py asserts structure (both directions named, a Coverage matrix
heading with L-numbered rows, the #351 reference) rather than prose wording,
per P16/P21, so the matrix cannot quietly disappear without a test noticing.
* docs(thoughtspot): add identifier-derivation limitations, tighten L2 wording
Review findings on the Task 8 README:
- Add a Known limitations section documenting identifiers.py's ASCII-only
normalise() (drops rather than transliterates non-[0-9a-z] characters,
e.g. Cafe -> caf, Urun -> r_n, CJK-only raises ValueError). This lived only
in the module docstring; a reader hitting non-English column names would
find nothing about it in the README. Kept out of the coverage matrix since
it is a different axis (identifier-derivation correctness, not an
uncarried TML construct) - the section says so explicitly.
- Tighten the L2 (row-level security) Limitation cell: one error-severity
issue is raised, its message naming every affected table - not one issue
per table, matching the TS_RLS_DROPPED pattern in test_issues.py.
* build(thoughtspot): match sibling converters' packaging and CI shape
pyproject.toml used setuptools + optional-dependencies where all eight
sibling converters use hatchling + PEP 735 dependency-groups, and was
missing license/readme/authors/pytest/uv config. The built wheel was
missing License-Expression, Author-email, Project-URL, and the embedded
README, and leaked pytest/hypothesis into distribution metadata via
Requires-Dist. Converted to the sibling shape and dropped hypothesis
entirely (unused on this branch).
CI ran `pip install -e ".[dev]"` and never touched uv, so the committed
uv.lock had no consumer. Switched to `uv sync` + `uv run pytest` and
widened the version matrix to 3.10-3.14 to match omni (the sibling with
the same requires-python floor). Regenerated uv.lock. SHA-pinned actions
left unchanged.
* fix(thoughtspot): correct KD2's upstream-warning claim; harden _yaml contract
keys.py (I1): the KD2 message unconditionally claimed "Ossie validation
will report a to_columns coverage warning" for every disqualified
relationship. That's wrong for an empty to_columns (upstream's schema
requires minItems: 1, so the document fails validation outright before
any coverage check runs — a schema ERROR, not a warning) and often wrong
for residual-predicate joins whose columns already cover the derived key
(the canonical SCD-2 shape). Split the message: the "not a declared key"
half is unconditional, the upstream-prediction half is appended only when
`not any(set(key) <= set(rel.to_columns) for key in seen)` — mirroring
validate.py:159-165 exactly. Empty to_columns gets its own ERROR-severity
issue with a remedy that doesn't say "Expected".
_yaml.py: load() now wraps a parse failure in ConversionError (I4),
matching stash.py's X4 never-a-bare-traceback contract. dump() now passes
allow_unicode=True (I5), so a non-ASCII label round-trips as a literal
character instead of an escaped one — every Ossie document Plans C/D
emit passes through this function.
Minor: keys.py module docstring no longer says databricks' orientation
swap is "silent" (it calls _warn()); documented Relationship's frozen=True
does not imply safe hashability when to_columns is a list; stash.py's X8
comment now states the guid/obj_id/fqn check is top-level only. Added a
test for restore()'s default (no witness_key) shape.
Tests: 107 passing (102 + 5 new: 2 keys.py, 3 _yaml.py; the stash.py
default-shape test replaces no prior assertion so nets to +1 net file
but the count above already reflects all additions).
* fix(thoughtspot): fold diacritics in normalise(); close a column-ref ambiguity gap
identifiers.py (R14, revising an earlier ruling): normalise() now applies
Unicode NFKD decomposition before the ASCII lowercase-and-substitute fold.
An earlier ruling treated the ASCII-only behaviour as a stated boundary on
the grounds that transliteration is a product decision — right about
transliteration (Japanese -> romaji), wrong about canonical decomposition,
which is stdlib and needs no policy choice. "Café" -> "cafe", "Ürün" ->
"urun", "Zürich" -> "zurich", "İstanbul" -> "istanbul", "naïve" -> "naive"
now fold correctly; a script with no ASCII decomposition (CJK, Cyrillic)
still raises, and conventional expansions (German "Müller" -> "mueller")
remain an open, separate question. Updated the module/function docstrings,
the README's Known limitations section, and re-pinned the limitation
tests to their new (narrower) expected values.
identifiers.py (M3): split_column_ref's ambiguity guard counted "::" via
str.count, which is non-overlapping — a run of three consecutive colons
(table ending in ':' immediately before the '::' delimiter) counts as one
match and silently mis-split. format_column_ref("ORDERS:", "Col") and
format_column_ref("ORDERS", ":Col") both produce the identical string
"[ORDERS:::Col]" and are genuinely ambiguous; the guard now also rejects
a captured column starting with ':'. Added tests for both origins.
Tests: 112 passing (107 + 5 new: 5 identifiers.py — 2 M3 cases; the R14
limitation tests replace existing pinned cases rather than adding, net
+3 from the expanded parametrize list).
* docs(thoughtspot): future-tense unimplemented claims; name the external ruleset
README.md (I6): "Converts between..." and "Each row raises a structured
ConverterIssue... nothing is dropped silently" both describe behaviour
that does not exist yet — neither conversion direction is implemented,
and no code raises any of L1-L6. Changed to future tense ("will
convert", "is required to raise... once the conversion directions
land"). This is exactly the shape of complaint discussion #325 raises
against an unfulfilled README promise.
README.md (I7): the Rules section named "the ThoughtSpot skills
repository" as the normative source for ~30 rule identifiers
(ID1-ID4, X1-X9, KD1-KD3, NM1-NM6, and more) with no URL — unresolvable
from inside this ASF repo, and no sibling converter defers its
normative behaviour to an external vendor-controlled document. Named
the repository (thoughtspot-agent-skills) explicitly, stated it is not
ASF-hosted, acknowledged the vendor-neutrality gap plainly, and recorded
the intent to contribute the mapping tables into this repository rather
than vendoring them now (a larger change needing its own review).
README.md (M9): L2's severity justification was a commercial-trend
argument ("the mechanism customers are actively migrating onto") in an
ASF repo. Restated as the technical property: RLS is unrepresentable in
Ossie core and is security-bearing.
test_packaging.py (M4): the ASF-header test globbed only src/**/*.py and
tests/**/*.py, leaving pyproject.toml, .gitignore, README.md, and the CI
workflow ungated. Added a check for all four, matching on the licence
text itself since each file uses a different comment syntax.
Tests: 113 passing (112 + 1 new).
* docs(thoughtspot): fix README dev commands for the hatchling/uv packaging
The Development section still said `pip install -e ".[dev]"` /
`python -m pytest tests/ -v`, left over from before the switch to
hatchling + PEP 735 dependency-groups. That install now warns the
package has no `dev` extra and skips pytest, so the next line fails
with ModuleNotFoundError. Replace with the uv invocation CI actually
runs.
* fix(thoughtspot): flip DIALECT_IS_REGISTERED now that ossie#351 merged
apache/ossie#351 merged 2026-09-01: THOUGHTSPOT is now a registered Dialect
enum member and is in SKIP_SQL_VALIDATION. Flip DIALECT_IS_REGISTERED to
True, rename FALLBACK_DIALECT to PORTABLE_DIALECT to reflect its new role
(emitted alongside THOUGHTSPOT for portable expressions per P8, not instead
of it), and update the README status section and both tests that encoded
the old assumption.
* feat(thoughtspot): expression catalog vocabulary and spec-coverage gate
Task 1 of the expression-translation plan (Plan B). Adds Classification,
Variant and Construct (_types.py), an empty CATALOG to be populated across
Tasks 3-7, and spec_construct_names() — an oracle that parses the upstream
core-spec/expression_language.md so a construct added upstream fails this
package's build instead of silently going unsupported.
spec_construct_names() returns 137 names, not the plan's target of 146; see
task-1-report.md for the family-by-family reconciliation and the concrete
gaps (Window's OVER-clause/aggregation prose, a few alias-pair rows). The
mapping document Tasks 3-9 are meant to transcribe from
(ts-ossie-function-mapping.md) does not exist in this repository yet.
* fix(thoughtspot): reconcile 137 vs 146 with an explicit divergence set
spec_construct_names() (137, parsed live from upstream core-spec) and the
mapping document's rule-E1 census (146, one row per construct incl. a few
described only in prose) count by different units. Per coordinator ruling,
keep the strict spec->catalog gate (catches an upstream addition) and add
CONVENTION_DIVERGENCES: the exact 9 constructs the mapping document counts
separately that the spec never gives a discrete table row - verified row by
row against the mapping document (now located at
thoughtspot-agent-skills/docs/ossie/ts-ossie-function-mapping.md, a
different repo, per the coordinator). test_the_two_counts_reconcile pins
137 + 9 == 146 so the two counts cannot drift apart silently.
Corrects this task's own earlier speculation: CEIL/CEILING, TRUNC/TRUNCATE
and TRUE/FALSE are each one row in the mapping document too (not split),
so they are a CATALOG-key spelling question for Tasks 3-8, not divergences.
Documented prominently in catalog.py's new "Spelling" docstring section.
* fix(thoughtspot): correct Spelling audit, drop dead code, gate empty templates
Review findings on Task 1:
- Spelling docstring omitted a AND b / a OR b (-> "expr1 AND expr2" /
"expr1 OR expr2", from the Boolean Functions table) and misattributed
IS DISTINCT FROM / IS NOT DISTINCT FROM to the top-level summary table
when they actually come from the Null-Safe Comparison code fence. Full
section rewritten grouped by extractor function and re-audited against
a live spec_construct_names() run plus the mapping document's literal
row text, with an explicit rule: the live function output is the
oracle, never this list or the mapping document's prose.
- Deleted unused _SUPPORTED_LIST_CUE_RE and corrected the docstring claim
of a "Supported ...:" prose-cue exclusion mechanism that doesn't exist
- argument-vocabulary bullet lists are simply invisible to
_extract_tables(), which only reads pipe-prefixed lines.
- Construct.__post_init__ now rejects a DIRECT/PASSTHROUGH row with no
template, so a family task can't add one that passes validation and
the coverage gate but fails only when something tries to emit it.
New tests/expressions/test_types.py covers all four branches.
* feat(thoughtspot): expression emitters for direct, passthrough and unmappable
Adds emit_direct/emit_passthrough/emit_unmappable — one renderer per
Classification, built before the family tasks (3-8) populate CATALOG, so
each family has something real to render into.
Uses the corrected signatures (object_ref required, keyword-only) rather
than the brief's own Interfaces block, which disagreed with its test
snippets on this point:
emit_direct(construct, args) -> str
emit_passthrough(construct, args, log, *, object_ref, has_parameter=False,
partition_column=None) -> str
emit_unmappable(construct, log, *, object_ref) -> None
- E2/E7: emit_direct substitutes positionally via str.format and rejects
an argument-count mismatch rather than tolerating it.
- E4/E7/E12: emit_passthrough renders the SQL body as a quoted template
string (never substituting args into it — ThoughtSpot resolves the
placeholders itself), always raises a WARNING issue naming the function
and the object, and refuses (has_parameter=True) a call that would
carry a runtime ThoughtSpot parameter (E9).
- E8: an optional partition_column wraps the passthrough call in
group_aggregate ( <passthrough> , query_groups ( ) + { <col> } ,
query_filters ( ) ), guaranteeing the partition column reaches GROUP BY.
- E12: emit_unmappable raises an ERROR issue naming the construct and
returns nothing — never a silent drop.
121 passed + 2 xfailed -> 132 passed + 2 xfailed (11 new tests), verified
on the Python 3.10 floor and 3.13.
* fix(thoughtspot): enforce the E8 wrapper instead of relying on caller convention
Review finding (Important): the partition_column kwarg was correct in shape
but nothing cross-checked it against the template, so a family task (3-8)
forgetting to pass it on a PARTITION BY row would silently emit an unwrapped
pass-through that is only sometimes correct — a silent wrong answer, not an
error. emit_passthrough now checks "partition by" in the template
(case-insensitive) against partition_column in both directions and raises
ValueError naming the construct on either mismatch. Two new tests, one per
direction.
Review finding (Minor): pinned that emit_passthrough's has_parameter refusal
never also logs a misleading WARNING, via an explicit
assert log.as_dicts() == [] in the existing E9 test.
132 passed + 2 xfailed -> 134 passed + 2 xfailed (2 new tests), verified on
the Python 3.10 floor and 3.13. Both xfails confirmed still xfailed (-rxX),
not xpassed.
* fix(thoughtspot): make the E8 PARTITION BY guard whitespace-tolerant
Review follow-up: the guard was a plain substring match on "partition by",
so PARTITION BY (two spaces) or a newline between the words silently made
carries_partition_by False -- defeating the guard in both directions (a
correctly-omitted partition_column leaves the row unwrapped; a correctly-
supplied one trips the opposite check as a false blocker). Switched to
re.search(r"partition\s+by", ..., re.IGNORECASE) and added a test with
irregular internal whitespace that fails against the old substring check
and passes against the regex.
134 passed + 2 xfailed -> 135 passed + 2 xfailed (1 new test), verified on
the Python 3.10 floor and 3.13. Both xfails confirmed still xfailed.
* feat(thoughtspot): catalog entries for aggregate functions and type conversion
* feat(thoughtspot): catalog entries for date/time functions
* feat(thoughtspot): catalog entries for string functions
* fix(thoughtspot): reject a passthrough template that wraps itself
Construct.__post_init__ now rejects a PASSTHROUGH row whose template already
contains its own variant call (e.g. 'sql_string_op ( "LOWER({0})" , {0} )'
instead of the bare 'LOWER({0})'). emit_passthrough builds the
variant(...) wrapper itself, so a template that already contains it would
double-wrap at emission time - a bug that reads as fine in a static glance at
the catalog and is wrong the moment it runs. Task 5 caught exactly this in its
own first draft; this convention had been written down nowhere, so make it
impossible instead of merely discouraged before two more families land.
Also fixes the pre-existing test_types.py example that (accidentally)
exercised the double-wrapped form and asserted it was valid.
* feat(thoughtspot): catalog entries for mathematical and conditional functions
* feat(thoughtspot): catalog entries for operators and constructs
* fix(thoughtspot): escape IN/NOT IN braces; force external dispatch for unary +/-
Task 7 review (executing emit_direct against every DIRECT row rather than
reading them) found two live bugs:
- IN/NOT IN embedded ThoughtSpot's literal { ... } set syntax unescaped in a
Python format-string template, so emit_direct crashed with
ValueError: unexpected '{' in field name on their own correct 3-arg usage.
Fixed by doubling the braces ({{ }}); verified the rendered output has
single braces again.
- The unary -x/+x row's -{0} template silently negated a parsed +x node
(right arg count, wrong semantics, no exception). Replaced with a
descriptive template that forces external dispatch, matching the TRUE/
FALSE and CAST per-type precedent, rather than returning a plausible but
wrong formula.
Adds a permanent catalog-wide sweep test (test_emit.py) that calls
emit_direct against every DIRECT row in CATALOG with its own natural arity
and asserts none raises - the exact check that caught the IN/NOT IN bug,
confirmed by running it against the unescaped templates first (2 failures,
both reported) and then against the fix (0 failures, 103 DIRECT rows).
* feat(thoughtspot): catalog entries for window functions
Task 8 of Plan B (the last of six catalog-population tasks): adds the 14
Window functions rows (5 direct / 9 passthrough / 0 unmappable), completing
the 146-row catalog. Removes the two strict xfail markers in
test_catalog_covers_the_spec.py now that the coverage gate passes for real.
Nine of the fourteen rows are passthrough because a ThoughtSpot window
formula cannot declare its own PARTITION BY (E13) - the partition is always
completed from the query's own dimensions. FIRST_VALUE/LAST_VALUE are the
one exception (a genuine explicit partition and order axis), and stay
direct along with RANK, PERCENT_RANK and the frame-clause boundaries.
CUME_DIST stays passthrough despite looking like PERCENT_RANK's twin -
rank_percentile divides by a different denominator and is not a
substitute. FIRST_VALUE/LAST_VALUE's direct templates carry ThoughtSpot's
literal `{ }` list syntax for the axis argument, escaped as `{{ }}` per
Task 7's IN/NOT IN fix (verified by calling emit_direct directly, not just
inspecting the template string).
* feat(thoughtspot): reverse-direction inventory (ThoughtSpot -> Ossie)
Task 9 of Plan B (final task): the other half of a bidirectional converter
- ThoughtSpot functions with no counterpart in the 146-row spec CATALOG.
Adds expressions/reverse.py: REVERSE (79 entries) + translate_thoughtspot(),
covering all three reverse-direction sub-sections of
ts-ossie-function-mapping.md (conditional aggregates/arithmetic helpers;
window/LOD/semi-additive functions; runtime/display/calendar concepts).
Rule E10 (prefer composition over the stash): 69 of 79 named constructs
(87%) compose fully or partially; only the semi-additive family and the
runtime-identity/parameter/markup/fiscal-calendar constructs are pure
stashes. Rule E11/P8 satisfied via three helpers (thoughtspot_dialect_entry,
portable_dialect_entry, custom_extensions_fragment) since
translate_thoughtspot's own str|None return can't carry a dialects[] entry.
283 passed, 0 xfailed (232 baseline + 51 new), on Python 3.10 and 3.13.
CATALOG unchanged at 146 (108/37/1). No tools/ts-cli code read or ported.
See task-9-report.md for the full compose/stash breakdown and judgment
calls (object_ref/connection_dialect as added kwargs, to_date format-token
scope, group_* shorthand arity assumption, is_weekend's DAYOFWEEK base).
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(ci): trigger thoughtspot CI when the upstream spec file changes
catalog.py's spec_construct_names() reads core-spec/expression_language.md
directly to gate the build against upstream construct additions, but the
workflow's paths filter only watched converters/thoughtspot/**. A PR that
added a construct upstream without touching any converter file would not
trigger this workflow at all, silencing the exact gate built to catch it.
* fix(thoughtspot): custom_extensions_fragment must be keyed by column, not vendor
write_stash(obj, payload) treats payload as the *contents* of the object's
THOUGHTSPOT entry (it owns the vendor-key wrapping itself, rule X1) — but
custom_extensions_fragment returned {VENDOR_KEY: {...}}, nesting the vendor
key inside its own entry and producing an invalid write_stash payload. Worse,
the docstring says the caller merges fragments across an object's columns
via {**f1, **f2}; keying by VENDOR_KEY instead of the column made that merge
lossy — a second column's fragment silently discarded the first's.
Reworks the signature to custom_extensions_fragment(column, name, args) so
merging across columns is lossless, and adds a test asserting two columns'
fragments both survive a merge and round-trip through write_stash/read_stash
intact.
Also fixes a cosmetic double-space in both custom_extensions_fragment and
thoughtspot_dialect_entry's zero-argument rendering ("ts_username ( )" ->
"ts_username ( )") while touching this call-reconstruction logic.
* fix(thoughtspot): correct the template contract and two catalog defects it hid
Construct.template's docstring claimed every DIRECT/PASSTHROUGH row is a
complete, substitutable template. That is false for ~20 rows that carry a
prose/dispatch template instead (CAST, EXTRACT/DATE_TRUNC/DATEADD, TRUE/FALSE,
both CASE forms, the column/metric reference, the unary row, several window
rows) and for 14 of 37 passthrough rows that are exemplars — a caller-supplied
value baked in as a literal while declaring a satisfiable arity (PERCENTILE_
CONT/DISC's 0.75, NTILE's 4, LAG/LEAD's offset 1, and others). Both are safe
by design, but neither was documented, so a uniform .format() dispatcher
built off the stated contract would mishandle both classes silently for the
exemplar case. Corrects the docstring on Construct.template and adds the same
exemplar convention to emit_passthrough's docstring.
Two real defects the false contract was hiding:
- emit_passthrough had no argument-count check, while emit_direct (and
reverse.py's _render) already have the identical one. A zero-placeholder
exemplar template (TIMESTAMP_NTZ's hardcoded 2024-01-15 literal) silently
accepted any number of args and appended them as unconsumed sql_*_op
arguments, keeping the hardcoded literal in the rendered SQL. Adds the
check, mirroring emit_direct's, plus tests reproducing the bug and pinning
the exemplar's own correct (zero-argument) rendering.
- The Window aggregation passthrough template contained a literal U+2026
ellipsis ("ROWS BETWEEN …") copied verbatim from the mapping document's own
prose shorthand for "a frame clause goes here". It passed __post_init__,
the E8 partition cross-check and declared a satisfiable 3-argument arity,
so emit_passthrough rendered it as-is — a warehouse SQL syntax error at
query time, far removed from the converter. Replaced with a concrete, valid
exemplar frame (ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, the same
cumulative-aggregate boundary the Frame clause row already maps cumulative_*
to), with tests asserting no ellipsis remains and that the row renders
valid SQL through emit_passthrough.
Also adds the E7 "typed sibling applies for a non-numeric aggregate" caveat
to LAG/LEAD/NTH_VALUE's notes — they already carry sql_number_aggregate_op as
their documented default and return their argument's own type, exactly like
the OVER clause and Window aggregation rows that already carry this caveat,
but the note was missing on these three. No variant changed; this documents
the caveat uniformly rather than changing behaviour.
* docs(thoughtspot): remove citations to a private, deleted workspace from src/
An ASF reviewer sees this source tree first, and one instance reached an end
user: an issue_message a user reads cited task-9-report.md, a document that
lives in a gitignored directory deleted at run end and exists in no
repository. Same treatment for the other task-N-report.md citations (reverse.py,
_types.py, catalog.py), the "learnings report P6/P7/P8" citations (constants.py,
_yaml.py, reverse.py), and catalog.py's "internal Tableau mapping" reference —
each removed while keeping the substance the citation was standing in for.
Also rewrites ~30 "Task N" / "Plan A-D" / "the brief" internal-process
narration sites across catalog.py, reverse.py, emit.py and __init__.py into
present-tense descriptions of what the code is, not how it was built —
including catalog.py's module docstring and its "READ THIS BEFORE TASKS 3-8"
banner, both stale on arrival (the catalog is 146/146, not "until Task 8
lands the rest").
Kept: BL-170 (8 references) and se-thoughtspot (13 references) are genuine
live-verification evidence, not internal process narration — added both to
README.md's Rules section so an outside reader knows what they refer to.
README.md's Status section described about 15% of the branch: it listed only
the five foundation modules and never mentioned the expression layer (the
146-construct catalog, the emitters, and the 79-row reverse inventory — the
majority of the diff, ~3,100 lines). Extends Status to describe it, and
corrects a present-tense claim ("Expressions are emitted under THOUGHTSPOT...")
to future tense — nothing emits a dialect entry yet; reverse.py only provides
the building blocks a document-level emitter will use.
Minor, folded in while touching this module's public surface:
expressions/__init__.py exported nothing from reverse.py (REVERSE,
ReverseConstruct, ReverseDisposition, translate_thoughtspot, the E11 helpers,
stash_runtime_parameter), while every earlier module's symbols were listed —
the reverse-direction inventory was otherwise invisible to a consumer
importing the package normally.
No behavioural change: prose and docstrings only, plus the __init__.py
export list. 288 tests still pass.
* fix(thoughtspot): remove remaining internal-process leakage; add passthrough arity sweep
The prior ASF cleanup wave used a case-sensitive grep and missed two survivors
of the same class: a private-workspace file citation in catalog.py, and
"family tasks (3-8)" narration in emit.py. Re-swept case-insensitively across
src/ and found one more, in reverse.py ("Plans C/D"). Extended the same sweep
to tests/, where every test file's module docstring (and several inline
comments) still opened with "Task N" narration; rewrote all of them to
describe coverage instead of provenance.
Also added test_every_passthrough_catalog_row_renders_with_its_own_natural_arity,
the symmetric counterpart to the existing DIRECT sweep, closing the gap where
emit_passthrough's argument-count guard was verified against all 37
PASSTHROUGH rows only by a throwaway script. partition_column need is derived
from each row's own template text (the same PARTITION BY regex emit_passthrough
itself uses), not a hardcoded list.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* feat(thoughtspot): shallow formula tokenizer for reference rewriting and call splitting
* fix(thoughtspot): stop quote state leaking into bracket bodies, delegate ref splitting, block keyword call names
- _scan suppressed quote toggling inside [TABLE::Column] bodies, so an
apostrophe in a display name (routine ThoughtSpot data) desynchronised
quote tracking and silently dropped or mis-rewrote later references.
Fixed with a bracket-type stack: quote toggling is suppressed only while
the innermost open bracket is '[', not '(' or '{'.
- find_column_refs, is_bare_column_ref, and rewrite_column_refs now
delegate to identifiers.split_column_ref instead of a bare
str.split('::', 1), so an ambiguous reference raises loudly instead of
silently taking the first delimiter.
- split_call now rejects a call head containing an operator/control-flow
keyword as one of its space-separated words, so 'true and count (...)'
is correctly not treated as a call while 'unique count (...)' still is.
- Documented in the module docstring that backslash is not an escape
character in ThoughtSpot's grammar, so a backslash-adjacent quote
splitting early is expected behaviour, not a bug.
* fix(thoughtspot): loosen column-ref table grammar to match its own ambiguity rule; make bracket-stack pop type-aware
- identifiers._COLUMN_REF's table group excluded any colon, stricter than
the ambiguity rule split_column_ref's own docstring describes. A table
name with a single ':' (e.g. one produced by format_column_ref) failed
to round-trip. Made the table group a lazy match up to the first '::'
instead; both documented ambiguity checks (trailing/leading colon runs,
more than one '::') are unaffected.
- formula._scan's bracket-type stack popped unconditionally on any closer,
so a mismatched or stray closer could pop the wrong entry and
desynchronise which bracket type is considered innermost, incorrectly
un-suppressing quote handling. The pop now only fires when the closer
matches the type of the top entry.
* docs(thoughtspot): drop review-process wording from a test comment
The comment described where the case came from rather than what it tests. It
predates this work (6207d70) and survived three prior cleanup sweeps, all of
which used hand-written patterns.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* feat(thoughtspot): bidirectional datatype map with declared losses named
* docs(thoughtspot): remove citations to an internal revision id
Three comments cited 'R14', which identifies a decision record that exists in
no repository. The substance of each is kept; only the unresolvable label goes.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* feat(thoughtspot): TML document set with the guid/brace/ordering/YAML-1.2 serialisation invariants
* fix(thoughtspot): sanitise dump_document_set filenames and strip nested guids
Filenames minted from a table/model name could previously escape the
intended output directory (e.g. a name of ../../etc/passwd) or collide
silently between distinct or missing names. Filename stems are now
sanitised against POSIX and Windows path hazards, with a deterministic
counter suffix when sanitisation makes two distinct names collide.
dump_document previously stripped only the document-root guid. A guid
nested anywhere in the body (e.g. on a column entry) passed straight
through, which is the actual import failure mode the rule exists to
prevent: a nested guid is silently ignored on import and ThoughtSpot
creates a duplicate object instead of updating the one that exists.
guid is now stripped at every depth of the body, without mutating the
caller's document, while fqn is left untouched.
* fix(thoughtspot): reserve minted filenames by their final form, and cap length
The counter-disambiguation added for filename collisions keyed its dedupe set
on the sanitised base name rather than the filename actually emitted, so a
counter-suffixed name could still land on another document's real name and
silently overwrite it. dump_document_set now reserves the exact filename it
mints, advancing the counter and regenerating until a truly free one is
found, for every document including the first and the model's.
Also caps each filename's stem to fit within a 255-byte filesystem component
limit, truncating on a UTF-8 character boundary and leaving room for the
suffix and the disambiguating counter so a long name cannot itself reopen
the collision this fix closes.
Rewrote docstrings that referenced internal planning process language rather
than describing the code's own behaviour and rationale.
* test(thoughtspot): add a self-policing guard against unresolvable references
Six hand-written grep sweeps each missed leakage the previous one caught, most
recently an identifier shape and an ordinary-English process usage that no
single sweep's pattern covered. Replaces the pattern with a permanent
tests/test_shipped_references.py: a fail-closed scan for any uppercase
letter+digit token not in a justified allowlist or the still-open
mapping-document-rule-id set, plus a hand-curated blocklist for ordinary
process language that has no identifier shape to key off.
First run surfaced 13 BL-170 citations (reworded to state the live-verified
finding directly) and 9 further identifier-shaped process citations (F5, M3
x3, M4, P6, P8 x2, P21) that were not part of the known mapping-doc rule-id
families — same reworded treatment. Two literal uses of "transcribed" were
also process leakage; two more were legitimate ("a miscopied catalog row")
and were reworded only to stay clear of the blocklist word. 45 mapping-doc
rule ids (240 citations, 26 files) are allowlisted provisionally pending the
open decision on vendoring that document. 18 unrelated technical tokens
(INT64, UTF-8, SCD-2, the L1-L6 loss codes README already defines, ...) are
allowlisted with individual justification.
406 -> 409 tests.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* feat(thoughtspot): convert TML columns to Ossie fields with verbatim dialect entries
Adds tml_to_ossie.py: expression_entries() carries a ThoughtSpot formula's exact
expr string into the THOUGHTSPOT dialect entry unmodified, and adds a portable
ANSI_SQL sibling only for a bare column reference the resolver can place -
never a guessed translation. attribute_dataset() places a computed column in
the one dataset every one of its references resolves to, and refuses to guess
when references disagree, are unresolvable, or are absent entirely.
convert_field() builds an Ossie field from a Model columns[] entry, keeping the
normalised identifier and the exact display label in separate fields.
447 tests passing (409 baseline + 38 new), verified on the 3.10 floor and 3.13.
* fix(thoughtspot): give convert_field a formulas map instead of an inlined expr
convert_field previously required its caller to inline a computed column's
formula expr onto the column dict under an "expr" key before calling it,
while its planned sibling convert_metric was designed to take a formulas
map directly. Align convert_field to the same shape: it now takes
`formulas: dict[str, dict]` (formulas[] keyed by id) and reads
formulas[column["formula_id"]]["expr"] itself, so a caller assembling a
model uses one lookup convention instead of two, and no modified copy of
the input document needs to be built.
- formula_id present and resolvable: converts, expr carried verbatim.
- formula_id with no matching formulas[] entry: logs TS-FIELD-FORMULA-MISSING
naming the column and the missing id, returns None (not a KeyError).
- neither column_id nor formula_id: unchanged behaviour, logs
TS-FIELD-NO-SOURCE and returns None (same contract as before formula_id
existed).
Updated tests/test_tml_to_ossie_fields.py for the new signature and added
three tests for the formulas-map cases above, including a verbatim-under-
adversarial-input check re-run through the new lookup path.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): log the genuine silent losses review round 2 found
_physical_datatype collapsed two different None outcomes into one silent
return: a column with no data_type at all (nothing to drop, datatype is
optional in Ossie) and a column whose data_type is present but unrecognised
by datatypes.to_ossie (a genuine loss - the warehouse told us the type and
it was dropped on the floor). Only the second now logs
TS-FIELD-DATATYPE-UNMAPPED, naming the column, table, and the unmapped
type; the first stays silent on purpose.
convert_field's formula_id lookup guarded a missing id but not a formulas[]
entry present under that id with no expr key of its own - same class of
problem (no expression text to read), so it gets the same treatment: an
issue naming the column and the formula id, and None, not a KeyError.
Added tests for both datatype outcomes, for the malformed-formula-entry
case, and for the two previously-uncovered ai_context shapes (a bare
free-text string, and synonyms + free text combined) plus the
neither-present omission case - asserting the actual returned shape, not
just presence.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* feat(thoughtspot): convert TML measures to Ossie metrics, composing column aggregations
Adds convert_metric to tml_to_ossie.py: a column_id + aggregation metric becomes
AGG(dataset.field); a scalar formula_id + aggregation composes into AGG(<scalar
expr>); an aggregate formula_id (sum(...), etc.) treats the column aggregation as
the documented no-op and carries the verbatim expr unchanged. Aggregate detection
reads ThoughtSpot's native call names off the same expression catalog used to
compose the wrapped rendering, so the two cannot drift apart. Metrics have no
label, so a display name that normalises differently is stashed as tml_name.
* fix(thoughtspot): stash the source metric shape for round-trip fidelity
Records shape (column_aggregation | scalar_formula_plus_aggregation | formula)
in the metric's stash alongside tml_name, using the pinned enum spellings, so a
return trip can reconstruct the physical-column and scalar-formula-plus-
aggregation shapes instead of collapsing every metric into one. formula is
omitted rather than written: it is also the reverse direction's own default
when no stash is present, so recording it changes nothing about the
reconstruction while making the payload heavier.
* fix(thoughtspot): stop silent double aggregation on group_aggregate and sql_*_aggregate_op
The outer-call-only aggregate check missed constructs the catalog holds beyond
the eight TML aggregation rows: group_aggregate (ThoughtSpot's own performant
grouped-aggregation pattern) and the sql_*_aggregate_op pass-through family, so
a formula built from either composed a second aggregation on top and produced
a wrong number with no warning.
Two-layer fix. (a) Broadened the aggregate-name set with group_aggregate and
every Variant ending in "_aggregate_op" (derived from the enum, not listed by
hand). (b) Added formula.find_call_names, a whole-expression, any-depth call
scanner, and use it (_contains_aggregate_call) instead of the outer-call-only
check before composing — this also catches an aggregate nested inside a scalar
wrapper (round(sum(x), 2)) that no set of names alone could. Discarding a
load-bearing column aggregation because the expression already aggregates now
always logs a warning naming why, whether the aggregate was the expression's
own outer call or nested inside one.
Also: expression_entries and _physical_datatype hardcoded "field" in issue
text and codes even when building a metric. Both now take a kind parameter
("field" by default, so convert_field's behaviour and codes are byte-for-byte
unchanged) and convert_metric passes kind="metric" at every call site.
* fix(thoughtspot): only warn when an aggregate is nested, not when it is the outer call
An aggregate as a formula's own outer call (sum(...), group_aggregate(...), ...)
plus a redundant column-level aggregation is ThoughtSpot's documented, routine
no-op -- the previous warning fired on that common, correct shape and would
have trained readers to ignore the issue log. Narrowed to warn only when the
outer call is not itself an aggregate but one is nested inside it
(round(sum(x), 2)), which is the one shape a reader might expect composition
and not get. Added _outer_call_is_aggregate for the (silent) outer-call check;
_contains_aggregate_call is now only consulted once that check is negative.
Strengthened the existing no-op regression test to assert on the issue log
(it previously asserted only on the expression), and added explicit coverage
for the boundary: sum/average/unique count/group_aggregate as the outer call
produce no warning; round(sum(x),2) and a scalar-wrapped group_aggregate each
produce exactly one; a genuinely scalar formula still composes.
Also corrects find_call_names' docstring and its test's comment, which
overstated what the function does: names that are also operator keywords
(not, if) are excluded even though both are real catalog functions, and the
keyword handling is weaker than split_call's (only a leading run is stripped,
so a keyword after a genuine word is not caught). No behaviour change.
* feat(thoughtspot): complete the TML -> Ossie direction with keys, stash and entry point
Adds the assembler that ties Tasks 1-5 together: datasets built from
model_tables[] + Table/SQL-View documents, the cross-model resolver
convert_field/convert_metric take as `resolve`, joins converted to
relationships (inline and referencing shapes) with KD1 key derivation
(including the ONE_TO_MANY orientation flip), the model/dataset/relationship
custom_extensions[THOUGHTSPOT] stash, and the public convert() entry point
returning OssieConversion(model, issues).
A malformed reference (an ambiguous column_id or join condition) is caught
per object, logged, and skipped rather than aborting the whole conversion.
* fix(thoughtspot): stash unconsumed column properties instead of dropping them
convert_field/convert_metric never returned a fragment for a column's
ThoughtSpot-only properties (index_type, value_casing, ...), so the
assembler had nothing to merge and they vanished with no issue -- a direct
breach of "nothing dropped silently".
Fixed fail-closed in the assembly, without touching either converter's
signature: stash the complement of the properties keys the converter
actually consumed (column_type, synonyms, ai_context, and aggregation for
metrics), under that object's own column_properties extension key, so a
future ThoughtSpot property is preserved by construction rather than
requiring another enumeration edit.
While re-verifying identity leakage for this change, found that copying an
unconsumed property's value wholesale can carry a nested identity key a
top-level-only guard cannot see (a documented ThoughtSpot shape: a custom
map reference nested inside an otherwise plain property). Extended the
identity check to scan nested values too, dropping and logging rather than
stashing when found.
* fix(thoughtspot): validate relationship targets, centralize the identity guard, stash unsurfaced columns
Three defects an independent review found and reproduced:
- A relationship's target dataset was never checked for existence, only its
source -- a join to a table this model failed to build (missing document,
bad alias) emitted a Relationship upstream's own validator rejects
outright, with no issue logged. Now checked the same way the source
already was: dropped, logged, rest of the model stays valid.
- The nested identity scan added for column properties covered only that
one caller. Moved the scan into stash.write_stash itself -- the single
point every stashed payload passes through -- via a new
stash.find_forbidden_key(value, forbidden=None), so no present or future
caller (model-scope parameters/filters/column_groups/lesson_plans/
action_object_associations/constraints/model-level joins_with included)
can bypass it by nesting identity content instead of putting it at a
payload's own top level.
- Table columns the Model doesn't surface (by column_id, field or metric
alike) vanished with no stash and no issue. Now preserved verbatim under
the dataset's unsurfaced_columns, per the pinned mapping document.
Also: stash.read_stash now rejects an unrecognised custom_extensions shape
version (_v) instead of partially reading it as the current shape.
Behavioural decision: the boundary (write_stash) still raises on any
forbidden key found -- that is what makes the guard impossible to bypass --
but the assembly now catches it (_write_stash_safely), drops the one
contaminated top-level stash field, logs which object and which key, and
keeps converting the rest of the model. The payload content is TML data the
converter did not construct, not a programming error, so it must not abort
an otherwise-fine conversion -- consistent with the column-property path,
which already dropped and logged rather than raising.
Committed the key-derivation edge cases (mixed equality+residual, a
self-join, a top-level OR, MANY_TO_MANY) that were previously only
scratch-verified during development.
* fix(thoughtspot): read SQL View columns from sql_view_columns[], not columns[]
A SQL View document's physical columns live under a different key entirely
(sql_view_columns[], bound to a query output alias via sql_output_column),
not a differently-shaped entry under columns[]. Every reader of "this
dataset's physical columns" -- datatype lookup via table_lookup, and the
unsurfaced-column check -- read columns[] unconditionally, so every SQL
View column vanished: no datatype resolved for a surfaced column, and an
unsurfaced one was lost with no stash and no issue. This has been broken
since the assembler was written; nothing exercised a sql_view document
until now.
Fixed via three small, kind-aware helpers (_raw_physical_columns,
_normalize_physical_column, _normalized_physical_columns) so table_lookup
and the unsurfaced-column pass both read the right key for the document
kind they were actually handed, translating a SQL View column's
sql_output_column into the db_column_name-shaped field the datatype lookup
already expects, while preserving the column verbatim (with its real
sql_output_column key) when it lands in unsurfaced_columns for reverse-
direction fidelity.
sql_output_column itself is now also stashed under the dataset's
sql_output_columns (DatasetLevel schema key: field name -> alias), for
every surfaced field on a SQL View -- unconditionally, not only when it
differs from the field's name, because there is no safe way to re-derive a
query output alias the way a table's db_column_name might be guessed at.
Confirmed source is already correct for a SQL View: it is the raw
sql_query string directly, matching the mapping document's row -- no
change needed there.
Also: routed convert_metric's own stash.write_stash call through
_write_stash_safely, so the wrapper's docstring claim ("every stash site in
this module") is true rather than aspirational; the payload is hardcoded
scalars today so this is currently a no-op, but the next author who adds
TML-derived content to it no longer silently reinstates the
abort-the-whole-conversion behavior the wrapper exists to remove.
* feat(thoughtspot): build Table TML documents from Ossie datasets
Adds build_table(dataset, log) -> TmlDocument, the first module of the
Ossie -> TML direction: one Ossie dataset becomes one Table or SQL View
document, so the Model document (a later module) has something to
reference by name.
- db_column_name is always written, even equal to the display name.
- db_column_properties.data_type is always written; a datatype-less
field infers one rather than omitting the compulsory block.
- Each of the four datatypes whose round trip is lossy (Float, Time,
DateTimeTz, Opaque) raises an issue naming the loss.
- source splits into db/schema/db_table for a table, or is read as a
verbatim query for a SQL view; a two- or one-part source that is
neither shape raises an issue instead of guessing.
- The BOOLEAN/BOOL and DOUBLE/FLOAT warehouse spelling is restored from
a field's own stash when present, and defaults otherwise.
- A dataset's stashed connection_name, source_parts, table_properties,
sql_output_columns and unsurfaced_columns are all honoured, with
staleness checked against the live source where a witness exists.
A round trip against the forward direction (tml_to_ossie.convert) found
a real, previously invisible gap no hand-written fixture had exposed:
the forward direction matches a physical column by its display name
only, so a surfaced column's true db_column_name is never captured
when it differs from that display name. build_table now assumes the
two agree in that case -- correct in the common case -- and reports an
INFO issue for it rather than staying silent, since the assumption can
be wrong.
The datatype map's conditional Time -> DATE_TIME mapping (only when the
underlying column is timestamp-backed) is deliberately left
unimplemented here too: the forward direction never emits Time at all,
so a Time-typed field only ever reaches this module hand-authored, with
no stashed ThoughtSpot column and no other storage-format signal on an
Ossie Field to condition on. VARCHAR is written unconditionally instead,
with the declared loss still reported.
589 tests = 556 baseline + 33 new.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): a bare field's portable expression names the warehouse column, not the Ossie identifier
resolve() built the ANSI_SQL sibling from the Ossie field's own
display-derived identifier ("orders.amount"), not the physical column the
mapping document is explicit about ("the identifier is the physical
column; the display name comes from label/name"). Confident, well-formed
SQL naming a column the warehouse does not have, in every model where a
display name differs from its physical column -- the normal case in any
curated model.
resolve() still uses the model's ATTRIBUTE-surfaced-column index
(attribute_index) to decide WHETHER a reference is one the model actually
surfaces, unchanged; only the value it returns once that gate passes now
looks up the table's own physical column list for the real warehouse
reference (db_column_name for a Table, sql_output_column for a SQL View,
via the existing kind-aware _normalize_physical_column translation). A
computed field's own references resolve through the identical path, since
[TABLE::Column] always names a physical column regardless of what formula
is referencing it.
Also, per the datatype map's own words ("the connection's spelling is
recorded in the field stash's data_type key so the return trip re-emits
the same one"): a physical field/metric's data_type is now stashed under
that documented key whenever it is not the canonical spelling for its
mapped Ossie datatype (BOOL vs BOOLEAN, FLOAT vs DOUBLE) -- never for the
canonical spelling itself, so the empty-payload rule stays reachable.
And: db_column_name, lost because the forward direction matches a
physical column by display name only, is now stashed under a new
field/metric-level db_column_name key when it differs from the display
name (Table-backed columns only -- a SQL View's own sql_output_columns
dataset-level key already covers the same fact). Not in the pinned payload
schema; verified by a real round trip (convert() then build_table()) that
this key is necessary but not yet sufficient on its own: the reverse
direction's _physical_identity prioritises the THOUGHTSPOT bracket
(display name only) and does not yet consult it, so db_column_name still
round-trips as an explicitly logged assumption (TS-FIELD-DB-COLUMN-NAME-ASSUMED)
rather than silently -- closing that consumption gap is reverse-direction
work, out of scope here.
* fix(thoughtspot): read the stashed warehouse column name back, and three more table-build gaps
- Critical: _physical_identity now reads the field-level stash key the
forward direction writes for a Table column's true db_column_name and
prefers it over the THOUGHTSPOT bracket's display name. Previously it
derived db_column_name from the bracket unconditionally and never
called read_stash for it, so every surfaced column whose display name
differs from its warehouse name round-tripped to a Table document
naming a column that does not exist. The fallback (display name ==
db_column_name) is now used, and reported, only when the stash
genuinely carries nothing -- upgraded from INFO to WARNING, since the
consequence is a column binding that may not exist.
- Dataset-level ai_context has no home in a Table or SQL View document;
build_table now raises TS-DATASET-AI-CONTEXT-UNSUPPORTED for it
instead of dropping it with no trace.
- The new db_column_name stash key is now a shared constant
(FIELD_STASH_DB_COLUMN_NAME in constants.py) imported by both
directions, so they cannot spell it differently. Every other stash key
shared between tml_to_ossie.py and ossie_to_thoughtspot.py is still a
bare string literal independently written on each side -- roughly 15
keys (source_parts, connection_name, unsurfaced_columns,
sql_output_columns, table_properties, tml_name, table_name, tml_object,
data_type, and the relationship/model-scope keys) share the same
drift exposure. Not fixed here; flagged as a follow-up.
- Kind detection ("does this source look like a query?") no longer uses
"contains whitespace" as its first test -- that misclassified a quoted
identifier with an embedded space (SALES.PUBLIC."ORDER TABLE") as a
query, emitting an unimportable sql_view with no issue. A three-part
dotted identifier (quoted segments included) is now tried first, and
only a genuine non-identifier shape falls back to the whitespace test.
Verified with a round trip: a five-column table whose display names all
differ from their db_column_names (plus one computed formula and one
unsurfaced column), through tml_to_ossie.convert and back through
build_table, reproduces every column exactly with zero issues raised.
607 tests = 600 baseline + 7 new.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* refactor(thoughtspot): centralize every custom_extensions[THOUGHTSPOT] payload key
Closes the follow-up flagged in 35fa490: every stash-payload key besides
db_column_name (FIELD_STASH_DB_COLUMN_NAME) was a bare string literal typed
independently in tml_to_ossie.py (the writer) and ossie_to_thoughtspot.py
(the reader), or independently retyped at two call sites within
tml_to_ossie.py itself. A one-character spelling disagreement between any
of those pairs makes the reader silently find nothing -- no error, just a
value that stops round-tripping.
Mechanical refactor only, no behaviour change: 34 new constants added to
constants.py, grouped by which Ossie object each key's custom_extensions
entry attaches to (model / dataset / relationship / field-metric), matching
FIELD_STASH_DB_COLUMN_NAME's existing naming and documentation style. Every
occurrence of each key as a stash-payload dict literal is replaced with the
constant, in both converter modules and in every test that hardcodes one
(dataset_stash=/field_stash= fixtures, and read_stash()/_own_stash()
assertions) -- a test hardcoding a key string is the same drift risk, and
the place a wrong key is most likely to be enshrined as correct.
Centralized, by scope:
- Shared: STASH_TML_NAME (model + metric; defensively read at dataset scope)
- Model: unattributed_formulas, unrepresentable_joins, model_properties,
parameters, filters, column_groups, lesson_plans,
action_object_associations, constraints, model_joins_with
- Dataset: tml_object, alias, table_name, connection_name, sql_query,
source_parts (+ its db/schema/db_table sub-keys), unsurfaced_columns,
sql_output_columns, table_properties
- Relationship (also reused, unchanged, inside unrepresentable_joins[]
entries): type, cardinality, join_shape, referencing_join, on_expression,
residual_predicates
- Field/metric: data_type, column_properties (joining db_column_name)
- Metric-only: shape
Deliberately NOT centralized, checked rather than assumed:
- The shape-version key (_v) and the vendor name (VENDOR_KEY) -- already
handled: _v is encapsulated entirely inside stash.py's own
read_stash/write_stash pair (one file, adjacent lines), and VENDOR_KEY
already has a shared constant.
- "from"/"to"/"name"/"expr" inside unrepresentable_joins[]/
unattributed_formulas[] entries -- each typed at exactly one site today
(no reader exists yet for either list) and each mirrors an identically-
named core Ossie/TML schema field (Relationship.from/to,
formulas[].name/expr), so there is no independent second typing to drift
against. Left as literals; worth a second look once a Model-scope reader
is written.
- model_properties' own nested keys (is_bypass_rls, join_progressive,
spotter_config.is_spotter_enabled) and column_properties' internal
vocabulary -- copied verbatim from TML's own property names via a single
typing site (a tuple, or a fail-closed complement), not independently
retyped anywhere.
Verified by construction: grepping both converter modules for every
identified key name as a bare string literal outside constants.py returns
only genuine TML-schema reads/writes (e.g. join.get("cardinality"),
db_column_properties.data_type on an emitted Table column) -- zero
remaining stash-payload literals. Both modules import the shared constants
from the same constants.py; no key's string value is defined twice.
607 tests, unchanged from baseline -- no test was rewritten to accommodate
this change, only literal keys swapped for the constant that already
carries that value.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* feat(thoughtspot): build the Model TML document under the R3-R9 import invariants
Adds build_model (Ossie semantic_model -> ThoughtSpot Model TML) and
to_thoughtspot_expression (dialect selection mirroring Task 4's own
expression_entries, preferring a verbatim THOUGHTSPOT entry and falling
back to structurally translating a catalog-matched ANSI_SQL expression).
Every formula/metric gets a formulas[] + columns[] pair (R3), a metric is
always a formula and never column_id + aggregation (R4, including the
column_aggregation-shaped arrival case), display names are deduplicated
across columns[]/formulas[] via a case-preserving allocator (R6/ID4),
column_type/synonyms land under properties (R7), is_hidden/
was_auto_generated are never emitted (R8), and a brace-carrying expr is a
block scalar (R9). Formula ids are derived from the normalised display
name so a THOUGHTSPOT-verbatim cross-reference resolves against a formula
this converter itself generates.
Also promotes tml_to_ossie.py's private metric-shape constants to
constants.py (METRIC_SHAPE_*) so the writer and reader share one spelling,
and adds primary_key/unique_keys "unused key" detection to build_model --
a real gap found via round-tripping the construct-mapping document's own
worked example.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): rewrite formula cross-references to match normalised ids
build_model regenerates every formula's id from the normalised form of its
own display name (formula_net_amount), but a THOUGHTSPOT-verbatim
cross-reference elsewhere in the model was written against the source
document's own id text ([formula_Net Amount]), which need not match. Left
unrewritten, the reference dangles and ThoughtSpot parses it as search
tokens rather than failing at parse time, so the emitted document fails
import on first attempt.
Fixed by deferring the R9 block-scalar wrap and doing one final pass over
the fully-assembled formulas[] list, once every formula's id is known:
build a normalised-name -> id map from that same list, then rewrite every
embedded [formula_X] reference through it (formula._bracketed_spans, X8's
identity scan already used the same way elsewhere in this package). A
reference matching nothing being built is left as written and logged as
an ERROR rather than silently dropped or guessed. Both the id-minting side
(_formula_id_from) and the reference side (_rewrite_formula_references)
share the same _normalise_or_self fold, so they cannot independently
drift.
Also logs an INFO issue (TS-MODEL-METRIC-AGGREGATION-CONVENTION) when a
metric's surfacing column gets the conventional aggregation property set
over an already-aggregating expression -- Ossie's Metric object has no way
to record whether the source column carried this property, so the value
is always re-derived, and a round trip built for fidelity should surface
that judgment rather than making it silently.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): rename FULL_OUTER to OUTER in every emitted join type
ThoughtSpot accepts only INNER, LEFT_OUTER, RIGHT_OUTER, OUTER for a join
type and rejects both FULL OUTER and FULL_OUTER identically; OUTER is its
own full outer join, so the rename is semantics-preserving and never a
loss. build_model was missing this rename entirely -- a relationship
stashed with type: FULL_OUTER emitted FULL_OUTER verbatim into the
model's inline join, which would fail on import. Applied at both call
sites that write a join type (relationships and unrepresentable_joins[]),
case/whitespace-insensitively so any source spelling normalises the same
way; every other type value passes through unchanged, with no issue
logged for the rename since nothing is lost. This module never writes a
Table joins_with[] entry (build_table has no relationship visibility to
add one there), so that context needs no corresponding fix.
Also closes three minors from the same review:
- _formula_id_from now calls _normalise_or_self instead of repeating its
try/except, so the id-minting and reference-rewriting sides cannot
independently drift onto two different fold rules.
- An unrecognised METRIC_STASH_SHAPE value now logs a warning and falls
back to the documented default, rather than silently falling through.
- Tests import the exported METRIC_SHAPE_* constants instead of
hardcoding the shape strings they represent.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): drop is_hidden/was_auto_generated from emitted columns
R8 forbids a generated model from ever carrying is_hidden: true or
was_auto_generated: true -- a hidden column cannot be surfaced again
without a manual edit on the target instance, and re-asserting
was_auto_generated on a column this build did not itself generate would
misrepresent its provenance. Both properties reach the emitted document
through the generic column_properties stash (neither is explicitly
consumed by tml_to_ossie.py, so both fall through to the catch-all that
preserves any property this converter did not otherwise read), and
nothing filtered them back out before this fix.
_drop_never_emit_true_properties removes both from a restored
column_properties payload before it is merged into the emitted
properties dict, at both call sites (fields and metrics). Only a true
value is dropped and logged, at WARNING -- matching this module's other
declared-loss severities, since a column silently losing its visibility
or provenance flag is a real difference the model owner needs to see,
not a benign note. A stashed false is simply omitted with no issue:
false (or absent) is ThoughtSpot's own default for both properties, so
leaving the key out loses nothing. The stash itself is untouched -- it
is the Ossie document's own record of what the source TML held, and
only the emitted TML side ever drops the flag.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* feat(thoughtspot): complete the Ossie -> TML direction with joins and stale-stash detection
Adds the public ossie_to_thoughtspot.convert() entry point (TmlConversion,
mirroring tml_to_ossie.OssieConversion in reverse) and applies X5's
stash-if-present-and-still-current-else-derive rule via stash.restore(),
rather than plain stash-if-present, to the two constructs whose stash
shadows a live, editable Ossie value:
- A relationship's on_expression (+ residual predicates): witnessed by a
snapshot of from_columns/to_columns taken when the stash was written.
A retargeted relationship (from_columns/to_columns edited since) drops
the stale condition and re-derives the plain equality join instead of
silently keeping the old narrowing, with an issue recording it.
- A field's stashed warehouse data_type spelling (BOOL/FLOAT): witnessed
by the Ossie datatype it was recorded against. An edited datatype drops
the stale spelling and re-derives the canonical one.
Also fixes a real round-trip bug found by driving both public entry
points back to back: a physical column referenced only by a MEASURE
metric's column_id (never by any ATTRIBUTE field) was excluded from
unsurfaced_columns as "already surfaced", but nothing else preserved its
definition -- the reverse direction regenerated a Table missing it while
the metric's own formula still referenced it, a dangling column
reference in an otherwise-valid document. Fixed by keying the
unsurfaced-columns computation on which columns became ATTRIBUTE fields
(attribute_index) rather than on every column_id any Model column names.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): a formula cross-reference is not a runtime parameter
find_parameter_refs treated any bracketed name with no `::` as a runtime
parameter, but a formula cross-reference ([formula_Name], R3's id form) is
the same textual shape and a first-class ThoughtSpot construct, not a
parameter — the model builder already recognises exactly this prefix when
rewriting references. The two modules had independent, disagreeing notions
of the convention.
Shares one definition in formula.py (FORMULA_REFERENCE_PREFIX,
is_formula_reference, find_formula_refs), which both tml_to_ossie.py and
ossie_to_thoughtspot.py now read rather than each keeping its own literal.
expression_entries still suppresses the portable ANSI_SQL sibling for a
cross-reference — it is not portable without inlining the referenced
formula, a transformation this converter does not attempt — but now says
so under its own code (TS-EXPR-FORMULA-REFERENCE, INFO) instead of the
false TS-EXPR-PARAM claim. An expression carrying both a cross-reference
and a genuine parameter logs both, each under its own code.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): audit and enforce X5's stash-currency rule across every key
FIELD_STASH_DB_COLUMN_NAME was written with no witness and read with no
currency check: retargeting a field's bracket reference to a different
physical column silently kept the OLD warehouse name, producing a duplicate
display name the emitted document would not import. Fixed with a witness
(the display name db_column_name was recorded against), matching the
pattern already used for FIELD_STASH_DATA_TYPE and a relationship's
on_expression.
Rather than stopping at that one instance, audited every stash key read on
the Ossie -> TML direction and classified each in a new
STASH_KEY_CLASSIFICATION table (constants.py): does it shadow a value
derivable from the live Ossie document (needs a witness/currency check), or
is it information that exists nowhere else (stash-if-present is correct)?
A new test (test_stash_key_classification.py) fails if a key read anywhere
in ossie_to_thoughtspot.py has no entry in the table, and if a
SHADOWS_DERIVABLE key has no witness constant or documented self-check --
so the next key added has to declare an answer rather than default to the
unsafe one.
The audit found two more unwitnessed shadowing keys, both fixed the same
way: STASH_TML_NAME at metric and model scope (a renamed metric/model kept
serving its stale pre-rename display name -- fixed via a self-verifying
check, no separate witness needed, since the stashed name's own normalised
form is the comparison), and DATASET_STASH_TML_OBJECT (a dataset whose
source was rewritten from a query to a table reference, or back, kept the
stale table/sql_view kind -- fixed with a witness on the source string).
Also: a display-name collision resolved by the allocator is now logged
(naming both the original and the allocated name) instead of silent; an
unattributed formula (references span two+ datasets) is now restored fully
surfaced with a normal columns[] entry -- TML never required dataset
attribution for a formula's surfacing entry in the first place, so nothing
stopped this -- rather than re-emitted as an orphan formulas[] entry that
ThoughtSpot's own visibility rule would hide, while the old issue claimed
only column properties were lost; and a parameter or formula cross-reference
used twice in one expression is now named once in its issue, not twice.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): unsurfaced-column membership is a second, orthogonal axis
DATASET_STASH_UNSURFACED_COLUMNS is INFORMATION_ONLY in its per-entry
content (a physical column's warehouse name/type has no Ossie-side
counterpart), but that classification says nothing about whether an entry
still BELONGS: a field added or retargeted onto a column that was
unsurfaced when the stash was written makes that column surfaced now.
Blindly restoring the stale entry alongside the live field's own build
duplicated it -- a duplicate Table/SQL-View column name, which does not
import. Reproduced and fixed: the reverse direction now drops any restored
unsurfaced-column entry whose name collides with a column already built
from a live field, silently -- the column is still present once, so there
is nothing to name in an issue.
The taxonomy could not express this (content-safe, membership-stale), so
it gained a second, orthogonal axis: STASH_KEY_CLASSIFICATION still answers
"can the value disagree with the live document", and the new
STASH_KEYS_WITH_DERIVABLE_MEMBERSHIP frozenset separately answers "can a
list entry be superseded by something the live document now covers".
Checked every other list-shaped INFORMATION_ONLY key against the second
question directly: DATASET_STASH_SQL_OUTPUT_COLUMNS is consulted only as a
per-field dict lookup (never appended as a block, so a stale entry is
simply never looked up, not duplicated) and does not qualify;
MODEL_STASH_UNATTRIBUTED_FORMULAS and MODEL_STASH_UNREPRESENTABLE_JOINS
flow through the shared display-name allocator or tolerate multiple joins
without an import-breaking collision. Extended
test_stash_key_classification.py with a consistency gate (a
membership-derivable key must be classified INFORMATION_ONLY) and verified
by construction that both it and the original completeness gate still fail
closed.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* test(thoughtspot): TPC-DS and minimal fixture pairs
Adds tests/fixtures/tpcds/ (5 core datasets, 4 core relationships, 5 core
metrics mirroring examples/tpcds_semantic_model.yaml exactly, plus a SQL
View dataset and two extra relationships/four extra metrics covering the
constructs that have broken during this build: a display name differing
from its db_column_name, a YAML 1.1 boolean-token column name, a
brace-carrying window formula, a formula cross-reference, a
connection-specific BOOL column, a SQL View output alias differing from
its column name, a physical column the Model does not surface, a
non-equality join condition, a composite-key relationship, and one metric
of each of the three TML shapes) and tests/fixtures/minimal/ (the smallest
one-model-plus-N-tables split). Each expected.ossie.yaml was generated by
running tml_to_ossie.convert() over its TML and reviewed by hand, then
also independently checked against validation/validate.py and the
upstream JSON schema.
tests/test_fixtures.py asserts every TML fixture loads, both
expected.ossie.yaml documents validate against core-spec/ossie-schema.json,
converting each fixture set reproduces its expected document exactly, and
the TPC-DS fixture set exercises every construct listed above.
725 -> 750 tests.
* feat(thoughtspot): CLI entry point for both directions
Adds `ossie-thoughtspot to-ossie`/`to-tml` (argparse only) and the
`[project.scripts]` entry point, now that `cli:main` exists to point it
at. Issues are always emitted as a JSON array -- to `--issues` when
given, to stderr otherwise -- and never mixed with document output;
exit code is tied to IssueLog.has_errors(), not to the mere presence of
a warning/info issue. `to-tml` writes one file per document via
`tml.dump_document_set` (tables before the model) into an output
directory, with a resolve-then-compare containment check as a second,
independent guard against a document name escaping that directory.
Neither direction overwrites an existing target without `--force`, and
a conflict on any one target refuses the whole run before any file is
written.
* test(thoughtspot): round-trip both directions, asserting preservation and translation separately
* fix(thoughtspot): restore a referencing join's own name and stop duplicating a field's description onto its Table column
Ossie -> TML -> Ossie renamed a Table-referenced relationship once its
join round-tripped through TML's nameless inline join syntax, because
the stashed original name was written but never read on the way out.
_join_entry_for_relationship now restores the referencing_join pointer
and the Table's own joins_with[] entry from that stash, with a currency
check against the relationship's live name so a renamed relationship
falls back to the old behaviour instead of restoring a stale reference.
_physical_table_column/_physical_sql_view_column no longer copy a
field's description onto its physical Table column -- every field
reaching those functions is already Model-surfaced, and the Model
columns[] entry is the description's only correct home.
* test(thoughtspot): property-based round-trip over adversarial identifiers
Adds a Hypothesis-driven Ossie -> TML -> Ossie property suite that draws
dataset and field names from a pool deliberately including YAML 1.1 boolean
tokens, names colliding only after normalise(), NFKD-foldable and
non-foldable non-ASCII, "::"-containing names, whitespace padding,
punctuation-only text, very long text, and the empty string. Each document
also crosses the real YAML 1.2 codec on both legs, not just the pure
conversion functions.
Two real, previously uncaught ValueError crashes were found and fixed:
- ossie_to_thoughtspot.py's _physical_identity/_field_physical_display_name
let an ambiguous [TABLE::Column] bracket's split_column_ref failure
propagate out of a document conversion uncaught; now caught, reported as
TS-FIELD-COLUMN-REF-MALFORMED, and the field is treated as a formula
rather than crashing the whole conversion.
- tml_to_ossie.py's convert() (the model's own top-level name) and
_index_attribute_columns() called identifiers.normalise() unguarded on a
display name with no ASCII alphanumerics to fold onto; now caught, with a
reported TS-MODEL-NAME-UNNORMALISABLE fallback for the former and a silent
skip (already re-reported downstream by convert()'s own Phase 3 guard) for
the latter.
hypothesis is dev-group only (never in [project.dependencies]), matching
converters/databricks' own placement. 793 baseline + 5 targeted regression
tests for the two fixes + 3 property tests = 801 passing.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* docs(thoughtspot): README, packaging and CI for the bidirectional converter
Rewrites the converter README to document both directions, the CLI, the
custom_extensions[THOUGHTSPOT] payload, and a dedicated section on what
expression translation does not attempt and why (the specification's own
pass-through default, no converter in this repo re-renders one SQL dialect
into another, and the reference converter behaves the same way).
Cleanup for public ASF review: de-identifies ~18 provenance comments that
named the internal test instance (se-thoughtspot) while keeping the
verification dates and evidence; trims the catalog.py module docstring and
several narrative note= entries toward what a maintainer needs. Removes two
custom_extensions stash keys that were written and never read (sql_query,
which duplicates the live source field; residual_predicates, already fully
contained in the verbatim on_expression) -- the payload shape version is
not bumped, since no reader ever required either key's presence. Verified
pyproject.toml's [project.scripts] entry and hypothesis test extra were
already correct; regenerated uv.lock (no changes). Added the THOUGHTSPOT
row to converters/README.md and core-spec/spec.md's custom_extensions
vendor table.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* docs(thoughtspot): generate expression, reverse, datatype and payload references from code
The expression mapping, reverse inventory, datatype map and vendor payload
were originally hand-authored design documents. The code now implements that
mapping, so it is the single source of truth: tools/generate_reference_docs.py
reads CATALOG, REVERSE, datatypes.py and constants.py's STASH_KEY_CLASSIFICATION
back out into four committed docs/*.md files, and
tests/test_reference_docs_current.py regenerates on every run and compares
byte-for-byte so a stale doc fails the suite instead of silently drifting.
The generator is dev/tooling only (stdlib + this package's own modules, no
new dependency, not a wheel package, not a console-script entry point) so the
package's only runtime dependency stays PyYAML. test_shipped_references.py
now also scans docs/**/*.md and tools/**/*.py for internal-process language
and unresolvable rule-id citations, same as every other shipped file.
* fix(thoughtspot): state the substance of internal rule citations, drop the label
The converter's comments, docstrings, and a handful of issue codes cited
short rule identifiers (E1-E13, X1-X9, R1-R11, ID1-ID4, A1-A12, KD1-KD3,
NM1-NM6, G1-G15) from a hand-authored design reference that is not part of
this repository and will not be. Every citation of that shape has been
rewritten in place to state what the rule actually requires -- most were
already fully explained by their surrounding sentence and just needed the
label dropped; a handful (the moving_*/cumulative_* partition loss, LAG's
aggregation/ORDER BY constraints, and write_stash's foreign-vendor
scenario) had real reasoning added back in.
A small number of internal issue codes (`E7-PASSTHROUGH`, `E12-UNMAPPABLE`,
`E10-DAYOFWEEK-BASE`, and 12 siblings in reverse.py/emit.py) also embedded
the citation as data, not prose. These are renamed to the `TS-EXPR-*`
convention every other issue code in this converter already uses -- no
test asserts the old literal values, so this is a safe alignment, not a
behavioural change.
docs/*.md are regenerated from the now-citation-free source via
tools/generate_reference_docs.py. README.md's "Rules" section is rewritten
to reflect that the shipping decision is resolved rather than pending.
The seven now-empty families are removed from
test_shipped_references.py's provisional allowlist, so the guard fails
closed on any of them reappearing; the "I" (invariant) family is
untouched -- resolving it is a separate, unscoped decision. Verified the
guard actually fires by temporarily reintroducing one citation and
confirming the suite failed before reverting it.
804 tests before, 804 after -- no behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): close the last mapping-doc rule-id family (I)
The previous cleanup rewrote citations to eight rule-id families
(A/E/G/ID/KD/NM/R/X) in place and removed them from
MAPPING_DOC_RULE_IDS, but left the "I" (invariant) family as an
explicitly provisional exception -- I1/I4/I5 appeared 5 times in
tests/test_keys.py and tests/test_yaml.py.
Those 5 comments already stated the substance of the rule inline, so
the fix is the same as before: drop the label, keep the sentence
(capitalizing the following word where the label led the sentence).
MAPPING_DOC_RULE_IDS is now permanently empty. The name and its two
guard tests (test_allowed_token_sets_do_not_overlap,
test_no_unresolvable_identifier_shaped_tokens) stay -- deleting them
would drop the test count below 804 -- but the surrounding comments no
longer describe anything as provisional or pending, and say plainly
that repopulating the set needs a fresh, reasoned decision, not a
silent addition.
Verified the guard still bites: reintroduced "I5" into
tests/test_yaml.py, confirmed test_no_unresolvable_identifier_shaped_tokens
failed naming that exact token and line, then reverted. A follow-up
sweep -- importing the module and running its own TOKEN_SHAPE_RE
across its own _shipped_files(), minus only ALLOWED_TOKENS -- found no
further unnamed families.
804 tests before and after -- no behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): swap ONE_TO_MANY relationship endpoints per spec
core-spec/spec.yaml requires a Relationship's `from` to name the many
side and `to` the one side, but Ossie has no cardinality field of its
own -- direction alone is the encoding. TML's join `from`/`to` do not
carry that convention; `cardinality` does, and ONE_TO_MANY is the one
value where TML's declared `from` is the one side and `to` is the many
side, backwards for Ossie's spec. A ONE_TO_MANY join was previously
emitted unswapped, silently inverting the relationship it represents;
upstream's key-coverage validator cannot catch this because it skips
datasets with no declared key.
`_relationship_from_join` (tml_to_ossie.py) now swaps a ONE_TO_MANY
join's from/to/from_columns/to_columns once built, re-deriving an
inline join's synthesized name from the swapped endpoints (a
referencing-shaped join's name is untouched -- it comes from the
Table's own joins_with[] entry, not from/to naming). `_convert_join`'s
key-candidate derivation now reads directly off the (possibly-swapped)
relationship's own to/to_columns instead of special-casing
ONE_TO_MANY separately, since the swap already puts the key on the
right side.
Round-tripping back to TML requires recovering the original,
unswapped shape (which dataset's model_tables[] entry the join is
nested under, and the `with` target), so two new stash keys carry it:
RELATIONSHIP_STASH_ENDPOINTS_SWAPPED and its witness
(_WITNESS = [from, to, from_columns, to_columns] as emitted). The
witness lets `_join_entry_for_relationship` (ossie_to_thoughtspot.py)
undo the swap only while nothing has retargeted the relationship since
-- otherwise the live shape is trusted as-is and a new
TS-JOIN-ENDPOINTS-SWAP-STALE issue records why, mirroring the existing
on_expression/referencing_join staleness pattern.
Adds a ONE_TO_MANY join to the TPC-DS fixture (store has many
store_returns, a real TPC-DS FK not previously modeled) since neither
fixture exercised this cardinality -- the gap that let the defect ship.
Regenerating the fixture's expected.ossie.yaml only added the new
sr_store_sk column/unsurfaced-column entry and the new
store_returns_sv_to_store relationship; no other dataset's derived
keys changed.
Adds unit coverage for the swap (forward), the un-swap and its
staleness fallback (reverse), and a dedicated TML -> Ossie -> TML
round-trip test for the new fixture join. 812 tests pass (804 baseline
+ 8 new); docs/vendor-payload.md regenerated via
tools/generate_reference_docs.py to catalog the two new stash keys.
* fix(thoughtspot): degrade non-Latin field/metric names instead of dropping them
identifiers.normalise() raises for a display name with no ASCII form
(CJK, Cyrillic, Greek). convert_field/convert_metric called it
unconditionally, so the raise propagated out of convert()'s per-column
try/except and was misreported as TS-COLUMN-REF-MALFORMED -- and the
column was dropped rather than converted. A fully non-Latin model lost
every field and metric it had.
_field_or_metric_identifier() extends the graceful-degradation pattern
already used for the model name to fields and metrics: fall back to the
underlying warehouse db_column_name (almost always ASCII, and unique
within its table) when there is one, else an allocator-suffixed
placeholder ("field"/"field_2", ...) so two colliding fallbacks stay
distinct. Each fallback is reported under its own TS-FIELD-NAME-
UNNORMALISABLE / TS-METRIC-NAME-UNNORMALISABLE code, separate from a
genuinely malformed column_id/join reference. A field's exact display
name is already recoverable via `label`; a metric's is stashed via the
existing STASH_TML_NAME key.
_index_attribute_columns (the Phase 2 cross-reference gate) is
simplified to track membership only, dropping a second, independent
identifier computation that had to somehow stay in sync with
convert_field's own -- and that used to silently exclude a non-Latin-
named column from the gate entirely, so a cross-reference to it
resolved to nothing even once the field itself was fixed. convert()'s
Phase 3 now records each field's real identifier in built_field_names
as it's built, which Phase 3.5's SQL View sql_output_columns stash
reads instead.
Adds a non-Latin (CJK) field to the tpcds fixture (customer.カナ名),
regenerated expected.ossie.yaml, plus unit tests for the fallback
value, fallback-collision avoidance, and the corrected issue code.
Dataset names are deliberately never normalised (verified, unchanged):
a Dataset's `name` is the verbatim model_tables[] reference name, which
every column_id/join in that dataset must match exactly -- there is no
separate label field to recover a display name from the way a Field or
Metric has, so folding it would break every reference into that
dataset instead of just relabelling it.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* test(thoughtspot): realign the TPC-DS fixture with the reference model
Upstream #363 renamed date_dim's month field from d_month_name to d_moy and
redescribed it as 'Month of year (1-12)', updating three sibling converters'
fixtures. This branch predated that change, so its fixture had drifted from the
reference it exists to mirror.
Also corrects the datatype: a month-of-year holding 1-12 is INT64, not VARCHAR.
The reference deliberately omits an explicit datatype there to show the field is
optional, which a TML fixture cannot mirror — a physical column always carries a
type — so INT64 is the faithful choice.
Diffed all five datasets, the relationships and the metrics against the current
reference; this was the only divergence. date_dim now matches field-for-field on
names.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* feat(thoughtspot): one semantic model per document, no wrapper
Follows apache/ossie#383, which moved the semantic model's fields to the
document root and removed the `semantic_model` wrapper. `ossie-schema.json`
now requires `version`, `name` and `datasets` at the root and forbids
additional properties, so the converter's emitted documents stopped
validating the moment the branch was rebased.
Emit side is `{"version": ..., **semantic_model}`. The spread cannot clobber
`version`: the model dict only ever holds name/description/datasets/
relationships/metrics plus the `custom_extensions` the stash writes, and no
path on it emits a `version` key.
Read side rejects the old wrapper by name rather than lifting its first
entry, matching the treatment in #396. Reading entry zero would convert
silently while discarding any later model, and a document old enough to
carry the wrapper may have moved on elsewhere too; naming the one thing the
reader must change is more useful than a best-effort guess.
`datasets` is `minItems: 1` upstream, so the emit side now reports
TS-MODEL-NO-DATASETS at ERROR when a model yields none. Without it the two
legs disagreed: `to-ossie` exited 0 having written a document that fails the
schema and that this converter's own `to-tml` refuses. Both legs now exit 1
on the same input.
Both fixtures were regenerated by running the migrated converter over their
own TML and asserting the result deep-equals the old expected document with
its wrapper lifted -- so the 1,151 changed fixture lines are a re-rooting,
not a content change.
823 tests pass. The README's mapping row is now pinned by a test, having
been verified by nothing: reverting it left the whole suite green.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* ci(thoughtspot): widen path filters to core-spec/** and examples/**
apache/ossie#396 added these two triggers to all fourteen sibling converter
workflows. This one was not in that PR -- the branch is unmerged, so there
was nothing to migrate -- which left it the only converter whose CI does not
re-run when the spec moves.
That gap is not cosmetic here. catalog.py's spec_construct_names() parses
core-spec/expression_language.md directly as its upstream-spec coverage
oracle, so a spec edit can fail this converter's tests without touching a
single converter file. The narrower `core-spec/expression_language.md`
trigger this replaces covered that one file; `core-spec/**` covers it and
matches the siblings.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): variadic arity, identifier collisions, case-folded filenames
Addresses the five findings in @jbonofre's review of #364. All five reproduce;
each fix is verified against the reporter's own failing case.
Expression catalog (findings 1-3), fixed as a class rather than row by row.
Six DIRECT rows took any number of arguments and were rendered by a fixed
template: CONCAT, GREATEST, LEAST, IN, NOT IN copied the specification's `...`
ellipsis into the template, where `str.format` never fills it, so a
two-argument CONCAT emitted `concat ( a , b , ... )`; COALESCE hardcoded three
arguments, so `COALESCE(a, b)` -- the case its own note calls common -- raised.
NOT IN carried the same ellipsis defect and was not in the review.
`Construct` now declares `variadic=Variadic(style, min_args, tail_from)`.
JOIN substitutes the argument tail at a `{*}` marker; FOLD applies a binary
template right-associatively, which is what COALESCE needs because ThoughtSpot's
`ifnull` has no n-ary form. Arity is checked as a minimum, and a literal `...`
in any template is now refused at construction, so no new row can reintroduce it.
The sweep that missed this derived each row's arity FROM its template and then
asserted only that emission did not raise -- guard and test read the same
string, so no template could ever disagree with itself. It now sweeps variadic
rows across a range of arities and asserts on the OUTPUT: no literal `...`, no
unfilled placeholder, balanced parens and braces. Mutation-checked both ways --
restoring the original CONCAT template fails the sweep on its output, and fails
at construction once the ban is back.
Identifier collisions (finding 4). Normalisation is many-to-one, so "Order Date"
and "Order-Date" both became `order_date` in one dataset with nothing logged.
Resolved where the owning scope is first known -- per dataset for fields, the
flat `metrics[]` for metrics -- rather than inside `_field_or_metric_identifier`,
whose allocator is deliberately model-wide: routing the common case through that
would rename `orders.amount` and `customers.amount` to `amount` and `amount_2`.
Comparison folds case, matching `Allocator`, because Ossie resolves identifiers
case-insensitively. Renames are reported, not applied quietly.
Filename collisions (finding 5). The check used a case-sensitive set, so on
macOS and Windows `Store_Sales` and `STORE_SALES` were one file: the second
write destroyed the first and the CLI exited 0. Reproduced as real data loss
(3 documents emitted, 2 on disk); comparison now folds case, emitted names keep
theirs.
824 tests pass. Both fixtures and emitted output validate against the current
upstream schema; the TPC-DS fixture round-trips end to end.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): eight defects found by a max-level review of the converter
All eight reproduce; each is verified against its own failing case and pinned
by a regression test. 848 tests pass (was 824).
SILENT DOUBLE AGGREGATION. `_AGGREGATE_CALL_NAMES` knew `group_aggregate`
alone, above a comment asserting "there is exactly one such construct", while
this package's own reverse inventory registered four shorthands (group_sum,
group_count, group_stddev, group_variance) and two further *_aggregate_op
variants. A MEASURE column using any of them was classified as a scalar formula
and had its own `aggregation` composed on top: `sum ( group_sum ( ... ) )`, no
issue raised, wrong number. The vocabulary is now derived from one source.
CARDINALITY SURVIVED A STALE SWAP WITNESS. An Ossie relationship has no
cardinality field -- `from` is the many side, `to` the one side, so direction IS
cardinality. When the endpoint-swap witness went stale the swap was correctly
abandoned but the stashed cardinality was still read, declaring the join
backwards; ThoughtSpot uses cardinality for fan-out, so the model returned
multiplied rows. Gated on the stale-swap condition, which also honours that
issue's own promise to emit "exactly as a hand-authored relationship would".
A PARENTHESISED JOIN CONDITION LOST ITS RELATIONSHIP. `formula._scan` tracks
paren depth, so `( [A::x] = [B::y] and [A::p] = [B::q] )` -- an ordinary TML
spelling -- put every `and` at depth 1: nothing split, no equality pair matched,
and the whole relationship was demoted to an unrepresentable-join stash entry,
leaving the datasets disconnected and `derive_keys` with no candidate.
AN ALIASED SELF-JOIN EMITTED DUPLICATE TABLE DOCUMENTS. N datasets over one
warehouse table produced N Table documents of the same name; filename
disambiguation then hid the collision and the import created a duplicate Table
object. One document per distinct table now, columns unioned. Found while
fixing it: two model_tables[] entries with no distinguishing alias are
ambiguous on import and were unreported -- now an ERROR.
Also: `zip` truncated a join condition when from_columns/to_columns disagreed,
dropping predicates silently (now refused, matching upstream #375's equal-arity
rule); `_restore_ai_context` overwrote a faithfully stashed synonym_type; shared
list objects made PyYAML emit anchor/alias pairs into every generated document
and both fixtures; `_safe_target_path`'s condition let through the one case it
existed to catch; `read_stash` and a non-UTF-8 input file escaped the CLI's
handler as bare tracebacks.
Fixtures regenerated, proven deep-equal to their previous parsed content. Both
fixtures and emitted output validate against the current upstream schema; the
TPC-DS fixture round-trips end to end.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): correct the reverse inventory and declare its exemplar rows
Closes the rest of the max-level review. 851 tests pass (was 848).
MOVING-WINDOW FRAME BOUNDS WERE INVERTED AT ONE END. `moving_*`'s two offsets
count in OPPOSITE directions -- argument 2 is rows BACKWARD (positive ->
PRECEDING), argument 3 is rows FORWARD (positive -> FOLLOWING) -- and both
shared a single "positive means PRECEDING" helper. Every window with a non-zero
forward count came out wrong, not only the LEAD idiom the review named:
`moving_average(m, 2, 3, d)` emitted `BETWEEN 2 PRECEDING AND 3 PRECEDING`,
which is valid SQL over the wrong five rows -- silently wrong, not a loud
failure -- and `(m, 0, 5, d)` emitted `BETWEEN CURRENT ROW AND 5 PRECEDING`,
which no engine runs.
The existing test asserted the defect and its comment recorded the misreading
as fact ("negative end flips to FOLLOWING"), so it had to be settled from
outside this repo. Two independent mappings agree: the Tableau mapping renders
`WINDOW_AVG(x, -3, 0)` as `moving_average(x, 3, 0, attr)`, and the Looker
mapping labels `moving_average(sum(REVENUE), 3, -1, date)` a "3-Row Moving
Average" -- which it is only if -1 means 1 PRECEDING. Both are cited in the
test.
A FISCAL LITERAL HIJACKED EVERY FUNCTION. The trailing-`fiscal` check ran for
any name, so `concat([T::label], 'fiscal')` -- a string function with a string
literal -- was routed into an ERROR-severity total-loss stash, discarding a
composable CONCAT and failing the conversion's exit code through has_errors().
Now gated on date functions, derived from both inventories rather than
hand-listed: the first hand-list drafted here already omitted two names this
module's own tests exercise.
`sql_number_op` was missing from the dialect family while being a declared
`Variant`, so it fell through to a silent None where its ten siblings log
TS-EXPR-DIALECT-UNKNOWN.
ELEVEN EXEMPLAR ROWS NOW SAY SO. A passthrough template carrying a baked-in
constant renders without error, so the arity guard cannot tell it from a
complete one, and `docs/expression-mapping.md` -- which ships -- offered
`NTILE(4)` as the mapping for `NTILE(n)` with nothing marking the 4 as
illustrative. Each now declares `exemplar_literals`, the document renders
"**example only**", and construction refuses a new undeclared one: an
allowlist, not a blocklist.
The group-aggregate vocabulary moves to constants.py, whose own header states
the rule it exists for. The live double-aggregation guard was importing it from
the reverse inventory, a module the README describes as unwired.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): SQL View merge read the wrong column key; output paths could collide
Both findings from @jbonofre's second review, and both are defects in code from
this PR's previous two commits rather than pre-existing ones. 859 tests pass.
SQL VIEW COLUMNS WERE DROPPED BY THE MERGE I ADDED. `_deduplicate_table_documents`
merged only `columns`, but a SQL View keeps its output columns under
`sql_view_columns`. Two aliased datasets over one SQL View therefore lost the
second's fields entirely, `build_model` then raised TS-MODEL-COLUMN-ID-MISSING
for the fields that were dropped, and the body-divergence comparison fired
spuriously because it excluded only `columns` too. So the fix for duplicate
Table documents was worse than the defect it replaced.
Reading the wrong column key is this converter's most repeated mistake -- three
separate silent failures now -- so the kind-to-key rule is named once, in
`_column_key_for`, and both call sites use it. The divergence comparison ignores
both keys, and a kind mismatch is itself a divergence.
A RUN COULD DESTROY ITS OWN OUTPUT. `--issues` and `-o` resolving to one path
passed the does-it-already-exist check on a fresh run; `_write_issues` then
replaced the converted document with the JSON issue log and the command exited
0. The same collision was reachable between `--issues` and a generated `to-tml`
filename. Both commands now reject a repeated target before writing anything,
compared after `resolve()` so two spellings of one file are caught, and
regardless of `--force` -- which licenses overwriting files that were already
there, not destroying an output this run just produced.
Both fixes are mutation-checked: reverting either fails the new tests.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): case-folded path guard, the full group_* family, public citations
Three findings from a four-lens review of the previous commits, two of them
defects in those commits' own claimed scope. 864 tests pass.
THE COLLISION GUARD I ADDED DID NOT FOLD CASE, so it did not close the defect
it was written for: `-o out.yaml --issues OUT.YAML` still resolved to one file
on macOS and Windows, the issue log still overwrote the converted document, and
the command still exited 0. The commit one earlier had fixed exactly this class
in `dump_document_set`; the CLI guard, written a day later, did not carry it.
FOUR OF NINE GROUPED AGGREGATES WERE STILL DOUBLE-AGGREGATED. The vocabulary was
sourced from the reverse inventory, whose rule is "name nothing the mapping
document does not name" -- right for traceability, wrong for this guard, whose
source of truth is ThoughtSpot's function list. `group_max`, `group_min`,
`group_average` and `group_unique_count` were missing, so a metric using one
emitted `sum ( group_max ( ... ) )`.
And the test could not see it: it parametrised itself FROM the constant under
test, so it passed for any subset -- the guard-cannot-fire class it was written
to prevent. It now carries an explicit list as an independent oracle, and a
separate assertion that the constant covers it. Mutation-checked: removing the
four names fails five tests.
PRIVATE-REPO CITATIONS REMOVED from the frame-bound test. They were the sole
stated evidence for a behaviour change on a public ASF PR, and pointed at a
repository no ASF reader can open -- in a package that ships a guard against
that very thing. Replaced with ThoughtSpot's public formula reference, whose
argument names (`num_backward`, `num_forward`) carry the whole argument. The
"two independent sources" claim is also withdrawn: one of the two examples has
a forward count of 0 and renders identically under both rules, so it
distinguished nothing.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* test(thoughtspot): pin three fixes that no test could fail on
A four-lens review of this PR's fix commits found that three fixes -- two of
them reported to the reviewer as complete and verified -- had no test that
could fail. Reverting each left the full suite green:
- case-folded filename deconfliction (the reviewer's own finding, which I
reported as "reproduced as real data loss: 3 documents emitted, 2 on disk")
- `primary_key` no longer aliasing `unique_keys[0]` in emitted YAML
- `_resolve_name_collision` entirely -- the helper, both call sites and both
issue codes, reachable by no test at all
Each now has a test that fails when the fix is reverted, checked by doing so.
The YAML one is the instructive case, and the error was mine. Its existing
guard read the COMMITTED FIXTURE's text, so it pinned the artifact rather than
the code -- and I "mutation-tested" it by editing that fixture, which proved
only that the test reads a file. The new test derives keys and asserts on
freshly dumped output, plus object identity, which is where the defect lives.
That is the same shape as the catalog sweep this PR already replaced for
deriving its expectation from the thing it was checking. Third occurrence in
this package; the first two were found by reviewers, this one by mutation.
872 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): refuse three unsafe table merges instead of corrupting output
`_deduplicate_table_documents` merged on name alone. Each case below was
silently corrupted rather than reported, and the first two were introduced by
the fix that added the function.
A TABLE AND A SQL VIEW ARE NO LONGER MERGED. They are different ThoughtSpot
objects, and merging them wrote `sql_view_columns` into a `table:` document --
not valid TML. Reverting the whole dedupe produced two well-formed documents,
so the fix had made previously-valid output invalid. Two kinds under one name
is a modelling error in the source, so it is now an ERROR and both documents
survive.
A CONFLICTING COLUMN DEFINITION IS NOW AN ERROR. Columns are unioned by name
because two aliases may surface different subsets of one table, but a
same-named column with a different body is a contradiction, not a subset. The
first won silently, so the second dataset's field read the wrong warehouse
column -- `amount` bound to AMT_USD while its dataset meant AMT_EUR.
UNNAMED DOCUMENTS ARE NO LONGER MERGED TOGETHER. `_table_name`'s last resort is
a shared literal, so two documents that reached it were treated as one object.
The divergence warning also now names the keys that actually differ instead of
guessing at a fixed list of three, and no longer claims the second document's
columns are discarded when they are merged in.
All four guards are mutation-checked. Worth recording that my first attempt at
that check was wrong: the unnamed-document guard lives in two places, I
neutralised one, saw the test still pass, and nearly concluded the guard was
dead code. It is not -- the second mutation fails the test.
877 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): a formula cross-reference no longer rebinds to another formula
TML's `formulas[].id` and `formulas[].name` are INDEPENDENT -- a formula renamed
after creation keeps its original id. The Ossie -> TML leg minted ids from
display names and resolved a verbatim `[formula_X]` reference by matching X
against normalised NAMES, so when some other formula's name normalised to the
referenced id's tail, the reference silently followed it.
Reproduced end to end. A model whose `Total` is `[formula_margin] + 1`, where
`formula_margin` is the id of "Net Margin" and a separate formula is named
"Margin":
in Total = (sum(a) - sum(b)) + 1
out Total = (sum(a) * 0.5) + 1 exit 0, reverse-leg issue log empty
The source id is now preserved. `tml_to_ossie` stashes the TML `formulas[].id`
on the field or metric it builds; `ossie_to_thoughtspot` mints from that stash
in preference to re-deriving from the display name, and the reference rewriter
treats a reference that already names an emitted id as resolved instead of
falling through to the name map. Ids round-trip verbatim, so the binding is
preserved rather than re-guessed.
The new stash key is classified INFORMATION_ONLY deliberately: Ossie has no
formula-id concept, so there is no live value it can diverge from -- and that
is the point, since renaming a metric in Ossie must NOT change the id or every
reference written against it breaks.
One fixture assertion changed with it: `test_the_three_metric_shapes_are_all_present`
asserted a bare aggregate carries NO stash, conflating "this shape needs no
marker" with "this object needs no stash". It now asserts no SHAPE is stashed.
Mutation-checked. 879 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): relationship names are de-collided, so the document validates
A TML join carries no name of its own, so this converter synthesises one as
`<from>_to_<to>`. Two joins between the same pair of tables -- a composite key
plus an as-of/range residual, which is the ordinary SCD-2 shape -- therefore
produced two relationships of one name. Fields and metrics were already
de-collided here; relationships were the one scope with no guard.
Verified against upstream's own tool, on the converter's own output:
before Validation FAILED with 1 error(s):
[Unique] Duplicate relationship name 'a_to_b' in model 'both_model'
after Validation PASSED
The converter emitted that rejected document with exit 0 and nothing logged.
The rename now goes through the same `_resolve_name_collision` the other two
scopes use and is reported at WARNING.
Mutation-checked. 881 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): a renamed dataset no longer dangles every reference to itself
`build_model` read the stashed dataset alias with no currency check, while
`column_id` and every join target were built from the LIVE dataset name. So
renaming a dataset emitted a model whose `model_tables[]` carried the old alias
and whose references named the new one -- pointing at nothing:
model_tables: [{name: EMPLOYEES, alias: emp}, {name: EMPLOYEES, alias: mgr}]
column_ids: ['emp::employee_id', 'manager::employee_name'] <- dangles
issues: nothing about it
The alias is SELF-VERIFYING: `tml_to_ossie._build_dataset` writes the Ossie
dataset's `name` from it, so `stashed == dataset["name"]` is the whole check --
the same shape `RELATIONSHIP_STASH_REFERENCING_JOIN` already uses. When they
disagree the live name wins and the divergence is reported.
The root cause was a misclassification, so the classification is corrected too:
`DATASET_STASH_ALIAS` was INFORMATION_ONLY, defined as "no Ossie-native
counterpart at all ... nothing there could have diverged from it". It has one,
and it diverges. It is SHADOWS_DERIVABLE now, listed among the self-verifying
keys, which is what makes the enforcement table agree with the code again.
Mutation-checked. 884 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): one warehouse column surfaced twice is emitted once
Showing the same date as "Sale Date" and, with a format pattern, as
"Sale Month" is ordinary modelling, but it is still ONE physical column. Both
body builders appended an entry per field, so:
sql_view_columns:
- {name: d, sql_output_column: d} <- alias GUESSED and lowercased
- {name: d, sql_output_column: D} <- duplicate name
- {name: amt, sql_output_column: AMT}
Two defects in one. The duplicate entry, and -- worse -- a corrupted alias: only
one of the two fields carried the stashed `sql_output_column`, so the other had
one derived from its display name. Lowercased, it binds to nothing on a
case-sensitive warehouse. Both were silent.
Entries are now folded by column name. Where the two agree, the first stands and
nothing is reported -- they are the same column. Where they disagree, a value the
SOURCE document recorded beats one this converter guessed, so the stashed alias
wins; a disagreement between two equally-trusted values is reported rather than
resolved by position.
Mutation-checked. 886 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): stop inventing display-name collisions that ThoughtSpot has not got
`_DisplayNameAllocator` assigns TML DISPLAY NAMES, but folded each candidate
through `identifiers.normalise` -- an OSSIE IDENTIFIER rule, which lowercases,
strips punctuation and drops non-ASCII after NFKD. Applied to the wrong side of
the converter it invented collisions between names that are perfectly distinct
in ThoughtSpot:
Order Amount / Order-Amount both folded to order_amount
Cafe / Café both folded to cafe
収益 2024 / 売上 2024 both folded to n_2024
One of each pair was renamed to `<name>_2`, and the issue explaining it asserted
"to satisfy ThoughtSpot's uniqueness requirement" -- a requirement that does not
apply, since those are different strings to ThoughtSpot. The allocator is also
model-wide while the forward leg de-collides fields per dataset, so two fields
in DIFFERENT datasets could trigger it.
The fold is now a plain casefold: case is the only thing ThoughtSpot ignores.
Strictly narrower, so it renames less, and everything it still renames is a
genuine collision.
The property test that covered this generated punctuation variants and asserted
they collide, pinning the defect. It now generates only case and whitespace
variants, and a new property pins the other direction -- that punctuation
variants are left alone.
Mutation-checked. 888 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): build the fiscal gate after the inventory it reads, not during
`_FISCAL_CAPABLE_FUNCTIONS` was a module-level constant evaluated partway down
`reverse.py` -- while `REVERSE` kept being populated for another hundred lines
below it. Twelve date functions registered after that point were therefore
excluded from the gate, and a fiscal call on one of them raised an uncaught
`ValueError` where it had previously reported a declared loss:
month, year_name, day_of_week, month_number_of_quarter,
day_number_of_quarter, week_number_of_month, week_number_of_quarter,
is_weekend, start_of_hour, start_of_min, date
Deriving the set was the right instinct -- the hand-list it replaced was
already missing two names its own tests exercised -- but deriving it before its
source existed made it worse, not better. It is computed on first call now.
The token match was also unbounded, so `min(?:ute)?` matched the AGGREGATE
`min`, taking `min_if`, `cumulative_min` and `moving_min` with it, and `time`
matched inside "runtime". All four were classed fiscal-capable, which is the
same over-inclusion the gate exists to prevent. Tokens are word-shaped now,
with `start_of_*`/`end_of_*` admitted by prefix since `start_of_min`
abbreviates "minute" to a string that cannot be a token without re-admitting
the aggregate.
The old test used `year`, `quarter_number` and `diff_months` -- all registered
BEFORE the constant, so it was blind to the ordering by construction. The new
one is parametrised from late-registered names for that reason.
Mutation-checked: pinning the set to an early-only source fails 22 tests.
913 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* test(thoughtspot): generate document SHAPES, not just names
The existing hypothesis properties generate adversarial NAMES -- unicode,
display-name collisions, warehouse names. Every structural defect found in
review was a SHAPE: a table aliased to itself, two datasets over one SQL View,
a parenthesised join condition, case-colliding document names. A name generator
cannot reach any of them, which is why reviewers kept finding what 800-odd
tests did not.
This generator varies dataset count, table vs SQL View, aliasing several
datasets onto one source, relationships (including composite), already
-aggregated metrics, and case-varying source names. What it asserts is the
package's own stated contract, which holds over any shape: nothing is silently
dropped -- every field survives or an issue names it -- no two documents would
be written to one path, and a declared relationship survives or is reported.
It earned its place on the first run, twice over. It surfaced that an Ossie
relationship's NAME is silently lost for an inline join (TML joins carry no
name, so the return leg re-derives `<from>_to_<to>`; only the "referencing"
shape stashes one). And it caught a bad assertion of my own: the property
originally compared relationships by name, which is too strict for exactly that
reason -- it now compares structure.
6,000 examples across three properties, all passing. 917 tests total.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): unique formula ids, and stop erroring on a valid self-join
Three findings from a review of the previous nine commits. Two are regressions
those commits introduced — the pattern this round was meant to break.
DUPLICATE `formulas[].id`, EMITTED SILENTLY. `formulas[].id` has two sources
sharing one namespace, and neither checked the other: an id PRESERVED from the
source stash, and one MINTED from the display name. Narrowing
`_DisplayNameAllocator`'s fold to what ThoughtSpot actually treats as equal --
correct in itself — removed the only thing that had been masking the minted
case, so "Order Amount" and "Order-Amount" both emitted `formula_order_amount`
with no issue. A hand-authored metric minting an id equal to a preserved one
collided the same way. Either makes every `[formula_X]` reference ambiguous,
and ThoughtSpot parses an ambiguous bracket reference as search tokens rather
than failing — so the import succeeds and the model is wrong. Ids now come
from one allocator across both sources, the surfacing column's `formula_id` is
rewritten in lockstep, and a rename is reported.
AN ORDINARY SELF-JOIN NOW FAILED. The column-conflict check added last commit
compared whole entries, but a Table's `columns[]` mixes field-derived entries
(name / db_column_name / db_column_properties) with verbatim
`unsurfaced_columns` stash entries carrying raw TML keys. A column surfaced
through one alias and unsurfaced through the other compared unequal: ERROR,
non-zero exit, on a document that was fine — and one that converted cleanly
before the fix. Only the keys that decide WHICH warehouse column is read are
compared now, which is what the message always claimed.
THE CLI CASEFOLD FIX HAD NO TEST. `-o out.yaml --issues OUT.YAML` was pinned by
nothing, so the guard could regress silently. Its sibling in
`dump_document_set` was pinned; this one was missed by the commit that went
looking for exactly this.
All three mutation-checked. 923 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): preserved formula ids win by reservation, not by luck of order
Three findings from a review of the previous commit, two of them defects that
commit introduced. Fifth consecutive round in which the fixes carried new
defects; recording that plainly rather than describing this as converged.
A PRESERVED ID WAS DECIDED BY PROCESSING ORDER. `_allocate_formula_id` renamed
whichever formula it reached second, so a preserved id -- the identity a source
cross-reference was written against -- could be handed to a newly minted one.
The field loop runs before the metric loop, so adding an ordinary formula field
to a round-tripped document was enough:
formula_margin Margin [t::a] + 1 <- the new field took it
formula_margin_2 Net Margin sum ( [t::a] ) <- the preserved one renamed
Double Net = 2 * [formula_margin] <- now 2 x the wrong formula
Preserved ids are reserved in a pass before any minting, and are never renamed.
The test that was meant to cover this asserted `ids[0] == "formula_margin"` and
passed only because the preserved metric happened to be listed first; it now
checks both orderings.
THE THIRD SOURCE OF IDS BYPASSED THE ALLOCATOR. A formula spanning two datasets
is stashed as name+expr only -- its id dropped -- and re-minted on return. That
loop appended directly, so a pure round trip of a VALID document still emitted
duplicate ids, silently. The allocator's own docstring said ids come from "two
places".
A TEST SURVIVED ITS FIX BEING REVERTED. `test_the_surfacing_column_follows_the_
renamed_id` asserted `referenced <= emitted`, which cannot see the failure: when
the column keeps the old id, that id is still emitted by the OTHER formula, so
the subset holds while the column surfaces the wrong expression. It asserts the
pair now.
Also widened `_COLUMN_BINDING_KEYS` to include `db_column_properties`: narrowing
it to fix the self-join false positive went further than needed and made a
datatype disagreement -- which binds a field to the wrong data just as surely as
a wrong column name -- completely silent. Confirmed the wider key does not
re-break the self-join.
All four mutation-checked, including the one that previously survived.
926 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): emit the compulsory `with` on a referencing join
Found by importing a converted REAL model into a live ThoughtSpot cluster.
`with` is mandatory on every `model_tables[].joins[]` entry. The referencing
branch emitted only `referencing_join` — but that pointer names the Table's
`joins_with[]` entry, it does not replace the join target:
emitted {"referencing_join": "C_DIM_RETAPP_PRODUCTS"}
real TML {"with": "DIM_RETAPP_PRODUCTS", "referencing_join": "C_DIM_RETAPP_PRODUCTS"}
ThoughtSpot: "Compulsory Field worksheet->model_tables(2nd)->joins(1st)->with
is not populated."
Nothing caught this, and the reason matters more than the defect. The document
passes the Ossie schema. It passes upstream's `validation/validate.py`. Both
validate the OSSIE side, and the missing field is on the ThoughtSpot side. And
the round-trip tests compared the converter's output against HAND-AUTHORED
FIXTURES THAT OMITTED `with` TOO — so code and tests shared one
misunderstanding and agreed with each other. 926 tests passed throughout.
Both fixtures now carry `with`, matching what a real export contains, and a new
test asserts that every emitted joins[] entry has one, over every fixture rather
than at two hardcoded sites.
Caught on the first real model imported. Structural round-tripping of 30 real
models -- 161 tables, 1,471 columns, 146 joins, all passing upstream's validator
-- did not surface it, because no amount of Ossie-side validation can.
927 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* feat(thoughtspot): preserve obj_id, ThoughtSpot's portable object handle
`guid`, `fqn` and `obj_id` were grouped together as "instance-local identity
never travels in a portable document". They are not the same kind of thing.
`guid` is a raw cluster UUID. `fqn` is a reference to one -- and this repo's own
schema reference already records that a viz-level `fqn` is DROPPED on import,
leaving the object with no data source. Both are correctly refused.
`obj_id` is the opposite: ThoughtSpot introduced it precisely so objects can be
referenced across environments, and it survives import. It is a readable handle
-- `SampleRetail-Apparel-LH-58435d2b`, the display name plus the GUID's first
segment -- not a bare identifier. Discarding it meant a converted model
re-imported as a NEW object beside the one it came from, rather than updating
it, which breaks the promote-between-environments workflow this converter exists
to serve.
It is now stashed under its own payload key and restored at the document root.
The forbidden-key scan still names `obj_id`, deliberately: that scan stops one
riding along unnoticed inside a block copied wholesale from source TML, while
the stash preserves it on purpose, under a distinct key. The two are not in
conflict and the comment now says so.
`guid` is still stripped at every depth, and the test asserts that.
Observed on a real export while investigating why a converted model could not be
re-imported: the source carried `obj_id: SampleRetail-Apparel-LH-58435d2b` and
the converted document carried nothing.
Mutation-checked. 928 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): a bare group_aggregate keeps its column aggregation
Not every already-aggregating formula makes the surfacing column's
`aggregation` inert, and the converter treated them as if it did.
Established by domain review and confirmed against a live cluster:
sum ( ... ) column aggregation is a NO-OP
group_sum / group_average ( ... ) NO-OP -- these behave like ordinary formulas
sum ( group_aggregate ( ... ) ) NO-OP -- wrapped, so the outer call rules
group_aggregate ( ... ) BARE USED -- like a raw column, ThoughtSpot may
apply the column's aggregation to it
The code grouped all four as "the documented no-op", naming
`group_aggregate ( ... )` in that comment explicitly, and discarded the value
with nothing logged -- deliberately, since warning on the genuinely inert
shapes would train readers to ignore the issue log. So the one shape where the
property is load-bearing lost it silently: a changed answer, not a changed
spelling.
Ossie's metric expression has nowhere to put it, so it is preserved verbatim in
the stash and restored, rather than discarded or folded into the expression --
folding would change what the formula means, since wrapping a bare
`group_aggregate` is exactly what makes the property inert.
Also recorded, because it was nearly acted on as a defect: `sum([SALES])` with
`aggregation: SUM` is genuinely a no-op. A converted model carrying both was
imported into a live cluster and returned numbers identical to its source. An
earlier reading of a `NESTED_AGGREGATE_NOT_SUPPORTED` error as a converter
defect was wrong -- the nesting was in the QUERY, which wrapped an
already-aggregated formula in SUM().
Mutation-checked. 932 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): keep formulas that no column surfaces, and keep them hidden
A model's `formulas[]` may hold entries no `columns[]` entry references. Those
are HIDDEN in the ThoughtSpot UI, and the usual reason to write one is that
another formula uses it -- a date-parameter model is the common case, where
`_startDate`/`_endDate` compute a window the visible formulas then apply.
The converter walks `columns[]`, so a formula no column surfaces was never
visited and was dropped outright. Even the unattributed-formula stash could not
catch them: that only holds formulas that WERE visited and could not be
attributed to a dataset.
On the real model that surfaced this, 32 of 41 formulas were lost, and the 9
visible ones referencing them came back with dangling `[formula__startDate]`
references. Round-tripping it now:
dangling references 4 -> NONE
reverse-leg ERRORs 4 -> none (the run exited 1 before)
surfacing columns 45 -> 45 unchanged
They are stashed with their `id` -- that is what the surviving references name,
and re-minting an id from the display name is exactly how the references came
to point at nothing -- and restored as `formulas[]` entries with NO surfacing
column. Giving them one, as the unattributed-formula path deliberately does,
would make a helper the modeller kept private visible to users.
Mutation-checked. 936 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): stop warning on the two most common correct paths
Measured across the 30-real-model corpus: 1,473 warnings and errors before,
622 after. None of the difference is a check being weakened -- both codes fired
on paths where nothing is wrong, which is exactly how an issue log stops being
read, a risk this package's own comments name elsewhere.
TS-FIELD-DB-COLUMN-NAME-ASSUMED, 593 -> 92. It said "assumed" about something
the document establishes: the forward direction stashes `db_column_name` ONLY
when the warehouse name differs from the display name, so its ABSENCE means
they agreed. For a field this converter produced that is a recorded fact, not a
guess. It remains a real assumption -- and still reports -- for a field with no
ThoughtSpot stash at all, where nothing ever recorded the relationship.
TS-EXPR-UNRESOLVED, WARNING -> INFO (346 occurrences). A formula may reference
any column of a joined table, including one the model does not surface as a
column of its own; that is ordinary modelling, and there is then no Ossie field
for a portable ANSI_SQL sibling to name. Nothing is lost -- the THOUGHTSPOT
dialect entry carries the expression verbatim -- and it is the same fact
TS-EXPR-THOUGHTSPOT-ONLY already records at INFO, so the WARNING was
inconsistent as well as noisy. The message now says what is and is not affected.
Re-running the corpus also confirms the unsurfaced-formula fix: models losing
structure 1 -> 0, and 30/30 still pass upstream's validator.
936 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* docs(thoughtspot): declare the exemplars a bare keyword hid from the gate
The exemplar gate added earlier catches a baked-in NUMBER or QUOTED STRING in a
passthrough template. It cannot see a baked-in AGGREGATE, because that is a bare
keyword. Five rows were in that state:
DISTINCT aggregate modifier SUM(DISTINCT {0})
DENSE_RANK() OVER (...) ... order by sum({0}) desc
NTILE(n) OVER (...) ... ORDER BY SUM({0})
CUME_DIST() OVER (...) ... ORDER BY SUM({0})
Window aggregation -- AGG(expr) SUM({0}) OVER (...)
In each the aggregate is illustrative, exactly as NTILE's literal 4 is: the
specification orders by whatever the caller chose. The generated document
presented `SUM` as the mapping, with no marker, for a row whose own spec_name
says `AGG(expr)`. Two of these rows already SAID so in their prose note -- the
Window aggregation note calls its frame "an exemplar, the same convention as
NTILE's literal 4" -- while the machine-readable declaration the document is
built from did not, so the note and the table disagreed.
NTILE also shows why a non-empty declaration is not enough on its own: it
declared `n`, satisfied the literal gate, and left its SUM unmentioned. The new
check therefore requires that some declared exemplar NAMES an aggregate, not
merely that the tuple is non-empty.
LAG/LEAD, separately: both declared `default` as an exemplar literal while their
template had no default argument at all -- a declaration describing something
absent rather than something illustrative. The template now shows the third
argument, so the declaration describes the template.
`baked_literals`, `baked_aggregates` and `declares_an_aggregate_exemplar` are
now module-level in `_types.py`, so the construction-time gate and the two
catalog-wide tests apply one rule rather than three copies of it.
Verified by mutation, both halves, since the guard lives in two places:
undeclaring NTILE's aggregate raises at catalog construction; with the gate
neutralised as well, the new test still fails on it.
No emitted output changes: no exemplar row is reachable from the conversion
path, because `_match_ansi_call` matches only single-argument `NAME(expr)` keys
and `COUNT(DISTINCT expr)`. These rows are the reference document's content.
937 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* test(thoughtspot): pin the README's construct counts to the catalog
The README opens its expression section with four bare numbers -- "all 146
constructs ... 108 with a native equivalent, 37 as a `sql_*_op` pass-through
... and 1 (`EXISTS_IN()`)". Nothing derived them from the catalog, so adding or
reclassifying a row left the sentence quietly wrong. They are correct today
(146 = 108 + 37 + 1, checked); this stops them drifting.
Each count is asserted PRESENT before it is compared, so rewording the sentence
fails the test rather than silently leaving the counts unchecked -- the
fail-open shape a guard like this otherwise takes. Mutation-checked both ways:
changing 108 to 109 fails on the value, replacing "108" with "one hundred and
eight" fails on the absence.
938 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* fix(thoughtspot): reserve unsurfaced formula ids against minted ones
`formulas[].id` has four sources sharing one namespace: preserved from a
field/metric stash, preserved from the model-scope unsurfaced-formula stash,
minted from a display name, and minted for an unattributed formula. The opening
sweep that reserves preserved ids before any minting read only the first. The
unsurfaced ids were therefore never reserved -- `_allocate_formula_id` returns
early for a preserved id and never adds it, and the unsurfaced loop runs after
both minting loops -- so a minted id could take one.
Reproduced, not reasoned about. A document carrying a hidden helper
`formula_revenue` plus a field named "Revenue" (which mints exactly that slug):
emitted formula ids: ['formula_revenue', 'formula_revenue']
collision issue raised? False
Two `formulas[]` entries, one id, no issue, exit 0. Every `[formula_revenue]`
reference in the model is then ambiguous, and ThoughtSpot parses an ambiguous
bracket reference as search tokens rather than failing -- so it imports and
means something else. This is the exact failure `_allocate_formula_id` exists to
prevent; it was simply not being told about one of the four sources.
Reachable by hand-editing the Ossie document, which is the point of a portable
format. A pure round trip does not hit it: there every id is preserved from the
one source document, where they were already unique.
After: the minted id moves to `formula_revenue_2`, the PRESERVED id keeps
`formula_revenue` (it is the identity the surviving cross-references name), and
TS-MODEL-FORMULA-ID-COLLISION is raised.
Found by an audit of what this package's comments claim against what its code
does -- the stale comment ("all THREE sources") and the defect were the same
mistake written down twice.
Mutation-checked: reverting the reservation fails all three new tests. A second,
belt-and-braces `taken.add` in the preserved branch was written first and then
removed -- reverting it alone changed nothing, so it was redundant with the
sweep and would have shipped as an uncovered line.
941 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* test(thoughtspot): close six guards that passed on an empty collection
An audit for tests whose every assertion sits inside a loop, verified by
breaking the thing each one guards. Six were vacuous; the mutation each one
survived is recorded beside it.
The serious one: `test_the_surfacing_column_follows_the_renamed_id` walked
`columns[]` and skipped entries with no `formula_id`. Deleting the lockstep
assignment outright -- so every RENAMED formula's column lost the link that
this project's own TML invariants call compulsory -- left the FULL SUITE green,
936 passed, nothing else caught it either. It now iterates `formulas[]` and
requires each to produce its surfacing column, so an absent key fails.
`test_shipped_references` accumulated eight globs into one list and asserted
only that the total was non-empty -- which `tests/**/*.py` guarantees by
matching the test module itself, so the guard could not fire at all. Renaming
`docs/` passed; moving `src/`, `tools/`, `docs/`, `README.md` and
`pyproject.toml` together also passed, with the non-test shipped surface empty.
Each required pattern is now asserted to match.
`test_every_stash_key_constant_is_a_real_constants_attribute` was written
against exactly this ("a typo in the regex would otherwise silently check
nothing") and had it: breaking the regex made the scan return nothing and the
test passed, while three siblings failed instead.
`test_every_shadows_derivable_key_has_a_witness_constant...` enforced the
witness rule on nothing when no key was SHADOWS_DERIVABLE. Reclassifying them
all failed only the generated-docs check -- so following that failure's own
remedy (regenerate) reached a green suite with the rule gone.
`test_stays_inside_the_output_directory` is a path-traversal guard that proved
nothing when its input vanished: making `dump_document_set` skip
traversal-shaped names instead of sanitising them passed all six malicious
cases. It now asserts both documents are emitted first.
`test_every_referencing_join_carries_the_compulsory_with_field` had three `or
[]` defaults in a row, so emitting no joins at all passed. Now counted.
Two in `test_datatypes.py` walked `OSSIE_DATATYPES`; emptying it passed both.
Every one re-checked against the mutation that defeated it. No production
behaviour changes here -- the surfacing-column defect the first mutation
simulates does not exist in the code, it was simply untested.
941 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* docs(thoughtspot): explain the placeholder notation the reference documents use
`docs/expression-mapping.md` used four brace notations and explained none of
them: 138 `{0}`-style positional slots, five `{*}` variadic tails, six escaped
`{{`/`}}` pairs, and a dozen literal `{ [attr] }` ThoughtSpot sets. The last
three matter most because they LOOK alike and do unrelated jobs -- `{{ [T::date] }}`
renders as `{ [T::date] }`, which is not a placeholder at all -- and a reader had
no way to tell them apart. `docs/reverse-inventory.md` had the same gap for its
61 positional slots. The meanings existed only in source comments the reader of
a generated document never sees.
Both documents now open with a "Reading the notation" table, which also covers
the two non-brace conventions: the `per-... — see note` dispatch cells and the
`**example only**` exemplar marker.
The legend is DERIVED from the rendered body and applied centrally in
`generate_all()`, not declared per document. So it lists exactly the notations
that document uses -- `datatype-map.md` and `vendor-payload.md` use none and
correctly get no legend -- and a document that grows its first `{0}` gets the
row without anyone remembering.
The test asserts both directions, plus that the detectors matched something at
all. That vacuity guard earned itself immediately: the first version of this
test PASSED when the `{*}` row's key was renamed, because the detector half and
the row half drifted apart while each stayed internally consistent. The two are
now checked against each other where both are in hand, and that mutation fails.
942 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* docs(thoughtspot): correct thirteen comments that describe the code wrongly
An audit of every checkable claim in the package's comments and docstrings
against the code beside it. Thirteen were wrong. No behaviour changes; the code
was right in all thirteen and the prose was not.
The one most likely to mislead: `catalog.py` states the trigonometry conversion
directions BACKWARDS -- "SIN/COS/TAN convert degrees->radians on the way in
(`* 180 / pi`)" when `* 180 / pi` is radians->degrees, and the file's own
DEGREES/RADIANS rows two hundred lines later say exactly that. The comment even
warns the two are "easy to transpose by mistake".
Counts that had drifted:
- "the ten `sql_*_op` names" (x3 in reverse.py) -- eleven are registered; the
ten matches the `Variant` enum, which has no `sql_date_op`.
- "one of the two places this converter uses a witness copy" -- five.
- `_AGGREGATE_CALL_NAMES` "plus `group_aggregate`" -- plus nine `group_*`
names. This is the same "there is exactly one such construct" error that the
comment fourteen lines above it records as a fixed bug.
- `tml.py`'s "Four invariants live here so no caller has to carry them" --
three are enforced here; the block-scalar rule is only SUPPORTED here and is
applied by `ossie_to_thoughtspot._maybe_block_scalar`, so it is precisely the
one a caller can still get wrong.
Enumerations that name things they do not cover:
- `emit.py`'s PARTITION BY guard listed eight rows; four carry PARTITION BY in
a template. The OVER row is a dispatch string, CUME_DIST has none, and
RANK/PERCENT_RANK are DIRECT rows the guard never sees -- their fallbacks
live in prose.
- `_forward_call_names` claims "every ThoughtSpot call name the FORWARD
catalog renders"; its regex is anchored at the template start and excludes
digits, so it misses `rank_percentile`, `asin`/`acos`/`atan`, `log10` and
every nested call. 50 of them. Harmless for its one consumer, and the
docstring now says what it actually returns.
Claims contradicted by a later change:
- Two docstrings say `_DisplayNameAllocator` folds the way they do. It folds on
a plain casefold and never calls `identifiers.normalise` -- deliberately, and
its own class docstring says so. Stale since the commit that stopped this
converter inventing collisions ThoughtSpot has not got.
- `tml_to_ossie.convert` says the resolver needs "every ATTRIBUTE column's
identifier"; the index deliberately stores membership only, as the function
building it documents at length.
- `stash.py` names the CLI's handler as `(ConversionError, OSError)`; it is
three-wide.
- `keys.py` cited `validation/validate.py:159-165`, which is an unrelated
list-duplicate helper. Now named by function, not line, since that is what
drifted.
- `identifiers.Allocator`'s docstring credits its case-insensitivity to the
`.casefold()` in `allocate`. Every candidate is already lowercase by then
(`normalise` lowercases), so that call is a no-op on this path; `normalise`
is what delivers the guarantee.
942 tests pass; the generated reference documents are byte-identical, as
comment-only edits should leave them.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
* test(thoughtspot): pin the last three loops that an empty collection satisfied
The remainder of the fail-open sweep. These were lower priority than the six in
cc5db0e because a sibling test happened to fail on the same mutation -- but
"another test noticed" is not the same as this test working, and the sibling can
be deleted or narrowed later without anyone seeing the cover disappear.
`test_every_table_or_sql_view_document_is_present` compares two sets, and
`set() == set()` holds, so a fixture with no table documents satisfied it and
the two sibling tests that walk the same collection. Pinned once for all three.
`test_column_type_is_never_a_bare_root_key` reads columns through a helper that
ends `or []`, so a model emitting no columns passed without checking one.
Not changed, deliberately: the ~16 tests whose loop iterable is a literal list
written inline in the test body. Those can only go empty if someone edits the
assertion out by hand, which no guard can prevent, and the six catalog
`test_classifications` tests already have a sibling pinning an exact row count.
942 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <no...@anthropic.com>
---
.github/workflows/converter-thoughtspot-ci.yml | 71 +
converters/README.md | 1 +
converters/thoughtspot/.gitignore | 25 +
converters/thoughtspot/README.md | 311 +++
converters/thoughtspot/docs/datatype-map.md | 76 +
converters/thoughtspot/docs/expression-mapping.md | 219 ++
converters/thoughtspot/docs/reverse-inventory.md | 143 ++
converters/thoughtspot/docs/vendor-payload.md | 112 +
converters/thoughtspot/pyproject.toml | 71 +
.../thoughtspot/src/ossie_thoughtspot/__init__.py | 20 +
.../thoughtspot/src/ossie_thoughtspot/_yaml.py | 97 +
.../thoughtspot/src/ossie_thoughtspot/cli.py | 283 +++
.../thoughtspot/src/ossie_thoughtspot/constants.py | 628 +++++
.../thoughtspot/src/ossie_thoughtspot/datatypes.py | 108 +
.../thoughtspot/src/ossie_thoughtspot/errors.py | 22 +
.../src/ossie_thoughtspot/expressions/__init__.py | 72 +
.../src/ossie_thoughtspot/expressions/_types.py | 302 +++
.../src/ossie_thoughtspot/expressions/catalog.py | 1962 +++++++++++++++
.../src/ossie_thoughtspot/expressions/emit.py | 274 +++
.../src/ossie_thoughtspot/expressions/reverse.py | 1034 ++++++++
.../thoughtspot/src/ossie_thoughtspot/formula.py | 337 +++
.../src/ossie_thoughtspot/identifiers.py | 151 ++
.../thoughtspot/src/ossie_thoughtspot/issues.py | 96 +
.../thoughtspot/src/ossie_thoughtspot/keys.py | 161 ++
.../src/ossie_thoughtspot/ossie_to_thoughtspot.py | 2563 +++++++++++++++++++
.../thoughtspot/src/ossie_thoughtspot/stash.py | 184 ++
.../thoughtspot/src/ossie_thoughtspot/tml.py | 271 +++
.../src/ossie_thoughtspot/tml_to_ossie.py | 2567 ++++++++++++++++++++
converters/thoughtspot/tests/conftest.py | 21 +
.../tests/expressions/test_catalog_aggregate.py | 138 ++
.../expressions/test_catalog_covers_the_spec.py | 62 +
.../tests/expressions/test_catalog_datetime.py | 238 ++
.../expressions/test_catalog_math_conditional.py | 262 ++
.../tests/expressions/test_catalog_operators.py | 233 ++
.../tests/expressions/test_catalog_string.py | 191 ++
.../tests/expressions/test_catalog_window.py | 270 ++
.../thoughtspot/tests/expressions/test_emit.py | 379 +++
.../thoughtspot/tests/expressions/test_reverse.py | 690 ++++++
.../thoughtspot/tests/expressions/test_types.py | 85 +
.../tests/fixtures/minimal/customers.table.tml | 33 +
.../tests/fixtures/minimal/expected.ossie.yaml | 86 +
.../minimal/minimal_orders_model.model.tml | 46 +
.../tests/fixtures/minimal/orders.table.tml | 44 +
.../tests/fixtures/tpcds/customer.table.tml | 52 +
.../tests/fixtures/tpcds/date_dim.table.tml | 45 +
.../tests/fixtures/tpcds/expected.ossie.yaml | 604 +++++
.../tests/fixtures/tpcds/item.table.tml | 49 +
.../tests/fixtures/tpcds/store.table.tml | 59 +
.../fixtures/tpcds/store_returns_sv.sql_view.tml | 53 +
.../tests/fixtures/tpcds/store_sales.table.tml | 90 +
.../fixtures/tpcds/tpcds_retail_model.model.tml | 283 +++
converters/thoughtspot/tests/test_cli.py | 406 ++++
converters/thoughtspot/tests/test_constants.py | 41 +
converters/thoughtspot/tests/test_datatypes.py | 118 +
converters/thoughtspot/tests/test_fixtures.py | 326 +++
converters/thoughtspot/tests/test_formula.py | 368 +++
converters/thoughtspot/tests/test_identifiers.py | 146 ++
converters/thoughtspot/tests/test_issues.py | 72 +
converters/thoughtspot/tests/test_keys.py | 165 ++
.../thoughtspot/tests/test_ossie_to_thoughtspot.py | 1329 ++++++++++
.../tests/test_ossie_to_thoughtspot_model.py | 1560 ++++++++++++
.../tests/test_ossie_to_thoughtspot_tables.py | 746 ++++++
converters/thoughtspot/tests/test_packaging.py | 62 +
converters/thoughtspot/tests/test_readme.py | 93 +
.../tests/test_reference_docs_current.py | 174 ++
converters/thoughtspot/tests/test_roundtrip.py | 1088 +++++++++
.../thoughtspot/tests/test_roundtrip_properties.py | 571 +++++
.../thoughtspot/tests/test_shape_properties.py | 211 ++
.../thoughtspot/tests/test_shipped_references.py | 276 +++
converters/thoughtspot/tests/test_stash.py | 189 ++
.../tests/test_stash_key_classification.py | 173 ++
converters/thoughtspot/tests/test_tml.py | 432 ++++
converters/thoughtspot/tests/test_tml_to_ossie.py | 1583 ++++++++++++
.../thoughtspot/tests/test_tml_to_ossie_fields.py | 594 +++++
.../thoughtspot/tests/test_tml_to_ossie_metrics.py | 580 +++++
converters/thoughtspot/tests/test_yaml.py | 93 +
.../thoughtspot/tools/generate_reference_docs.py | 691 ++++++
converters/thoughtspot/uv.lock | 638 +++++
core-spec/spec.md | 1 +
79 files changed, 28900 insertions(+)
diff --git a/.github/workflows/converter-thoughtspot-ci.yml b/.github/workflows/converter-thoughtspot-ci.yml
new file mode 100644
index 00000000..5be52ae3
--- /dev/null
+++ b/.github/workflows/converter-thoughtspot-ci.yml
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Converters ThoughtSpot CI
+on:
+ push:
+ branches: [ "main" ]
+ paths:
+ - 'converters/thoughtspot/**'
+ # core-spec/** is the convention every sibling workflow adopted in #396.
+ # It is load-bearing here rather than merely conventional: catalog.py's
+ # spec_construct_names() parses core-spec/expression_language.md directly
+ # as its upstream-spec coverage oracle, so a spec edit can fail this
+ # converter's tests without touching a single converter file.
+ - 'core-spec/**'
+ - 'examples/**'
+ - '.github/workflows/converter-thoughtspot-ci.yml'
+ pull_request:
+ branches: [ "main" ]
+ paths:
+ - 'converters/thoughtspot/**'
+ - 'core-spec/**'
+ - 'examples/**'
+ - '.github/workflows/converter-thoughtspot-ci.yml'
+
+jobs:
+ build:
+ runs-on: ubuntu-latest
+ strategy:
+ matrix:
+ python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
+
+ steps:
+ - name: Checkout project
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+ - name: Set up Python ${{ matrix.python-version }}
+ uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+ with:
+ python-version: ${{ matrix.python-version }}
+
+ - name: Install uv
+ run: |
+ curl -LsSf https://astral.sh/uv/install.sh | sh
+ echo "${HOME}/.local/bin" >> "${GITHUB_PATH}"
+
+ - name: Sync dependencies
+ working-directory: converters/thoughtspot
+ run: |
+ uv sync
+
+ - name: Unit Tests
+ working-directory: converters/thoughtspot
+ run: |
+ uv run pytest
diff --git a/converters/README.md b/converters/README.md
index 07b586c0..83c84465 100644
--- a/converters/README.md
+++ b/converters/README.md
@@ -87,6 +87,7 @@ The Ossie specification currently defines extensions for the following vendors:
| `WISDOM` | WisdomAI domain |
| `NVIDIA_GSF` | NVIDIA Generative Semantic Fabric standalone YAML |
| `SIGMA` | Sigma Computing data model |
+| `THOUGHTSPOT` | ThoughtSpot TML (Model + Table/SQL View) |
Each vendor may define custom extensions (via the `custom_extensions` field in the Ossie spec) to carry vendor-specific metadata that does not have an equivalent in the core specification.
diff --git a/converters/thoughtspot/.gitignore b/converters/thoughtspot/.gitignore
new file mode 100644
index 00000000..d6ae8114
--- /dev/null
+++ b/converters/thoughtspot/.gitignore
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+__pycache__/
+*.py[cod]
+.pytest_cache/
+*.egg-info/
+dist/
+build/
+.venv/
+venv/
diff --git a/converters/thoughtspot/README.md b/converters/thoughtspot/README.md
new file mode 100644
index 00000000..f17d9e71
--- /dev/null
+++ b/converters/thoughtspot/README.md
@@ -0,0 +1,311 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Apache Ossie ThoughtSpot Converter
+
+Bidirectional, offline conversion between an [Apache Ossie](https://github.com/apache/ossie)
+semantic model and ThoughtSpot TML. No ThoughtSpot connection required.
+
+- **ThoughtSpot TML → Ossie** (`to-ossie`): reads a Model TML document plus the Table and
+ SQL View documents it references, and emits one Ossie semantic model.
+- **Ossie → ThoughtSpot TML** (`to-tml`): reads one Ossie semantic model and emits the
+ corresponding set of TML documents.
+
+A single Ossie semantic model corresponds to **1 + N TML documents**, not one file: one
+`model:` document plus one `table:` or `sql_view:` document per dataset. The converter
+reads and writes the set. File-to-file only — nothing here calls a ThoughtSpot API.
+
+The `THOUGHTSPOT` dialect is registered upstream — apache/ossie#351 merged 2026-09-01.
+Both conversion directions are implemented and tested: example-based unit tests, an
+exact-document comparison against a shared TPC-DS fixture set (the same retail schema
+every sibling converter round-trips), a round-trip suite asserting preservation and
+translation separately, and a Hypothesis property-based suite over adversarial
+identifiers.
+
+## Installation
+
+```bash
+pip install apache-ossie-thoughtspot # once published to PyPI
+# or, from a checkout of this directory:
+pip install -e .
+```
+
+The only runtime dependency is `PyYAML`. Python 3.10+.
+
+## Usage
+
+### Command line
+
+```bash
+ossie-thoughtspot to-ossie <tml-file>... -o <out.yaml> [--issues <issues.json>] [--force]
+ossie-thoughtspot to-tml <ossie.yaml> -o <out-dir> [--issues <issues.json>] [--force]
+```
+
+`to-ossie` takes the Model TML document plus every Table/SQL View document it references
+and writes one Ossie YAML file. `to-tml` takes one Ossie YAML document and writes the
+corresponding TML document set — one file per document, tables before the model — into
+an output directory it creates if needed.
+
+`-o`/`--output` is required in both directions: `to-tml` writes a set of files that has
+no single-file stdout representation, so unlike some sibling converters there is no
+"default: stdout" fallback. Neither subcommand overwrites an existing output file unless
+`--force` is given.
+
+Every declared loss or degradation the conversion records is written as a JSON array of
+issues — to `--issues` when given, to stderr otherwise — never mixed into the document
+output. The process exits `1` when that issue log contains an ERROR-severity issue, `0`
+otherwise: a conversion that only warned or informed about a declared loss is still a
+successful conversion.
+
+### Python API
+
+```python
+from ossie_thoughtspot import tml, tml_to_ossie, ossie_to_thoughtspot, _yaml
+
+# ThoughtSpot TML -> Ossie
+texts = [(path, open(path).read()) for path in ("model.model.tml", "orders.table.tml")]
+result = tml_to_ossie.convert(tml.load_document_set(texts))
+ossie_yaml = _yaml.dump(result.model) # result.issues: IssueLog
+
+# Ossie -> ThoughtSpot TML
+ossie_document = _yaml.load(open("model.yaml").read())
+result = ossie_to_thoughtspot.convert(ossie_document)
+for filename, text in tml.dump_document_set(result.documents):
+ ... # result.issues: IssueLog
+```
+
+`result.issues` is an `IssueLog`: `has_errors()`, `count_by_severity()`, `as_dicts()`.
+Every declared loss raises an issue here — see [Coverage matrix](#coverage-matrix) and
+[Expression translation](#expression-translation-what-is-not-translated-and-why) below
+for what gets declared and why.
+
+## Mapping
+
+| Ossie | ThoughtSpot TML | Notes |
+|---|---|---|
+| the semantic model (the document root) | one Model document + the Table/SQL View documents it references | One semantic model per document, its fields at the root beside `version`; a document still carrying the removed `semantic_model` wrapper is a hard failure naming it |
+| `dataset` | `table:`/`sql_view:` document, surfaced via the Model's `model_tables[]` entry | One dataset per participating `model_tables[]` entry, not per physical table — a self-join or a table used twice gets two datasets sharing one `source` |
+| `dataset.source` | `db`.`schema`.`db_table`, or `sql_query` for a SQL View | A dotted part is stashed individually (`source_parts`) when the joined form would be ambiguous |
+| `dataset.fields` | Table `columns[]` (physical) or Model `formulas[]` + surfacing `columns[]` entry (computed) | A computed field is attributed to the one dataset every column reference in its expression resolves to; ambiguous or cross-dataset references raise an issue instead of guessing |
+| `relationship` | `model_tables[].joins[]` (inline) or Table `joins_with[]` (referencing) | `from_columns`/`to_columns` are the join's equality pairs; a join with a non-equality residual (range/ASOF) narrows the same pairs, with the verbatim condition stashed — see [the payload section](#the-custom_extensionsthoughtspot-payload) below |
+| `dataset.primary_key` / `unique_keys` | *(not native to TML)* | TML declares no keys; Ossie's are derived from to-one relationships targeting the dataset |
+| `metric` | Model `formulas[]` + surfacing `columns[]` entry with `column_type: MEASURE` | Three TML shapes compose into one metric: a bare aggregate formula, a scalar formula plus the surfacing column's `aggregation`, or a physical column plus `aggregation` |
+| `field`/`metric` `expression.dialects` | `formulas[].expr` or a physical `db_column_name` | See [Expression translation](#expression-translation-what-is-not-translated-and-why) |
+| `custom_extensions[THOUGHTSPOT]` | TML fields with no Ossie equivalent | See [The `custom_extensions[THOUGHTSPOT]` payload](#the-custom_extensionsthoughtspot-payload) below |
+
+## The `custom_extensions[THOUGHTSPOT]` payload
+
+TML carries properties Ossie's core specification has no field for — a Connection name,
+search-indexing settings, a display column's warehouse name when it differs from its
+label, a join's exact type, and more. `TML → Ossie` stashes each one under a single
+`custom_extensions` entry (`vendor_name: THOUGHTSPOT`) attached to the Ossie object it
+came from; `Ossie → TML` reads the same entry back to reconstruct the original TML
+property, so `TML → Ossie → TML` is lossless for everything TML itself can express.
+
+```yaml
+custom_extensions:
+- vendor_name: THOUGHTSPOT
+ data: '{"_v": 1, "connection_name": "My Snowflake", "tml_object": "table"}'
+```
+
+`data` is always a single JSON-encoded string (never a nested object — the core
+specification requires this), carrying:
+
+- `_v`, a shape version bumped only when the payload's *shape* changes, never for a
+ value change — an unrecognised version is a hard failure rather than a silent
+ misread of a shape this converter has never seen.
+- Never a `guid`, `obj_id`, or `fqn` at any depth — object identity is instance-local
+ and is refused outright rather than carried, at write time.
+- Foreign-vendor `custom_extensions` entries on the same object pass through untouched
+ in both directions.
+- An object with nothing to stash gets no `custom_extensions[THOUGHTSPOT]` entry at
+ all, so a converted document stays as small as its content requires.
+
+`Ossie → TML` restores a stashed value only when it is still current: several keys
+(a physical column's warehouse name, a relationship's verbatim join condition, a
+dataset's TML object kind) are recorded alongside a witness copy of the live value they
+were stashed next to, and the stash is used only when that witness still matches the
+document's current value — an edit to the Ossie document since the stash was written
+(retargeting a relationship, renaming a field) makes the stash stale for that key, and
+the value is re-derived instead of silently reapplied to the wrong thing.
+
+The full key vocabulary is documented in `src/ossie_thoughtspot/constants.py`, grouped
+by which Ossie object each key's entry attaches to (model, dataset, relationship,
+field/metric).
+
+## Expression translation: what is not translated, and why
+
+**A ThoughtSpot formula is captured verbatim under a `THOUGHTSPOT` dialect entry. A
+portable `ANSI_SQL` sibling is added only when the whole expression is a single bare
+column reference — the one shape where portability is certain. Every other shape — a
+function call, an operator expression, a runtime parameter, a formula cross-reference —
+is recorded THOUGHTSPOT-only, with an issue explaining why no portable sibling was
+produced.** No SQL dialect is ever re-rendered into another dialect in this converter,
+in either direction.
+
+This is a deliberate design position, not an oversight, and it is worth stating plainly
+rather than leaving it to be inferred from reading `tml_to_ossie.py`:
+
+1. **The specification's own default is pass-through.** `core-spec/spec.md` defines
+ `dialects[]` as a list of `{dialect, expression}` pairs precisely so a value a
+ converter cannot translate can still travel, tagged with the dialect it is valid in.
+ Emitting an untranslated ThoughtSpot expression under `THOUGHTSPOT` and stopping
+ there is using the mechanism the specification provides for exactly this case, not
+ working around a gap in it.
+2. **No converter in this repository re-renders an expression from one SQL dialect into
+ another.** Every sibling converter that meets a dialect it does not natively speak
+ either passes the expression through under its own vendor dialect or falls back to
+ `ANSI_SQL` when the source already provides one — none parses a foreign dialect's
+ grammar and re-emits it in a different one. A hand-rolled reimplementation of that
+ translation, done once per converter, is exactly the kind of duplicated, easy-to-get-
+ subtly-wrong logic a shared SQL parser would exist to prevent — and no such shared
+ parser exists in this project today. Building one is out of scope for a single
+ converter to take on unilaterally.
+3. **The reference converter (`converters/databricks`) tags its own vendor dialect and
+ reads that first**, falling back to `ANSI_SQL` only when the source document already
+ provides it — it does not translate a foreign dialect into its own either. This
+ converter follows the same shape: prefer `THOUGHTSPOT`, add `ANSI_SQL` only when it
+ can be produced with certainty, never invent a translation.
+
+**What "certain" means in practice.** A bare column reference (`[TABLE::Column]`) is the
+one shape this converter resolves without ambiguity: the reference names a physical or
+computed field this document already knows how to place, so the `ANSI_SQL` sibling is
+just that field's own dataset-qualified name — no expression semantics are being
+translated at all, only a reference being resolved. Everything past that — even a
+composition ThoughtSpot's own documentation says is exactly equivalent to a portable
+form — is left THOUGHTSPOT-only. `src/ossie_thoughtspot/expressions/reverse.py` records,
+for testing and future use, which of ThoughtSpot's native functions compose into a
+portable Ossie expression and which do not (`ReverseDisposition`: compose fully,
+compose partially, resolve only to a dialect entry, or have no Ossie form at all) — but
+this inventory is not yet called from the shipped `TML → Ossie` conversion path itself.
+The current release is more conservative than what that inventory shows is possible: it
+never guesses, so it never translates something it has not resolved to a certainty.
+
+**The complementary direction.** `Ossie → TML` faces the reverse problem: rendering an
+Ossie specification construct as a ThoughtSpot formula. There, `expressions/catalog.py`
+maps all 146 constructs the Ossie expression language defines to a ThoughtSpot rendering
+— 108 with a native equivalent, 37 as a `sql_*_op` pass-through (opaque,
+warehouse-dialect-specific SQL ThoughtSpot cannot introspect, logged at WARNING every
+time), and 1 (`EXISTS_IN()`) with no representation ThoughtSpot has a slot for at all
+(logged at ERROR, never silently dropped). This direction *can* translate constructs to
+their ThoughtSpot equivalents because the Ossie expression language — unlike an
+arbitrary ThoughtSpot formula — is the one grammar this converter fully parses; nothing
+here reads or re-renders raw ThoughtSpot formula syntax, or any other vendor's SQL.
+
+Read together with the [coverage matrix](#coverage-matrix) below, this is the
+converter's whole answer to "what does not survive a round trip and why": expressions
+pass through by declared design; every other construct's loss is declared per row.
+
+## Coverage matrix
+
+Every construct this converter does not carry, with its consequence. Each row raises a
+structured `ConverterIssue` at conversion time — nothing is dropped silently.
+
+| # | Construct | Limitation | Consequence |
+|---|---|---|---|
+| L1 | Object identity (`guid`, `obj_id`, `fqn`) | Not carried — instance-local by construction | A round-tripped document imports as a new object |
+| L2 | Row-level security (`rls_rules`) | Not carried — rule expressions name instance-local groups. **ERROR severity**: a single issue is raised, its message naming every affected table | RLS is unrepresentable in Ossie core and is security-bearing; rules must be re-applied in the target for each table named in the error |
+| L3 | Presentation artifacts (Answers, Liveboards, charts) | Out of scope — Ossie models semantics, not visualisations | No loss to the semantic model |
+| L4 | Spotter coaching objects | Separate object types; `ai_context.examples` is not interchangeable | Coaching must be re-created in the target |
+| L5 | Aggregate-model associations (`aggregated_models`) | Entries are GUIDs of other Models — instance-local | Query routing is silently disabled; the issue is the only signal |
+| L6 | Worksheets, Views, Sets, Alerts, Model Aliases | Predecessors or layers, not models | Convert the Model the alias points at instead |
+
+## Known limitations
+
+Separate from the coverage matrix above — this covers identifier derivation
+correctness, not TML constructs.
+
+`identifiers.py`'s `normalise()` folds diacritics via Unicode NFKD decomposition before
+lowercasing and substituting, so accented Latin normalises correctly: `"Café"` ->
+`"cafe"`, `"Ürün"` -> `"urun"`, `"Zürich"` -> `"zurich"`. A character with **no ASCII
+decomposition** (Cyrillic, CJK, and similarly non-Latin scripts) is still dropped, not
+transliterated, and a name with no ASCII alphanumerics surviving still raises
+`ValueError`. There is also an open question NFKD does not settle: some accented Latin
+folds to a *conventional* ASCII expansion rather than the bare decomposed letter —
+German `"Müller"` decomposes to `"Muller"` here, not the conventional `"Mueller"` — and
+choosing between them is a product decision left to a later change.
+
+## Rules
+
+Earlier revisions of this converter's comments and docstrings cited short, letter-plus-
+number rule identifiers drawn from an internal construct/expression mapping reference
+that is not part of this repository and is not publicly readable — a citation of that
+shape in the shipped source was therefore unresolvable from inside this repository
+alone. That has been resolved: every such citation has been rewritten to state the
+substance it stood for directly, in place, so nothing shipped here depends on material
+outside this repository. `tests/test_shipped_references.py` enforces this going
+forward — it fails the suite if a citation of that shape reappears in any shipped
+file.
+
+**Before declaring any expression untranslatable, consult the function mapping.** Many
+window and LOD constructs have exact native equivalents; declaring one untranslatable
+without checking is an error.
+
+## Generated reference documentation
+
+`docs/` holds four Markdown reference documents, generated from this converter's own
+code rather than hand-authored — the code is the single source of truth for the
+mapping each one describes, so a document maintained separately would only be able to
+drift from it:
+
+- [`docs/expression-mapping.md`](docs/expression-mapping.md) — every Ossie
+ specification construct, its classification and its ThoughtSpot rendering, generated
+ from `expressions/catalog.py`'s `CATALOG`.
+- [`docs/reverse-inventory.md`](docs/reverse-inventory.md) — every ThoughtSpot-only
+ function with no specification counterpart, and how it composes (or does not) back
+ into a portable Ossie expression, generated from `expressions/reverse.py`'s `REVERSE`.
+- [`docs/datatype-map.md`](docs/datatype-map.md) — the bidirectional datatype map and
+ which types are declared lossy, generated from `datatypes.py`.
+- [`docs/vendor-payload.md`](docs/vendor-payload.md) — every
+ `custom_extensions[THOUGHTSPOT]` key, its scope and how it is treated on the return
+ trip, generated from `constants.py`'s `STASH_KEY_CLASSIFICATION`.
+
+`tools/generate_reference_docs.py` produces all four; it is dev/tooling only — not a
+runtime dependency, not part of the wheel, not a `[project.scripts]` entry point.
+Regenerate with:
+
+```bash
+uv run --python 3.13 python tools/generate_reference_docs.py
+```
+
+`tests/test_reference_docs_current.py` regenerates on every test run and compares the
+result against the committed files byte-for-byte, so a `docs/*.md` file that has
+drifted from the code it describes fails the suite rather than going unnoticed.
+
+## Development
+
+```bash
+uv run --python 3.13 pytest tests/ -v
+```
+
+`uv run` syncs the `dev` dependency group (declared via PEP 735 `[dependency-groups]`,
+not an extra — `pytest`, plus `jsonschema` and `hypothesis` for the schema-validation and
+property-based suites) and runs the tests in one step — see
+`.github/workflows/converter-thoughtspot-ci.yml` for the CI invocation this mirrors,
+run across every Python version this package declares support for.
+
+## Future effort
+
+The Apache Ossie specification is still evolving. As it adds or changes fields, this
+converter will be updated to track them — extending the mapping and coverage in both
+directions to keep the conversion current and to support as much as the format allows
+over time. `expressions/reverse.py`'s classified inventory of ThoughtSpot-only functions
+is a candidate foundation for a future, more ambitious `TML → Ossie` composition
+strategy, once that expansion is deliberately taken on rather than folded into this
+release.
diff --git a/converters/thoughtspot/docs/datatype-map.md b/converters/thoughtspot/docs/datatype-map.md
new file mode 100644
index 00000000..4cef32a3
--- /dev/null
+++ b/converters/thoughtspot/docs/datatype-map.md
@@ -0,0 +1,76 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+<!-- GENERATED FILE -- do not edit by hand.
+ Produced by `tools/generate_reference_docs.py` from:
+ - `src/ossie_thoughtspot/datatypes.py`
+ Regenerate with:
+ uv run --python 3.13 python tools/generate_reference_docs.py
+ tests/test_reference_docs_current.py fails the suite if this file
+ drifts from what the generator currently produces. -->
+
+# Ossie <-> ThoughtSpot Datatype Map
+
+The bidirectional Ossie <-> ThoughtSpot TML datatype map. The map is **not injective** — several Ossie types collapse onto one TML spelling and cannot be told apart on the way back; see "Not injective" below.
+
+## The closed Ossie datatype enum
+
+`Boolean`, `Date`, `DateTime`, `DateTimeTz`, `Decimal`, `Float`, `Integer`, `Opaque`, `String`, `Time`
+
+## Ossie -> TML
+
+| Ossie datatype | TML `data_type` (default) | Notes |
+|---|---|---|
+| `Boolean` | `BOOLEAN` | connection-dependent spelling — `BOOLEAN` by default, `BOOL` when the connection's own TML spells it that way |
+| `Date` | `DATE` | exact, single spelling |
+| `DateTime` | `DATE_TIME` | exact, single spelling |
+| `DateTimeTz` | `DATE_TIME` | ThoughtSpot has no offset-aware column type; the value becomes DATE_TIME and returns as DateTime. |
+| `Decimal` | `DOUBLE` | exact, single spelling |
+| `Float` | `DOUBLE` | connection-dependent spelling — `DOUBLE` by default, `FLOAT` when the connection's own TML spells it that way; ThoughtSpot has one approximate numeric type, so Float and Decimal both become DOUBLE and return as Decimal. |
+| `Integer` | `INT64` | exact, single spelling |
+| `Opaque` | `VARCHAR` | Opaque is Ossie's marker for a type outside the portable vocabulary; it becomes VARCHAR and returns as String. |
+| `String` | `VARCHAR` | exact, single spelling |
+| `Time` | `VARCHAR` | ThoughtSpot has no time-of-day column type; the value becomes VARCHAR. |
+
+A column with no declared `datatype` at all infers `INT64` rather than raising — `datatype` is optional in Ossie, but TML rejects a column with no `db_column_properties` block at all.
+
+## TML -> Ossie
+
+| TML `data_type` | Ossie datatype |
+|---|---|
+| `BOOL` | `Boolean` |
+| `BOOLEAN` | `Boolean` |
+| `DATE` | `Date` |
+| `DATE_TIME` | `DateTime` |
+| `DOUBLE` | `Decimal` |
+| `FLOAT` | `Float` |
+| `INT64` | `Integer` |
+| `VARCHAR` | `String` |
+
+A TML `data_type` outside this map returns no Ossie datatype at all — `datatype` is optional in Ossie, so omitting it is preferred over inventing one.
+
+## Not injective — declared losses
+
+| Ossie datatype | Why the round trip is lossy |
+|---|---|
+| `DateTimeTz` | ThoughtSpot has no offset-aware column type; the value becomes DATE_TIME and returns as DateTime. |
+| `Float` | ThoughtSpot has one approximate numeric type, so Float and Decimal both become DOUBLE and return as Decimal. |
+| `Opaque` | Opaque is Ossie's marker for a type outside the portable vocabulary; it becomes VARCHAR and returns as String. |
+| `Time` | ThoughtSpot has no time-of-day column type; the value becomes VARCHAR. |
+
diff --git a/converters/thoughtspot/docs/expression-mapping.md b/converters/thoughtspot/docs/expression-mapping.md
new file mode 100644
index 00000000..a1d979d6
--- /dev/null
+++ b/converters/thoughtspot/docs/expression-mapping.md
@@ -0,0 +1,219 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+<!-- GENERATED FILE -- do not edit by hand.
+ Produced by `tools/generate_reference_docs.py` from:
+ - `src/ossie_thoughtspot/expressions/catalog.py`
+ - `src/ossie_thoughtspot/expressions/_types.py`
+ Regenerate with:
+ uv run --python 3.13 python tools/generate_reference_docs.py
+ tests/test_reference_docs_current.py fails the suite if this file
+ drifts from what the generator currently produces. -->
+
+# Ossie -> ThoughtSpot Expression Mapping
+
+Every construct the Ossie expression language specification defines, mapped to its ThoughtSpot rendering (`Ossie -> ThoughtSpot`, the direction `expressions/catalog.py` drives). Rows follow the source's own definition order, which groups related constructs together (aggregates, then type conversion, date/time, string, math/conditional, operators, window functions) — that grouping exists only as source comments, not as data the code carries, so it is not reproduced as separate sections here.
+
+## Reading the notation
+
+| Notation | Means |
+|---|---|
+| `{0}`, `{1}`, `{2}` | A positional argument slot, filled left to right from the construct's own arguments. |
+| `{*}` | The variadic tail: every remaining argument, joined with ` , `, inside the one call. Only on rows whose construct takes any number of arguments. |
+| `{{` and `}}` | An ESCAPED literal brace — it renders as a single `{` or `}`. Templates are filled with Python's `str.format`, which requires a literal brace to be doubled. |
+| `{ ... }` (brace, space) | NOT a placeholder. ThoughtSpot's own literal set syntax, as in `{ [attr] }`; it reaches the emitted formula unchanged. |
+| `per-... — see note` | The row has no single rendering: what it emits depends on a value not known when this document is generated. The row's Note describes the real dispatch. |
+| **example only** | The template bakes one caller-supplied value in as an illustrative constant. Rebuild the template per occurrence rather than reading the constant as the mapping. |
+
+## Coverage
+
+| Classification | Count | Share |
+|---|---|---|
+| direct | 108 | 74% |
+| passthrough | 37 | 25% |
+| unmappable | 1 | 1% |
+| **Total** | **146** | **100%** |
+
+## Constructs with no discrete specification table row
+
+9 `CATALOG` rows are real, intended constructs that `spec_construct_names()` cannot key on directly, because the upstream specification describes them in prose or a code fence rather than a table row with a `Syntax` column. Each is keyed via `CONVENTION_DIVERGENCES` instead, with the reason recorded per construct.
+
+| Construct | Why it has no discrete spec table row |
+|---|---|
+| `-x / +x (unary)` | unary +/- is named only in the 'Operator Precedence' list (core-spec/expression_language.md:142), never a table row |
+| `CASE expr WHEN v1 THEN r1 ... END (simple)` | simple CASE is described only in the CASE Expression code fence (core-spec/expression_language.md:508-513) alongside searched CASE; the top-level summary table's single bare 'CASE WHEN' token covers the searched form and does not extend to this one |
+| `Parentheses — expression grouping` | its Supported SQL Constructs row (core-spec/expression_language.md:120) carries no backtick token in either cell, the only marker the top-table extraction keys on |
+| `DISTINCT aggregate modifier` | the DISTINCT modifier is described only in the Conditional Aggregations prose/code block (core-spec/expression_language.md:219-230), never a table row |
+| `Column / metric reference — field, dataset.field` | its Supported SQL Constructs row (core-spec/expression_language.md:108) carries no backtick token in either cell, same reason as Parentheses |
+| `EXISTS_IN()` | named only in the Reason column of the excluded 'Not Supported in Expressions' table (core-spec/expression_language.md:131), never in a table of its own |
+| `OVER (PARTITION BY ... ORDER BY ...) clause` | the generic OVER syntax template (core-spec/expression_language.md:548-560) is a fenced code block, not a table |
+| `Frame clause — ROWS BETWEEN ... / RANGE BETWEEN ...` | frame options are a bullet list under the OVER syntax section (core-spec/expression_language.md:556-560), not a table |
+| `Window aggregation — AGG(expr) OVER (...)` | the Window Aggregations section (core-spec/expression_language.md:583-599) is prose and code examples, not a table |
+
+## Every construct
+
+| Ossie construct | Classification | ThoughtSpot rendering | Notes |
+|---|---|---|---|
+| `SUM(expr)` | direct | `sum ( {0} )` | — |
+| `COUNT(expr)` | direct | `count ( {0} )` | Counts non-null values on both sides. |
+| `COUNT(*)` | direct | `count ( {0} )` | ThoughtSpot has no count(*); the row count is count() over a column known to be non-null. The converter uses the dataset's primary_key when the model declares one, and raises an issue rather than guessing a column when it does not. |
+| `COUNT(DISTINCT expr)` | direct | `unique count ( {0} )` | A space, not an underscore. count_distinct(...) is rejected by the formula parser. |
+| `AVG(expr)` | direct | `average ( {0} )` | — |
+| `MIN(expr)` | direct | `min ( {0} )` | ThoughtSpot min is aggregate-only — it never compares two columns row-wise. Scalar two-argument minima are LEAST, a separate row. |
+| `MAX(expr)` | direct | `max ( {0} )` | Aggregate-only, as MIN. |
+| `STDDEV(expr)` | direct | `stddev ( {0} )` | Sample standard deviation on both sides. |
+| `STDDEV_POP(expr)` | passthrough | `STDDEV_POP({0})` — pass-through via `sql_number_aggregate_op` | ThoughtSpot stddev is sample-only; there is no population form, and substituting it would change the divisor from n-1 to n. |
+| `STDDEV_SAMP(expr)` | direct | `stddev ( {0} )` | Specification alias for STDDEV (:171). |
+| `VARIANCE(expr)` | direct | `variance ( {0} )` | Sample variance on both sides. |
+| `VAR_POP(expr)` | passthrough | `VAR_POP({0})` — pass-through via `sql_number_aggregate_op` | Same divisor reason as STDDEV_POP. |
+| `VAR_SAMP(expr)` | direct | `variance ( {0} )` | Specification alias for VARIANCE (:174). |
+| `MEDIAN(expr)` | direct | `median ( {0} )` | — |
+| `PERCENTILE_CONT(p) WITHIN GROUP (ORDER BY expr)` | passthrough | `PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY {0})` — pass-through via `sql_number_aggregate_op` — **example only**: `p` shown as a literal, rebuild the template per occurrence | No native percentile function. p is a literal in the specification's syntax, so it is baked into the template rather than passed as a placeholder. p = 0.5 is the one case with a native equivalent — median ( [x] ) — and the converter should prefer it. |
+| `PERCENTILE_DISC(p) WITHIN GROUP (ORDER BY expr)` | passthrough | `PERCENTILE_DISC(0.75) WITHIN GROUP (ORDER BY {0})` — pass-through via `sql_number_aggregate_op` — **example only**: `p` shown as a literal, rebuild the template per occurrence | As PERCENTILE_CONT; the discrete/interpolated distinction is preserved only because the template is emitted verbatim. |
+| `APPROX_COUNT_DISTINCT(expr)` | passthrough | `APPROX_COUNT_DISTINCT({0})` — pass-through via `sql_int_aggregate_op` | ThoughtSpot's unique count ( [x] ) is the exact-semantics alternative: same answer to within the sketch's ~2% error, at exact-count cost. The converter emits the pass-through by default — the specification chose approximate deliberately — and offers the exact form as a documented downgrade. |
+| `APPROX_PERCENTILE(expr, p)` | passthrough | `APPROX_PERCENTILE({0}, 0.5)` — pass-through via `sql_number_aggregate_op` — **example only**: `p` shown as a literal, rebuild the template per occurrence | p baked into the template as for the exact percentiles. |
+| `CAST` | direct | `per-type — see the target-type table below` | 5 of the 8 specified target types are direct; the other three — BOOLEAN, TIMESTAMP and TIME — fall back to a pass-through. |
+| `TRY_CAST` | direct | `the same functions as CAST` | ThoughtSpot's to_integer / to_double / to_string already return NULL on failure, which is exactly TRY_CAST semantics — so the two rows share a mapping and it is CAST, not TRY_CAST, that is the imprecise one. A strict CAST that must error rather than null is not expressible; the converter records that in the issue log when the source distinguishes them. |
+| `CURRENT_DATE or CURRENT_DATE()` | direct | `today ( )` | Both specification spellings map to the same function. |
+| `CURRENT_TIMESTAMP or CURRENT_TIMESTAMP()` | direct | `now ( )` | — |
+| `CURRENT_TIME or CURRENT_TIME()` | direct | `time ( now ( ) )` | ThoughtSpot has no current-time function, but time ( ) extracts the time part of a datetime, so the composition is exact. |
+| `YEAR(date_expr)` | direct | `year ( {0} )` | — |
+| `QUARTER(date_expr)` | direct | `quarter_number ( {0} )` | The function is quarter_number, not quarter. |
+| `MONTH(date_expr)` | direct | `month_number ( {0} )` | Not month ( ) — ThoughtSpot's month returns the month NAME ('January'); month_number returns 1-12, which is what the specification means. Mapping to month would silently change the column's type from integer to string. |
+| `DAY(date_expr)` | direct | `day ( {0} )` | Day of month, 1-31 on both sides. |
+| `DAYOFYEAR(date_expr)` | direct | `day_number_of_year ( {0} )` | The function is day_number_of_year, not day_of_year. |
+| `HOUR(timestamp_expr)` | direct | `hour_of_day ( {0} )` | The function is hour_of_day, not hour. |
+| `MINUTE(timestamp_expr)` | passthrough | `MINUTE({0})` — pass-through via `sql_int_op` | No native minute-of-hour extractor; add_minutes and diff_minutes exist but neither extracts. |
+| `SECOND(timestamp_expr)` | passthrough | `SECOND({0})` — pass-through via `sql_int_op` | As MINUTE. |
+| `EXTRACT` | direct | `per-part — see the date-part table below` | Rewritten to the part's own ThoughtSpot function; there is no generic extractor. 8 of the 11 specified parts are direct (YEAR->year, QUARTER->quarter_number, MONTH->month_number, WEEK->week_number_of_year, DAY->day, DAYOFWEEK->day_number_of_week, DAYOFYEAR->day_number_of_year, HOUR->hour_of_day); MINUTE, SECOND and MILLISECOND fall back to sql_int_op. |
+| `DATE_PART` | direct | `per-part — see the date-part table below` | Identical treatment to EXTRACT; the two spellings collapse onto one rewrite (:276-279). |
+| `DATE_TRUNC(part, date_expr)` | direct | `per-precision — see the truncation table below` | ThoughtSpot has no date_trunc. The start_of_* family covers 7 of the 8 specified precisions ('year'->start_of_year, 'quarter'->start_of_quarter, 'month'->start_of_month, 'week'->start_of_week, 'day'->date ( ), 'hour'->start_of_hour, 'minute'->start_of_min — the function is start_of_min, not start_of_minute); 'second' falls back to sql_date_time_op. The specification says week truncation is Monda [...]
+| `DATEADD(part, amount, date_expr)` | direct | `per-part add_* — see the arithmetic table below` | Argument order differs: ThoughtSpot is add_days ( [d] , n ), the specification is DATEADD(day, n, d). Every specified part is reachable: day->add_days, week->add_weeks, month->add_months, year->add_years, minute->add_minutes, second->add_seconds, plus two by arithmetic on a coarser unit since there is no native add_quarters or add_hours: quarter->add_months ( [d] , 3 * n ), hour->add_minut [...]
+| `DATEDIFF(part, start_date, end_date)` | direct | `per-part diff_* — see the arithmetic table below` | Argument order is reversed: ThoughtSpot is diff_days ( [end] , [start] ) — end first. Getting this wrong silently negates every duration in the model. day->diff_days, week->diff_weeks, month->diff_months, quarter->diff_quarters, year->diff_years, hour->diff_hours, minute->diff_minutes, second->diff_time (returns seconds). |
+| `DATE '2024-01-15'` | direct | `to_date ( '{0}' , 'yyyy-MM-dd' )` | A bare '2024-01-15' in a ThoughtSpot formula is parsed as arithmetic (2024 - 1 - 15), so the typed literal must always be wrapped. to_date takes exactly two arguments, so the converter supplies the ISO format model; {0} is the literal date string. |
+| `TIMESTAMP_NTZ '2024-01-15 10:30:00'` | passthrough | `CAST('2024-01-15 10:30:00' AS TIMESTAMP)` — pass-through via `sql_date_time_op` — **example only**: `the timestamp value` shown as a literal, rebuild the template per occurrence | to_date returns a DATE and drops the time part, so there is no native way to construct a wall-clock timestamp. A zero-placeholder template is a documented form of the pass-through — the document's own worked example is recorded verbatim here; a real occur [...]
+| `TIME '10:30:00'` | passthrough | `CAST('10:30:00' AS TIME)` — pass-through via `sql_date_time_op` — **example only**: `the time value` shown as a literal, rebuild the template per occurrence | ThoughtSpot has no TIME column type — time ( ) extracts a time FROM a datetime, it does not construct one — so the pass-through returns DATETIME and the date part is whatever the warehouse defaults to. Flagged with an issue for that reason, not only for the dialect. |
+| `TO_DATE(string)` | direct | `to_date ( {0} , 'yyyy-MM-dd' )` | The single-argument ISO form. ThoughtSpot's to_date is strictly two-argument, so the converter supplies 'yyyy-MM-dd'. |
+| `TO_TIMESTAMP(string)` | passthrough | `TO_TIMESTAMP({0})` — pass-through via `sql_date_time_op` | to_date is date-only; parsing to a timestamp would drop the time silently. |
+| `TO_DATE(string, format)` | direct | `to_date ( {0} , <translated format> )` | EXPERIMENTAL. Format tokens are translated, not passed through — see the format-token table. ThoughtSpot accepts Java/LDML tokens (yyyy-MM-dd) and strptime %-codes, which between them cover the specification's entire portable core. |
+| `TO_TIMESTAMP(string, format)` | passthrough | `TO_TIMESTAMP({0}, 'YYYY-MM-DD HH24:MI:SS')` — pass-through via `sql_date_time_op` — **example only**: `format` shown as a literal, rebuild the template per occurrence | EXPERIMENTAL. Date-only to_date again. The format model inside the template is the warehouse's, not Ossie's, so the token translation table does not apply — this is the sharpest case of the pass-through caveat. |
+| `TO_CHAR(date_expr, format)` | passthrough | `TO_CHAR({0}, 'YYYY-MM')` — pass-through via `sql_string_op` — **example only**: `format` shown as a literal, rebuild the template per occurrence | EXPERIMENTAL. ThoughtSpot has no general date formatter. Single-token formats do have native equivalents and the converter prefers them: 'YYYY' -> year_name ( [d] ), 'MONTH' -> month ( [d] ), 'DAY' -> day_of_week ( [d] ). Those three return locale-dependent text on both sides. |
+| `CONCAT(str1, str2, ...)` | direct | `concat ( {*} )` | N-ary on both sides. + does not concatenate in ThoughtSpot — it is numeric-only and the parser rejects string operands, so both \|\| and CONCAT land here. |
+| `LENGTH(str)` | direct | `strlen ( {0} )` | Characters, not bytes, on both sides. |
+| `LOWER(str)` | passthrough | `LOWER({0})` — pass-through via `sql_string_op` | There is no native lower in ThoughtSpot. |
+| `UPPER(str)` | passthrough | `UPPER({0})` — pass-through via `sql_string_op` | There is no native upper in ThoughtSpot. LOWER/UPPER are the most-used functions in the whole passthrough set, and their absence is also what forces ILIKE and case-insensitive comparison into pass-throughs. |
+| `TRIM(str)` | passthrough | `TRIM({0})` — pass-through via `sql_string_op` | There is no native trim in ThoughtSpot — live-verified 2026-07-29, rejected with 'Search did not find "trim ("'. The whole trim family is a pass-through, not just the one-sided forms. |
+| `LTRIM(str)` | passthrough | `LTRIM({0})` — pass-through via `sql_string_op` | No native ltrim — live-verified 2026-07-29. |
+| `RTRIM(str)` | passthrough | `RTRIM({0})` — pass-through via `sql_string_op` | As LTRIM. |
+| `LEFT(str, n)` | direct | `left ( {0} , {1} )` | — |
+| `RIGHT(str, n)` | direct | `right ( {0} , {1} )` | — |
+| `SUBSTRING(str, start, length)` | direct | `substr ( {0} , {1} - 1 , {2} )` | Index base differs. ANSI SUBSTRING is 1-based; ThoughtSpot's substr is 0-based. The -1 is mandatory and is the single most likely off-by-one in the whole mapping. When start is an expression rather than a literal, the arithmetic is emitted rather than folded. |
+| `REPLACE(str, from, to)` | passthrough | `REPLACE({0}, {1}, {2})` — pass-through via `sql_string_op` | There is no native replace in ThoughtSpot — live-verified 2026-07-29, rejected with 'Search did not find "replace ("'. |
+| `SPLIT_PART(str, delimiter, part)` | passthrough | `SPLIT_PART({0}, {1}, {2})` — pass-through via `sql_string_op` | ThoughtSpot has no tokenising function at all — not split, split_part or an nth-occurrence search — so there is no composition to fall back on. |
+| `POSITION(substr IN str)` | direct | `strpos ( {1} , {0} )` | Operand order is reversed (haystack first in ThoughtSpot) and the specification's infix IN form becomes a comma. 1-based, returning 0 when absent, on both sides. |
+| `CHARINDEX(substr, str)` | direct | `strpos ( {1} , {0} )` | Specification alias for POSITION (:419) with the operands already in prefix order; the reversal is the same. |
+| `CONTAINS(str, substr)` | direct | `contains ( {0} , {1} )` | Returns boolean on both sides. |
+| `STARTSWITH(str, prefix)` | direct | `strpos ( {0} , {1} ) = 1` | There is no native starts_with — live-verified 2026-07-29. Still direct because the composition is exact and uses only native functions: strpos is 1-based, so a true prefix sits at position 1. |
+| `ENDSWITH(str, suffix)` | direct | `substr ( {0} , strlen ( {0} ) - strlen ( {1} ) , strlen ( {1} ) ) = {1}` | There is no native ends_with — live-verified 2026-07-29. Direct by composition, as STARTSWITH. |
+| `REGEXP_LIKE(str, pattern)` | passthrough | `REGEXP_LIKE({0}, {1})` — pass-through via `sql_bool_op` | Boolean return, so not sql_string_op. ThoughtSpot has no regular-expression support of any kind. |
+| `REGEXP_EXTRACT(str, pattern)` | passthrough | `REGEXP_SUBSTR({0}, {1})` — pass-through via `sql_string_op` | The function name inside the template is dialect-specific — Snowflake spells it REGEXP_SUBSTR, others REGEXP_EXTRACT — so the converter selects it from the connection's dialect and raises an issue when the dialect is unknown. |
+| `REGEXP_REPLACE(str, pattern, replacement)` | passthrough | `REGEXP_REPLACE({0},{1},{2})` — pass-through via `sql_string_op` | Name is portable; the pattern dialect (POSIX vs PCRE, backreference syntax) is not. |
+| `REGEXP_COUNT(str, pattern)` | passthrough | `REGEXP_COUNT({0}, {1})` — pass-through via `sql_int_op` | Integer return. |
+| `ABS(x)` | direct | `abs ( {0} )` | — |
+| `ROUND(x, d)` | direct | `round ( {0} , {1} )` | — |
+| `FLOOR(x)` | direct | `floor ( {0} )` | — |
+| `CEIL(x)` | direct | `ceil ( {0} )` | Specification alias pair CEIL(x) / CEILING(x); both spellings map to ceil. |
+| `TRUNC(x, d)` | passthrough | `TRUNC({0}, {1})` — pass-through via `sql_double_op` | Specification alias pair TRUNC(x, d) / TRUNCATE(x, d). ThoughtSpot has no truncation function. floor agrees with TRUNC only for x >= 0 and d = 0, and round disagrees at every half-value, so neither is a safe substitute. |
+| `MOD(x, y)` | direct | `mod ( {0} , {1} )` | Sign-of-result for negative operands follows the warehouse on both sides. |
+| `SIGN(x)` | direct | `if ( {0} > 0 ) then 1 else if ( {0} < 0 ) then -1 else 0` | No native sign, but the three-way result is exactly expressible as an if chain. The else 0 is required — ThoughtSpot rejects an if chain with no else. |
+| `POWER(x, y)` | direct | `pow ( {0} , {1} )` | The function is pow. power is rejected by the parser. |
+| `SQRT(x)` | direct | `sqrt ( {0} )` | — |
+| `EXP(x)` | direct | `exp ( {0} )` | — |
+| `LN(x)` | direct | `ln ( {0} )` | — |
+| `LOG(base, x)` | direct | `safe_divide ( ln ( {1} ) , ln ( {0} ) )` | ThoughtSpot has fixed-base log2 and log10 only; base is a runtime argument here, not a literal known at catalog time, so the general change-of-base composition is the one template that is exact for every base. safe_divide rather than / guards base = 1. |
+| `LOG10(x)` | direct | `log10 ( {0} )` | — |
+| `SIN(x)` | direct | `sin ( {0} * 180 / 3.14159265358979 )` | ThoughtSpot trigonometry is in degrees; the specification is in radians. The conversion is mandatory — a bare sin ( {0} ) returns the sine of x degrees and is wrong for every non-zero input. |
+| `COS(x)` | direct | `cos ( {0} * 180 / 3.14159265358979 )` | Degrees, as SIN. |
+| `TAN(x)` | direct | `tan ( {0} * 180 / 3.14159265358979 )` | Degrees, as SIN. |
+| `ASIN(x)` | direct | `( asin ( {0} ) * 3.14159265358979 / 180 )` | Inverse functions convert the other way: ThoughtSpot returns degrees, the specification expects radians. |
+| `ACOS(x)` | direct | `( acos ( {0} ) * 3.14159265358979 / 180 )` | Degrees -> radians, as ASIN. |
+| `ATAN(x)` | direct | `( atan ( {0} ) * 3.14159265358979 / 180 )` | Degrees -> radians, as ASIN. |
+| `ATAN2(y, x)` | passthrough | `ATAN2({0}, {1})` — pass-through via `sql_double_op` | atan2 is not a two-argument atan — it is quadrant-aware and defined where x = 0. Composing it from atan plus sign tests is possible but the branch table is easy to get wrong at the axes, so the pass-through is the honest mapping. |
+| `RADIANS(degrees)` | direct | `{0} * 3.14159265358979 / 180` | No native radians; the arithmetic is exact and dialect-free. |
+| `DEGREES(radians)` | direct | `{0} * 180 / 3.14159265358979` | No native degrees; as RADIANS. |
+| `PI()` | direct | `3.14159265358979` | No native pi. The literal is emitted at the precision ThoughtSpot's own documented composites use; sql_double_op ( "pi()" ) is available where full warehouse precision matters. |
+| `GREATEST(x, y, ...)` | direct | `greatest ( {*} )` | Not max. ThoughtSpot's max is an aggregate; greatest is the row-wise N-ary function. Mapping GREATEST to max would collapse the column to one value and also flip it from attribute to measure. |
+| `LEAST(x, y, ...)` | direct | `least ( {*} )` | Not min, for the same reason as GREATEST. |
+| `IF(condition, true_result, false_result)` | direct | `if ( {0} ) then {1} else {2}` | The parentheses around the condition are mandatory for TML import — without them the parser reports "Expecting keyword '('". Applies to every condition shape, including a bare BOOL column reference. |
+| `IFF(condition, true_result, false_result)` | direct | `if ( {0} ) then {1} else {2}` | Specification alias for IF. |
+| `NULLIF(expr1, expr2)` | direct | `nullif ( {0} , {1} )` | — |
+| `COALESCE(expr1, expr2, ...)` | direct | `ifnull ( {0} , {1} )` | ThoughtSpot's ifnull is strictly two-argument, so an N-ary COALESCE becomes a right-nested chain. Two arguments is the common case and needs no nesting. |
+| `IFNULL(expr, default)` | direct | `ifnull ( {0} , {1} )` | — |
+| `NVL(expr, default)` | direct | `ifnull ( {0} , {1} )` | Specification alias for two-argument COALESCE. |
+| `NVL2(expr, not_null_result, null_result)` | direct | `if ( isnotnull ( {0} ) ) then {1} else {2}` | No native three-way null function; the composition is exact. |
+| `ZEROIFNULL(expr)` | direct | `ifnull ( {0} , 0 )` | — |
+| `NULLIFZERO(expr)` | direct | `nullif ( {0} , 0 )` | — |
+| `+` | direct | `{0} + {1}` | Numeric only. ThoughtSpot's + rejects string operands, so a + that concatenates on the source side must become concat ( ). The specification does not overload +, so this only bites when translating a dialect expression. |
+| `-` | direct | `{0} - {1}` | — |
+| `*` | direct | `{0} * {1}` | — |
+| `/` | direct | `{0} / {1}` | Both yield NULL (or a warehouse error) on divide-by-zero. ThoughtSpot's safe_divide returns 0, not NULL, so it is not a faithful substitute and is used only where the source itself guards the denominator. |
+| `%` | direct | `mod ( {0} , {1} )` | ThoughtSpot has no % operator — the modulo is the function. |
+| `-x / +x (unary)` | direct | `per-spelling — see note` | CONVENTION_DIVERGENCE: unary +/- is named only in the 'Operator Precedence' list, never a table row. This row merges two Ossie spellings that need DIFFERENT output — unary minus is -[x] (negation), unary plus is the identity ([x] unchanged) — so a single {0}-substitutable template would be wrong for whichever spelling didn't produce it: an earlier draft used template="-{0}", which is correct for -x but silently negates a parsed + [...]
+| `=` | direct | `{0} = {1}` | — |
+| `<>` | direct | `{0} <> {1}` | — |
+| `!=` | direct | `{0} != {1}` | ThoughtSpot accepts both inequality spellings, so the two rows are independent and both direct. |
+| `<` | direct | `{0} < {1}` | — |
+| `>` | direct | `{0} > {1}` | — |
+| `<=` | direct | `{0} <= {1}` | — |
+| `>=` | direct | `{0} >= {1}` | — |
+| `expr1 AND expr2` | direct | `{0} and {1}` | Lower-case, infix. |
+| `expr1 OR expr2` | direct | `{0} or {1}` | Lower-case, infix. |
+| `NOT expr` | direct | `not ( {0} )` | Function form with parentheses, not a prefix operator — not [x] does not parse. |
+| `BETWEEN` | direct | `{0} between {1} and {2}` | Inclusive on both sides. |
+| `IN` | direct | `{0} in {{ {*} }}` | Literal lists only on both sides — no subqueries. The curly-brace delimiter is confirmed, live-verified 2026-07-29: the round-parenthesis form is rejected with 'Expecting one of the valid keywords, such as, "ts_var", "{"'. It forces >- block-scalar YAML. The braces are doubled ({{ }}) in the template because emit_direct renders via str.format, which reads a single literal brace as the start of a field name (see test_emit.py's catalog-wide sweep). |
+| `NOT IN` | direct | `not ( {0} in {{ {*} }} )` | Emitted as a negated in rather than a not in keyword — the bare keyword form is not reliably accepted. Braces doubled for str.format, as IN above. |
+| `str LIKE pattern` | direct | `per-pattern-shape — see note` | Prefix ('foo%') -> strpos ( {0} , 'foo' ) = 1; suffix ('%foo') -> substr ( {0} , strlen ( {0} ) - strlen ( 'foo' ) , strlen ( 'foo' ) ) = 'foo'; contains ('%foo%') -> contains ( {0} , 'foo' ). Only contains is a native function — starts_with and ends_with do not exist (live-verified 2026-07-29), so the first two shapes are compositions of native functions, same as the STARTSWITH/ENDSWITH rows. These three shapes are the ove [...]
+| `str ILIKE pattern` | passthrough | `{0} ILIKE {1}` — pass-through via `sql_bool_op` | Case-insensitive matching has no native form, and the usual workaround — fold both sides with lower — is itself a pass-through, so there is nothing to compose from. |
+| `IS NULL` | direct | `isnull ( {0} )` | — |
+| `IS NOT NULL` | direct | `isnotnull ( {0} )` | Native, so not composed as not ( isnull ( ) ). |
+| `IS DISTINCT FROM` | direct | `if ( isnull ( {0} ) and isnull ( {1} ) ) then false else if ( isnull ( {0} ) or isnull ( {1} ) ) then true else {0} != {1}` | No native null-safe comparison, but the three-case truth table is exactly expressible. The nesting order matters: both-null must be tested before either-null. |
+| `IS NOT DISTINCT FROM` | direct | `if ( isnull ( {0} ) and isnull ( {1} ) ) then true else if ( isnull ( {0} ) or isnull ( {1} ) ) then false else {0} = {1}` | The negation of the row above, written directly rather than wrapped in not ( ) — one fewer nesting level for the parser. |
+| `CASE WHEN` | direct | `if ( c1 ) then r1 else if ( c2 ) then r2 else d` | The searched CASE WHEN c1 THEN r1 ... ELSE d END form. No native CASE; the chain is else if, two words. The final else is mandatory and must be type-matched — else 0 for a measure, else '' for an attribute. Omitting it raises 'Unknown data type', and a CASE with no ELSE (legal in the specification, yielding NULL) therefore needs one synthesised. The branch count is unbounded, so the template uses symbolic c1/r1/ [...]
+| `CASE expr WHEN v1 THEN r1 ... END (simple)` | direct | `if ( [expr] = v1 ) then r1 else if ( [expr] = v2 ) then r2 else d` | CONVENTION_DIVERGENCE: the simple CASE form is described only in the CASE Expression code fence, never a table row. Expanded to the searched form with an explicit equality per branch. expr is repeated per branch, so a converter should hoist an expensive expr into its own formula first. Symbolic template, as CASE WHEN above, for the same unbounded-branch-count reason. |
+| str1 \|\| str2 | direct | `concat ( {0} , {1} )` | ThoughtSpot has no concatenation operator at all — + is numeric-only — so \|\| and CONCAT share one target. |
+| `Parentheses — expression grouping` | direct | `( {0} )` | CONVENTION_DIVERGENCE: its Supported SQL Constructs row carries no backtick token in either cell, the only marker the top-table extraction keys on. Precedence is the standard SQL ordering on the Ossie side. The converter emits explicit parentheses around every rewritten sub-expression rather than relying on the two languages agreeing about precedence — cheap, and it removes a whole class of silent arithmetic errors. |
+| `TRUE, FALSE` | direct | `true / false` | The Boolean Functions table's Syntax cell merges TRUE and FALSE into one comma-joined entry, matching what spec_construct_names() extracts. Which of the two lower-case literals is emitted depends on which the source wrote — TRUE -> true, FALSE -> false — resolved per-occurrence, out of this catalog's scope (same as CAST's per-type dispatch). A bare BOOL column reference used as a condition still needs its parentheses: if ( [T::flag] ) then ... [...]
+| `DISTINCT aggregate modifier` | passthrough | `SUM(DISTINCT {0})` — pass-through via `sql_number_aggregate_op` — **example only**: `the aggregate` shown as a literal, rebuild the template per occurrence | CONVENTION_DIVERGENCE: described only in the Conditional Aggregations prose/code block, never a table row. The specification allows DISTINCT on SUM as well as COUNT. ThoughtSpot has exactly one distinct-aware aggregate — unique count — which is COUNT(DISTINCT) and already has its own [...]
+| `Column / metric reference — field, dataset.field` | direct | `[TABLE::Column], or [Formula Name] for a metric` | CONVENTION_DIVERGENCE: its Supported SQL Constructs row carries no backtick token in either cell, same reason as Parentheses. Always rewritten from resolved metadata, never passed through textually — the rewrite, the case-sensitivity rules and the display-name-versus-identifier problem are out of this catalog's scope. |
+| `EXISTS_IN()` | unmappable | — | CONVENTION_DIVERGENCE: named only in the Reason column of the excluded 'Not Supported in Expressions' table, never in a table of its own. The single unmappable row in the whole 146-row catalog: named at :131 as the sanctioned way to filter on a subquery, but defined nowhere in the specification — no signature, no argument order, no semantics, absent from every function table. Even given a signature, ThoughtSpot's nearest capability is a sql_bool_op subq [...]
+| `ROW_NUMBER() OVER (...)` | passthrough | `ROW_NUMBER() OVER (PARTITION BY {0} ORDER BY {1})` — pass-through via `sql_int_aggregate_op` | ThoughtSpot's rank is competition rank, not a row number, so it is not a substitute. Wrap in group_aggregate so the partition column reaches the GROUP BY even when the user's search omits it. |
+| `RANK() OVER (...)` | direct | `rank ( sum ( [m] ) , 'desc' )` | direct for one shape only, and the boundary is proven rather than asserted: the global, ORDER BY-only form over an aggregate. Live-confirmed 2026-07-30: rank ( sum ( [m] ) , 'desc' ) and 'asc' both validate, and the arity is enforced at exactly two — a third argument in any shape (bare attribute, { [attr] }, or query_groups ( )) is rejected with 'Function rank expects only 2 arguments', so an explicit PARTITION BY is prov [...]
+| `DENSE_RANK() OVER (...)` | passthrough | `dense_rank() over (order by sum({0}) desc)` — pass-through via `sql_int_aggregate_op` — **example only**: `the ordering aggregate`, `the sort direction` shown as literals, rebuild the template per occurrence | ThoughtSpot's rank skips ranks after a tie; dense ranking has no native form — live-confirmed 2026-07-30, dense_rank ( ... ) rejected with 'Search did not find "dense_rank ( sum ("'. Passthrough is correct: no native ThoughtSpot construc [...]
+| `NTILE(n) OVER (...)` | passthrough | `NTILE(4) OVER (ORDER BY SUM({0}))` — pass-through via `sql_int_aggregate_op` — **example only**: `n`, `the ordering aggregate` shown as literals, rebuild the template per occurrence | n is a literal, baked into the template, as the aggregate percentiles are. So is the SUM: the specification orders an NTILE by whatever the caller chose, and this row illustrates with one aggregate rather than mapping only that one. |
+| `PERCENT_RANK() OVER (...)` | direct | `1 - rank_percentile ( sum ( [m] ) , 'asc' ) / 100` | ThoughtSpot's rank_percentile is documented as (1.0 - PERCENT_RANK() OVER (ORDER BY ...)) * 100, so the inverse is exact. Two adjustments are both required: the scale (ThoughtSpot 0-100, specification 0-1) and the inversion. Dropping either produces a plausible-looking column that is wrong everywhere. Same shape restriction as RANK, and the same live-proven boundary — rank_percentile is also fi [...]
+| `CUME_DIST() OVER (...)` | passthrough | `CUME_DIST() OVER (ORDER BY SUM({0}))` — pass-through via `sql_number_aggregate_op` — **example only**: `the ordering aggregate` shown as a literal, rebuild the template per occurrence | rank_percentile is NOT a substitute, despite PERCENT_RANK's row looking equivalent: PERCENT_RANK divides by n - 1 and starts at 0; CUME_DIST divides by n and ends at 1. They agree on no row of a tie-free window except the last, so there is no native fallback at [...]
+| `LAG(expr, offset, default) OVER (...)` | passthrough | `LAG({0}, 1, 0) OVER (PARTITION BY {1} ORDER BY {2})` — pass-through via `sql_number_aggregate_op` — **example only**: `offset`, `default` shown as literals, rebuild the template per occurrence | Reclassified direct -> passthrough 2026-07-30. The native idiom moving_sum ( [m] , n , -n , [ord] ) is real and validates (a frame of n PRECEDING to n PRECEDING) but is not equivalent to any OVER shape: moving_sum has no partition slot, a [...]
+| `LEAD(expr, offset, default) OVER (...)` | passthrough | `LEAD({0}, 1, 0) OVER (PARTITION BY {1} ORDER BY {2})` — pass-through via `sql_number_aggregate_op` — **example only**: `offset`, `default` shown as literals, rebuild the template per occurrence | Mirror of LAG, reclassified for the same reason and on the same date. The native downgrade is moving_sum ( [m] , -n , n , [ord] ) — ThoughtSpot's start/end arguments use opposite sign conventions, so a forward offset is a negative start [...]
+| `FIRST_VALUE(expr) OVER (...)` | direct | `first_value ( sum ( [m] ) , query_groups ( ) , {{ [T::date] }} )` | The section's exception, and the only window row whose direct verdict survived the 2026-07-30 rework — first_value takes a genuine explicit partition argument and a genuine explicit order axis, so the formula does define its own window. Live-confirmed 2026-07-30: query_groups ( ), a fixed single-column { [attr] }, a multi-column { [a] , [b] }, the grand-total { } and the dynam [...]
+| `LAST_VALUE(expr) OVER (...)` | direct | `last_value ( sum ( [m] ) , query_groups ( ) , {{ [T::date] }} )` | Same conditions, same live evidence and same fallback as FIRST_VALUE. last_value_in_period and first_value_in_period also validate in the identical three-argument shape and are the period-completeness variants (see the reverse-direction table) — out of this row's scope. Braces doubled on the axis argument for the same str.format reason as FIRST_VALUE. |
+| `NTH_VALUE(expr, n) OVER (...)` | passthrough | `NTH_VALUE({0}, 2) OVER (ORDER BY {1})` — pass-through via `sql_number_aggregate_op` — **example only**: `n` shown as a literal, rebuild the template per occurrence | ThoughtSpot's semi-additive functions reach only the first and last values of the axis — live-confirmed 2026-07-30, nth_value ( ... ) rejected with 'Search did not find "nth_value ( sum ("'. n is a literal, baked into the template, as NTILE's. Variant recorded here is the do [...]
+| `OVER (PARTITION BY ... ORDER BY ...) clause` | passthrough | `per-clause-shape — see note` — pass-through via `sql_number_aggregate_op` | CONVENTION_DIVERGENCE: the generic OVER syntax template is a fenced code block, not a table. Reclassified direct -> passthrough 2026-07-30. The previous verdict claimed a clean structural rewrite — 'PARTITION BY attrs becomes the group_aggregate grouping argument; ORDER BY becomes the window function's trailing attribute arguments' — but that holds [...]
+| `Frame clause — ROWS BETWEEN ... / RANGE BETWEEN ...` | direct | `per-frame-shape — see note` | CONVENTION_DIVERGENCE: frame options are a bullet list under the OVER syntax section, not a table. direct for the frame boundaries only — deliberately scoped, so the partition loss is counted once, on the OVER clause row, and not twice. ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW -> cumulative_*. Bounded ROWS frames -> moving_* with n PRECEDING -> positive n, CURRENT ROW -> 0, n FOLLOWI [...]
+| `Window aggregation — AGG(expr) OVER (...)` | passthrough | `SUM({0}) OVER (PARTITION BY {1} ORDER BY {2} ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)` — pass-through via `sql_number_aggregate_op` — **example only**: `the aggregate`, `the window frame` shown as literals, rebuild the template per occurrence | CONVENTION_DIVERGENCE: the Window Aggregations section is prose and code examples, not a table. Reclassified direct -> passthrough 2026-07-30, inheriting the OVER row's proble [...]
+
diff --git a/converters/thoughtspot/docs/reverse-inventory.md b/converters/thoughtspot/docs/reverse-inventory.md
new file mode 100644
index 00000000..96dacc2f
--- /dev/null
+++ b/converters/thoughtspot/docs/reverse-inventory.md
@@ -0,0 +1,143 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+<!-- GENERATED FILE -- do not edit by hand.
+ Produced by `tools/generate_reference_docs.py` from:
+ - `src/ossie_thoughtspot/expressions/reverse.py`
+ Regenerate with:
+ uv run --python 3.13 python tools/generate_reference_docs.py
+ tests/test_reference_docs_current.py fails the suite if this file
+ drifts from what the generator currently produces. -->
+
+# ThoughtSpot -> Ossie Reverse Inventory
+
+ThoughtSpot's own native functions with no counterpart in the Ossie specification (`ThoughtSpot -> Ossie`, the reverse of the expression mapping above), and how each reaches — or does not reach — a portable Ossie expression. This inventory is not yet called from the shipped `TML -> Ossie` conversion path; see `converters/thoughtspot/README.md`'s "Expression translation" section for the converter's current, more conservative default.
+
+## Reading the notation
+
+| Notation | Means |
+|---|---|
+| `{0}`, `{1}`, `{2}` | A positional argument slot, filled left to right from the construct's own arguments. |
+
+## Coverage
+
+| Disposition | Count | Share | Meaning |
+|---|---|---|---|
+| compose | 53 | 63% | A full, portable Ossie expression is produced. |
+| partial | 8 | 10% | A real Ossie expression is produced, but it is provably incomplete. |
+| dialect | 11 | 13% | Resolves to the Ossie `dialects[]` mechanism, not a portable expression. |
+| stash | 12 | 14% | No Ossie expression exists at all; preserved verbatim for round-trip only. |
+| **Total** | **84** | **100%** | |
+
+## Cross-cutting dispatch, not name-keyed
+
+Two checks apply before an ordinary lookup by name into `REVERSE`, so they are not rows of the table below:
+
+- **Fiscal-calendar argument.** Any call whose last argument is `'fiscal'`, `fiscal` stashes unconditionally, for any function name at all, before the name is looked up.
+- **Hyperlink markup.** A `concat` call whose string arguments contain `{caption}` or `{/caption}` is redirected to the `concat (hyperlink markup)` row below; plain `concat` has a specification counterpart already covered by `CATALOG` and is not this module's concern.
+
+## Every entry
+
+| ThoughtSpot construct | Disposition | Composes to | Issue | Notes |
+|---|---|---|---|---|
+| `sum_if` | compose | `SUM(CASE WHEN {0} THEN {1} END)` | — | sum_if ( cond , x ) -> SUM(CASE WHEN cond THEN x END). |
+| `count_if` | compose | `COUNT(CASE WHEN {0} THEN {1} END)` | — | count_if ( cond , x ) -> COUNT(CASE WHEN cond THEN x END). |
+| `average_if` | compose | `AVG(CASE WHEN {0} THEN {1} END)` | — | average_if ( cond , x ) -> AVG(CASE WHEN cond THEN x END). |
+| `min_if` | compose | `MIN(CASE WHEN {0} THEN {1} END)` | — | min_if ( cond , x ) -> MIN(CASE WHEN cond THEN x END). |
+| `max_if` | compose | `MAX(CASE WHEN {0} THEN {1} END)` | — | max_if ( cond , x ) -> MAX(CASE WHEN cond THEN x END). |
+| `stddev_if` | compose | `STDDEV(CASE WHEN {0} THEN {1} END)` | — | stddev_if ( cond , x ) -> STDDEV(CASE WHEN cond THEN x END). |
+| `variance_if` | compose | `VARIANCE(CASE WHEN {0} THEN {1} END)` | — | variance_if ( cond , x ) -> VARIANCE(CASE WHEN cond THEN x END). |
+| `unique_count_if` | compose | `COUNT(DISTINCT CASE WHEN {0} THEN {1} END)` | — | unique_count_if ( cond , x ) -> COUNT(DISTINCT CASE WHEN cond THEN x END). |
+| `unique count` | compose | `COUNT(DISTINCT {0})` | — | ThoughtSpot's own spelling has a space, not an underscore. |
+| `safe_divide` | compose | `COALESCE({0} / NULLIF({1}, 0), 0)` | — | The zero-not-null result is preserved by the explicit COALESCE. |
+| `pow` | compose | `POWER({0}, {1})` | — | — |
+| `log2` | compose | `LOG(2, {0})` | — | — |
+| `strlen` | compose | `LENGTH({0})` | — | — |
+| `strpos` | compose | `POSITION({1} IN {0})` | — | ThoughtSpot strpos(s, sub) -> Ossie POSITION(sub IN s); operand order reverses. |
+| `substr` | compose | `SUBSTRING({0}, {1} + 1, {2})` | — | ThoughtSpot's substr is 0-based; the +1 is mandatory going this way. |
+| `left` | compose | `LEFT({0}, {1})` | — | — |
+| `right` | compose | `RIGHT({0}, {1})` | — | — |
+| `sin` | compose | `SIN(RADIANS({0}))` | — | ThoughtSpot trigonometry is in degrees; the conversion reverses. |
+| `cos` | compose | `COS(RADIANS({0}))` | — | ThoughtSpot trigonometry is in degrees; the conversion reverses. |
+| `tan` | compose | `TAN(RADIANS({0}))` | — | ThoughtSpot trigonometry is in degrees; the conversion reverses. |
+| `asin` | compose | `DEGREES(ASIN({0}))` | — | ThoughtSpot's inverse trig functions return degrees. |
+| `acos` | compose | `DEGREES(ACOS({0}))` | — | ThoughtSpot's inverse trig functions return degrees. |
+| `atan` | compose | `DEGREES(ATAN({0}))` | — | ThoughtSpot's inverse trig functions return degrees. |
+| `to_integer` | compose | `CAST({0} AS INTEGER)` | — | — |
+| `to_double` | compose | `CAST({0} AS DOUBLE)` | — | — |
+| `to_string` | compose | `CAST({0} AS VARCHAR)` | — | — |
+| `to_date` | compose | `TO_DATE({0}, {1})` | `TS-EXPR-FORMAT-TOKENS-PASSTHROUGH` · INFO | Judgment call: format-token reversal is deferred until an expression parser exists to do the translation. |
+| `if` | compose | `CASE WHEN {0} THEN {1} ELSE {2} END` | — | if ( c ) then a else b -> CASE WHEN c THEN a ELSE b END, or IF(c, a, b). |
+| `rank` | compose | dynamic — see `_compose_rank` in `reverse.py` | — | Global, ORDER-BY-only shape only — rank's arity is fixed at exactly two (live-confirmed), so there is never a partition to lose in this direction. |
+| `rank_percentile` | compose | dynamic — see `_compose_rank_percentile` in `reverse.py` | — | Scale (0-100 -> 0-1) and inversion both reverse. |
+| `moving_sum` | partial | dynamic — see `_compose_moving.<locals>._compose` in `reverse.py` | `TS-EXPR-PARTIAL-PARTITION` · WARNING | Frame and order translate exactly; the partition does not — a ThoughtSpot window formula cannot declare its own PARTITION BY, and the specification has no way to express that limitation. |
+| `cumulative_sum` | partial | dynamic — see `_compose_cumulative.<locals>._compose` in `reverse.py` | `TS-EXPR-PARTIAL-PARTITION` · WARNING | Frame and order translate exactly; the partition does not — a ThoughtSpot window formula cannot declare its own PARTITION BY, and the specification has no way to express that limitation. |
+| `moving_average` | partial | dynamic — see `_compose_moving.<locals>._compose` in `reverse.py` | `TS-EXPR-PARTIAL-PARTITION` · WARNING | Frame and order translate exactly; the partition does not — a ThoughtSpot window formula cannot declare its own PARTITION BY, and the specification has no way to express that limitation. |
+| `cumulative_average` | partial | dynamic — see `_compose_cumulative.<locals>._compose` in `reverse.py` | `TS-EXPR-PARTIAL-PARTITION` · WARNING | Frame and order translate exactly; the partition does not — a ThoughtSpot window formula cannot declare its own PARTITION BY, and the specification has no way to express that limitation. |
+| `moving_max` | partial | dynamic — see `_compose_moving.<locals>._compose` in `reverse.py` | `TS-EXPR-PARTIAL-PARTITION` · WARNING | Frame and order translate exactly; the partition does not — a ThoughtSpot window formula cannot declare its own PARTITION BY, and the specification has no way to express that limitation. |
+| `cumulative_max` | partial | dynamic — see `_compose_cumulative.<locals>._compose` in `reverse.py` | `TS-EXPR-PARTIAL-PARTITION` · WARNING | Frame and order translate exactly; the partition does not — a ThoughtSpot window formula cannot declare its own PARTITION BY, and the specification has no way to express that limitation. |
+| `moving_min` | partial | dynamic — see `_compose_moving.<locals>._compose` in `reverse.py` | `TS-EXPR-PARTIAL-PARTITION` · WARNING | Frame and order translate exactly; the partition does not — a ThoughtSpot window formula cannot declare its own PARTITION BY, and the specification has no way to express that limitation. |
+| `cumulative_min` | partial | dynamic — see `_compose_cumulative.<locals>._compose` in `reverse.py` | `TS-EXPR-PARTIAL-PARTITION` · WARNING | Frame and order translate exactly; the partition does not — a ThoughtSpot window formula cannot declare its own PARTITION BY, and the specification has no way to express that limitation. |
+| `group_aggregate` | compose | dynamic — see `_dispatch_group_aggregate` in `reverse.py` | — | Shape dispatch on the grouping/filter arguments — see _compose_grouped. |
+| `group_sum` | compose | dynamic — see `_make_group_shorthand_dispatch.<locals>._dispatch` in `reverse.py` | — | Shorthand for group_aggregate(sum(m), ...) — same shape dispatch. Judgment call: the (m, grouping, filter) 3-argument shape is assumed by analogy with group_aggregate's live-confirmed form; the shorthand family's own arity was not independently live-tested. |
+| `group_count` | compose | dynamic — see `_make_group_shorthand_dispatch.<locals>._dispatch` in `reverse.py` | — | Shorthand for group_aggregate(count(m), ...) — same shape dispatch. Judgment call: the (m, grouping, filter) 3-argument shape is assumed by analogy with group_aggregate's live-confirmed form; the shorthand family's own arity was not independently live-tested. |
+| `group_stddev` | compose | dynamic — see `_make_group_shorthand_dispatch.<locals>._dispatch` in `reverse.py` | — | Shorthand for group_aggregate(stddev(m), ...) — same shape dispatch. Judgment call: the (m, grouping, filter) 3-argument shape is assumed by analogy with group_aggregate's live-confirmed form; the shorthand family's own arity was not independently live-tested. |
+| `group_variance` | compose | dynamic — see `_make_group_shorthand_dispatch.<locals>._dispatch` in `reverse.py` | — | Shorthand for group_aggregate(variance(m), ...) — same shape dispatch. Judgment call: the (m, grouping, filter) 3-argument shape is assumed by analogy with group_aggregate's live-confirmed form; the shorthand family's own arity was not independently live-tested. |
+| `group_max` | compose | dynamic — see `_make_group_shorthand_dispatch.<locals>._dispatch` in `reverse.py` | — | Shorthand for group_aggregate(max(m), ...) — same shape dispatch. Judgment call: the (m, grouping, filter) 3-argument shape is assumed by analogy with group_aggregate's live-confirmed form; the shorthand family's own arity was not independently live-tested. |
+| `group_min` | compose | dynamic — see `_make_group_shorthand_dispatch.<locals>._dispatch` in `reverse.py` | — | Shorthand for group_aggregate(min(m), ...) — same shape dispatch. Judgment call: the (m, grouping, filter) 3-argument shape is assumed by analogy with group_aggregate's live-confirmed form; the shorthand family's own arity was not independently live-tested. |
+| `group_average` | compose | dynamic — see `_make_group_shorthand_dispatch.<locals>._dispatch` in `reverse.py` | — | Shorthand for group_aggregate(avg(m), ...) — same shape dispatch. Judgment call: the (m, grouping, filter) 3-argument shape is assumed by analogy with group_aggregate's live-confirmed form; the shorthand family's own arity was not independently live-tested. |
+| `group_unique_count` | compose | dynamic — see `_make_group_shorthand_dispatch.<locals>._dispatch` in `reverse.py` | — | Shorthand for group_aggregate(count_distinct(m), ...) — same shape dispatch. Judgment call: the (m, grouping, filter) 3-argument shape is assumed by analogy with group_aggregate's live-confirmed form; the shorthand family's own arity was not independently live-tested. |
+| `last_value` | stash | — | `TS-EXPR-SEMI-ADDITIVE` · ERROR | The window clause itself round-trips; only the roll-up declaration is lost. |
+| `first_value` | stash | — | `TS-EXPR-SEMI-ADDITIVE` · ERROR | The window clause itself round-trips; only the roll-up declaration is lost. |
+| `last_value_in_period` | stash | — | `TS-EXPR-SEMI-ADDITIVE` · ERROR | The window clause itself round-trips; only the roll-up declaration is lost. |
+| `first_value_in_period` | stash | — | `TS-EXPR-SEMI-ADDITIVE` · ERROR | The window clause itself round-trips; only the roll-up declaration is lost. |
+| `sql_string_op` | dialect | dynamic — see `_dispatch_sql_op` in `reverse.py` | — | Resolves to the Ossie dialects[] mechanism for the connection's own dialect, not a portable expression — the right home for raw warehouse SQL. |
+| `sql_int_op` | dialect | dynamic — see `_dispatch_sql_op` in `reverse.py` | — | Resolves to the Ossie dialects[] mechanism for the connection's own dialect, not a portable expression — the right home for raw warehouse SQL. |
+| `sql_number_op` | dialect | dynamic — see `_dispatch_sql_op` in `reverse.py` | — | Resolves to the Ossie dialects[] mechanism for the connection's own dialect, not a portable expression — the right home for raw warehouse SQL. |
+| `sql_double_op` | dialect | dynamic — see `_dispatch_sql_op` in `reverse.py` | — | Resolves to the Ossie dialects[] mechanism for the connection's own dialect, not a portable expression — the right home for raw warehouse SQL. |
+| `sql_bool_op` | dialect | dynamic — see `_dispatch_sql_op` in `reverse.py` | — | Resolves to the Ossie dialects[] mechanism for the connection's own dialect, not a portable expression — the right home for raw warehouse SQL. |
+| `sql_date_op` | dialect | dynamic — see `_dispatch_sql_op` in `reverse.py` | — | Resolves to the Ossie dialects[] mechanism for the connection's own dialect, not a portable expression — the right home for raw warehouse SQL. |
+| `sql_date_time_op` | dialect | dynamic — see `_dispatch_sql_op` in `reverse.py` | — | Resolves to the Ossie dialects[] mechanism for the connection's own dialect, not a portable expression — the right home for raw warehouse SQL. |
+| `sql_string_aggregate_op` | dialect | dynamic — see `_dispatch_sql_op` in `reverse.py` | — | Resolves to the Ossie dialects[] mechanism for the connection's own dialect, not a portable expression — the right home for raw warehouse SQL. |
+| `sql_int_aggregate_op` | dialect | dynamic — see `_dispatch_sql_op` in `reverse.py` | — | Resolves to the Ossie dialects[] mechanism for the connection's own dialect, not a portable expression — the right home for raw warehouse SQL. |
+| `sql_number_aggregate_op` | dialect | dynamic — see `_dispatch_sql_op` in `reverse.py` | — | Resolves to the Ossie dialects[] mechanism for the connection's own dialect, not a portable expression — the right home for raw warehouse SQL. |
+| `sql_date_time_aggregate_op` | dialect | dynamic — see `_dispatch_sql_op` in `reverse.py` | — | Resolves to the Ossie dialects[] mechanism for the connection's own dialect, not a portable expression — the right home for raw warehouse SQL. |
+| `<runtime parameter reference>` | stash | — | `TS-EXPR-RUNTIME-PARAMETER` · ERROR | Synthetic key — not a callable name. See stash_runtime_parameter(). |
+| `ts_username` | stash | — | `TS-EXPR-RUNTIME-IDENTITY` · ERROR | `ts_username` resolves signed-in-user identity at query time; an interchange document that carried it would describe an access-control decision, not semantics. Preserved verbatim for roundtrip. |
+| `ts_groups` | stash | — | `TS-EXPR-RUNTIME-IDENTITY` · ERROR | `ts_groups` resolves signed-in-user identity at query time; an interchange document that carried it would describe an access-control decision, not semantics. Preserved verbatim for roundtrip. |
+| `ts_groups_int` | stash | — | `TS-EXPR-RUNTIME-IDENTITY` · ERROR | `ts_groups_int` resolves signed-in-user identity at query time; an interchange document that carried it would describe an access-control decision, not semantics. Preserved verbatim for roundtrip. |
+| `ts_org` | stash | — | `TS-EXPR-RUNTIME-IDENTITY` · ERROR | `ts_org` resolves signed-in-user identity at query time; an interchange document that carried it would describe an access-control decision, not semantics. Preserved verbatim for roundtrip. |
+| `ts_email_domain` | stash | — | `TS-EXPR-RUNTIME-IDENTITY` · ERROR | `ts_email_domain` resolves signed-in-user identity at query time; an interchange document that carried it would describe an access-control decision, not semantics. Preserved verbatim for roundtrip. |
+| `ts_var` | stash | — | `TS-EXPR-RUNTIME-IDENTITY` · ERROR | `ts_var` resolves signed-in-user identity at query time; an interchange document that carried it would describe an access-control decision, not semantics. Preserved verbatim for roundtrip. |
+| `concat (hyperlink markup)` | stash | — | `TS-EXPR-HYPERLINK-MARKUP` · ERROR | Synthetic key, reached only via the content-pattern check in translate_thoughtspot -- plain concat (no markup) is out of this module's scope entirely. |
+| `month` | compose | `TO_CHAR({0}, 'MONTH')` | `TS-EXPR-LOCALE-DEPENDENT` · WARNING | Name-returning form, distinct from month_number/year/day_number_of_week. |
+| `year_name` | compose | `TO_CHAR({0}, 'YYYY')` | `TS-EXPR-LOCALE-DEPENDENT` · WARNING | Name-returning form, distinct from month_number/year/day_number_of_week. |
+| `day_of_week` | compose | `TO_CHAR({0}, 'DAY')` | `TS-EXPR-LOCALE-DEPENDENT` · WARNING | Name-returning form, distinct from month_number/year/day_number_of_week. |
+| `month_number_of_quarter` | compose | `MOD(MONTH({0}) - 1, 3) + 1` | — | — |
+| `day_number_of_quarter` | compose | `DATEDIFF(day, DATE_TRUNC('quarter', {0}), {0}) + 1` | — | — |
+| `week_number_of_month` | compose | `DATEDIFF(week, DATE_TRUNC('month', {0}), {0}) + 1` | `TS-EXPR-WEEK-START-ASSUMED` · WARNING | `week_number_of_month` is correct only if the target engine's week start agrees with the specification's fixed Monday start; ThoughtSpot's week start is an instance setting. Verify alignment before relying on this column. |
+| `week_number_of_quarter` | compose | `DATEDIFF(week, DATE_TRUNC('quarter', {0}), {0}) + 1` | `TS-EXPR-WEEK-START-ASSUMED` · WARNING | `week_number_of_quarter` is correct only if the target engine's week start agrees with the specification's fixed Monday start; ThoughtSpot's week start is an instance setting. Verify alignment before relying on this column. |
+| `is_weekend` | compose | `DATE_PART('dayofweek', {0}) IN (6, 7)` | `TS-EXPR-DAYOFWEEK-BASE` · WARNING | `is_weekend`'s member list (6, 7) uses ThoughtSpot's own DAYOFWEEK base (1 = Monday); the specification does not fix a base and engines disagree — confirm the target engine's base agrees before relying on this column. |
+| `start_of_hour` | compose | `DATE_TRUNC('hour', {0})` | — | — |
+| `start_of_min` | compose | `DATE_TRUNC('minute', {0})` | — | — |
+| `date` | compose | `DATE_TRUNC('day', {0})` | — | — |
+| `time` | compose | `CAST({0} AS TIME)` | — | — |
+| `greatest` | compose | dynamic — see `_compose_variadic.<locals>._compose` in `reverse.py` | — | Never MAX — that would turn a row-wise attribute into an aggregate measure. |
+| `least` | compose | dynamic — see `_compose_variadic.<locals>._compose` in `reverse.py` | — | Never MIN, for the same reason. |
+
diff --git a/converters/thoughtspot/docs/vendor-payload.md b/converters/thoughtspot/docs/vendor-payload.md
new file mode 100644
index 00000000..6a26f22b
--- /dev/null
+++ b/converters/thoughtspot/docs/vendor-payload.md
@@ -0,0 +1,112 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+<!-- GENERATED FILE -- do not edit by hand.
+ Produced by `tools/generate_reference_docs.py` from:
+ - `src/ossie_thoughtspot/constants.py`
+ Regenerate with:
+ uv run --python 3.13 python tools/generate_reference_docs.py
+ tests/test_reference_docs_current.py fails the suite if this file
+ drifts from what the generator currently produces. -->
+
+# The `custom_extensions[THOUGHTSPOT]` Payload
+
+TML carries properties Ossie's core specification has no field for. `TML -> Ossie` stashes each one under a single `custom_extensions` entry attached to the Ossie object it came from; `Ossie -> TML` reads the same entry back. This page is generated from `constants.py`'s own key vocabulary and `STASH_KEY_CLASSIFICATION` — the table `test_stash_key_classification.py` enforces every key read on the `Ossie -> TML` direction must appear in.
+
+## Envelope
+
+Every `custom_extensions` entry this converter writes uses `vendor_name` `THOUGHTSPOT`. `data` is a single JSON-encoded string (never a nested object) whose own top-level `_v` field is the shape version (`1` today) — bumped only when the payload's shape changes, never for a value change; an unrecognised version is a hard failure rather than a silent misread.
+
+## Payload keys
+
+| Key | Scope | Classification | Treatment on the return trip |
+|---|---|---|---|
+| `tml_name` | Shared | shadows_derivable | Restored only if reconstructing it from the live document still agrees with the stashed value (self-verifying — no separate witness key); disagreement re-derives instead. |
+| `column_aggregation_value` | Metric | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `tml_obj_id` | Model | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `formula_id` | Field | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `db_column_name` | Field | shadows_derivable | Restored only if its witness companion key still matches the live document's current value; a mismatch means the document changed since the stash was written, so the value is re-derived instead. |
+| `data_type` | Field | shadows_derivable | Restored only if its witness companion key still matches the live document's current value; a mismatch means the document changed since the stash was written, so the value is re-derived instead. |
+| `column_properties` | Field | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `shape` | Metric | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `tml_object` | Dataset | shadows_derivable | Restored only if its witness companion key still matches the live document's current value; a mismatch means the document changed since the stash was written, so the value is re-derived instead. |
+| `source_parts` | Dataset | shadows_derivable | Restored only if reconstructing it from the live document still agrees with the stashed value (self-verifying — no separate witness key); disagreement re-derives instead. |
+| `connection_name` | Dataset | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `table_name` | Dataset | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `alias` | Dataset | shadows_derivable | Restored only if reconstructing it from the live document still agrees with the stashed value (self-verifying — no separate witness key); disagreement re-derives instead. |
+| `table_properties` | Dataset | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `unsurfaced_columns` | Dataset | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. Each list entry is additionally checked against live coverage before being restored: an entry now covered by a live field is dropped rather than duplicated. |
+| `sql_output_columns` | Dataset | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `on_expression` | Relationship | shadows_derivable | Restored only if its witness companion key still matches the live document's current value; a mismatch means the document changed since the stash was written, so the value is re-derived instead. |
+| `type` | Relationship | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `cardinality` | Relationship | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `endpoints_swapped` | Relationship | shadows_derivable | Restored only if its witness companion key still matches the live document's current value; a mismatch means the document changed since the stash was written, so the value is re-derived instead. |
+| `referencing_join` | Relationship | shadows_derivable | Restored only if reconstructing it from the live document still agrees with the stashed value (self-verifying — no separate witness key); disagreement re-derives instead. |
+| `join_shape` | Relationship | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `unsurfaced_formulas` | Model | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `unattributed_formulas` | Model | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `unrepresentable_joins` | Model | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `model_properties` | Model | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `parameters` | Model | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `filters` | Model | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `column_groups` | Model | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `lesson_plans` | Model | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `action_object_associations` | Model | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `constraints` | Model | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+| `model_joins_with` | Model | information_only | Restored as-is whenever present — nothing on the Ossie side could have diverged from it. |
+
+## Witness companion keys
+
+A `SHADOWS_DERIVABLE` key's stashed value is checked for currency before being restored; these are the witness copies that check does it against (see `stash.restore`).
+
+| Witness key | Checks currency for |
+|---|---|
+| `tml_object_source_witness` | `tml_object` |
+| `data_type_ossie_datatype_witness` | `data_type` |
+| `db_column_name_display_name_witness` | `db_column_name` |
+| `endpoints_swapped_witness` | `endpoints_swapped` |
+| `on_expression_equality_witness` | `on_expression` |
+
+## Nested keys under `source_parts`
+
+| Sub-key | Full path |
+|---|---|
+| `db` | `source_parts.db` |
+| `db_table` | `source_parts.db_table` |
+| `schema` | `source_parts.schema` |
+
+## `METRIC_STASH_SHAPE` value vocabulary
+
+| Constant | Value |
+|---|---|
+| `METRIC_SHAPE_COLUMN_AGGREGATION` | `column_aggregation` |
+| `METRIC_SHAPE_FORMULA` | `formula` |
+| `METRIC_SHAPE_SCALAR_FORMULA_PLUS_AGGREGATION` | `scalar_formula_plus_aggregation` |
+
+## Reclassified on a stash-only carrier
+
+The same key name, reclassified when it is read off a stash-only carrier (an `unrepresentable_joins[]` or `unattributed_formulas[]` entry) that has no independent Relationship/Metric/Field object of its own to diverge from.
+
+| Key | Classification (primary carrier) | Classification (stash-only carrier) |
+|---|---|---|
+| `on_expression` | shadows_derivable | information_only |
+| `type` | information_only | information_only |
+| `cardinality` | information_only | information_only |
+| `column_properties` | information_only | information_only |
+
diff --git a/converters/thoughtspot/pyproject.toml b/converters/thoughtspot/pyproject.toml
new file mode 100644
index 00000000..fe09593a
--- /dev/null
+++ b/converters/thoughtspot/pyproject.toml
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[dependency-groups]
+dev = [
+ "pytest>=8.0",
+ # Dev/test-only: validates convert()'s output against core-spec/ossie-schema.json
+ # in tests/test_tml_to_ossie.py. Guarded there by pytest.importorskip("jsonschema")
+ # so its absence never fails the suite -- the package's only *runtime* dependency
+ # stays PyYAML. Same dev-group placement as converters/nvidia's pyproject.toml.
+ "jsonschema>=4.26.0",
+ # Dev/test-only: drives tests/test_roundtrip_properties.py. Guarded there by
+ # pytest.importorskip("hypothesis"), same discipline as jsonschema above --
+ # the package's only runtime dependency stays PyYAML. Same dev-group
+ # placement as converters/databricks' own pyproject.toml.
+ "hypothesis>=6.0",
+]
+
+[project]
+name = "apache-ossie-thoughtspot"
+version = "0.1.0"
+description = "Convert between ThoughtSpot TML and the Apache Ossie semantic model"
+authors = [{ name = "Apache Software Foundation", email = "dev@ossie.apache.org" }]
+requires-python = ">=3.10"
+readme = "README.md"
+license = "Apache-2.0"
+keywords = [
+ "Apache Ossie",
+ "Ossie",
+ "ThoughtSpot"
+]
+dependencies = [
+ "PyYAML>=6.0",
+]
+
+[project.scripts]
+ossie-thoughtspot = "ossie_thoughtspot.cli:main"
+
+[project.urls]
+homepage = "https://ossie.apache.org/"
+repository = "https://github.com/apache/ossie/"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/ossie_thoughtspot"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.uv]
+required-version = ">=0.9.0"
+default-groups = [
+ "dev"
+]
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/__init__.py b/converters/thoughtspot/src/ossie_thoughtspot/__init__.py
new file mode 100644
index 00000000..0c7aba10
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/__init__.py
@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Bidirectional converter between ThoughtSpot TML and the Apache Ossie semantic model."""
+
+__version__ = "0.1.0"
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/_yaml.py b/converters/thoughtspot/src/ossie_thoughtspot/_yaml.py
new file mode 100644
index 00000000..72c2aaa4
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/_yaml.py
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""YAML codec with the boolean resolver narrowed to YAML 1.2.
+
+PyYAML implements YAML 1.1, in which `on`, `off`, `yes`, `no`, `y` and `n`
+resolve to booleans. TML uses such tokens as ordinary strings, so a bare
+`yaml.safe_load` corrupts them silently. Only the boolean resolver is
+narrowed here — no other YAML 1.1/1.2 divergence (e.g. octal/sexagesimal
+number parsing) is addressed.
+
+Both directions matter. The loader stops 1.1 bool tokens becoming booleans; the
+dumper quotes them on the way out so the next reader — which may be a 1.1
+implementation — cannot re-resolve them.
+"""
+import re
+
+import yaml
+
+from .errors import ConversionError
+
+#: YAML 1.2 core schema: only these spellings are booleans.
+_YAML12_BOOL = re.compile(r"^(?:true|True|TRUE|false|False|FALSE)$")
+
+#: Bare scalars YAML 1.1 resolves as booleans and YAML 1.2 does not.
+_YAML11_ONLY_BOOLS = frozenset(
+ {"y", "Y", "yes", "Yes", "YES", "n", "N", "no", "No", "NO",
+ "on", "On", "ON", "off", "Off", "OFF"}
+)
+
+
+class Yaml12Loader(yaml.SafeLoader):
+ """SafeLoader with the YAML 1.1 boolean resolver narrowed to the 1.2 set."""
+
+
+# Drop the inherited bool resolver outright, then reinstate the 1.2-only one.
+# Mutating in place would affect SafeLoader itself, so rebuild the mapping.
+Yaml12Loader.yaml_implicit_resolvers = {
+ key: [(tag, regexp) for tag, regexp in resolvers if tag != "tag:yaml.org,2002:bool"]
+ for key, resolvers in yaml.SafeLoader.yaml_implicit_resolvers.items()
+}
+Yaml12Loader.add_implicit_resolver("tag:yaml.org,2002:bool", _YAML12_BOOL, list("tTfF"))
+
+
+class Yaml12Dumper(yaml.SafeDumper):
+ """SafeDumper that quotes strings a YAML 1.1 reader would take for booleans."""
+
+
+def _represent_str(dumper: yaml.SafeDumper, data: str) -> yaml.ScalarNode:
+ style = "'" if data in _YAML11_ONLY_BOOLS else None
+ return dumper.represent_scalar("tag:yaml.org,2002:str", data, style=style)
+
+
+Yaml12Dumper.add_representer(str, _represent_str)
+
+
+def load(text: str) -> object:
+ """Parse YAML text under YAML 1.2 boolean rules.
+
+ A malformed document raises `ConversionError` naming the failure, never a
+ bare `yaml.YAMLError` traceback — the same never-a-bare-traceback contract
+ `stash.py` holds for malformed `custom_extensions` JSON.
+ """
+ try:
+ return yaml.load(text, Loader=Yaml12Loader)
+ except yaml.YAMLError as exc:
+ raise ConversionError(f"malformed YAML: {exc}") from exc
+
+
+def dump(data: object) -> str:
+ """Serialise to YAML, preserving insertion order and quoting 1.1 bool tokens.
+
+ `allow_unicode=True` so a non-ASCII value (e.g. a display label) emits as
+ a literal character rather than a `\\xXX`/`\\uXXXX` escape — Ossie
+ documents are human-read YAML.
+ """
+ return yaml.dump(
+ data,
+ Dumper=Yaml12Dumper,
+ sort_keys=False,
+ default_flow_style=False,
+ allow_unicode=True,
+ )
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/cli.py b/converters/thoughtspot/src/ossie_thoughtspot/cli.py
new file mode 100644
index 00000000..3cd09cc5
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/cli.py
@@ -0,0 +1,283 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Command-line interface for the Apache Ossie <-> ThoughtSpot TML converter.
+
+ ossie-thoughtspot to-ossie <tml-file>... -o <out.yaml> [--issues <issues.json>]
+ ossie-thoughtspot to-tml <ossie.yaml> -o <out-dir> [--issues <issues.json>]
+
+`to-ossie` reads a Model TML document plus the Table/SQL View documents it references
+and writes one Apache Ossie semantic model. `to-tml` reads one Apache Ossie semantic
+model and writes the corresponding TML document set -- one file per document, tables
+before the model -- into an output directory.
+
+`-o`/`--output` is required in both directions: `to-ossie` writes exactly one file, and
+`to-tml` writes a set of files that has no single-file stdout representation, so unlike
+some sibling converters there is no "default: stdout" fallback here.
+
+Every declared loss or degradation the conversion records is written as a JSON array of
+issues -- to `--issues` when given, to stderr otherwise -- and never mixed into the
+document output. The process exits 1 when that issue log contains an ERROR-severity
+issue, 0 otherwise: a conversion that only warned or informed about a declared loss is
+still a successful conversion, and failing the exit code on a warning would just teach
+scripts to ignore it.
+
+Neither subcommand overwrites an existing output file unless `--force` is given; without
+it, a target that already exists refuses the run before anything is written.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from . import _yaml, tml, tml_to_ossie, ossie_to_thoughtspot
+from .errors import ConversionError
+from .issues import IssueLog
+
+
+def _build_parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser(
+ prog="ossie-thoughtspot",
+ description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ )
+ sub = parser.add_subparsers(dest="command")
+ sub.required = True # set as attribute (the add_subparsers kwarg is 3.7+)
+
+ to_ossie = sub.add_parser(
+ "to-ossie", help="ThoughtSpot TML documents -> one Apache Ossie semantic model"
+ )
+ to_ossie.add_argument(
+ "tml_files",
+ nargs="+",
+ metavar="TML_FILE",
+ help="the Model TML document plus every Table/SQL View document it references",
+ )
+ to_ossie.add_argument(
+ "-o", "--output", required=True, metavar="FILE", help="output Apache Ossie YAML file"
+ )
+ to_ossie.add_argument(
+ "--issues",
+ metavar="FILE",
+ help="write the issue log as JSON to this path (default: stderr)",
+ )
+ to_ossie.add_argument(
+ "--force",
+ action="store_true",
+ help="overwrite the output file (and --issues file) if either already exists; "
+ "without this flag, an existing target refuses the run before writing anything",
+ )
+
+ to_tml = sub.add_parser(
+ "to-tml", help="one Apache Ossie semantic model -> ThoughtSpot TML documents"
+ )
+ to_tml.add_argument("ossie_file", metavar="OSSIE_FILE", help="Apache Ossie YAML document")
+ to_tml.add_argument(
+ "-o",
+ "--output",
+ required=True,
+ metavar="DIR",
+ help="output directory; one TML file per document is written here (tables before "
+ "the model), created if it does not already exist",
+ )
+ to_tml.add_argument(
+ "--issues",
+ metavar="FILE",
+ help="write the issue log as JSON to this path (default: stderr)",
+ )
+ to_tml.add_argument(
+ "--force",
+ action="store_true",
+ help="overwrite output files (and --issues file) if any already exist; without "
+ "this flag, an existing target refuses the run before writing anything",
+ )
+ return parser
+
+
+def _safe_target_path(directory: Path, filename: str) -> Path:
+ """`directory / filename`, refusing to resolve outside `directory`.
+
+ `tml.dump_document_set` already sanitises `filename` (a document's `name` is
+ user-controlled TML content, and a hostile one -- `../../etc/passwd` -- previously
+ produced a path that escaped its output directory), so this should never trigger in
+ practice. It exists anyway because this module is the first caller that actually
+ writes these documents to disk, and a defence that lives only in the dumper is not
+ one this module can verify it still has. Both paths are resolved before comparing --
+ a naive string-prefix check is wrong on a filesystem with symlinks in play, e.g.
+ macOS where `/tmp` is a symlink to `/private/tmp`.
+ """
+ resolved_directory = directory.resolve()
+ target = (resolved_directory / filename).resolve()
+ # `==`, not `!=`: a filename resolving to the output directory ITSELF is the
+ # case this guard exists to catch, and the old `target != resolved_directory
+ # and ...` spelling made that case short-circuit to False and pass.
+ if target == resolved_directory or resolved_directory not in target.parents:
+ raise ConversionError(
+ f"refusing to write {filename!r}: it resolves outside the output directory "
+ f"{resolved_directory}"
+ )
+ return target
+
+
+def _colliding(paths: list[Path]) -> list[Path]:
+ """Any path this run would write more than once.
+
+ Compared after `resolve()`, because two spellings of one file (a relative
+ path and an absolute one, a symlinked directory) are the same file on disk.
+ Checked BEFORE any write: `--issues` pointing at `-o` passed the
+ does-it-already-exist check on a fresh run, and then `_write_issues`
+ replaced the converted document with the JSON issue log and the command
+ exited 0. `--force` does not license this -- it permits overwriting files
+ that were already there, not destroying one of this run's own outputs.
+ """
+ # Case-folded as well as resolved. `Path.resolve()` does not fold case, and
+ # the default filesystem on macOS and Windows does -- so `-o out.yaml
+ # --issues OUT.YAML` named one file, slipped past this very guard, and the
+ # issue log overwrote the converted document with exit 0. `tml.py` already
+ # folds for exactly this reason; this guard was written a day later and did
+ # not, which is how the defect it exists to prevent survived its own fix.
+ seen: dict[str, Path] = {}
+ counts: dict[str, int] = {}
+ for path in paths:
+ key = str(path.resolve()).casefold()
+ seen.setdefault(key, path.resolve())
+ counts[key] = counts.get(key, 0) + 1
+ return sorted(seen[key] for key, count in counts.items() if count > 1)
+
+
+def _refuse_collision(paths: list[Path]) -> str:
+ names = ", ".join(str(p) for p in paths)
+ return (
+ f"refusing to write the same path twice in one run: {names}. "
+ f"--issues must name a different file from the converted output."
+ )
+
+
+def _existing(paths: list[Path]) -> list[Path]:
+ return [p for p in paths if p.exists()]
+
+
+def _refuse_overwrite(paths: list[Path]) -> str:
+ names = ", ".join(str(p) for p in paths)
+ return f"refusing to overwrite existing file(s): {names} (pass --force to overwrite)"
+
+
+def _write_issues(issues: IssueLog, issues_path: Path | None) -> None:
+ """Issues as a JSON array -- to `issues_path` when given, to stderr otherwise.
+
+ Always written, even when empty: a caller scripting against this output should be
+ able to rely on the shape (a JSON array) rather than on whether anything was said.
+ """
+ text = json.dumps(issues.as_dicts(), indent=2)
+ if issues_path is not None:
+ issues_path.parent.mkdir(parents=True, exist_ok=True)
+ issues_path.write_text(text + "\n", encoding="utf-8")
+ else:
+ print(text, file=sys.stderr)
+
+
+def _cmd_to_ossie(args: argparse.Namespace) -> int:
+ output_path = Path(args.output)
+ issues_path = Path(args.issues) if args.issues else None
+
+ try:
+ texts = [(path, Path(path).read_text(encoding="utf-8")) for path in args.tml_files]
+ document_set = tml.load_document_set(texts)
+ result = tml_to_ossie.convert(document_set)
+ except (ConversionError, OSError, UnicodeDecodeError) as e:
+ print(f"Error: {e}", file=sys.stderr)
+ return 1
+
+ targets = [output_path] + ([issues_path] if issues_path else [])
+ collisions = _colliding(targets)
+ if collisions:
+ print(f"Error: {_refuse_collision(collisions)}", file=sys.stderr)
+ return 1
+ if not args.force:
+ existing = _existing(targets)
+ if existing:
+ print(f"Error: {_refuse_overwrite(existing)}", file=sys.stderr)
+ return 1
+
+ try:
+ output_path.parent.mkdir(parents=True, exist_ok=True)
+ output_path.write_text(_yaml.dump(result.model), encoding="utf-8")
+ _write_issues(result.issues, issues_path)
+ except OSError as e:
+ print(f"Error: {e}", file=sys.stderr)
+ return 1
+
+ return 1 if result.issues.has_errors() else 0
+
+
+def _cmd_to_tml(args: argparse.Namespace) -> int:
+ output_dir = Path(args.output)
+ issues_path = Path(args.issues) if args.issues else None
+
+ if output_dir.exists() and not output_dir.is_dir():
+ print(f"Error: output path {output_dir} exists and is not a directory", file=sys.stderr)
+ return 1
+
+ try:
+ text = Path(args.ossie_file).read_text(encoding="utf-8")
+ ossie_document = _yaml.load(text)
+ if not isinstance(ossie_document, dict):
+ raise ConversionError(f"{args.ossie_file} is not an Ossie document: expected a mapping")
+ result = ossie_to_thoughtspot.convert(ossie_document)
+ files = tml.dump_document_set(result.documents)
+ # Path safety belongs here, before any write: a document name is user-controlled
+ # TML/Ossie content, and `dump_document_set` sanitises filenames for exactly this
+ # reason (see `_safe_target_path`).
+ targets = [(_safe_target_path(output_dir, name), text_) for name, text_ in files]
+ except (ConversionError, OSError, UnicodeDecodeError) as e:
+ print(f"Error: {e}", file=sys.stderr)
+ return 1
+
+ all_targets = [path for path, _ in targets] + ([issues_path] if issues_path else [])
+ collisions = _colliding(all_targets)
+ if collisions:
+ print(f"Error: {_refuse_collision(collisions)}", file=sys.stderr)
+ return 1
+ if not args.force:
+ existing = _existing(all_targets)
+ if existing:
+ print(f"Error: {_refuse_overwrite(existing)}", file=sys.stderr)
+ return 1
+
+ try:
+ output_dir.mkdir(parents=True, exist_ok=True)
+ for path, text_ in targets:
+ path.write_text(text_, encoding="utf-8")
+ _write_issues(result.issues, issues_path)
+ except OSError as e:
+ print(f"Error: {e}", file=sys.stderr)
+ return 1
+
+ return 1 if result.issues.has_errors() else 0
+
+
+def main(argv: list[str] | None = None) -> int:
+ args = _build_parser().parse_args(argv)
+ if args.command == "to-ossie":
+ return _cmd_to_ossie(args)
+ return _cmd_to_tml(args)
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/constants.py b/converters/thoughtspot/src/ossie_thoughtspot/constants.py
new file mode 100644
index 00000000..6a00e57f
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/constants.py
@@ -0,0 +1,628 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Vocabulary constants.
+
+VENDOR_KEY and DIALECT hold the same string today and are deliberately separate
+names. They are governed differently upstream: the vendor key needs no spec
+change because `Vendor` is an `examples` list that accepts any string, while
+the dialect is a closed enum — it was a pending apache/ossie#351 change, now
+merged (see DIALECT_IS_REGISTERED).
+"""
+
+from enum import Enum
+
+#: `custom_extensions[].vendor_name` value for ThoughtSpot-owned entries.
+VENDOR_KEY = "THOUGHTSPOT"
+
+#: Expression-language dialect label. A registered member of the Ossie Dialect enum.
+DIALECT = "THOUGHTSPOT"
+
+#: apache/ossie#351 merged 2026-09-01: THOUGHTSPOT is now in the closed `Dialect`
+#: enum (core-spec/ossie-schema.json) and in validation/validate.py's
+#: SKIP_SQL_VALIDATION set, so emitting DIALECT no longer fails schema validation.
+DIALECT_IS_REGISTERED = True
+
+#: Dialect emitted alongside DIALECT (not instead of it) for portable expressions, so a
+#: consumer that does not implement our dialect still gets something it can execute.
+PORTABLE_DIALECT = "ANSI_SQL"
+
+#: Ossie spec series this converter targets, matched on major.minor. Not an exact
+#: version: upstream's first release is proposed as 0.3.0, not 0.2.0.
+SPEC_SERIES = "0.2"
+
+#: The exact `version` this converter writes at the root of every document it
+#: emits (`{"version": DOCUMENT_VERSION, "name": ..., "datasets": [...]}`). Unlike
+#: SPEC_SERIES (major.minor, used to check an *incoming* document's rough
+#: compatibility), ossie-schema.json pins `version` to this exact string as a
+#: `const` (`ossie-schema.json:8-12`), so a document that emits anything else
+#: fails schema validation outright. Bump in lockstep with core-spec/'s own
+#: `version` if it ever moves -- the same discipline converters/databricks'
+#: `OSSIE_VERSION` constant documents.
+DOCUMENT_VERSION = "0.2.0.dev0"
+
+#: ThoughtSpot's grouped-aggregation calls, mapped to the plain aggregate each
+#: is shorthand for. `group_aggregate` is the general form and has no single
+#: aggregate, so it is named separately below.
+#:
+#: Here rather than beside the reverse inventory that first recorded them,
+#: because BOTH directions need them and for different jobs: the reverse
+#: inventory composes them into windowed SQL, and `tml_to_ossie` must recognise
+#: them as *already aggregated* so it does not compose a metric's own
+#: `aggregation` on top. When only the inventory knew them, the forward
+#: direction emitted `sum ( group_sum ( ... ) )` -- silently doubled.
+GROUP_SHORTHAND_AGGREGATES = {
+ "group_sum": "SUM",
+ "group_count": "COUNT",
+ "group_stddev": "STDDEV",
+ "group_variance": "VARIANCE",
+ # The four below were missing while this map served only the reverse
+ # inventory, whose rule is "name nothing the mapping document does not".
+ # That rule is right for traceability and wrong for the double-aggregation
+ # guard, whose source of truth is ThoughtSpot's own function list: a
+ # `group_max(...)` formula was classified as a scalar and had its column
+ # `aggregation` composed on top, emitting `sum ( group_max ( ... ) )`.
+ "group_max": "MAX",
+ "group_min": "MIN",
+ "group_average": "AVG",
+ "group_unique_count": "COUNT_DISTINCT",
+}
+
+#: Every grouped-aggregation call name: the shorthands plus the general form.
+GROUP_AGGREGATE_CALL_NAMES = frozenset({"group_aggregate", *GROUP_SHORTHAND_AGGREGATES})
+
+#: Shape version of the custom_extensions payload. Bump when the
+#: payload's shape changes, never for a value change.
+STASH_VERSION = 1
+
+#: Field/metric-level custom_extensions[THOUGHTSPOT] key holding the
+#: warehouse column's own name, stashed by the TML -> Ossie direction only
+#: when it differs from the column's display name (Table-backed columns
+#: only -- a SQL View's own sql_output_columns dataset-level key already
+#: covers the same fact). Not yet in the pinned payload schema. Shared here,
+#: rather than written as a literal in each direction separately, because
+#: tml_to_ossie.py (the writer) and ossie_to_thoughtspot.py (the reader)
+#: must agree on the exact spelling and nothing else enforces that.
+FIELD_STASH_DB_COLUMN_NAME = "db_column_name"
+
+#: Formulas present in `model.formulas[]` that NO `columns[]` entry surfaces --
+#: internal helpers, referenced by other formulas but not exposed to users. A
+#: date-parameter model is the common case: `_startDate`, `_endDate` and the
+#: like compute a window that the visible formulas then use.
+#:
+#: They were dropped outright: the converter walks `columns[]`, so a formula no
+#: column surfaces is never visited at all, and even the unattributed-formula
+#: stash only catches ones that WERE visited and could not be attributed. In one
+#: real model that was 32 of 41 formulas -- and the 9 visible ones that
+#: referenced them came back with dangling `[formula__startDate]` references, so
+#: the emitted document would not import.
+#:
+#: Stashed with their `id`, because that is what the surviving references name.
+#: Restored as `formulas[]` entries with NO surfacing column, which is what they
+#: were: giving them one would make an internal helper user-visible.
+MODEL_STASH_UNSURFACED_FORMULAS = "unsurfaced_formulas"
+
+#: A surfacing column's `aggregation` that is LOAD-BEARING and has no home in
+#: the Ossie metric's own expression, so it is preserved verbatim instead.
+#:
+#: The one shape this is for: a formula whose outer call is a BARE
+#: `group_aggregate ( ... )`. Every other already-aggregating shape makes the
+#: column's aggregation a genuine no-op -- `sum ( ... )`, and the
+#: `group_sum`/`group_average` shorthands, all behave that way (verified on a
+#: live cluster: a model carrying `sum([SALES])` WITH `aggregation: SUM`
+#: returns the same number as the source). A bare `group_aggregate` does not:
+#: like a raw column, ThoughtSpot may APPLY the column aggregation to it. It
+#: was being discarded with the others and with nothing logged, which silently
+#: changes the number.
+METRIC_STASH_COLUMN_AGGREGATION = "column_aggregation_value"
+
+#: The source TML `obj_id` -- ThoughtSpot's own PORTABLE object handle, e.g.
+#: `SampleRetail-Apparel-LH-58435d2b` (display name, then the first segment of
+#: the GUID). Stashed under a distinct payload key so the forbidden-key scan,
+#: which looks for a literal `obj_id`, does not mistake this deliberate copy for
+#: leaked identity.
+#:
+#: Why this is kept when `guid` and `fqn` are not, though all three were once
+#: grouped as "instance-local identity": they are not the same kind of thing.
+#: `guid` is a raw cluster UUID and `fqn` is a reference to one -- and a
+#: viz-level `fqn` is DROPPED on import, leaving the object with no data source.
+#: `obj_id` is the identifier ThoughtSpot introduced precisely so objects can be
+#: referenced across environments, and it survives import. Discarding it meant a
+#: converted model re-imported as a NEW object rather than updating the one it
+#: came from, which is the ordinary promote-between-environments workflow.
+MODEL_STASH_OBJ_ID = "tml_obj_id"
+
+#: The source TML `formulas[].id` of a computed field or metric, stashed
+#: verbatim. TML's `formulas[].id` and `formulas[].name` are INDEPENDENT -- a
+#: formula renamed after creation keeps its original id -- but the Ossie -> TML
+#: leg mints ids from display names and so resolved a verbatim `[formula_X]`
+#: cross-reference by matching X against normalised NAMES. When some other
+#: formula's name normalised to the referenced id's tail, the reference
+#: silently followed it: a metric defined as `(sum(a) - sum(b)) + 1` came back
+#: as `(sum(a) * 0.5) + 1`, exit 0, nothing logged. Preserving the id is what
+#: makes the binding survive; matching on names can only ever approximate it.
+FIELD_STASH_FORMULA_ID = "formula_id"
+
+#: The witness copy for FIELD_STASH_DB_COLUMN_NAME: the physical column's
+#: own display name (the bracket's column part, e.g. "Amount") as it stood
+#: the moment db_column_name was stashed. Ossie -> TML compares this against
+#: the CURRENT bracket reference's column part: agreement means nobody
+#: retargeted the field to a different physical column since, so the
+#: stashed warehouse name is still trustworthy; disagreement means the
+#: field now names a different column and the stashed warehouse name
+#: describes the wrong one -- it is dropped rather than misapplied to the
+#: new column.
+FIELD_STASH_DB_COLUMN_NAME_WITNESS = "db_column_name_display_name_witness"
+
+# ---------------------------------------------------------------------------
+# The rest of the custom_extensions[THOUGHTSPOT] payload vocabulary.
+#
+# Every name below is a key of the JSON object stash.write_stash serialises
+# and stash.read_stash parses back -- the same channel FIELD_STASH_DB_COLUMN_NAME
+# above already covers for one key. tml_to_ossie.py (the writer) and
+# ossie_to_thoughtspot.py (the reader) must agree on each spelling exactly, and
+# nothing but this module enforces that; several of these are, today, written
+# by only one side (tml_to_ossie.py has no Model/Metric-reading counterpart yet
+# in ossie_to_thoughtspot.py) -- they are named here anyway so the reader that
+# is eventually written consumes the same literal, not a freshly retyped guess.
+#
+# Grouped by which Ossie object each key's custom_extensions entry attaches to
+# -- Model, Dataset, Relationship, or Field/Metric -- because that grouping is
+# itself part of the payload's schema (see the design doc's SemanticModelLevel /
+# DatasetLevel / RelationshipLevel / FieldLevel / MetricLevel $defs).
+# ---------------------------------------------------------------------------
+
+#: Exact ThoughtSpot display name, stashed whenever identifier normalisation produced
+#: a different Ossie identifier. Shared across every scope that can suffer
+#: this divergence: Model (the document root's `name`) and Metric (a Metric has no
+#: `label` field to carry the display name the way a Field does). Also
+#: defensively checked at Dataset scope by `_table_name` in
+#: ossie_to_thoughtspot.py -- but a Dataset's own `name` is the verbatim
+#: model_tables[] alias-or-name and is never run through normalisation, so
+#: nothing writes this key there today; that check is symmetry with the other
+#: two scopes, not a reachable path.
+STASH_TML_NAME = "tml_name"
+
+# --- Model scope (attached to the document root) ---------------------------
+
+#: Formula-backed ATTRIBUTE columns whose references span two or more
+#: datasets, so no single Ossie dataset can own the field. Preserved verbatim
+#: (each entry carries at least `name` and `expr`) alongside an issue.
+MODEL_STASH_UNATTRIBUTED_FORMULAS = "unattributed_formulas"
+
+#: Joins with no equality pair at all (a pure range or pure constant
+#: condition), which cannot become a Relationship because Ossie's
+#: `from_columns`/`to_columns` are required and non-empty. Preserved
+#: verbatim, alongside an issue. Each entry reuses the RELATIONSHIP_STASH_*
+#: keys below for the facts a real Relationship's own custom_extensions
+#: entry would have carried, since it describes the same kind of TML join.
+MODEL_STASH_UNREPRESENTABLE_JOINS = "unrepresentable_joins"
+
+#: ThoughtSpot Model-only properties with no Ossie equivalent
+#: (`is_bypass_rls`, `join_progressive`, `spotter_config.is_spotter_enabled`),
+#: copied verbatim under their own TML property names.
+MODEL_STASH_MODEL_PROPERTIES = "model_properties"
+
+#: Verbatim `model.parameters[]`. No Ossie equivalent; formulas referencing
+#: them are not portable.
+MODEL_STASH_PARAMETERS = "parameters"
+
+#: Verbatim `model.filters[]`.
+MODEL_STASH_FILTERS = "filters"
+
+#: Verbatim `model.column_groups[]` -- the search-bar data-panel folder structure.
+MODEL_STASH_COLUMN_GROUPS = "column_groups"
+
+#: Verbatim `model.lesson_plans[]` -- the in-product guided-lesson strings
+#: attached to a Model. No Ossie equivalent.
+MODEL_STASH_LESSON_PLANS = "lesson_plans"
+
+#: Verbatim `model.action_object_associations[]` -- custom actions bound to
+#: the Model by display name only.
+MODEL_STASH_ACTION_OBJECT_ASSOCIATIONS = "action_object_associations"
+
+#: Verbatim `model.constraints` block (rolling date-window conditions per table).
+MODEL_STASH_CONSTRAINTS = "constraints"
+
+#: Verbatim Model-level `joins_with[]` data-augmentation joins. Named
+#: "model_" rather than reusing the bare TML key `joins_with` because a
+#: *Table* document has its own, differently-scoped `joins_with[]`
+#: (referencing-join definitions) -- the two must not collide under one
+#: stash key.
+MODEL_STASH_MODEL_JOINS_WITH = "model_joins_with"
+
+# --- Dataset scope (attached to a `datasets[]` entry) -----------------------
+
+#: Whether the source TML document was a `table:` or a `sql_view:` --
+#: authoritative over `_derive_kind`'s own whitespace/dotted-identifier
+#: heuristic whenever present, since it also determines which shape
+#: `unsurfaced_columns` was captured in.
+DATASET_STASH_TML_OBJECT = "tml_object"
+
+#: The witness copy for DATASET_STASH_TML_OBJECT: the dataset's own
+#: `source` string as it stood the moment `tml_object` was stashed.
+#: Ossie -> TML compares this against the CURRENT `source`: agreement means
+#: nobody edited it since (a query rewritten as a table reference, or vice
+#: versa), so the stashed kind is still trustworthy; disagreement means the
+#: stash is stale and `_derive_kind` re-guesses from the current `source`
+#: instead of trusting a kind that used to describe a different value.
+DATASET_STASH_TML_OBJECT_WITNESS = "tml_object_source_witness"
+
+#: `model_tables[].alias`, when one physical table participates more than once.
+DATASET_STASH_ALIAS = "alias"
+
+#: The underlying table object name that `alias` (above) aliases.
+DATASET_STASH_TABLE_NAME = "table_name"
+
+#: ThoughtSpot Connection display name (case-sensitive, never a GUID).
+#: Required to emit a Table document; when absent it must be supplied by the
+#: caller as `build_table`'s own `connection_name` argument.
+DATASET_STASH_CONNECTION_NAME = "connection_name"
+
+#: `db`/`schema`/`db_table` recorded individually when the dotted `source`
+#: form would be ambiguous. A nested object; see the three keys below for its
+#: own contents.
+DATASET_STASH_SOURCE_PARTS = "source_parts"
+
+#: `source_parts.db`.
+DATASET_STASH_SOURCE_PARTS_DB = "db"
+
+#: `source_parts.schema`.
+DATASET_STASH_SOURCE_PARTS_SCHEMA = "schema"
+
+#: `source_parts.db_table`.
+DATASET_STASH_SOURCE_PARTS_DB_TABLE = "db_table"
+
+#: Verbatim Table/SQL-View physical-column entries the Model does not
+#: surface. Not semantic model content, but required to regenerate the
+#: source document exactly.
+DATASET_STASH_UNSURFACED_COLUMNS = "unsurfaced_columns"
+
+#: `{Ossie field name: sql_output_column alias}`, for every surfaced SQL
+#: View column -- there is no safe way to re-derive a query output alias
+#: from an Ossie field's own identifier the way a Table's db_column_name
+#: might be guessed at.
+DATASET_STASH_SQL_OUTPUT_COLUMNS = "sql_output_columns"
+
+#: ThoughtSpot Table-only properties with no Ossie equivalent (mirrors
+#: MODEL_STASH_MODEL_PROPERTIES's `spotter_config` shape, at Dataset scope).
+DATASET_STASH_TABLE_PROPERTIES = "table_properties"
+
+# --- Relationship scope (attached to a `relationships[]` entry) ------------
+#
+# Also reused, unchanged, inside MODEL_STASH_UNREPRESENTABLE_JOINS entries --
+# a join that could not become a Relationship at all still needs the same
+# facts recorded, under the same names, because it is the same kind of TML
+# join fact either way.
+
+#: ThoughtSpot's join-type vocabulary (`INNER`, `LEFT_OUTER`, `RIGHT_OUTER`,
+#: `OUTER`), identical in a Model inline join and a Table `joins_with[]` entry.
+RELATIONSHIP_STASH_TYPE = "type"
+
+#: ThoughtSpot's join cardinality (`MANY_TO_ONE`, `ONE_TO_ONE`, `ONE_TO_MANY`,
+#: `MANY_TO_MANY`). Always the exact TML value, verbatim, regardless of
+#: whether `RELATIONSHIP_STASH_ENDPOINTS_SWAPPED` (below) also fired for this
+#: relationship -- the two facts are independent: this one is never stale
+#: (TML's cardinality has no Ossie-side counterpart to disagree with), while
+#: whether the endpoint swap it triggered is still trustworthy is a separate,
+#: witnessed question.
+RELATIONSHIP_STASH_CARDINALITY = "cardinality"
+
+#: Whether this relationship's `from`/`to`/`from_columns`/`to_columns` were
+#: swapped relative to TML's own declared join direction. core-spec/spec.yaml
+#: requires a Relationship's `from` to name the many side and `to` the one
+#: side, but TML's `from`/`to` (the model_tables[] entry a join is declared
+#: under, and its `with`/`destination` target) do not themselves encode which
+#: side is which -- `cardinality` does. Only a `ONE_TO_MANY` join has TML's
+#: `from` naming the one side and `to` naming the many side -- the wrong way
+#: around for Ossie's spec -- so only that cardinality ever sets this `True`
+#: and swaps the emitted relationship's endpoints to compensate. `MANY_TO_ONE`
+#: and `ONE_TO_ONE` are already oriented correctly and never set it.
+RELATIONSHIP_STASH_ENDPOINTS_SWAPPED = "endpoints_swapped"
+
+#: The witness copy for RELATIONSHIP_STASH_ENDPOINTS_SWAPPED: `[from, to,
+#: from_columns, to_columns]` exactly as emitted -- i.e. already swapped --
+#: at the moment the marker was stashed. `Ossie -> TML` compares this against
+#: the relationship's CURRENT `from`/`to`/`from_columns`/`to_columns`:
+#: agreement means nobody retargeted the relationship since, so it is safe to
+#: undo the swap and recover the TML join's original `from`/`to`/columns
+#: (and, with them, which dataset's `model_tables[]` entry the join is
+#: nested under); disagreement means the relationship was edited since the
+#: stash was written, so the swap is not undone -- the live shape is trusted
+#: instead, exactly as a hand-authored relationship with no stash at all
+#: would be -- and an issue records it.
+RELATIONSHIP_STASH_ENDPOINTS_SWAPPED_WITNESS = "endpoints_swapped_witness"
+
+#: Which TML join shape produced this relationship -- `"referencing"` (a
+#: named Table `joins_with[]` entry the Model points at), `"inline"` (defined
+#: directly in `model_tables[].joins[]`), or `"referencing_with_inline_attrs"`
+#: (both: a `referencing_join` plus a `type`/`cardinality` override).
+RELATIONSHIP_STASH_JOIN_SHAPE = "join_shape"
+
+#: The Table `joins_with[]` entry name, when `join_shape` is `"referencing"`
+#: (or the hybrid).
+RELATIONSHIP_STASH_REFERENCING_JOIN = "referencing_join"
+
+#: The verbatim join condition. Required whenever the condition is not a
+#: pure equality (range / ASOF / constant joins), because `from_columns`/
+#: `to_columns` then carry only part of it.
+RELATIONSHIP_STASH_ON_EXPRESSION = "on_expression"
+
+#: The witness copy for `RELATIONSHIP_STASH_ON_EXPRESSION`: `[from_columns,
+#: to_columns]` exactly as they stood the moment `on_expression` was
+#: stashed (only ever written alongside it, i.e. only when residual
+#: predicates exist). `Ossie -> TML` compares this against the relationship's
+#: CURRENT `from_columns`/`to_columns` -- agreement means nobody retargeted
+#: the relationship since the stash was written, so the verbatim
+#: `on_expression` (and the residual narrowing it carries) is still current
+#: and is restored; disagreement means the stash is stale, so both are
+#: dropped and the plain equality condition is re-derived from the live
+#: from_columns/to_columns instead, with an issue recording it. This is one of
+#: the FIVE witness copies this converter keeps -- the others being
+#: FIELD_STASH_DB_COLUMN_NAME_WITNESS, DATASET_STASH_TML_OBJECT_WITNESS,
+#: RELATIONSHIP_STASH_ENDPOINTS_SWAPPED_WITNESS and FIELD_STASH_DATA_TYPE_WITNESS.
+#: What this one witnesses: "a relationship's verbatim on_expression".
+RELATIONSHIP_STASH_ON_EXPRESSION_WITNESS = "on_expression_equality_witness"
+
+# --- Field/metric scope (attached to a `fields[]` or `metrics[]` entry) -----
+#
+# FIELD_STASH_DB_COLUMN_NAME above is the original of this group; the rest
+# follow its naming even though, like it, they are written for a Metric just
+# as often as for a Field -- `_physical_column_stash` and
+# `_unconsumed_properties` in tml_to_ossie.py build both keys identically
+# regardless of which of the two the caller is converting.
+
+#: The exact ThoughtSpot `db_column_properties.data_type` spelling, recorded
+#: only when it is not the canonical spelling `datatypes.to_tml` would emit
+#: by default for the Ossie datatype (`BOOLEAN` vs `BOOL`, `FLOAT` vs
+#: `DOUBLE`), so the return trip re-emits the same one.
+FIELD_STASH_DATA_TYPE = "data_type"
+
+#: The witness copy for FIELD_STASH_DATA_TYPE: the Ossie `datatype` value
+#: (`"Boolean"` or `"Float"` -- the only two `_CANONICAL_TML_SPELLING` ever
+#: stashes a non-canonical spelling for) as it stood the moment the spelling
+#: was recorded. `Ossie -> TML` compares this against the field's CURRENT
+#: `datatype`: agreement means nobody edited the field's declared type since,
+#: so the exact warehouse spelling is still trustworthy and is restored;
+#: disagreement -- the field now declares a different datatype -- means the
+#: spelling is stale (it names a warehouse type for the *old* datatype, not
+#: this one) and is dropped, falling back to the canonical spelling
+#: `datatypes.to_tml` derives for the current value instead.
+FIELD_STASH_DATA_TYPE_WITNESS = "data_type_ossie_datatype_witness"
+
+#: ThoughtSpot column properties this converter did not read and consume
+#: elsewhere -- the fail-closed complement `_unconsumed_properties` builds,
+#: so a future ThoughtSpot-only property this module has never heard of is
+#: preserved rather than silently dropped. Also reused, unchanged, for the
+#: `column_properties` an unattributed formula's own source column carried
+#: (see MODEL_STASH_UNATTRIBUTED_FORMULAS) -- the same "properties this
+#: converter did not otherwise account for" concept, just attached to a
+#: preserved formula instead of a built Field or Metric.
+FIELD_STASH_COLUMN_PROPERTIES = "column_properties"
+
+# --- Metric-only scope -------------------------------------------------------
+
+#: Which of the three TML shapes (`column_aggregation`,
+#: `scalar_formula_plus_aggregation`, `formula`) produced this metric, so a
+#: return trip can reproduce the source shape instead of collapsing all three
+#: into one. `formula` is the default a document with no stash at all
+#: reconstructs as, so it is the one value never written.
+METRIC_STASH_SHAPE = "shape"
+
+#: METRIC_STASH_SHAPE's own value vocabulary -- shared here, not written as a
+#: literal by tml_to_ossie.py (the writer) and re-typed as a literal by
+#: ossie_to_thoughtspot.py (the reader), for the same reason every other name
+#: in this file is centralised: the two must agree on the exact spelling and
+#: nothing else enforces that. `METRIC_SHAPE_FORMULA` is also what a document
+#: with no `shape` stash at all defaults to on the way back -- see
+#: METRIC_STASH_SHAPE above for which shape is emitted by default.
+METRIC_SHAPE_COLUMN_AGGREGATION = "column_aggregation"
+METRIC_SHAPE_SCALAR_FORMULA_PLUS_AGGREGATION = "scalar_formula_plus_aggregation"
+METRIC_SHAPE_FORMULA = "formula"
+
+
+# ---------------------------------------------------------------------------
+# Stash-freshness classification.
+#
+# Every custom_extensions[THOUGHTSPOT] key above answers one question before
+# ossie_to_thoughtspot.py is allowed to read it: does the stashed value
+# shadow something this converter could otherwise derive from the live Ossie
+# document -- a field's own bracket reference, a metric's own name, a
+# dataset's own source -- or is it information that exists nowhere else in
+# the Ossie document at all?
+#
+# The first kind can go stale: a user edits the Ossie document (retargets a
+# field, renames a metric, rewrites a relationship, rewrites a dataset's
+# source) and the stash still describes the document as it was. A key in
+# that category needs a witness and a currency check -- reused only
+# when the two still agree, dropped and re-derived otherwise -- never plain
+# stash-if-present. The second kind cannot go stale, because there is
+# nothing on the Ossie side for it to disagree with; plain stash-if-present
+# is correct there and a witness would have nothing to compare against.
+#
+# STASH_KEY_CLASSIFICATION is the enforcement point. Three different stash
+# keys were found, independently, reading the unsafe way before this table
+# existed (FIELD_STASH_DB_COLUMN_NAME, FIELD_STASH_DATA_TYPE, and a
+# relationship's on_expression) — each found by generalising from the
+# instance before it, not by a rule anyone consulted. This table is that
+# rule, made structural: a key read anywhere in ossie_to_thoughtspot.py that
+# is missing here fails test_stash_key_classification.py, so the next key
+# has to declare an answer rather than default to the unsafe one. It does
+# not, by itself, prove the *code* honours a SHADOWS_DERIVABLE
+# classification with an actual witness -- that is still a review
+# discipline -- but it makes "someone forgot" a build failure instead of a
+# silent gap for a key already known to need one.
+# ---------------------------------------------------------------------------
+
+
+class StashKeyClass(Enum):
+ #: The stashed value could disagree with something the live Ossie
+ #: document itself says. Reading it MUST check currency: via
+ #: `stash.restore`'s witness/witness_key (FIELD_STASH_DB_COLUMN_NAME,
+ #: FIELD_STASH_DATA_TYPE, RELATIONSHIP_STASH_ON_EXPRESSION on a real
+ #: Relationship, DATASET_STASH_TML_OBJECT), or a self-verifying
+ #: reconstruction when the stashed value's own shape lets it check
+ #: itself against the live document with nothing extra stored
+ #: (DATASET_STASH_SOURCE_PARTS re-joins to compare against `source`;
+ #: STASH_TML_NAME re-normalises to compare against the live
+ #: identifier). Either way, a mismatch drops the stash, re-derives, and
+ #: logs why -- never keeps the stale value.
+ SHADOWS_DERIVABLE = "shadows_derivable"
+ #: The stashed value has no Ossie-native counterpart at all -- nothing
+ #: on the Ossie side represents it independently, so nothing there
+ #: could have diverged from it. Plain stash-if-present is correct.
+ INFORMATION_ONLY = "information_only"
+
+
+#: One entry per stash key read anywhere in ossie_to_thoughtspot.py.
+#: RELATIONSHIP_STASH_ON_EXPRESSION appears once, classified for its
+#: primary carrier (a real Relationship, where it shadows
+#: from_columns/to_columns) -- the same key read off a
+#: MODEL_STASH_UNREPRESENTABLE_JOINS entry has no independent Relationship
+#: object to diverge from and would be INFORMATION_ONLY in that context;
+#: see the read site's own docstring, not a second table entry, since the
+#: dict is keyed by string and cannot hold two classifications for one key.
+STASH_KEY_CLASSIFICATION: dict[str, "StashKeyClass"] = {
+ # -- Shared --
+ STASH_TML_NAME: StashKeyClass.SHADOWS_DERIVABLE,
+
+ # -- Field/metric scope --
+ # INFORMATION_ONLY deliberately: Ossie has no formula-id concept, so there
+ # is no live value this can diverge FROM. That is also the point of keeping
+ # it -- TML's id is independent of its name, so renaming the metric in Ossie
+ # must NOT change the id, or every cross-reference written against it breaks.
+ # Ossie has no object-identity concept, so nothing here can diverge from it.
+ # No Ossie counterpart: the metric's expression cannot carry it.
+ METRIC_STASH_COLUMN_AGGREGATION: StashKeyClass.INFORMATION_ONLY,
+ MODEL_STASH_OBJ_ID: StashKeyClass.INFORMATION_ONLY,
+ FIELD_STASH_FORMULA_ID: StashKeyClass.INFORMATION_ONLY,
+ FIELD_STASH_DB_COLUMN_NAME: StashKeyClass.SHADOWS_DERIVABLE,
+ FIELD_STASH_DATA_TYPE: StashKeyClass.SHADOWS_DERIVABLE,
+ FIELD_STASH_COLUMN_PROPERTIES: StashKeyClass.INFORMATION_ONLY,
+ METRIC_STASH_SHAPE: StashKeyClass.INFORMATION_ONLY,
+
+ # -- Dataset scope --
+ DATASET_STASH_TML_OBJECT: StashKeyClass.SHADOWS_DERIVABLE,
+ DATASET_STASH_SOURCE_PARTS: StashKeyClass.SHADOWS_DERIVABLE,
+ DATASET_STASH_CONNECTION_NAME: StashKeyClass.INFORMATION_ONLY,
+ DATASET_STASH_TABLE_NAME: StashKeyClass.INFORMATION_ONLY,
+ # SHADOWS_DERIVABLE, corrected from INFORMATION_ONLY. It does have an Ossie
+ # counterpart: `tml_to_ossie._build_dataset` writes the dataset's `name`
+ # FROM this alias, so the two diverge the moment anyone renames the dataset.
+ # Labelled INFORMATION_ONLY -- "nothing there could have diverged from it" --
+ # it was read without a currency check, and a rename emitted a model whose
+ # model_tables[] alias was stale while every reference used the new name.
+ # Self-verifying: `stashed == dataset["name"]` is the whole check.
+ DATASET_STASH_ALIAS: StashKeyClass.SHADOWS_DERIVABLE,
+ DATASET_STASH_TABLE_PROPERTIES: StashKeyClass.INFORMATION_ONLY,
+ DATASET_STASH_UNSURFACED_COLUMNS: StashKeyClass.INFORMATION_ONLY, # value only -- see STASH_KEYS_WITH_DERIVABLE_MEMBERSHIP below for its membership axis
+ DATASET_STASH_SQL_OUTPUT_COLUMNS: StashKeyClass.INFORMATION_ONLY, # per-field dict lookup, never appended -- checked, does not share unsurfaced_columns' hybrid
+
+ # -- Relationship scope --
+ RELATIONSHIP_STASH_ON_EXPRESSION: StashKeyClass.SHADOWS_DERIVABLE,
+ RELATIONSHIP_STASH_TYPE: StashKeyClass.INFORMATION_ONLY,
+ RELATIONSHIP_STASH_CARDINALITY: StashKeyClass.INFORMATION_ONLY,
+ # Witnessed against [from, to, from_columns, to_columns]: a ONE_TO_MANY
+ # join's endpoint swap is only undone while nothing has retargeted the
+ # relationship since it was stashed.
+ RELATIONSHIP_STASH_ENDPOINTS_SWAPPED: StashKeyClass.SHADOWS_DERIVABLE,
+ # Self-verifying (STASH_TML_NAME's own pattern, nothing extra stored):
+ # written equal to the relationship's own `name` at stash time, so
+ # agreement on read means nobody renamed the relationship since and the
+ # stash is still trustworthy; a mismatch means it was renamed, so the
+ # stashed Table joins_with[] reference is dropped rather than restored
+ # under the wrong, stale name.
+ RELATIONSHIP_STASH_REFERENCING_JOIN: StashKeyClass.SHADOWS_DERIVABLE,
+ # Which TML shape produced this relationship -- purely descriptive, no
+ # live Ossie counterpart to disagree with (mirrors METRIC_STASH_SHAPE).
+ RELATIONSHIP_STASH_JOIN_SHAPE: StashKeyClass.INFORMATION_ONLY,
+
+ # -- Model scope --
+ MODEL_STASH_UNSURFACED_FORMULAS: StashKeyClass.INFORMATION_ONLY,
+ MODEL_STASH_UNATTRIBUTED_FORMULAS: StashKeyClass.INFORMATION_ONLY,
+ MODEL_STASH_UNREPRESENTABLE_JOINS: StashKeyClass.INFORMATION_ONLY,
+ MODEL_STASH_MODEL_PROPERTIES: StashKeyClass.INFORMATION_ONLY,
+ MODEL_STASH_PARAMETERS: StashKeyClass.INFORMATION_ONLY,
+ MODEL_STASH_FILTERS: StashKeyClass.INFORMATION_ONLY,
+ MODEL_STASH_COLUMN_GROUPS: StashKeyClass.INFORMATION_ONLY,
+ MODEL_STASH_LESSON_PLANS: StashKeyClass.INFORMATION_ONLY,
+ MODEL_STASH_ACTION_OBJECT_ASSOCIATIONS: StashKeyClass.INFORMATION_ONLY,
+ MODEL_STASH_CONSTRAINTS: StashKeyClass.INFORMATION_ONLY,
+ MODEL_STASH_MODEL_JOINS_WITH: StashKeyClass.INFORMATION_ONLY,
+}
+
+#: Keys read only when attached to a stash-only carrier that has no
+#: independent Ossie object of its own (an unrepresentable_joins[] entry,
+#: an unattributed_formulas[] entry) -- the same key name as a
+#: SHADOWS_DERIVABLE entry above, but INFORMATION_ONLY in this context,
+#: because there is no live Relationship/Metric/Field for it to diverge
+#: from. Recorded separately rather than overwriting the primary
+#: classification above, so both contexts stay documented.
+STASH_ONLY_CARRIER_KEY_CLASSIFICATION: dict[str, "StashKeyClass"] = {
+ RELATIONSHIP_STASH_ON_EXPRESSION: StashKeyClass.INFORMATION_ONLY,
+ RELATIONSHIP_STASH_TYPE: StashKeyClass.INFORMATION_ONLY,
+ RELATIONSHIP_STASH_CARDINALITY: StashKeyClass.INFORMATION_ONLY,
+ FIELD_STASH_COLUMN_PROPERTIES: StashKeyClass.INFORMATION_ONLY,
+}
+
+# ---------------------------------------------------------------------------
+# A second, orthogonal axis StashKeyClass alone cannot express.
+#
+# StashKeyClass answers one question: can this key's stashed VALUE disagree
+# with something the live Ossie document says? DATASET_STASH_UNSURFACED_
+# COLUMNS answers that "no" correctly -- a physical column's own
+# db_column_name/data_type has no Ossie-side counterpart to check it
+# against, so INFORMATION_ONLY is the right answer for its *content*. But a
+# key that holds a LIST of entries has a second question INFORMATION_ONLY
+# does not cover at all: does each entry still BELONG in the list? For
+# unsurfaced_columns specifically, an entry belongs only while no live field
+# now covers the same physical column -- and that membership fact changes
+# the moment a field is added, or retargeted, onto a column that used to be
+# unsurfaced. Restoring a membership-stale entry verbatim (the value itself
+# is still perfectly accurate) alongside the live field's own build of the
+# same column duplicates it -- a duplicate Table/SQL-View column name, which
+# does not import. This was found live: a field retargeted onto a
+# previously-unsurfaced column produced exactly that duplicate, undetected
+# by the value-only classification above because the value itself was never
+# wrong.
+#
+# So "information-only in value" and "derivable in membership" are
+# independent facts about one key, and a single INFORMATION_ONLY /
+# SHADOWS_DERIVABLE answer cannot record both. STASH_KEYS_WITH_DERIVABLE_
+# MEMBERSHIP is the second axis: a key here is a *list*-shaped stash whose
+# entries can be superseded by something the live document now covers, and
+# whose read site MUST filter entries against that live coverage before
+# appending them -- silently, same as an ordinary derive-instead-of-stash
+# fallback, because a filtered-out entry was not lost, just no longer
+# needed. Every other list-shaped INFORMATION_ONLY key was checked against
+# this question directly, not assumed innocent: DATASET_STASH_SQL_OUTPUT_
+# COLUMNS is consulted only as a per-field dict lookup keyed by the live
+# field's own name (never appended as a block), so a field no longer
+# present just means the lookup is never made -- no duplication is
+# possible, and it does not belong here. MODEL_STASH_UNATTRIBUTED_FORMULAS
+# and MODEL_STASH_UNREPRESENTABLE_JOINS are appended into collections
+# (formulas[]/columns[], and a table's inline joins[]) that already run
+# every entry through the shared display-name allocator or accept multiple
+# joins between the same pair without an import-breaking collision, so a
+# name clash there is caught (and now logged -- see
+# TS-MODEL-DISPLAY-NAME-COLLISION) rather than silently duplicated. Every
+# scalar-valued INFORMATION_ONLY key (a single string, dict, or bool
+# assigned once, never merged with anything else the live document also
+# populates) has no membership question to ask at all.
+STASH_KEYS_WITH_DERIVABLE_MEMBERSHIP: frozenset[str] = frozenset({
+ DATASET_STASH_UNSURFACED_COLUMNS,
+})
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/datatypes.py b/converters/thoughtspot/src/ossie_thoughtspot/datatypes.py
new file mode 100644
index 00000000..616cb861
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/datatypes.py
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""The ThoughtSpot <-> Ossie datatype map, covering every data type each side supports.
+
+Three properties of the map shape this module's surface. It is **not injective** — Decimal
+and Float both become DOUBLE, and Time, DateTimeTz and Opaque collapse into types that
+cannot carry them — so `declared_loss` names the types whose round trip is lossy in one
+place rather than leaving each caller to rediscover them. Two types have a
+**connection-dependent spelling** (BOOLEAN/BOOL, DOUBLE/FLOAT), which the forward direction
+records so the return trip re-emits the same one. And `datatype` is **optional in Ossie but
+compulsory in TML** — ThoughtSpot rejects a table whose column has no
+`db_column_properties`, so `to_tml(None)` infers rather than raising.
+"""
+from __future__ import annotations
+
+#: The closed Ossie datatype enum (core specification).
+OSSIE_DATATYPES = frozenset({
+ "String", "Integer", "Decimal", "Float", "Boolean",
+ "Date", "Time", "DateTime", "DateTimeTz", "Opaque",
+})
+
+#: Ossie datatype -> the TML `data_type` written for it.
+_TO_TML = {
+ "String": "VARCHAR",
+ "Integer": "INT64",
+ "Decimal": "DOUBLE",
+ "Float": "DOUBLE",
+ "Boolean": "BOOLEAN",
+ "Date": "DATE",
+ "Time": "VARCHAR",
+ "DateTime": "DATE_TIME",
+ "DateTimeTz": "DATE_TIME",
+ "Opaque": "VARCHAR",
+}
+
+#: TML `data_type` -> the Ossie datatype emitted for it. Deliberately not the inverse of
+#: `_TO_TML`: DOUBLE comes back as Decimal and VARCHAR as String, which is what makes the
+#: types in `_DECLARED_LOSS` lossy.
+_TO_OSSIE = {
+ "VARCHAR": "String",
+ "INT64": "Integer",
+ "DOUBLE": "Decimal",
+ "FLOAT": "Float",
+ "BOOL": "Boolean",
+ "BOOLEAN": "Boolean",
+ "DATE": "Date",
+ "DATE_TIME": "DateTime",
+}
+
+#: Types whose `Ossie -> TML -> Ossie` trip cannot return the original, and why. None of
+#: these can be rescued by a stash: the stash is written from a TML document, and TML
+#: never held the distinction in the first place.
+_DECLARED_LOSS = {
+ "Float": "ThoughtSpot has one approximate numeric type, so Float and Decimal both "
+ "become DOUBLE and return as Decimal.",
+ "Time": "ThoughtSpot has no time-of-day column type; the value becomes VARCHAR.",
+ "DateTimeTz": "ThoughtSpot has no offset-aware column type; the value becomes "
+ "DATE_TIME and returns as DateTime.",
+ "Opaque": "Opaque is Ossie's marker for a type outside the portable vocabulary; it "
+ "becomes VARCHAR and returns as String.",
+}
+
+#: What a column with no declared datatype becomes. The Table TML reference advises
+#: preferring INT64 and letting ThoughtSpot report a mismatch, over omitting the block.
+_INFERRED = "INT64"
+
+
+def to_tml(datatype: str | None, *, boolean_spelling: str = "BOOLEAN",
+ float_spelling: str = "DOUBLE") -> str:
+ """The TML `data_type` for an Ossie datatype. `None` infers rather than raising."""
+ if datatype is None:
+ return _INFERRED
+ if datatype not in _TO_TML:
+ raise ValueError(f"{datatype!r} is not an Ossie datatype")
+ if datatype == "Boolean":
+ return boolean_spelling
+ if datatype == "Float":
+ return float_spelling
+ return _TO_TML[datatype]
+
+
+def to_ossie(tml_type: str) -> str | None:
+ """The Ossie datatype for a TML `data_type`, or `None` when there is no mapping.
+
+ `None` is a legitimate answer, not a failure: `datatype` is optional in Ossie, so
+ omitting it is strictly better than inventing one for a type outside the map.
+ """
+ return _TO_OSSIE.get(tml_type)
+
+
+def declared_loss(datatype: str) -> str | None:
+ """Why this datatype's round trip is lossy, or `None` when it is exact."""
+ return _DECLARED_LOSS.get(datatype)
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/errors.py b/converters/thoughtspot/src/ossie_thoughtspot/errors.py
new file mode 100644
index 00000000..229e6251
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/errors.py
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Hard failures. Distinct from ConverterIssue, which records a survivable loss."""
+
+
+class ConversionError(Exception):
+ """Raised when the converter cannot proceed — e.g. a malformed stash."""
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/expressions/__init__.py b/converters/thoughtspot/src/ossie_thoughtspot/expressions/__init__.py
new file mode 100644
index 00000000..5b3b870f
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/expressions/__init__.py
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Expression translation: the Ossie expression language <-> ThoughtSpot formulas.
+
+Public surface:
+ - `Classification`, `Variant`, `Construct` — the shared vocabulary the forward
+ catalog, the emitters and the reverse inventory all use.
+ - `CATALOG` — every specification construct mapped to a ThoughtSpot rendering.
+ - `spec_construct_names()` — the upstream-spec coverage oracle: reads
+ core-spec/expression_language.md directly so a construct added upstream fails
+ this package's build instead of silently going unsupported.
+ - `CONVENTION_DIVERGENCES` — constructs the mapping document counts one row per
+ construct that have no discrete row in the upstream spec.
+ - `emit_direct`, `emit_passthrough`, `emit_unmappable` — render a `Construct`
+ into an actual ThoughtSpot formula, one function per `Classification`.
+ - `REVERSE`, `ReverseConstruct`, `ReverseDisposition`, `translate_thoughtspot`,
+ `stash_runtime_parameter` — the reverse-direction inventory (ThoughtSpot
+ functions with no counterpart in the Ossie specification) and its translator.
+ - `thoughtspot_dialect_entry`, `portable_dialect_entry`,
+ `custom_extensions_fragment` — helpers a caller combines with
+ `translate_thoughtspot`'s result to satisfy roundtrip at the object level.
+"""
+from .catalog import CATALOG, CONVENTION_DIVERGENCES, spec_construct_names
+from .emit import emit_direct, emit_passthrough, emit_unmappable
+from .reverse import (
+ GROUP_AGGREGATE_CALL_NAMES,
+ REVERSE,
+ ReverseConstruct,
+ ReverseDisposition,
+ custom_extensions_fragment,
+ portable_dialect_entry,
+ stash_runtime_parameter,
+ thoughtspot_dialect_entry,
+ translate_thoughtspot,
+)
+from ._types import Classification, Construct, Variant
+
+__all__ = [
+ "CATALOG",
+ "CONVENTION_DIVERGENCES",
+ "Classification",
+ "Construct",
+ "GROUP_AGGREGATE_CALL_NAMES",
+ "REVERSE",
+ "ReverseConstruct",
+ "ReverseDisposition",
+ "Variant",
+ "custom_extensions_fragment",
+ "emit_direct",
+ "emit_passthrough",
+ "emit_unmappable",
+ "portable_dialect_entry",
+ "spec_construct_names",
+ "stash_runtime_parameter",
+ "thoughtspot_dialect_entry",
+ "translate_thoughtspot",
+]
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/expressions/_types.py b/converters/thoughtspot/src/ossie_thoughtspot/expressions/_types.py
new file mode 100644
index 00000000..b5a9ab9b
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/expressions/_types.py
@@ -0,0 +1,302 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""The small shared vocabulary the catalog, emitters and reverse map all use."""
+from dataclasses import dataclass
+import re
+
+from enum import Enum
+
+
+class Classification(str, Enum):
+ """How a specification construct reaches ThoughtSpot.
+
+ DIRECT a native ThoughtSpot equivalent exists, possibly as a documented
+ composition of native functions.
+ PASSTHROUGH requires a sql_*_op pass-through: warehouse-dialect-specific, and
+ opaque to ThoughtSpot's query planner.
+ UNMAPPABLE no representation; the converter raises an issue and preserves the
+ construct in custom_extensions. Never a silent drop.
+ """
+
+ DIRECT = "direct"
+ PASSTHROUGH = "passthrough"
+ UNMAPPABLE = "unmappable"
+
+
+class Variant(str, Enum):
+ """The sql_*_op family. The variant fixes the emitted column's type AND
+ its measure/attribute role. The scalar variants produce attributes; the
+ *_aggregate_op variants produce measures. Emitting sql_int_op where
+ sql_int_aggregate_op was needed yields a column that imports cleanly and then
+ aggregates wrongly — worse than a rejected import.
+ """
+
+ BOOL = "sql_bool_op"
+ DATE_TIME = "sql_date_time_op"
+ DOUBLE = "sql_double_op"
+ INT = "sql_int_op"
+ NUMBER = "sql_number_op"
+ STRING = "sql_string_op"
+ INT_AGGREGATE = "sql_int_aggregate_op"
+ NUMBER_AGGREGATE = "sql_number_aggregate_op"
+ # Declared for completeness of the *_aggregate_op family even though no
+ # catalog row targets them today: `tml_to_ossie` derives "what already
+ # aggregates" by filtering this enum on the `_aggregate_op` suffix, so a
+ # missing member there is a missed double-aggregation guard, not merely an
+ # absent rendering option. Both are named in the reverse inventory.
+ STRING_AGGREGATE = "sql_string_aggregate_op"
+ DATE_TIME_AGGREGATE = "sql_date_time_aggregate_op"
+
+
+class VariadicStyle(str, Enum):
+ """How a construct that takes any number of arguments is rendered.
+
+ A fixed `{0} , {1}` template cannot express one: it either rejects the real
+ arity or -- worse -- bakes a literal `...` into the output, which is how
+ `concat ( {0} , {1} , ... )` came to emit `concat ( [T::A] , [T::B] , ... )`
+ as if that were a formula.
+ """
+
+ #: `{*}` is replaced by the argument tail, joined with " , ". The whole
+ #: list lives inside one call: `concat ( a , b , c )`.
+ JOIN = "join"
+ #: The template is BINARY and applied right-associatively over the whole
+ #: list, for a ThoughtSpot function that has no n-ary form:
+ #: `ifnull ( a , ifnull ( b , c ) )`.
+ FOLD = "fold"
+
+
+@dataclass(frozen=True)
+class Variadic:
+ """The arity rule for a construct whose argument count is not fixed.
+
+ `min_args` fewest arguments that render to something meaningful (2 for
+ every construct here -- a one-argument CONCAT or COALESCE is
+ the argument itself, not a call).
+ `tail_from` index at which the repeating tail begins, for JOIN. `IN` keeps
+ argument 0 positional (the column) and repeats from 1.
+ """
+
+ style: VariadicStyle
+ min_args: int = 2
+ tail_from: int = 0
+
+
+#: A `{0}`-style placeholder, removed before scanning for baked-in constants so
+#: the digits inside one are never mistaken for a literal.
+_ARGUMENT_PLACEHOLDER_RE = re.compile(r"\{\d+\}")
+#: A bare number or quoted string sitting in an argument position.
+_BAKED_LITERAL_RE = re.compile(r"(?<![\w.])\d+(?:\.\d+)?(?![\w.])|'[^']*'")
+#: An aggregate CALL baked into a template. `SUM` in
+#: `NTILE(4) OVER (ORDER BY SUM({0}))` stands for "whatever the caller is ranking
+#: by" exactly as the `4` stands for `n`, but it is a bare keyword rather than a
+#: number or a quoted string, so `_BAKED_LITERAL_RE` cannot see it. Five rows sat
+#: in that state undeclared while passing the literal gate, and the generated
+#: document offered `SUM` as the mapping for a row whose own `spec_name` says
+#: `AGG(expr)`.
+_BAKED_AGGREGATE_RE = re.compile(
+ r"(?<![\w.])(SUM|AVG|COUNT|MIN|MAX|MEDIAN|STDDEV|STDDEV_POP|STDDEV_SAMP"
+ r"|VARIANCE|VAR_POP|VAR_SAMP)\s*\(",
+ re.IGNORECASE,
+)
+
+
+def baked_literals(template: str | None) -> list[str]:
+ """Every bare number or quoted string `template` holds in an argument position."""
+ return _BAKED_LITERAL_RE.findall(_ARGUMENT_PLACEHOLDER_RE.sub("", template or ""))
+
+
+def baked_aggregates(template: str | None, spec_name: str) -> set[str]:
+ """Aggregate calls `template` bakes in that `spec_name` does not itself name.
+
+ A row whose specification IS the aggregate (`SUM(expr)`) is not baking an
+ exemplar in by writing `SUM` -- it is naming its own construct. A row whose
+ specification is `NTILE(n) OVER (...)`, `DENSE_RANK() OVER (...)` or
+ `Window aggregation -- AGG(expr) OVER (...)` and whose template nonetheless
+ says `SUM` has picked one aggregate to illustrate with, and has to say so.
+ """
+ body = _ARGUMENT_PLACEHOLDER_RE.sub("", template or "")
+ own = spec_name.split("(")[0].strip().upper()
+ return {
+ match.group(1).upper()
+ for match in _BAKED_AGGREGATE_RE.finditer(body)
+ if match.group(1).upper() != own
+ }
+
+
+def declares_an_aggregate_exemplar(exemplar_literals: tuple[str, ...]) -> bool:
+ """Whether some declared exemplar names an aggregate as the thing baked in.
+
+ Deliberately a substring test rather than an enum: these declarations are
+ free text destined for a generated document ("the ordering aggregate"), and
+ the check only has to be strong enough that a row baking `SUM` in cannot
+ satisfy it by declaring something unrelated, the way `NTILE` satisfied the
+ literal gate with `n` while its `SUM` went unmentioned.
+ """
+ return any("aggregate" in entry.lower() for entry in exemplar_literals)
+
+
+@dataclass(frozen=True)
+class Construct:
+ """One row of the function-mapping document.
+
+ `spec_name` the construct as the specification writes it, e.g. "SUM(expr)".
+ `template` for DIRECT, the ThoughtSpot formula with {0}, {1}... placeholders;
+ for PASSTHROUGH, the SQL body passed to the variant; None if UNMAPPABLE.
+
+ Not every DIRECT/PASSTHROUGH template is a complete, positionally
+ substitutable one — THREE shapes diverge from that default. Read
+ `variadic` and `exemplar_literals` before formatting this field: two of
+ the three fail loud on a naive `.format()` (a raised ValueError or
+ KeyError, or rejection at TML import), but the fold-variadic shape does
+ NOT, and silently drops arguments instead:
+
+ - A "dispatch" template — literal text such as "per-type — see note" or
+ "per-pattern-shape — see note" — for a row whose actual ThoughtSpot
+ rendering depends on a runtime value not known at catalog-construction
+ time (CAST's per-type table, the EXTRACT/DATE_PART/DATE_TRUNC/DATEADD
+ family, TRUE/FALSE, both CASE forms, the column/metric reference, the
+ unary +/- row, and several window rows). A caller building a uniform
+ `.format()` dispatcher off this field alone will hit these ~20 rows and
+ must special-case them; each row's `note` says so and describes the
+ real dispatch.
+ - A "variadic" template — six rows, flagged by a non-None `variadic`.
+ A JOIN-style one carries the `{*}` tail marker (`concat ( {*} )`,
+ `{0} in {{ {*} }}`) and `str.format` raises `KeyError('*')` on it, so
+ that half is loud. A FOLD-style one is the exception to this whole
+ paragraph: `COALESCE`'s template is the BINARY `ifnull ( {0} , {1} )`,
+ applied right-associatively by `_emit_variadic` because ThoughtSpot
+ has no n-ary `ifnull`. Formatting it directly with three arguments
+ returns `ifnull ( a , b )` and drops the third WITHOUT error — the one
+ place in this dataclass where reading `template` alone is silently
+ wrong rather than loudly wrong. `emit_direct` routes every variadic
+ row to `_emit_variadic`, so the converter itself is not exposed to
+ this; an external caller reading the catalog is.
+ - An "exemplar" PASSTHROUGH template — a complete, renderable body that
+ bakes ONE caller-supplied value in as a literal while still declaring a
+ satisfiable arity (`PERCENTILE_CONT`/`DISC`'s `0.75`, `NTILE`'s `4`,
+ `LAG`/`LEAD`'s offset `1`, and others — see `emit_passthrough`'s
+ docstring for the full convention). This kind renders without error, so
+ the arg-count guard alone does not distinguish it from a genuinely
+ complete template: treating the baked-in literal as universal instead of
+ rebuilding the template per real occurrence is silently wrong, not
+ loud — `PERCENTILE_CONT(0.9)` would render as a P75 measure that imports
+ and runs. Each such row's `note` names the baked-in value.
+ `variant` required for PASSTHROUGH, forbidden otherwise.
+ `note` the row's caveat, verbatim enough to be traceable to the document.
+ """
+
+ spec_name: str
+ classification: Classification
+ template: str | None = None
+ variant: Variant | None = None
+ note: str = ""
+ variadic: Variadic | None = None
+ exemplar_literals: tuple[str, ...] = ()
+
+ #: Names of the parameters this row's template bakes in as a LITERAL rather
+ #: than exposing as a placeholder -- `PERCENTILE_CONT`'s `p` rendered as
+ #: `0.75`, `NTILE`'s `n` as `4`, `LAG`'s `offset` as `1`. Such a row renders
+ #: without error, so the arity guard cannot tell it from a complete
+ #: template, and the generated reference document presented the example
+ #: value as if it were the mapping. Declaring it here is what lets the
+ #: document say so, and what lets the gate below refuse a NEW undeclared one.
+
+ def __post_init__(self) -> None:
+ if self.variadic is not None:
+ if self.classification is Classification.UNMAPPABLE:
+ raise ValueError(f"{self.spec_name}: an unmappable row has no arity to vary")
+ if self.variadic.style is VariadicStyle.JOIN and "{*}" not in (self.template or ""):
+ raise ValueError(
+ f"{self.spec_name}: a join-variadic template must carry the {{*}} tail marker"
+ )
+ if self.variadic.style is VariadicStyle.FOLD and "{*}" in (self.template or ""):
+ raise ValueError(
+ f"{self.spec_name}: a fold-variadic template is binary and takes no {{*}}"
+ )
+ # A literal "..." is how this defect shipped: every DIRECT variadic row
+ # carried the specification's own ellipsis notation straight into its
+ # template, where `str.format` never fills it, so a two-argument CONCAT
+ # rendered `concat ( a , b , ... )` -- syntactically invalid, and caught
+ # by nothing because the catalog sweep only asserted emission did not
+ # raise. Rejecting it here means a new row cannot reintroduce it.
+ # Fail-closed on baked-in literals: a PASSTHROUGH body carrying a bare
+ # numeric or quoted constant in an argument position is either an
+ # exemplar -- which must SAY so -- or a defect. Eleven rows were the
+ # former and said nothing, so the generated reference document offered
+ # `NTILE(4)` as the mapping for `NTILE(n)` with no sign that the 4 was
+ # illustrative. An allowlist, not a blocklist: a new row is refused
+ # until it declares itself.
+ if self.classification is Classification.PASSTHROUGH:
+ if not self.exemplar_literals:
+ baked = baked_literals(self.template)
+ if baked:
+ raise ValueError(
+ f"{self.spec_name}: template bakes in literal(s) {baked} while "
+ f"declaring no exemplar_literals. Name the parameter(s) the "
+ f"literal stands for, or parameterise the template"
+ )
+ # The same rule for a baked-in AGGREGATE, which the literal scan above
+ # cannot see. Checked independently of `exemplar_literals` being empty:
+ # `NTILE` declared `n` and so satisfied the literal gate while the `SUM`
+ # its template also bakes in went unmentioned, which is exactly the state
+ # this catches.
+ aggregates = baked_aggregates(self.template, self.spec_name)
+ if aggregates and not declares_an_aggregate_exemplar(self.exemplar_literals):
+ raise ValueError(
+ f"{self.spec_name}: template bakes in the aggregate(s) "
+ f"{sorted(aggregates)} while no declared exemplar names an "
+ f"aggregate. Add one that does (e.g. 'the ordering aggregate'), "
+ f"or parameterise the template"
+ )
+ if self.template and "..." in self.template:
+ raise ValueError(
+ f"{self.spec_name}: a template cannot contain a literal '...'; a construct "
+ f"that takes any number of arguments declares `variadic=` instead"
+ )
+ if self.classification is Classification.PASSTHROUGH and self.variant is None:
+ raise ValueError(f"{self.spec_name}: a passthrough row must name its variant")
+ if self.classification is not Classification.PASSTHROUGH and self.variant is not None:
+ raise ValueError(f"{self.spec_name}: only a passthrough row may name a variant")
+ if self.classification is Classification.UNMAPPABLE and self.template is not None:
+ raise ValueError(f"{self.spec_name}: an unmappable row has no template")
+ if self.classification is not Classification.UNMAPPABLE and not self.template:
+ raise ValueError(
+ f"{self.spec_name}: a {self.classification.value} row must have a template"
+ )
+ # A PASSTHROUGH template holds only the bare inner SQL body (e.g.
+ # "LOWER({0})") — emit_passthrough builds the `variant.value ( "..." , args )`
+ # wrapper itself. A template that already contains its own variant call
+ # (e.g. 'sql_string_op ( "LOWER({0})" , {0} )', copied verbatim from the
+ # mapping document's ThoughtSpot-column cell) double-wraps at emission time:
+ # `sql_string_op ( "sql_string_op ( ""LOWER({0})"" , {0} )" , {0} )`. That
+ # reads as fine in the catalog file and is wrong the moment it runs — a real
+ # transcription mistake this check exists to catch. Matching on
+ # "{variant} (" (the space and paren) rather than a bare substring guards
+ # against a coincidental token inside a legitimate body; a `sql_*_op` name
+ # is a ThoughtSpot-side synthetic formula-function name, so it cannot
+ # legitimately appear inside raw warehouse SQL either.
+ if self.classification is Classification.PASSTHROUGH:
+ marker = f"{self.variant.value} ("
+ if marker in self.template:
+ raise ValueError(
+ f"{self.spec_name}: passthrough template already contains "
+ f"'{marker}' — the template must hold only the bare inner SQL "
+ "body; emit_passthrough builds the variant(...) wrapper itself, "
+ "so this template would double-wrap at emission time"
+ )
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/expressions/catalog.py b/converters/thoughtspot/src/ossie_thoughtspot/expressions/catalog.py
new file mode 100644
index 00000000..321321e0
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/expressions/catalog.py
@@ -0,0 +1,1962 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""The catalog: every specification construct mapped to a ThoughtSpot rendering.
+
+`CATALOG` is organised into families below — Aggregate functions, Type conversion,
+Date/time functions, String functions, Mathematical + Conditional functions, Operators
+and constructs, and Window functions — one block per family, together covering the full
+specification. `spec_construct_names()` is an oracle read from the **upstream**
+`core-spec/expression_language.md`: it independently reports every construct the
+specification defines, so a construct added upstream in the future fails this package's
+build instead of silently going unsupported (see `test_catalog_covers_the_spec.py`).
+
+Extraction approach
+--------------------
+The spec document mixes three kinds of content that must be told apart:
+
+1. Genuine constructs — a named function, operator or literal form the
+ specification defines, almost always as one row of a markdown table whose
+ row carries a `Syntax` column (e.g. `SUM(expr)`), or, for a handful of
+ operators/keywords with no table of their own, a row of the top-level
+ "Supported SQL Constructs" table.
+2. Argument vocabularies — the `EXTRACT`/`DATE_PART` parts, the `DATE_TRUNC`
+ precisions, the `TO_DATE`/`TO_CHAR` format tokens and the `CAST` target
+ types. These describe values an argument may take, not constructs in
+ their own right, and must be excluded.
+3. Informative tables — the per-engine "Common Dialect Variations" table and
+ the "Cross-Reference: Tool Mappings" section describe *other products'*
+ spellings (Tableau, Looker Studio, DAX, and per-engine SQL). Names that
+ appear only there are not Ossie constructs.
+
+The exclusions are keyed off the document's own structure — a table's own header naming
+and section heading text ("Not Supported in Expressions", "Common Dialect Variations",
+"Cross-Reference") — rather than a hardcoded list of names to drop, so a renamed or added
+upstream construct cannot go silently unnoticed. `_extract_tables()` only looks at lines
+starting with "|", so the argument-vocabulary bullet lists are simply invisible to it.
+
+Spelling: `CATALOG` keys must match `spec_construct_names()` exactly
+------------------------------------------------------------------------------
+`spec_construct_names()` is the oracle for a `CATALOG` key's exact spelling, not any
+mapping document's prose. Several rows write their Ossie-side syntax differently than
+this parser extracts it; a `CATALOG` entry keyed on a mapping document's own wording
+instead of this function's output will read as an "invented" construct even though it is
+a real, intended row. When adding or editing a row, run `spec_construct_names()` and
+match a member of it exactly, rather than transcribing another document's column text.
+Grouped by which extractor produces the divergent spelling:
+
+- **`_extract_tables()`** (an ordinary table with a `Syntax` column):
+ - Alias pairs the spec merges into ONE table row keep this parser's single
+ extracted spelling: `CEIL(x)` (not `CEIL(x) / CEILING(x)`), `TRUNC(x, d)`
+ (not `.../ TRUNCATE(x, d)`).
+ - Two-alternative-syntax rows keep the spec's own joining word, "or":
+ `"CURRENT_DATE or CURRENT_DATE()"`, `"CURRENT_TIMESTAMP or
+ CURRENT_TIMESTAMP()"`, `"CURRENT_TIME or CURRENT_TIME()"`.
+ - The merged boolean-literal row (`BOOLEAN`'s `Syntax` cell) is one entry,
+ comma-joined: `"TRUE, FALSE"`.
+ - The Boolean Functions table's `AND`/`OR` rows keep the spec's own
+ `expr1`/`expr2` placeholder names: `"expr1 AND expr2"`, `"expr1 OR expr2"`.
+- **`_extract_summary_rows()`** (the top-level "Supported SQL Constructs"
+ table, bare backtick token): `BETWEEN`, `IN`, `NOT IN`, `IS NULL`, `IS NOT NULL`,
+ `CASE WHEN`, and the raw symbols `+ - * / % = <> != < > <= >=`.
+- **`_extract_null_safe_comparison_operators()`** (the "Null-Safe Comparison"
+ code fence, not the summary table): `IS DISTINCT FROM`, `IS NOT DISTINCT FROM`.
+- **`_extract_extraction_syntax_functions()`** (the "Alternative Extraction
+ Syntax" code fence, bare token, no argument list): `EXTRACT`, `DATE_PART`.
+- **`_extract_single_construct_headings()`** (a standalone heading with no
+ table, bare token): `CAST`, `TRY_CAST` (not `CAST(expression AS target_type)`).
+
+See `CONVENTION_DIVERGENCES` below for the (much shorter) list of constructs that have
+no entry in `spec_construct_names()` at all and are exempted instead.
+"""
+import re
+from pathlib import Path
+
+from ._types import Classification, Construct, Variadic, VariadicStyle, Variant
+
+# --------------------------------------------------------------------------
+# CATALOG: organised into families, one block per family below.
+# --------------------------------------------------------------------------
+CATALOG: dict[str, Construct] = {}
+
+# --------------------------------------------------------------------------
+# Aggregate functions — 18 rows: 12 direct / 6 passthrough / 0 unmappable.
+# Source: docs/ossie/ts-ossie-function-mapping.md, "Aggregate functions" section
+# (thoughtspot-agent-skills repo — not vendored here; prose above/below the table
+# read in full).
+# --------------------------------------------------------------------------
+CATALOG.update(
+ {
+ "SUM(expr)": Construct(
+ "SUM(expr)", Classification.DIRECT, template="sum ( {0} )",
+ ),
+ "COUNT(expr)": Construct(
+ "COUNT(expr)", Classification.DIRECT, template="count ( {0} )",
+ note="Counts non-null values on both sides.",
+ ),
+ "COUNT(*)": Construct(
+ "COUNT(*)", Classification.DIRECT, template="count ( {0} )",
+ note=(
+ "ThoughtSpot has no count(*); the row count is count() over a column "
+ "known to be non-null. The converter uses the dataset's primary_key "
+ "when the model declares one, and raises an issue rather than "
+ "guessing a column when it does not."
+ ),
+ ),
+ "COUNT(DISTINCT expr)": Construct(
+ "COUNT(DISTINCT expr)", Classification.DIRECT, template="unique count ( {0} )",
+ note=(
+ "A space, not an underscore. count_distinct(...) is rejected by the "
+ "formula parser."
+ ),
+ ),
+ "AVG(expr)": Construct(
+ "AVG(expr)", Classification.DIRECT, template="average ( {0} )",
+ ),
+ "MIN(expr)": Construct(
+ "MIN(expr)", Classification.DIRECT, template="min ( {0} )",
+ note=(
+ "ThoughtSpot min is aggregate-only — it never compares two columns "
+ "row-wise. Scalar two-argument minima are LEAST, a separate row."
+ ),
+ ),
+ "MAX(expr)": Construct(
+ "MAX(expr)", Classification.DIRECT, template="max ( {0} )",
+ note="Aggregate-only, as MIN.",
+ ),
+ "STDDEV(expr)": Construct(
+ "STDDEV(expr)", Classification.DIRECT, template="stddev ( {0} )",
+ note="Sample standard deviation on both sides.",
+ ),
+ "STDDEV_POP(expr)": Construct(
+ "STDDEV_POP(expr)", Classification.PASSTHROUGH,
+ template="STDDEV_POP({0})", variant=Variant.NUMBER_AGGREGATE,
+ note=(
+ "ThoughtSpot stddev is sample-only; there is no population form, and "
+ "substituting it would change the divisor from n-1 to n."
+ ),
+ ),
+ "STDDEV_SAMP(expr)": Construct(
+ "STDDEV_SAMP(expr)", Classification.DIRECT, template="stddev ( {0} )",
+ note="Specification alias for STDDEV (:171).",
+ ),
+ "VARIANCE(expr)": Construct(
+ "VARIANCE(expr)", Classification.DIRECT, template="variance ( {0} )",
+ note="Sample variance on both sides.",
+ ),
+ "VAR_POP(expr)": Construct(
+ "VAR_POP(expr)", Classification.PASSTHROUGH,
+ template="VAR_POP({0})", variant=Variant.NUMBER_AGGREGATE,
+ note="Same divisor reason as STDDEV_POP.",
+ ),
+ "VAR_SAMP(expr)": Construct(
+ "VAR_SAMP(expr)", Classification.DIRECT, template="variance ( {0} )",
+ note="Specification alias for VARIANCE (:174).",
+ ),
+ "MEDIAN(expr)": Construct(
+ "MEDIAN(expr)", Classification.DIRECT, template="median ( {0} )",
+ ),
+ "PERCENTILE_CONT(p) WITHIN GROUP (ORDER BY expr)": Construct(
+ "PERCENTILE_CONT(p) WITHIN GROUP (ORDER BY expr)", Classification.PASSTHROUGH,
+ template="PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY {0})",
+ exemplar_literals=("p",),
+ variant=Variant.NUMBER_AGGREGATE,
+ note=(
+ "No native percentile function. p is a literal in the specification's "
+ "syntax, so it is baked into the template rather than passed as a "
+ "placeholder. p = 0.5 is the one case with a native equivalent — "
+ "median ( [x] ) — and the converter should prefer it."
+ ),
+ ),
+ "PERCENTILE_DISC(p) WITHIN GROUP (ORDER BY expr)": Construct(
+ "PERCENTILE_DISC(p) WITHIN GROUP (ORDER BY expr)", Classification.PASSTHROUGH,
+ template="PERCENTILE_DISC(0.75) WITHIN GROUP (ORDER BY {0})",
+ exemplar_literals=("p",),
+ variant=Variant.NUMBER_AGGREGATE,
+ note=(
+ "As PERCENTILE_CONT; the discrete/interpolated distinction is "
+ "preserved only because the template is emitted verbatim."
+ ),
+ ),
+ "APPROX_COUNT_DISTINCT(expr)": Construct(
+ "APPROX_COUNT_DISTINCT(expr)", Classification.PASSTHROUGH,
+ template="APPROX_COUNT_DISTINCT({0})", variant=Variant.INT_AGGREGATE,
+ note=(
+ "ThoughtSpot's unique count ( [x] ) is the exact-semantics "
+ "alternative: same answer to within the sketch's ~2% error, at "
+ "exact-count cost. The converter emits the pass-through by default "
+ "— the specification chose approximate deliberately — and offers "
+ "the exact form as a documented downgrade."
+ ),
+ ),
+ "APPROX_PERCENTILE(expr, p)": Construct(
+ "APPROX_PERCENTILE(expr, p)", Classification.PASSTHROUGH,
+ template="APPROX_PERCENTILE({0}, 0.5)",
+ exemplar_literals=("p",), variant=Variant.NUMBER_AGGREGATE,
+ note="p baked into the template as for the exact percentiles.",
+ ),
+ }
+)
+
+# --------------------------------------------------------------------------
+# Type conversion — 2 rows: 2 direct / 0 passthrough / 0 unmappable.
+# Source: docs/ossie/ts-ossie-function-mapping.md, "Type conversion" section.
+#
+# CAST/TRY_CAST are direct rows whose target-type argument vocabulary is only
+# partly covered (5 of 8 types direct, 3 fall back to a pass-through) — the
+# per-type dispatch is not counted as its own construct (the target-type
+# table is an argument vocabulary, marked "not counted" in the mapping
+# document) and is not resolved here. Resolving a
+# `CAST` occurrence to an actual formula from its target type would need an
+# expression parser, which is out of scope, and whether to take on a sqlglot
+# dependency for that is still unresolved; `template` records the document's
+# own ThoughtSpot-column text for traceability rather than a
+# directly-substitutable formula.
+# --------------------------------------------------------------------------
+CATALOG.update(
+ {
+ "CAST": Construct(
+ "CAST", Classification.DIRECT,
+ template="per-type — see the target-type table below",
+ note=(
+ "5 of the 8 specified target types are direct; the other three — "
+ "BOOLEAN, TIMESTAMP and TIME — fall back to a pass-through."
+ ),
+ ),
+ "TRY_CAST": Construct(
+ "TRY_CAST", Classification.DIRECT,
+ template="the same functions as CAST",
+ note=(
+ "ThoughtSpot's to_integer / to_double / to_string already return "
+ "NULL on failure, which is exactly TRY_CAST semantics — so the two "
+ "rows share a mapping and it is CAST, not TRY_CAST, that is the "
+ "imprecise one. A strict CAST that must error rather than null is "
+ "not expressible; the converter records that in the issue log when "
+ "the source distinguishes them."
+ ),
+ ),
+ }
+)
+
+# --------------------------------------------------------------------------
+# Date/time functions — 24 rows: 17 direct / 7 passthrough / 0 unmappable.
+# Source: docs/ossie/ts-ossie-function-mapping.md, "Date/time functions" section
+# (thoughtspot-agent-skills repo — not vendored here; prose above/below the table
+# read in full).
+#
+# EXTRACT/DATE_PART date-parts, DATE_TRUNC precisions, DATEADD/DATEDIFF parts and
+# TO_DATE/TO_CHAR format tokens are argument vocabularies and get no
+# entry of their own (see the mapping document's "(not counted — arguments)"
+# sub-tables). EXTRACT, DATE_PART, DATE_TRUNC(part, date_expr),
+# DATEADD(part, amount, date_expr) and DATEDIFF(part, start_date, end_date) are
+# themselves still DIRECT rows in the 24 — the per-argument dispatch happens for
+# each, but the dispatch table itself is out of catalog scope (same pattern as
+# CAST/TRY_CAST in Type conversion above): `template` records the mapping document's own
+# ThoughtSpot-column text for traceability, and the real per-argument content
+# (which native function each part/precision rewrites to, and the argument-order
+# caveats) is recorded in `note`.
+# --------------------------------------------------------------------------
+CATALOG.update(
+ {
+ "CURRENT_DATE or CURRENT_DATE()": Construct(
+ "CURRENT_DATE or CURRENT_DATE()", Classification.DIRECT, template="today ( )",
+ note="Both specification spellings map to the same function.",
+ ),
+ "CURRENT_TIMESTAMP or CURRENT_TIMESTAMP()": Construct(
+ "CURRENT_TIMESTAMP or CURRENT_TIMESTAMP()", Classification.DIRECT, template="now ( )",
+ ),
+ "CURRENT_TIME or CURRENT_TIME()": Construct(
+ "CURRENT_TIME or CURRENT_TIME()", Classification.DIRECT,
+ template="time ( now ( ) )",
+ note=(
+ "ThoughtSpot has no current-time function, but time ( ) extracts "
+ "the time part of a datetime, so the composition is exact."
+ ),
+ ),
+ "YEAR(date_expr)": Construct(
+ "YEAR(date_expr)", Classification.DIRECT, template="year ( {0} )",
+ ),
+ "QUARTER(date_expr)": Construct(
+ "QUARTER(date_expr)", Classification.DIRECT, template="quarter_number ( {0} )",
+ note="The function is quarter_number, not quarter.",
+ ),
+ "MONTH(date_expr)": Construct(
+ "MONTH(date_expr)", Classification.DIRECT, template="month_number ( {0} )",
+ note=(
+ "Not month ( ) — ThoughtSpot's month returns the month NAME "
+ "('January'); month_number returns 1-12, which is what the "
+ "specification means. Mapping to month would silently change "
+ "the column's type from integer to string."
+ ),
+ ),
+ "DAY(date_expr)": Construct(
+ "DAY(date_expr)", Classification.DIRECT, template="day ( {0} )",
+ note="Day of month, 1-31 on both sides.",
+ ),
+ "DAYOFYEAR(date_expr)": Construct(
+ "DAYOFYEAR(date_expr)", Classification.DIRECT,
+ template="day_number_of_year ( {0} )",
+ note="The function is day_number_of_year, not day_of_year.",
+ ),
+ "HOUR(timestamp_expr)": Construct(
+ "HOUR(timestamp_expr)", Classification.DIRECT, template="hour_of_day ( {0} )",
+ note="The function is hour_of_day, not hour.",
+ ),
+ "MINUTE(timestamp_expr)": Construct(
+ "MINUTE(timestamp_expr)", Classification.PASSTHROUGH,
+ template="MINUTE({0})", variant=Variant.INT,
+ note=(
+ "No native minute-of-hour extractor; add_minutes and "
+ "diff_minutes exist but neither extracts."
+ ),
+ ),
+ "SECOND(timestamp_expr)": Construct(
+ "SECOND(timestamp_expr)", Classification.PASSTHROUGH,
+ template="SECOND({0})", variant=Variant.INT,
+ note="As MINUTE.",
+ ),
+ "EXTRACT": Construct(
+ "EXTRACT", Classification.DIRECT,
+ template="per-part — see the date-part table below",
+ note=(
+ "Rewritten to the part's own ThoughtSpot function; there is no "
+ "generic extractor. 8 of the 11 specified parts are direct "
+ "(YEAR->year, QUARTER->quarter_number, MONTH->month_number, "
+ "WEEK->week_number_of_year, DAY->day, "
+ "DAYOFWEEK->day_number_of_week, DAYOFYEAR->day_number_of_year, "
+ "HOUR->hour_of_day); MINUTE, SECOND and MILLISECOND fall back "
+ "to sql_int_op."
+ ),
+ ),
+ "DATE_PART": Construct(
+ "DATE_PART", Classification.DIRECT,
+ template="per-part — see the date-part table below",
+ note="Identical treatment to EXTRACT; the two spellings collapse onto one rewrite (:276-279).",
+ ),
+ "DATE_TRUNC(part, date_expr)": Construct(
+ "DATE_TRUNC(part, date_expr)", Classification.DIRECT,
+ template="per-precision — see the truncation table below",
+ note=(
+ "ThoughtSpot has no date_trunc. The start_of_* family covers 7 "
+ "of the 8 specified precisions ('year'->start_of_year, "
+ "'quarter'->start_of_quarter, 'month'->start_of_month, "
+ "'week'->start_of_week, 'day'->date ( ), 'hour'->start_of_hour, "
+ "'minute'->start_of_min — the function is start_of_min, not "
+ "start_of_minute); 'second' falls back to sql_date_time_op. "
+ "The specification says week truncation is Monday-start; "
+ "ThoughtSpot's week start is an instance setting, so the "
+ "converter verifies alignment and raises an issue when it cannot."
+ ),
+ ),
+ "DATEADD(part, amount, date_expr)": Construct(
+ "DATEADD(part, amount, date_expr)", Classification.DIRECT,
+ template="per-part add_* — see the arithmetic table below",
+ note=(
+ "Argument order differs: ThoughtSpot is add_days ( [d] , n ), "
+ "the specification is DATEADD(day, n, d). Every specified part "
+ "is reachable: day->add_days, week->add_weeks, "
+ "month->add_months, year->add_years, minute->add_minutes, "
+ "second->add_seconds, plus two by arithmetic on a coarser unit "
+ "since there is no native add_quarters or add_hours: "
+ "quarter->add_months ( [d] , 3 * n ), "
+ "hour->add_minutes ( [d] , 60 * n )."
+ ),
+ ),
+ "DATEDIFF(part, start_date, end_date)": Construct(
+ "DATEDIFF(part, start_date, end_date)", Classification.DIRECT,
+ template="per-part diff_* — see the arithmetic table below",
+ note=(
+ "Argument order is reversed: ThoughtSpot is "
+ "diff_days ( [end] , [start] ) — end first. Getting this wrong "
+ "silently negates every duration in the model. day->diff_days, "
+ "week->diff_weeks, month->diff_months, quarter->diff_quarters, "
+ "year->diff_years, hour->diff_hours, minute->diff_minutes, "
+ "second->diff_time (returns seconds)."
+ ),
+ ),
+ "DATE '2024-01-15'": Construct(
+ "DATE '2024-01-15'", Classification.DIRECT,
+ template="to_date ( '{0}' , 'yyyy-MM-dd' )",
+ note=(
+ "A bare '2024-01-15' in a ThoughtSpot formula is parsed as "
+ "arithmetic (2024 - 1 - 15), so the typed literal must always "
+ "be wrapped. to_date takes exactly two arguments, so the "
+ "converter supplies the ISO format model; {0} is the literal "
+ "date string."
+ ),
+ ),
+ "TIMESTAMP_NTZ '2024-01-15 10:30:00'": Construct(
+ "TIMESTAMP_NTZ '2024-01-15 10:30:00'", Classification.PASSTHROUGH,
+ template="CAST('2024-01-15 10:30:00' AS TIMESTAMP)",
+ exemplar_literals=("the timestamp value",),
+ variant=Variant.DATE_TIME,
+ note=(
+ "to_date returns a DATE and drops the time part, so there is "
+ "no native way to construct a wall-clock timestamp. A "
+ "zero-placeholder template is a documented form of the "
+ "pass-through — the document's own worked example is recorded "
+ "verbatim here; a real occurrence's literal value is "
+ "substituted per-occurrence when the template is built, out of "
+ "this catalog's scope (same as CAST's per-type dispatch)."
+ ),
+ ),
+ "TIME '10:30:00'": Construct(
+ "TIME '10:30:00'", Classification.PASSTHROUGH,
+ template="CAST('10:30:00' AS TIME)",
+ exemplar_literals=("the time value",),
+ variant=Variant.DATE_TIME,
+ note=(
+ "ThoughtSpot has no TIME column type — time ( ) extracts a "
+ "time FROM a datetime, it does not construct one — so the "
+ "pass-through returns DATETIME and the date part is whatever "
+ "the warehouse defaults to. Flagged with an issue for that "
+ "reason, not only for the dialect."
+ ),
+ ),
+ "TO_DATE(string)": Construct(
+ "TO_DATE(string)", Classification.DIRECT,
+ template="to_date ( {0} , 'yyyy-MM-dd' )",
+ note=(
+ "The single-argument ISO form. ThoughtSpot's to_date is "
+ "strictly two-argument, so the converter supplies 'yyyy-MM-dd'."
+ ),
+ ),
+ "TO_TIMESTAMP(string)": Construct(
+ "TO_TIMESTAMP(string)", Classification.PASSTHROUGH,
+ template="TO_TIMESTAMP({0})", variant=Variant.DATE_TIME,
+ note="to_date is date-only; parsing to a timestamp would drop the time silently.",
+ ),
+ "TO_DATE(string, format)": Construct(
+ "TO_DATE(string, format)", Classification.DIRECT,
+ template="to_date ( {0} , <translated format> )",
+ note=(
+ "EXPERIMENTAL. Format tokens are translated, not passed "
+ "through — see the format-token table. ThoughtSpot accepts "
+ "Java/LDML tokens (yyyy-MM-dd) and strptime %-codes, which "
+ "between them cover the specification's entire portable core."
+ ),
+ ),
+ "TO_TIMESTAMP(string, format)": Construct(
+ "TO_TIMESTAMP(string, format)", Classification.PASSTHROUGH,
+ template="TO_TIMESTAMP({0}, 'YYYY-MM-DD HH24:MI:SS')",
+ exemplar_literals=("format",),
+ variant=Variant.DATE_TIME,
+ note=(
+ "EXPERIMENTAL. Date-only to_date again. The format model "
+ "inside the template is the warehouse's, not Ossie's, so the "
+ "token translation table does not apply — this is the "
+ "sharpest case of the pass-through caveat."
+ ),
+ ),
+ "TO_CHAR(date_expr, format)": Construct(
+ "TO_CHAR(date_expr, format)", Classification.PASSTHROUGH,
+ template="TO_CHAR({0}, 'YYYY-MM')",
+ exemplar_literals=("format",), variant=Variant.STRING,
+ note=(
+ "EXPERIMENTAL. ThoughtSpot has no general date formatter. "
+ "Single-token formats do have native equivalents and the "
+ "converter prefers them: 'YYYY' -> year_name ( [d] ), "
+ "'MONTH' -> month ( [d] ), 'DAY' -> day_of_week ( [d] ). Those "
+ "three return locale-dependent text on both sides."
+ ),
+ ),
+ }
+)
+
+# --------------------------------------------------------------------------
+# String functions — 21 rows: 10 direct / 11 passthrough / 0 unmappable.
+# Source: docs/ossie/ts-ossie-function-mapping.md, "String functions" section
+# (thoughtspot-agent-skills repo — not vendored here; prose above/below the table
+# read in full).
+#
+# This family is over half passthrough, and the reasons run against intuition
+# rather than with it: LOWER/UPPER/TRIM/LTRIM/RTRIM/REPLACE are passthrough not
+# because they behave differently in ThoughtSpot but because ThoughtSpot has no
+# native equivalent at all (live-verified 2026-07-29: TRIM and REPLACE were
+# rejected with "Search did not find ..."). STARTSWITH/ENDSWITH run the other
+# way: also no native function, but their compositions use only native
+# functions (strpos/substr/strlen), so they stay direct. There is no
+# regular-expression support of any kind, so every REGEXP_* row is passthrough
+# with no native fallback.
+# --------------------------------------------------------------------------
+CATALOG.update(
+ {
+ "CONCAT(str1, str2, ...)": Construct(
+ "CONCAT(str1, str2, ...)", Classification.DIRECT,
+ template="concat ( {*} )",
+ variadic=Variadic(VariadicStyle.JOIN),
+ note=(
+ "N-ary on both sides. + does not concatenate in ThoughtSpot — "
+ "it is numeric-only and the parser rejects string operands, so "
+ "both || and CONCAT land here."
+ ),
+ ),
+ "LENGTH(str)": Construct(
+ "LENGTH(str)", Classification.DIRECT, template="strlen ( {0} )",
+ note="Characters, not bytes, on both sides.",
+ ),
+ "LOWER(str)": Construct(
+ "LOWER(str)", Classification.PASSTHROUGH,
+ template="LOWER({0})", variant=Variant.STRING,
+ note="There is no native lower in ThoughtSpot.",
+ ),
+ "UPPER(str)": Construct(
+ "UPPER(str)", Classification.PASSTHROUGH,
+ template="UPPER({0})", variant=Variant.STRING,
+ note=(
+ "There is no native upper in ThoughtSpot. LOWER/UPPER are the "
+ "most-used functions in the whole passthrough set, and their "
+ "absence is also what forces ILIKE and case-insensitive "
+ "comparison into pass-throughs."
+ ),
+ ),
+ "TRIM(str)": Construct(
+ "TRIM(str)", Classification.PASSTHROUGH,
+ template="TRIM({0})", variant=Variant.STRING,
+ note=(
+ "There is no native trim in ThoughtSpot — live-verified "
+ "2026-07-29, rejected with 'Search did not find \"trim (\"'. "
+ "The whole trim family is a pass-through, not just the "
+ "one-sided forms."
+ ),
+ ),
+ "LTRIM(str)": Construct(
+ "LTRIM(str)", Classification.PASSTHROUGH,
+ template="LTRIM({0})", variant=Variant.STRING,
+ note="No native ltrim — live-verified 2026-07-29.",
+ ),
+ "RTRIM(str)": Construct(
+ "RTRIM(str)", Classification.PASSTHROUGH,
+ template="RTRIM({0})", variant=Variant.STRING,
+ note="As LTRIM.",
+ ),
+ "LEFT(str, n)": Construct(
+ "LEFT(str, n)", Classification.DIRECT, template="left ( {0} , {1} )",
+ ),
+ "RIGHT(str, n)": Construct(
+ "RIGHT(str, n)", Classification.DIRECT, template="right ( {0} , {1} )",
+ ),
+ "SUBSTRING(str, start, length)": Construct(
+ "SUBSTRING(str, start, length)", Classification.DIRECT,
+ template="substr ( {0} , {1} - 1 , {2} )",
+ note=(
+ "Index base differs. ANSI SUBSTRING is 1-based; ThoughtSpot's "
+ "substr is 0-based. The -1 is mandatory and is the single most "
+ "likely off-by-one in the whole mapping. When start is an "
+ "expression rather than a literal, the arithmetic is emitted "
+ "rather than folded."
+ ),
+ ),
+ "REPLACE(str, from, to)": Construct(
+ "REPLACE(str, from, to)", Classification.PASSTHROUGH,
+ template="REPLACE({0}, {1}, {2})",
+ variant=Variant.STRING,
+ note=(
+ "There is no native replace in ThoughtSpot — live-verified "
+ "2026-07-29, rejected with 'Search did not find \"replace (\"'."
+ ),
+ ),
+ "SPLIT_PART(str, delimiter, part)": Construct(
+ "SPLIT_PART(str, delimiter, part)", Classification.PASSTHROUGH,
+ template="SPLIT_PART({0}, {1}, {2})",
+ variant=Variant.STRING,
+ note=(
+ "ThoughtSpot has no tokenising function at all — not split, "
+ "split_part or an nth-occurrence search — so there is no "
+ "composition to fall back on."
+ ),
+ ),
+ "POSITION(substr IN str)": Construct(
+ "POSITION(substr IN str)", Classification.DIRECT,
+ template="strpos ( {1} , {0} )",
+ note=(
+ "Operand order is reversed (haystack first in ThoughtSpot) and "
+ "the specification's infix IN form becomes a comma. 1-based, "
+ "returning 0 when absent, on both sides."
+ ),
+ ),
+ "CHARINDEX(substr, str)": Construct(
+ "CHARINDEX(substr, str)", Classification.DIRECT,
+ template="strpos ( {1} , {0} )",
+ note=(
+ "Specification alias for POSITION (:419) with the operands "
+ "already in prefix order; the reversal is the same."
+ ),
+ ),
+ "CONTAINS(str, substr)": Construct(
+ "CONTAINS(str, substr)", Classification.DIRECT,
+ template="contains ( {0} , {1} )",
+ note="Returns boolean on both sides.",
+ ),
+ "STARTSWITH(str, prefix)": Construct(
+ "STARTSWITH(str, prefix)", Classification.DIRECT,
+ template="strpos ( {0} , {1} ) = 1",
+ note=(
+ "There is no native starts_with — live-verified 2026-07-29. "
+ "Still direct because the composition is exact and uses only "
+ "native functions: strpos is 1-based, so a true prefix sits "
+ "at position 1."
+ ),
+ ),
+ "ENDSWITH(str, suffix)": Construct(
+ "ENDSWITH(str, suffix)", Classification.DIRECT,
+ template="substr ( {0} , strlen ( {0} ) - strlen ( {1} ) , strlen ( {1} ) ) = {1}",
+ note=(
+ "There is no native ends_with — live-verified 2026-07-29. "
+ "Direct by composition, as STARTSWITH."
+ ),
+ ),
+ "REGEXP_LIKE(str, pattern)": Construct(
+ "REGEXP_LIKE(str, pattern)", Classification.PASSTHROUGH,
+ template="REGEXP_LIKE({0}, {1})",
+ variant=Variant.BOOL,
+ note=(
+ "Boolean return, so not sql_string_op. ThoughtSpot has no "
+ "regular-expression support of any kind."
+ ),
+ ),
+ "REGEXP_EXTRACT(str, pattern)": Construct(
+ "REGEXP_EXTRACT(str, pattern)", Classification.PASSTHROUGH,
+ template="REGEXP_SUBSTR({0}, {1})",
+ variant=Variant.STRING,
+ note=(
+ "The function name inside the template is dialect-specific — "
+ "Snowflake spells it REGEXP_SUBSTR, others REGEXP_EXTRACT — so "
+ "the converter selects it from the connection's dialect and "
+ "raises an issue when the dialect is unknown."
+ ),
+ ),
+ "REGEXP_REPLACE(str, pattern, replacement)": Construct(
+ "REGEXP_REPLACE(str, pattern, replacement)", Classification.PASSTHROUGH,
+ template="REGEXP_REPLACE({0},{1},{2})",
+ variant=Variant.STRING,
+ note=(
+ "Name is portable; the pattern dialect (POSIX vs PCRE, "
+ "backreference syntax) is not."
+ ),
+ ),
+ "REGEXP_COUNT(str, pattern)": Construct(
+ "REGEXP_COUNT(str, pattern)", Classification.PASSTHROUGH,
+ template="REGEXP_COUNT({0}, {1})",
+ variant=Variant.INT,
+ note="Integer return.",
+ ),
+ }
+)
+
+# --------------------------------------------------------------------------
+# Mathematical + Conditional functions — 34 rows: 32 direct /
+# 2 passthrough / 0 unmappable.
+# Source: docs/ossie/ts-ossie-function-mapping.md, "Mathematical functions" and
+# "Conditional functions" sections (thoughtspot-agent-skills repo — not
+# vendored here; prose above/below the tables read in full).
+#
+# Nearly every row here is direct, several by composition: SIGN has
+# no native function but composes exactly as a three-way `if` chain — the
+# trailing `else 0` is mandatory, ThoughtSpot rejects an `if` with no `else`.
+# RADIANS/DEGREES are bare dialect-free arithmetic, not passthroughs. PI is a
+# literal at the precision ThoughtSpot's own documented trig composites use.
+# ThoughtSpot's trigonometry is degrees-native while the specification is
+# radians-native, so SIN/COS/TAN convert radians->degrees on the way IN
+# (`* 180 / pi`, applied to the argument) and ASIN/ACOS/ATAN convert
+# degrees->radians on the way OUT (`* pi / 180`, applied to the result) —
+# opposite directions, easy to transpose by mistake, and this comment had them
+# transposed. The templates were and are correct; cross-check against the
+# DEGREES/RADIANS rows below, which state the same two conversions as their
+# whole definition: DEGREES is `* 180 / pi`, RADIANS is `* pi / 180`.
+# GREATEST/LEAST are deliberately NOT mapped to max/min: ThoughtSpot's max/min
+# are aggregate-only, so that mapping would both collapse the row-wise N-ary
+# result to one value and flip it from attribute to measure.
+#
+# Only two rows are passthrough: TRUNC/TRUNCATE (no native truncation — floor
+# only agrees with it for x >= 0, d = 0, and round disagrees at every
+# half-value) and ATAN2 (quadrant-aware and defined where x = 0, so it is not
+# a two-argument ATAN composition, unlike every other inverse trig function
+# in this family).
+# --------------------------------------------------------------------------
+CATALOG.update(
+ {
+ "ABS(x)": Construct(
+ "ABS(x)", Classification.DIRECT, template="abs ( {0} )",
+ ),
+ "ROUND(x, d)": Construct(
+ "ROUND(x, d)", Classification.DIRECT, template="round ( {0} , {1} )",
+ ),
+ "FLOOR(x)": Construct(
+ "FLOOR(x)", Classification.DIRECT, template="floor ( {0} )",
+ ),
+ "CEIL(x)": Construct(
+ "CEIL(x)", Classification.DIRECT, template="ceil ( {0} )",
+ note="Specification alias pair CEIL(x) / CEILING(x); both spellings map to ceil.",
+ ),
+ "TRUNC(x, d)": Construct(
+ "TRUNC(x, d)", Classification.PASSTHROUGH,
+ template="TRUNC({0}, {1})", variant=Variant.DOUBLE,
+ note=(
+ "Specification alias pair TRUNC(x, d) / TRUNCATE(x, d). "
+ "ThoughtSpot has no truncation function. floor agrees with "
+ "TRUNC only for x >= 0 and d = 0, and round disagrees at every "
+ "half-value, so neither is a safe substitute."
+ ),
+ ),
+ "MOD(x, y)": Construct(
+ "MOD(x, y)", Classification.DIRECT, template="mod ( {0} , {1} )",
+ note="Sign-of-result for negative operands follows the warehouse on both sides.",
+ ),
+ "SIGN(x)": Construct(
+ "SIGN(x)", Classification.DIRECT,
+ template="if ( {0} > 0 ) then 1 else if ( {0} < 0 ) then -1 else 0",
+ note=(
+ "No native sign, but the three-way result is exactly "
+ "expressible as an if chain. The else 0 is required — "
+ "ThoughtSpot rejects an if chain with no else."
+ ),
+ ),
+ "POWER(x, y)": Construct(
+ "POWER(x, y)", Classification.DIRECT, template="pow ( {0} , {1} )",
+ note="The function is pow. power is rejected by the parser.",
+ ),
+ "SQRT(x)": Construct(
+ "SQRT(x)", Classification.DIRECT, template="sqrt ( {0} )",
+ ),
+ "EXP(x)": Construct(
+ "EXP(x)", Classification.DIRECT, template="exp ( {0} )",
+ ),
+ "LN(x)": Construct(
+ "LN(x)", Classification.DIRECT, template="ln ( {0} )",
+ ),
+ "LOG(base, x)": Construct(
+ "LOG(base, x)", Classification.DIRECT,
+ template="safe_divide ( ln ( {1} ) , ln ( {0} ) )",
+ note=(
+ "ThoughtSpot has fixed-base log2 and log10 only; base is a "
+ "runtime argument here, not a literal known at catalog time, "
+ "so the general change-of-base composition is the one "
+ "template that is exact for every base. safe_divide rather "
+ "than / guards base = 1."
+ ),
+ ),
+ "LOG10(x)": Construct(
+ "LOG10(x)", Classification.DIRECT, template="log10 ( {0} )",
+ ),
+ "SIN(x)": Construct(
+ "SIN(x)", Classification.DIRECT,
+ template="sin ( {0} * 180 / 3.14159265358979 )",
+ note=(
+ "ThoughtSpot trigonometry is in degrees; the specification is "
+ "in radians. The conversion is mandatory — a bare sin ( {0} ) "
+ "returns the sine of x degrees and is wrong for every "
+ "non-zero input."
+ ),
+ ),
+ "COS(x)": Construct(
+ "COS(x)", Classification.DIRECT,
+ template="cos ( {0} * 180 / 3.14159265358979 )",
+ note="Degrees, as SIN.",
+ ),
+ "TAN(x)": Construct(
+ "TAN(x)", Classification.DIRECT,
+ template="tan ( {0} * 180 / 3.14159265358979 )",
+ note="Degrees, as SIN.",
+ ),
+ "ASIN(x)": Construct(
+ "ASIN(x)", Classification.DIRECT,
+ template="( asin ( {0} ) * 3.14159265358979 / 180 )",
+ note=(
+ "Inverse functions convert the other way: ThoughtSpot returns "
+ "degrees, the specification expects radians."
+ ),
+ ),
+ "ACOS(x)": Construct(
+ "ACOS(x)", Classification.DIRECT,
+ template="( acos ( {0} ) * 3.14159265358979 / 180 )",
+ note="Degrees -> radians, as ASIN.",
+ ),
+ "ATAN(x)": Construct(
+ "ATAN(x)", Classification.DIRECT,
+ template="( atan ( {0} ) * 3.14159265358979 / 180 )",
+ note="Degrees -> radians, as ASIN.",
+ ),
+ "ATAN2(y, x)": Construct(
+ "ATAN2(y, x)", Classification.PASSTHROUGH,
+ template="ATAN2({0}, {1})", variant=Variant.DOUBLE,
+ note=(
+ "atan2 is not a two-argument atan — it is quadrant-aware and "
+ "defined where x = 0. Composing it from atan plus sign tests "
+ "is possible but the branch table is easy to get wrong at the "
+ "axes, so the pass-through is the honest mapping."
+ ),
+ ),
+ "RADIANS(degrees)": Construct(
+ "RADIANS(degrees)", Classification.DIRECT,
+ template="{0} * 3.14159265358979 / 180",
+ note="No native radians; the arithmetic is exact and dialect-free.",
+ ),
+ "DEGREES(radians)": Construct(
+ "DEGREES(radians)", Classification.DIRECT,
+ template="{0} * 180 / 3.14159265358979",
+ note="No native degrees; as RADIANS.",
+ ),
+ "PI()": Construct(
+ "PI()", Classification.DIRECT, template="3.14159265358979",
+ note=(
+ "No native pi. The literal is emitted at the precision "
+ "ThoughtSpot's own documented composites use; "
+ 'sql_double_op ( "pi()" ) is available where full warehouse '
+ "precision matters."
+ ),
+ ),
+ "GREATEST(x, y, ...)": Construct(
+ "GREATEST(x, y, ...)", Classification.DIRECT,
+ template="greatest ( {*} )",
+ variadic=Variadic(VariadicStyle.JOIN),
+ note=(
+ "Not max. ThoughtSpot's max is an aggregate; greatest is the "
+ "row-wise N-ary function. Mapping GREATEST to max would "
+ "collapse the column to one value and also flip it from "
+ "attribute to measure."
+ ),
+ ),
+ "LEAST(x, y, ...)": Construct(
+ "LEAST(x, y, ...)", Classification.DIRECT,
+ template="least ( {*} )",
+ variadic=Variadic(VariadicStyle.JOIN),
+ note="Not min, for the same reason as GREATEST.",
+ ),
+ "IF(condition, true_result, false_result)": Construct(
+ "IF(condition, true_result, false_result)", Classification.DIRECT,
+ template="if ( {0} ) then {1} else {2}",
+ note=(
+ "The parentheses around the condition are mandatory for TML "
+ "import — without them the parser reports \"Expecting keyword "
+ "'('\". Applies to every condition shape, including a bare "
+ "BOOL column reference."
+ ),
+ ),
+ "IFF(condition, true_result, false_result)": Construct(
+ "IFF(condition, true_result, false_result)", Classification.DIRECT,
+ template="if ( {0} ) then {1} else {2}",
+ note="Specification alias for IF.",
+ ),
+ "NULLIF(expr1, expr2)": Construct(
+ "NULLIF(expr1, expr2)", Classification.DIRECT,
+ template="nullif ( {0} , {1} )",
+ ),
+ "COALESCE(expr1, expr2, ...)": Construct(
+ "COALESCE(expr1, expr2, ...)", Classification.DIRECT,
+ template="ifnull ( {0} , {1} )",
+ variadic=Variadic(VariadicStyle.FOLD),
+ note=(
+ "ThoughtSpot's ifnull is strictly two-argument, so an N-ary "
+ "COALESCE becomes a right-nested chain. Two arguments is the "
+ "common case and needs no nesting."
+ ),
+ ),
+ "IFNULL(expr, default)": Construct(
+ "IFNULL(expr, default)", Classification.DIRECT,
+ template="ifnull ( {0} , {1} )",
+ ),
+ "NVL(expr, default)": Construct(
+ "NVL(expr, default)", Classification.DIRECT,
+ template="ifnull ( {0} , {1} )",
+ note="Specification alias for two-argument COALESCE.",
+ ),
+ "NVL2(expr, not_null_result, null_result)": Construct(
+ "NVL2(expr, not_null_result, null_result)", Classification.DIRECT,
+ template="if ( isnotnull ( {0} ) ) then {1} else {2}",
+ note="No native three-way null function; the composition is exact.",
+ ),
+ "ZEROIFNULL(expr)": Construct(
+ "ZEROIFNULL(expr)", Classification.DIRECT,
+ template="ifnull ( {0} , 0 )",
+ ),
+ "NULLIFZERO(expr)": Construct(
+ "NULLIFZERO(expr)", Classification.DIRECT,
+ template="nullif ( {0} , 0 )",
+ ),
+ }
+)
+
+# --------------------------------------------------------------------------
+# Operators and constructs — 33 rows: 30 direct / 2 passthrough /
+# 1 unmappable.
+# Source: docs/ossie/ts-ossie-function-mapping.md, "Operators and constructs"
+# section (thoughtspot-agent-skills repo — not vendored here; prose above/below
+# the table read in full).
+#
+# The document's own section header states that CASE (both forms) and the
+# boolean literals/operators are rowed HERE, not under Conditional functions —
+# confirmed by the arithmetic: 25 Math + 9 Conditional (the Mathematical +
+# Conditional functions family above) + 33 here would double-count CASE otherwise.
+#
+# spec_construct_names() extracts the BARE operator/keyword token for most of
+# this family, not the document's own "a + b"-style worked-example row header —
+# confirmed live before writing this block (see test_catalog_operators.py's
+# docstring for the full list). Six rows have no discrete spec table row at
+# all and are keyed via CONVENTION_DIVERGENCES instead: unary -x/+x, the
+# simple CASE form, Parentheses, the DISTINCT modifier, the column/metric
+# reference, and EXISTS_IN() itself.
+#
+# This family holds the single UNMAPPABLE row in the whole 146-row catalog:
+# EXISTS_IN() is named at :131 as the sanctioned way to filter on a subquery,
+# but the specification defines it nowhere — no signature, no argument order,
+# no semantics, absent from every function table. Construct.__post_init__
+# forbids a template or variant on an UNMAPPABLE row, so this is the one entry
+# in the whole file with neither.
+#
+# LIKE is direct despite ThoughtSpot having no native starts_with/ends_with:
+# the prefix/suffix/contains compositions it needs use only native functions,
+# the same reasoning as the String functions family's STARTSWITH/
+# ENDSWITH rows above. ILIKE is passthrough for the opposite reason —
+# case-insensitive matching has no native form, and the usual lower()-fold
+# workaround is itself a passthrough, so there is nothing to compose from. The
+# DISTINCT aggregate modifier is passthrough for every aggregate except COUNT,
+# which already has its own native unique count row (COUNT(DISTINCT expr),
+# in Aggregate functions above).
+# --------------------------------------------------------------------------
+CATALOG.update(
+ {
+ "+": Construct(
+ "+", Classification.DIRECT, template="{0} + {1}",
+ note=(
+ "Numeric only. ThoughtSpot's + rejects string operands, so a + "
+ "that concatenates on the source side must become concat ( ). "
+ "The specification does not overload +, so this only bites "
+ "when translating a dialect expression."
+ ),
+ ),
+ "-": Construct(
+ "-", Classification.DIRECT, template="{0} - {1}",
+ ),
+ "*": Construct(
+ "*", Classification.DIRECT, template="{0} * {1}",
+ ),
+ "/": Construct(
+ "/", Classification.DIRECT, template="{0} / {1}",
+ note=(
+ "Both yield NULL (or a warehouse error) on divide-by-zero. "
+ "ThoughtSpot's safe_divide returns 0, not NULL, so it is not "
+ "a faithful substitute and is used only where the source "
+ "itself guards the denominator."
+ ),
+ ),
+ "%": Construct(
+ "%", Classification.DIRECT, template="mod ( {0} , {1} )",
+ note="ThoughtSpot has no % operator — the modulo is the function.",
+ ),
+ "-x / +x (unary)": Construct(
+ "-x / +x (unary)", Classification.DIRECT,
+ template="per-spelling — see note",
+ note=(
+ "CONVENTION_DIVERGENCE: unary +/- is named only in the "
+ "'Operator Precedence' list, never a table row. This row "
+ "merges two Ossie spellings that need DIFFERENT output — "
+ "unary minus is -[x] (negation), unary plus is the identity "
+ "([x] unchanged) — so a single {0}-substitutable template "
+ "would be wrong for whichever spelling didn't produce it: "
+ "an earlier draft used template=\"-{0}\", which is correct "
+ "for -x but silently negates a parsed +x node (right arg "
+ "count, wrong semantics, no exception — the arg-count guard "
+ "cannot catch it). Forced external dispatch instead, the "
+ "same treatment as TRUE, FALSE below and CAST's per-type "
+ "table: the caller must choose -{0} or {0} "
+ "unchanged based on which spelling it parsed, rather than "
+ "getting a plausible-looking wrong answer from this row. "
+ "Unary minus is where the bare-date-literal trap "
+ "originates: '2024-05-01' unquoted is parsed as "
+ "2024 - 5 - 1. Date literals are always wrapped in "
+ "to_date ( )."
+ ),
+ ),
+ "=": Construct(
+ "=", Classification.DIRECT, template="{0} = {1}",
+ ),
+ "<>": Construct(
+ "<>", Classification.DIRECT, template="{0} <> {1}",
+ ),
+ "!=": Construct(
+ "!=", Classification.DIRECT, template="{0} != {1}",
+ note="ThoughtSpot accepts both inequality spellings, so the two rows are independent and both direct.",
+ ),
+ "<": Construct(
+ "<", Classification.DIRECT, template="{0} < {1}",
+ ),
+ ">": Construct(
+ ">", Classification.DIRECT, template="{0} > {1}",
+ ),
+ "<=": Construct(
+ "<=", Classification.DIRECT, template="{0} <= {1}",
+ ),
+ ">=": Construct(
+ ">=", Classification.DIRECT, template="{0} >= {1}",
+ ),
+ "expr1 AND expr2": Construct(
+ "expr1 AND expr2", Classification.DIRECT, template="{0} and {1}",
+ note="Lower-case, infix.",
+ ),
+ "expr1 OR expr2": Construct(
+ "expr1 OR expr2", Classification.DIRECT, template="{0} or {1}",
+ note="Lower-case, infix.",
+ ),
+ "NOT expr": Construct(
+ "NOT expr", Classification.DIRECT, template="not ( {0} )",
+ note=(
+ "Function form with parentheses, not a prefix operator — "
+ "not [x] does not parse."
+ ),
+ ),
+ "BETWEEN": Construct(
+ "BETWEEN", Classification.DIRECT,
+ template="{0} between {1} and {2}",
+ note="Inclusive on both sides.",
+ ),
+ "IN": Construct(
+ "IN", Classification.DIRECT,
+ template="{0} in {{ {*} }}",
+ variadic=Variadic(VariadicStyle.JOIN, tail_from=1),
+ note=(
+ "Literal lists only on both sides — no subqueries. The "
+ "curly-brace delimiter is confirmed, live-verified "
+ "2026-07-29: the round-parenthesis form is rejected with "
+ "'Expecting one of the valid keywords, such as, \"ts_var\", "
+ "\"{\"'. It forces >- block-scalar YAML. The braces are "
+ "doubled ({{ }}) in the template because emit_direct renders "
+ "via str.format, which reads a single literal brace as the "
+ "start of a field name (see test_emit.py's catalog-wide "
+ "sweep)."
+ ),
+ ),
+ "NOT IN": Construct(
+ "NOT IN", Classification.DIRECT,
+ template="not ( {0} in {{ {*} }} )",
+ variadic=Variadic(VariadicStyle.JOIN, tail_from=1),
+ note=(
+ "Emitted as a negated in rather than a not in keyword — the "
+ "bare keyword form is not reliably accepted. Braces doubled "
+ "for str.format, as IN above."
+ ),
+ ),
+ "str LIKE pattern": Construct(
+ "str LIKE pattern", Classification.DIRECT,
+ template="per-pattern-shape — see note",
+ note=(
+ "Prefix ('foo%') -> strpos ( {0} , 'foo' ) = 1; suffix "
+ "('%foo') -> substr ( {0} , strlen ( {0} ) - strlen ( 'foo' "
+ ") , strlen ( 'foo' ) ) = 'foo'; contains ('%foo%') -> "
+ "contains ( {0} , 'foo' ). Only contains is a native "
+ "function — starts_with and ends_with do not exist "
+ "(live-verified 2026-07-29), so the "
+ "first two shapes are compositions of native functions, "
+ "same as the STARTSWITH/ENDSWITH rows. These "
+ "three shapes are the overwhelming majority of LIKE use. "
+ "Interior wildcards and any _ single-character wildcard have "
+ "no native form and fall back to "
+ 'sql_bool_op ( "{0} LIKE {1}" , [s] , [pattern] ). '
+ "The per-pattern-shape dispatch is out of this catalog's "
+ "scope, same treatment as CAST's per-type dispatch "
+ "— the actual pattern literal is a runtime value, not known "
+ "at catalog-construction time."
+ ),
+ ),
+ "str ILIKE pattern": Construct(
+ "str ILIKE pattern", Classification.PASSTHROUGH,
+ template="{0} ILIKE {1}", variant=Variant.BOOL,
+ note=(
+ "Case-insensitive matching has no native form, and the "
+ "usual workaround — fold both sides with lower — is itself "
+ "a pass-through, so there is nothing to compose from."
+ ),
+ ),
+ "IS NULL": Construct(
+ "IS NULL", Classification.DIRECT, template="isnull ( {0} )",
+ ),
+ "IS NOT NULL": Construct(
+ "IS NOT NULL", Classification.DIRECT, template="isnotnull ( {0} )",
+ note="Native, so not composed as not ( isnull ( ) ).",
+ ),
+ "IS DISTINCT FROM": Construct(
+ "IS DISTINCT FROM", Classification.DIRECT,
+ template=(
+ "if ( isnull ( {0} ) and isnull ( {1} ) ) then false else "
+ "if ( isnull ( {0} ) or isnull ( {1} ) ) then true else "
+ "{0} != {1}"
+ ),
+ note=(
+ "No native null-safe comparison, but the three-case truth "
+ "table is exactly expressible. The nesting order matters: "
+ "both-null must be tested before either-null."
+ ),
+ ),
+ "IS NOT DISTINCT FROM": Construct(
+ "IS NOT DISTINCT FROM", Classification.DIRECT,
+ template=(
+ "if ( isnull ( {0} ) and isnull ( {1} ) ) then true else "
+ "if ( isnull ( {0} ) or isnull ( {1} ) ) then false else "
+ "{0} = {1}"
+ ),
+ note=(
+ "The negation of the row above, written directly rather "
+ "than wrapped in not ( ) — one fewer nesting level for the "
+ "parser."
+ ),
+ ),
+ "CASE WHEN": Construct(
+ "CASE WHEN", Classification.DIRECT,
+ template="if ( c1 ) then r1 else if ( c2 ) then r2 else d",
+ note=(
+ "The searched CASE WHEN c1 THEN r1 ... ELSE d END form. No "
+ "native CASE; the chain is else if, two words. The final "
+ "else is mandatory and must be type-matched — else 0 for a "
+ "measure, else '' for an attribute. Omitting it raises "
+ "'Unknown data type', and a CASE with no ELSE (legal in the "
+ "specification, yielding NULL) therefore needs one "
+ "synthesised. The branch count is unbounded, so the "
+ "template uses symbolic c1/r1/c2/r2/d names rather than "
+ "being forced into a fixed {0}/{1} scheme — the same "
+ "out-of-scope-dispatch treatment "
+ "as CAST's per-type table."
+ ),
+ ),
+ "CASE expr WHEN v1 THEN r1 ... END (simple)": Construct(
+ "CASE expr WHEN v1 THEN r1 ... END (simple)", Classification.DIRECT,
+ template="if ( [expr] = v1 ) then r1 else if ( [expr] = v2 ) then r2 else d",
+ note=(
+ "CONVENTION_DIVERGENCE: the simple CASE form is described "
+ "only in the CASE Expression code fence, never a table row. "
+ "Expanded to the searched form with an explicit equality "
+ "per branch. expr is repeated per branch, so a converter "
+ "should hoist an expensive expr into its own formula first. "
+ "Symbolic template, as CASE WHEN above, for the same "
+ "unbounded-branch-count reason."
+ ),
+ ),
+ "str1 || str2": Construct(
+ "str1 || str2", Classification.DIRECT, template="concat ( {0} , {1} )",
+ note=(
+ "ThoughtSpot has no concatenation operator at all — + is "
+ "numeric-only — so || and CONCAT share one target."
+ ),
+ ),
+ "Parentheses — expression grouping": Construct(
+ "Parentheses — expression grouping", Classification.DIRECT,
+ template="( {0} )",
+ note=(
+ "CONVENTION_DIVERGENCE: its Supported SQL Constructs row "
+ "carries no backtick token in either cell, the only marker "
+ "the top-table extraction keys on. Precedence is the "
+ "standard SQL ordering on the Ossie side. The converter "
+ "emits explicit parentheses around every rewritten "
+ "sub-expression rather than relying on the two languages "
+ "agreeing about precedence — cheap, and it removes a whole "
+ "class of silent arithmetic errors."
+ ),
+ ),
+ "TRUE, FALSE": Construct(
+ "TRUE, FALSE", Classification.DIRECT, template="true / false",
+ note=(
+ "The Boolean Functions table's Syntax cell merges TRUE and "
+ "FALSE into one comma-joined entry, matching what "
+ "spec_construct_names() extracts. Which of the two "
+ "lower-case literals is emitted depends on which the source "
+ "wrote — TRUE -> true, FALSE -> false — resolved "
+ "per-occurrence, out of this catalog's scope (same as "
+ "CAST's per-type dispatch). A bare BOOL column reference "
+ "used as a condition still needs its parentheses: "
+ "if ( [T::flag] ) then ... parses, if [T::flag] then ... "
+ "does not."
+ ),
+ ),
+ "DISTINCT aggregate modifier": Construct(
+ "DISTINCT aggregate modifier", Classification.PASSTHROUGH,
+ template="SUM(DISTINCT {0})",
+ exemplar_literals=("the aggregate",),
+ variant=Variant.NUMBER_AGGREGATE,
+ note=(
+ "CONVENTION_DIVERGENCE: described only in the Conditional "
+ "Aggregations prose/code block, never a table row. The "
+ "specification allows DISTINCT on SUM as well as COUNT. "
+ "ThoughtSpot has exactly one distinct-aware aggregate — "
+ "unique count — which is COUNT(DISTINCT) and already has "
+ "its own row. Every other DISTINCT aggregate is a "
+ "pass-through."
+ ),
+ ),
+ "Column / metric reference — field, dataset.field": Construct(
+ "Column / metric reference — field, dataset.field",
+ Classification.DIRECT,
+ template="[TABLE::Column], or [Formula Name] for a metric",
+ note=(
+ "CONVENTION_DIVERGENCE: its Supported SQL Constructs row "
+ "carries no backtick token in either cell, same reason as "
+ "Parentheses. Always rewritten from resolved metadata, "
+ "never passed through textually — the rewrite, the "
+ "case-sensitivity rules and the display-name-versus-"
+ "identifier problem are out of this catalog's scope."
+ ),
+ ),
+ "EXISTS_IN()": Construct(
+ "EXISTS_IN()", Classification.UNMAPPABLE,
+ note=(
+ "CONVENTION_DIVERGENCE: named only in the Reason column of "
+ "the excluded 'Not Supported in Expressions' table, never "
+ "in a table of its own. The single unmappable row in the "
+ "whole 146-row catalog: named at :131 as the sanctioned way "
+ "to filter on a subquery, but defined nowhere in the "
+ "specification — no signature, no argument order, no "
+ "semantics, absent from every function table. Even given a "
+ "signature, ThoughtSpot's nearest capability is a "
+ "sql_bool_op subquery template that requires a "
+ "fully-qualified warehouse table name, which is not "
+ "derivable from an Ossie expression."
+ ),
+ ),
+ }
+)
+
+# --------------------------------------------------------------------------
+# Window functions — 14 rows: 5 direct / 9 passthrough / 0 unmappable.
+# Source: docs/ossie/ts-ossie-function-mapping.md, "Window functions" section, plus
+# "Window rows live-confirmed — 2026-07-30" (thoughtspot-agent-skills repo — not
+# vendored here; prose above/below the table read in full).
+#
+# This is the hardest family, and the last one — it completes the 146-row catalog.
+# Three constraints govern it:
+#
+# - A raw aggregate cannot be nested inside a ThoughtSpot window function. The
+# argument must be a column reference or a group_aggregate ( ... ). Live-confirmed
+# both directions: the raw-aggregate form is rejected, the group_aggregate form
+# validates, for moving_* and cumulative_* alike.
+# - ThoughtSpot's ORDER BY column must be a physical column reference, not a
+# formula. A formula column in the sort position fails to resolve.
+# - A ThoughtSpot window formula cannot declare its own PARTITION BY; the
+# window shape is completed from the search context. There is no argument slot
+# for a partition and none can be added — live-confirmed by rejection,
+# 2026-07-30 (a fifth { [attr] } or query_groups ( ) argument to moving_sum,
+# and a third to cumulative_sum, are both rejected at the parser). rank /
+# rank_percentile are the stricter case: arity fixed at exactly two, enforced
+# ("Function rank expects only 2 arguments"), so they are always global. This
+# is why LAG, LEAD, the OVER clause and window aggregation moved
+# direct -> passthrough in the 2026-07-30 rework (52 live probes, 31 accepted /
+# 21 rejected) — a native idiom (moving_sum as the LAG/LEAD idiom) exists but
+# is NOT equivalent to any OVER shape, because it has no partition slot and
+# ThoughtSpot's partition is never empty.
+#
+# FIRST_VALUE/LAST_VALUE are the section's one exception: they take a genuine,
+# explicit partition argument and a genuine, explicit order axis — both
+# live-confirmed accepted, including a multi-column fixed partition — so the
+# formula does define its own window, and they stay direct. RANK/PERCENT_RANK and
+# the frame-clause boundaries also stay direct, with their native reach now proven
+# by rejection rather than asserted.
+#
+# PERCENT_RANK is direct via rank_percentile (both the 0-100 scale and the
+# inversion are required); CUME_DIST is NOT a rank_percentile substitute —
+# PERCENT_RANK divides by n-1 and starts at 0, CUME_DIST divides by n and ends at
+# 1 — so CUME_DIST stays passthrough with no native fallback at all.
+#
+# RANK, PERCENT_RANK, FIRST_VALUE and LAST_VALUE record the document's own worked
+# example verbatim — symbolic bracket names ([m], [dim], [ord], [attr], [T::date]),
+# not numbered {0}/{1} substitution slots — because resolving which actual column
+# fills each slot needs model metadata not known at catalog-construction time; same
+# out-of-scope-dispatch treatment as CASE WHEN's c1/r1 names and CAST's per-type
+# table. FIRST_VALUE/LAST_VALUE's worked example carries ThoughtSpot's literal
+# `{ [T::date] }` list syntax for the axis argument; since these are DIRECT rows
+# rendered via emit_direct's str.format, the literal braces are doubled ({{ }})
+# the same fix the IN/NOT IN rows above need for the same reason — verified here by
+# actually calling emit_direct and checking the rendered output has single braces
+# again (see test_catalog_window.py).
+#
+# Three of the 14 rows have no discrete row of their own in the upstream spec —
+# the OVER clause, the frame clause and window aggregation are keyed via the
+# pre-existing CONVENTION_DIVERGENCES entries (copied verbatim, not retyped, to
+# avoid an em-dash/ellipsis mismatch) rather than spec_construct_names(). The
+# other 11 key on spec_construct_names()'s own extraction from the "Ranking
+# Functions" and "Offset Functions" tables' Syntax column — confirmed live before
+# writing this block.
+# --------------------------------------------------------------------------
+CATALOG.update(
+ {
+ "ROW_NUMBER() OVER (...)": Construct(
+ "ROW_NUMBER() OVER (...)", Classification.PASSTHROUGH,
+ template="ROW_NUMBER() OVER (PARTITION BY {0} ORDER BY {1})",
+ variant=Variant.INT_AGGREGATE,
+ note=(
+ "ThoughtSpot's rank is competition rank, not a row number, so it "
+ "is not a substitute. Wrap in group_aggregate so the "
+ "partition column reaches the GROUP BY even when the user's "
+ "search omits it."
+ ),
+ ),
+ "RANK() OVER (...)": Construct(
+ "RANK() OVER (...)", Classification.DIRECT,
+ template="rank ( sum ( [m] ) , 'desc' )",
+ note=(
+ "direct for one shape only, and the boundary is proven rather "
+ "than asserted: the global, ORDER BY-only form over an "
+ "aggregate. Live-confirmed 2026-07-30: rank ( sum ( [m] ) , "
+ "'desc' ) and 'asc' both validate, and the arity is enforced at "
+ "exactly two — a third argument in any shape (bare attribute, "
+ "{ [attr] }, or query_groups ( )) is rejected with 'Function "
+ "rank expects only 2 arguments', so an explicit PARTITION BY is "
+ "provably not expressible. Two further live-proven "
+ "restrictions: the first argument must be aggregated (rank "
+ "( [m] , 'desc' ) -> 'Function rank expects 1st argument to be "
+ "aggregated'), so an Ossie ORDER BY <non-aggregated column> has "
+ "no native target either; and it may not be a "
+ "group_aggregate ( ... ), so the partition cannot be smuggled "
+ "in through the measure. Every non-covered shape falls back to "
+ "sql_int_aggregate_op ( \"RANK() OVER (PARTITION BY {0} ORDER "
+ "BY SUM({1}) DESC)\" , ... ), wrapped in group_aggregate. Query-context "
+ "caveat: rank carries no dynamic partition but it is "
+ "evaluated over the query's result rows, so the covered shape "
+ "is faithful to RANK() OVER (ORDER BY ...) only when the search "
+ "returns the grain the expression assumed — a query-time "
+ "semantic no import probe can observe, taken from ThoughtSpot's "
+ "formula documentation rather than this run. The direction "
+ "string is not validated at import ('descending' was "
+ "accepted), so acceptance proves the call shape, never the "
+ "ordering."
+ ),
+ ),
+ "DENSE_RANK() OVER (...)": Construct(
+ "DENSE_RANK() OVER (...)", Classification.PASSTHROUGH,
+ template="dense_rank() over (order by sum({0}) desc)",
+ exemplar_literals=("the ordering aggregate", "the sort direction"),
+ variant=Variant.INT_AGGREGATE,
+ note=(
+ "ThoughtSpot's rank skips ranks after a tie; dense ranking has "
+ "no native form — live-confirmed 2026-07-30, dense_rank ( ... ) "
+ "rejected with 'Search did not find \"dense_rank ( sum (\"'. "
+ "Passthrough is correct: no native ThoughtSpot construct produces "
+ "dense-rank semantics."
+ ),
+ ),
+ "NTILE(n) OVER (...)": Construct(
+ "NTILE(n) OVER (...)", Classification.PASSTHROUGH,
+ template="NTILE(4) OVER (ORDER BY SUM({0}))",
+ exemplar_literals=("n", "the ordering aggregate"),
+ variant=Variant.INT_AGGREGATE,
+ note=(
+ "n is a literal, baked into the template, as the aggregate "
+ "percentiles are. So is the SUM: the specification orders an "
+ "NTILE by whatever the caller chose, and this row illustrates "
+ "with one aggregate rather than mapping only that one."
+ ),
+ ),
+ "PERCENT_RANK() OVER (...)": Construct(
+ "PERCENT_RANK() OVER (...)", Classification.DIRECT,
+ template="1 - rank_percentile ( sum ( [m] ) , 'asc' ) / 100",
+ note=(
+ "ThoughtSpot's rank_percentile is documented as "
+ "(1.0 - PERCENT_RANK() OVER (ORDER BY ...)) * 100, so the "
+ "inverse is exact. Two adjustments are both required: the "
+ "scale (ThoughtSpot 0-100, specification 0-1) and the "
+ "inversion. Dropping either produces a plausible-looking "
+ "column that is wrong everywhere. Same shape restriction as "
+ "RANK, and the same live-proven boundary — rank_percentile is "
+ "also fixed at exactly two arguments ('Function rank_percentile "
+ "expects only 2 arguments', live-verified 2026-07-30), so it "
+ "too is global-only and an explicit PARTITION BY falls back to "
+ "sql_number_aggregate_op ( \"PERCENT_RANK() OVER (PARTITION BY "
+ "{0} ORDER BY SUM({1}))\" , ... ). Same evidence-class "
+ "caveat as RANK: the arity is probe-proven, the global-window "
+ "semantic is documentation-derived. CUME_DIST is deliberately "
+ "NOT given this same composition — see that row."
+ ),
+ ),
+ "CUME_DIST() OVER (...)": Construct(
+ "CUME_DIST() OVER (...)", Classification.PASSTHROUGH,
+ template="CUME_DIST() OVER (ORDER BY SUM({0}))",
+ exemplar_literals=("the ordering aggregate",),
+ variant=Variant.NUMBER_AGGREGATE,
+ note=(
+ "rank_percentile is NOT a substitute, despite PERCENT_RANK's "
+ "row looking equivalent: PERCENT_RANK divides by n - 1 and "
+ "starts at 0; CUME_DIST divides by n and ends at 1. They agree "
+ "on no row of a tie-free window except the last, so there is no "
+ "native fallback at all for this row."
+ ),
+ ),
+ "LAG(expr, offset, default) OVER (...)": Construct(
+ "LAG(expr, offset, default) OVER (...)", Classification.PASSTHROUGH,
+ template="LAG({0}, 1, 0) OVER (PARTITION BY {1} ORDER BY {2})",
+ exemplar_literals=("offset", "default"),
+ variant=Variant.NUMBER_AGGREGATE,
+ note=(
+ "Reclassified direct -> passthrough 2026-07-30. The "
+ "native idiom moving_sum ( [m] , n , -n , [ord] ) is real and "
+ "validates (a frame of n PRECEDING to n PRECEDING) but is not "
+ "equivalent to any OVER shape: moving_sum has no partition "
+ "slot, and ThoughtSpot completes the partition from the "
+ "query's own dimensions instead. So an Ossie LAG with a "
+ "PARTITION BY cannot be expressed, and one without a "
+ "PARTITION BY still cannot, because ThoughtSpot's partition is "
+ "not empty. The converter emits the pass-through by default "
+ "and offers the native moving_sum idiom as a documented "
+ "downgrade the user must accept: correct exactly when the "
+ "search's dimensions are the intended partition. The default "
+ "argument has no equivalent in the native idiom — ThoughtSpot "
+ "yields null outside the frame — a second reason the native "
+ "form is a downgrade (the pass-through carries default fine). "
+ "Subject to the same aggregation and physical-ORDER-BY-column "
+ "constraints as the rest of this family. Variant recorded here is the documented "
+ "default (sql_number_aggregate_op); the typed sibling applies "
+ "for a non-numeric expr — LAG returns its argument's own type, "
+ "not an aggregate, so a string-typed expr (LAG(order_status, "
+ "1) OVER (...)) needs the typed sibling, not this default, or "
+ "it imports cleanly and aggregates wrongly."
+ ),
+ ),
+ "LEAD(expr, offset, default) OVER (...)": Construct(
+ "LEAD(expr, offset, default) OVER (...)", Classification.PASSTHROUGH,
+ template="LEAD({0}, 1, 0) OVER (PARTITION BY {1} ORDER BY {2})",
+ exemplar_literals=("offset", "default"),
+ variant=Variant.NUMBER_AGGREGATE,
+ note=(
+ "Mirror of LAG, reclassified for the same reason and on the "
+ "same date. The native downgrade is "
+ "moving_sum ( [m] , -n , n , [ord] ) — ThoughtSpot's start/end "
+ "arguments use opposite sign conventions, so a forward offset "
+ "is a negative start (both live-confirmed 2026-07-30). Same "
+ "default limitation as LAG. Variant recorded here is the "
+ "documented default (sql_number_aggregate_op); the typed "
+ "sibling applies for a non-numeric expr, same reason as LAG's "
+ "note — LEAD returns its argument's own type, not an aggregate."
+ ),
+ ),
+ "FIRST_VALUE(expr) OVER (...)": Construct(
+ "FIRST_VALUE(expr) OVER (...)", Classification.DIRECT,
+ template="first_value ( sum ( [m] ) , query_groups ( ) , {{ [T::date] }} )",
+ note=(
+ "The section's exception, and the only window row whose direct "
+ "verdict survived the 2026-07-30 rework — first_value takes a "
+ "genuine explicit partition argument and a genuine explicit "
+ "order axis, so the formula does define its own window. "
+ "Live-confirmed 2026-07-30: query_groups ( ), "
+ "a fixed single-column { [attr] }, a multi-column "
+ "{ [a] , [b] }, the grand-total { } and the dynamic "
+ "query_groups ( ) - { [attr] } all validate in the partition "
+ "slot, so a static Ossie PARTITION BY list maps straight onto "
+ "it. The axis slot is typed and enforced — a bare column "
+ "reference is rejected with 'Function last_value expects 3rd "
+ "argument to be List', so the { } braces are mandatory (and "
+ "force >- block-scalar YAML on the document side; doubled here "
+ "as {{ }} because emit_direct renders via str.format, the same "
+ "fix the IN/NOT IN rows above need for the same reason — "
+ "verified by calling emit_direct and checking the rendered "
+ "output has single braces again). "
+ "Two boundaries remain: ThoughtSpot's first_value is a "
+ "semi-additive function over a date axis rather than a general "
+ "window function, so an OVER shape with a row frame other than "
+ "the whole partition falls back to "
+ "sql_number_aggregate_op ( \"FIRST_VALUE({0}) OVER (...)\" , "
+ "... ); and the axis column's type is not validated at "
+ "import (a VARCHAR axis was accepted), so acceptance proves "
+ "the call shape, not that the axis is temporal."
+ ),
+ ),
+ "LAST_VALUE(expr) OVER (...)": Construct(
+ "LAST_VALUE(expr) OVER (...)", Classification.DIRECT,
+ template="last_value ( sum ( [m] ) , query_groups ( ) , {{ [T::date] }} )",
+ note=(
+ "Same conditions, same live evidence and same fallback as "
+ "FIRST_VALUE. last_value_in_period and first_value_in_period "
+ "also validate in the identical three-argument shape and are "
+ "the period-completeness variants (see the reverse-direction "
+ "table) — out of this row's scope. Braces doubled on the axis "
+ "argument for the same str.format reason as FIRST_VALUE."
+ ),
+ ),
+ "NTH_VALUE(expr, n) OVER (...)": Construct(
+ "NTH_VALUE(expr, n) OVER (...)", Classification.PASSTHROUGH,
+ template="NTH_VALUE({0}, 2) OVER (ORDER BY {1})",
+ exemplar_literals=("n",),
+ variant=Variant.NUMBER_AGGREGATE,
+ note=(
+ "ThoughtSpot's semi-additive functions reach only the first "
+ "and last values of the axis — live-confirmed 2026-07-30, "
+ "nth_value ( ... ) rejected with 'Search did not find "
+ "\"nth_value ( sum (\"'. n is a literal, baked into the "
+ "template, as NTILE's. Variant recorded here is the "
+ "documented default (sql_number_aggregate_op); the typed "
+ "sibling applies for a non-numeric expr, same reason as LAG's "
+ "note — NTH_VALUE returns its argument's own type, not an "
+ "aggregate."
+ ),
+ ),
+ "OVER (PARTITION BY ... ORDER BY ...) clause": Construct(
+ "OVER (PARTITION BY ... ORDER BY ...) clause", Classification.PASSTHROUGH,
+ template="per-clause-shape — see note",
+ variant=Variant.NUMBER_AGGREGATE,
+ note=(
+ "CONVENTION_DIVERGENCE: the generic OVER syntax template is a "
+ "fenced code block, not a table. Reclassified direct -> "
+ "passthrough 2026-07-30. The previous verdict claimed a clean "
+ "structural rewrite — 'PARTITION BY attrs becomes the "
+ "group_aggregate grouping argument; ORDER BY becomes the "
+ "window function's trailing attribute arguments' — but that "
+ "holds for PARTITION BY alone and breaks the moment an "
+ "ORDER BY is present, which is most window use. There are two "
+ "disjoint targets and only one accepts a partition: an OVER "
+ "clause with a PARTITION BY and no ORDER BY/frame is "
+ "group_aggregate ( agg ( [m] ) , { [T::a] , [T::b] } , "
+ "query_filters ( ) ) and is lossless; an OVER clause with an "
+ "ORDER BY must target moving_*/cumulative_*, which have no "
+ "partition slot at all. Live-confirmed accepted: a "
+ "fixed single-column grouping { [T::pk] } inside "
+ "group_aggregate (as a moving_* and a cumulative_* argument), "
+ "and query_groups ( ) - { [attr] } / "
+ "query_groups ( ) + { [attr] } inside group_aggregate. "
+ "Live-confirmed rejected: moving_sum ( ... , [ord] , "
+ "{ [attr] } ) and moving_sum ( ... , [ord] , query_groups ( ) "
+ "), plus cumulative_sum ( ... , [ord] , { [attr] } ). Not "
+ "probed: a bare { } or a bare query_groups ( ) as the "
+ "group_aggregate grouping argument, and the query_groups ( ) "
+ "form of the cumulative_sum rejection — those three cells rest "
+ "on the formula reference, not this run. A partitioned, "
+ "ordered window therefore has no native home and the whole "
+ "clause is out of catalog scope for the general case — "
+ "template records the dispatch rather than one substitutable "
+ "body, same treatment as CAST's per-type table. Variant "
+ "recorded here is the documented default "
+ "(sql_number_aggregate_op); the typed sibling applies for a "
+ "non-numeric aggregate. The reverse direction is lossy for the "
+ "mirror-image reason — ThoughtSpot's ordered window functions "
+ "add the query's own dimensions to the partition dynamically, "
+ "which the specification cannot express."
+ ),
+ ),
+ "Frame clause — ROWS BETWEEN ... / RANGE BETWEEN ...": Construct(
+ "Frame clause — ROWS BETWEEN ... / RANGE BETWEEN ...", Classification.DIRECT,
+ template="per-frame-shape — see note",
+ note=(
+ "CONVENTION_DIVERGENCE: frame options are a bullet list under "
+ "the OVER syntax section, not a table. direct for the frame "
+ "boundaries only — deliberately scoped, so the partition loss "
+ "is counted once, on the OVER clause row, and not twice. "
+ "ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW -> "
+ "cumulative_*. Bounded ROWS frames -> moving_* with "
+ "n PRECEDING -> positive n, CURRENT ROW -> 0, n FOLLOWING -> "
+ "negative -n. All four boundary shapes were live-confirmed, "
+ "2026-07-30 (moving_sum ( [m] , 2 , 0 , [ord] "
+ "), ( ... , 1 , -1 , ... ), ( ... , -1 , 1 , ... ), "
+ "cumulative_sum ( [m] , [ord] )), and the positional signature "
+ "is enforced — moving_sum ( [m] , [ord] ) is rejected with "
+ "'Function moving_sum expects 2nd argument to be Numeric'. "
+ "RANGE frames fall back to sql_number_aggregate_op (the same "
+ "variant the window-aggregation row below falls back to): "
+ "ThoughtSpot's frames are row-positional, not value-ranged — "
+ "live-verified on gapped dates, moving_* counts surviving rows "
+ "regardless of the calendar distance between them — so a "
+ "RANGE frame over a gapped sort column would silently return "
+ "different numbers. A frame reaches ThoughtSpot natively "
+ "only when the accompanying OVER clause declares no "
+ "PARTITION BY; otherwise it is emitted verbatim inside the "
+ "pass-through template the OVER row selects. Per-shape "
+ "dispatch out of catalog scope, same treatment as CAST's "
+ "per-type table."
+ ),
+ ),
+ "Window aggregation — AGG(expr) OVER (...)": Construct(
+ "Window aggregation — AGG(expr) OVER (...)", Classification.PASSTHROUGH,
+ template="SUM({0}) OVER (PARTITION BY {1} ORDER BY {2} "
+ "ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)",
+ exemplar_literals=("the aggregate", "the window frame"),
+ variant=Variant.NUMBER_AGGREGATE,
+ note=(
+ "CONVENTION_DIVERGENCE: the Window Aggregations section is "
+ "prose and code examples, not a table. Reclassified direct -> "
+ "passthrough 2026-07-30, inheriting the OVER row's problem: "
+ "the specification allows every aggregate as a window "
+ "function, but every ordered ThoughtSpot target "
+ "(cumulative_*, moving_*) completes its partition from the "
+ "query. The unordered case remains lossless and is the "
+ "group_aggregate path on the OVER row. The native family is "
+ "also narrower than the specification's: cumulative_*/"
+ "moving_* cover SUM, AVG, MIN and MAX only — live-confirmed "
+ "2026-07-30 that moving_count, moving_stddev and "
+ "cumulative_count do not exist ('Search did not find "
+ "\"moving_count (\"' and siblings) — so a windowed COUNT, "
+ "MEDIAN, STDDEV or VARIANCE has a partitioned form via "
+ "group_count/group_stddev/group_variance and no ordered or "
+ "framed form of any kind. The frame is an exemplar, the same "
+ "convention as NTILE's literal 4 (see the Construct.template "
+ "docstring): ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW "
+ "is the cumulative-aggregate boundary the Frame clause row "
+ "above maps cumulative_* to, and is one concrete, valid frame "
+ "among the ones a real occurrence could carry — a caller "
+ "rebuilds the frame per occurrence, same as any other "
+ "exemplar row. The mapping document's own cell for this row "
+ "writes the frame as a literal ellipsis ('ROWS BETWEEN …'), "
+ "which is prose shorthand for 'a frame clause goes here', not "
+ "renderable SQL — transcribing it verbatim rendered warehouse "
+ "syntax errors at query time, so this template supplies a "
+ "concrete, valid frame instead. Variant recorded here is the "
+ "documented default (sql_number_aggregate_op); the typed "
+ "sibling applies for a non-numeric aggregate. The argument must still be "
+ "an aggregate — a raw column reference cannot be nested inside window "
+ "aggregation."
+ ),
+ ),
+ }
+)
+
+#: Constructs the mapping document (docs/ossie/ts-ossie-function-mapping.md in the
+#: thoughtspot-agent-skills repo) counts separately ("one row per
+#: construct") that core-spec/expression_language.md does not give a discrete
+#: table row of their own. Each entry records WHY it diverges. This is NOT an
+#: escape hatch for missing coverage: the 137 names in spec_construct_names() are
+#: still parsed from the live upstream file, so an upstream addition still fails
+#: the build. It exists because the mapping document's 146-row census counts by a
+#: different unit (one row per construct, including constructs the spec only
+#: describes in prose) than spec_construct_names() counts by (one entry per
+#: parseable table row / heading). Verified directly against the mapping
+#: document's actual row list — see catalog.py's docstring for the
+#: reconciliation (137 + 9 == 146).
+#:
+#: Two mechanisms an earlier pass mistakenly guessed would appear here do NOT:
+#: `CEIL(x)`/`CEILING(x)`, `TRUNC(x, d)`/`TRUNCATE(x, d)` and `TRUE`/`FALSE` are
+#: each ONE row in the mapping document too (not split), matching spec_name's
+#: single merged entry — a spelling-convention question addressed throughout
+#: this file (see the module docstring's "Spelling" note), not a divergence.
+CONVENTION_DIVERGENCES: dict[str, str] = {
+ "-x / +x (unary)": (
+ "unary +/- is named only in the 'Operator Precedence' list "
+ "(core-spec/expression_language.md:142), never a table row"
+ ),
+ "CASE expr WHEN v1 THEN r1 ... END (simple)": (
+ "simple CASE is described only in the CASE Expression code fence "
+ "(core-spec/expression_language.md:508-513) alongside searched CASE; "
+ "the top-level summary table's single bare 'CASE WHEN' token covers "
+ "the searched form and does not extend to this one"
+ ),
+ "Parentheses — expression grouping": (
+ "its Supported SQL Constructs row (core-spec/expression_language.md:120) "
+ "carries no backtick token in either cell, the only marker the "
+ "top-table extraction keys on"
+ ),
+ "DISTINCT aggregate modifier": (
+ "the DISTINCT modifier is described only in the Conditional "
+ "Aggregations prose/code block (core-spec/expression_language.md:219-230), "
+ "never a table row"
+ ),
+ "Column / metric reference — field, dataset.field": (
+ "its Supported SQL Constructs row (core-spec/expression_language.md:108) "
+ "carries no backtick token in either cell, same reason as Parentheses"
+ ),
+ "EXISTS_IN()": (
+ "named only in the Reason column of the excluded 'Not Supported in "
+ "Expressions' table (core-spec/expression_language.md:131), never in a "
+ "table of its own"
+ ),
+ "OVER (PARTITION BY ... ORDER BY ...) clause": (
+ "the generic OVER syntax template (core-spec/expression_language.md:548-560) "
+ "is a fenced code block, not a table"
+ ),
+ "Frame clause — ROWS BETWEEN ... / RANGE BETWEEN ...": (
+ "frame options are a bullet list under the OVER syntax section "
+ "(core-spec/expression_language.md:556-560), not a table"
+ ),
+ "Window aggregation — AGG(expr) OVER (...)": (
+ "the Window Aggregations section (core-spec/expression_language.md:583-599) "
+ "is prose and code examples, not a table"
+ ),
+}
+
+
+# --------------------------------------------------------------------------
+# spec_construct_names(): the upstream-spec oracle.
+# --------------------------------------------------------------------------
+
+_HEADING_RE = re.compile(r"^(#{1,6})\s+(.*)$")
+_CODE_SPAN_RE = re.compile(r"`([^`]+)`")
+
+# A section whose entire content is about something *other than* an Ossie
+# construct in its own right. Matched case-insensitively as a substring of
+# the heading text at any level, so a subsection ("### Common Dialect
+# Variations" under "## Dialect Extensions") is caught the same way as a
+# top-level one ("## Cross-Reference: Tool Mappings").
+_EXCLUDED_SECTION_MARKERS = (
+ "not supported", # "Not Supported in Expressions": SELECT/FROM/GROUP BY/... are
+ # explicitly things Ossie expressions do NOT support - the opposite of a construct.
+ "dialect variation", # "Common Dialect Variations": other engines' spellings of
+ # constructs already counted from their Ossie-standard table.
+ "cross-reference", # "Cross-Reference: Tool Mappings": Tableau / Looker Studio / DAX
+ # spellings, not Ossie constructs.
+)
+
+# A standalone H3 section that defines exactly one construct via prose/a code
+# fence rather than a table - e.g. "### CAST (REQUIRED)". Matches only when
+# the heading's own name is a single token (letters, digits, underscore, or a
+# markdown-escaped underscore "\_"), which is what distinguishes "CAST" or
+# "TRY\_CAST" from a descriptive multi-word heading like "Alternative
+# Extraction Syntax" or "Null-Safe Comparison" (those need bespoke handling
+# below, because more than one construct - or a construct whose name isn't
+# the heading text - lives in their body).
+_SINGLE_CONSTRUCT_HEADING_RE = re.compile(
+ r"^### ([A-Za-z0-9_]+(?:\\_[A-Za-z0-9_]+)*) \((?:REQUIRED|RECOMMENDED|EXPERIMENTAL)\)\s*$",
+ re.MULTILINE,
+)
+
+
+def _find_spec_path() -> Path:
+ """Walk up from this file to the repository root and locate the upstream spec."""
+ here = Path(__file__).resolve()
+ for parent in here.parents:
+ candidate = parent / "core-spec" / "expression_language.md"
+ if candidate.is_file():
+ return candidate
+ raise FileNotFoundError(
+ f"core-spec/expression_language.md not found by walking up from {here}"
+ )
+
+
+def _split_table_row(line: str) -> list[str]:
+ """Split a markdown table row on '|', but never inside a backtick code span.
+
+ The document contains at least one cell whose code span itself contains a
+ pipe character - the `||` concatenation operator, written as
+ `` `str1 || str2` `` - which a naive `str.split("|")` would shred into
+ extra empty cells. Backticks otherwise never appear as literal (non-code)
+ content in this document's tables, so "toggle on backtick" is a safe,
+ general rule rather than a special case for that one row.
+ """
+ line = line.strip()
+ if line.startswith("|"):
+ line = line[1:]
+ if line.endswith("|"):
+ line = line[:-1]
+ cells: list[str] = []
+ current: list[str] = []
+ in_code_span = False
+ for ch in line:
+ if ch == "`":
+ in_code_span = not in_code_span
+ current.append(ch)
+ elif ch == "|" and not in_code_span:
+ cells.append("".join(current).strip())
+ current = []
+ else:
+ current.append(ch)
+ cells.append("".join(current).strip())
+ return cells
+
+
+def _clean(cell: str) -> str:
+ """Strip markdown code-span backticks, leaving the underlying syntax text."""
+ return cell.replace("`", "").strip()
+
+
+def _is_section_excluded(heading_stack: dict[int, str]) -> bool:
+ return any(
+ marker in heading.lower()
+ for heading in heading_stack.values()
+ for marker in _EXCLUDED_SECTION_MARKERS
+ )
+
+
+def _extract_tables(text: str) -> tuple[set[str], set[str], list[tuple[str, str]]]:
+ """One pass over the document collecting constructs from ordinary tables.
+
+ Returns:
+ names: constructs whose spec_name is a `Syntax` column value.
+ identifier_tokens: the identifying (first) column's backtick tokens for every
+ such table row, used to de-duplicate against the top-level summary table.
+ summary_rows: (construct_cell, notes_cell) pairs from the top-level
+ "Supported SQL Constructs" table, processed by the caller once every
+ detailed table has been seen.
+ """
+ names: set[str] = set()
+ identifier_tokens: set[str] = set()
+ summary_rows: list[tuple[str, str]] = []
+
+ heading_stack: dict[int, str] = {}
+ lines = text.splitlines()
+ i, n = 0, len(lines)
+ while i < n:
+ line = lines[i]
+
+ heading_match = _HEADING_RE.match(line)
+ if heading_match:
+ level = len(heading_match.group(1))
+ for lvl in [lvl for lvl in heading_stack if lvl >= level]:
+ del heading_stack[lvl]
+ heading_stack[level] = heading_match.group(2).strip()
+ i += 1
+ continue
+
+ stripped = line.strip()
+
+ # A fenced code block is never itself a table; skip its body outright.
+ # (The few constructs defined only inside a fence are picked up by the
+ # bespoke passes in spec_construct_names(), keyed by heading shape.)
+ if stripped.startswith("```"):
+ i += 1
+ while i < n and not lines[i].strip().startswith("```"):
+ i += 1
+ i += 1
+ continue
+
+ if not stripped.startswith("|"):
+ i += 1
+ continue
+
+ if _is_section_excluded(heading_stack):
+ # Skip the whole table without even parsing it.
+ while i < n and lines[i].strip().startswith("|"):
+ i += 1
+ continue
+
+ table_lines = []
+ while i < n and lines[i].strip().startswith("|"):
+ table_lines.append(lines[i])
+ i += 1
+ if len(table_lines) < 2:
+ continue # a header with no separator row is not a real table
+
+ header = [_split_table_row(table_lines[0])]
+ header_lower = [c.lower() for c in header[0]]
+ data_rows = [_split_table_row(r) for r in table_lines[2:]]
+
+ # A table whose identifying column is literally "Token" is a
+ # format-token argument vocabulary (TO_CHAR/TO_DATE's `format` argument).
+ if header_lower and header_lower[0] == "token":
+ continue
+
+ if "syntax" in header_lower:
+ syntax_idx = header_lower.index("syntax")
+ form_idx = header_lower.index("form") if "form" in header_lower else None
+ for row in data_rows:
+ if len(row) <= syntax_idx:
+ continue
+ if (
+ form_idx is not None
+ and len(row) > form_idx
+ and row[form_idx].strip().lower() == "cast"
+ ):
+ # A "Cast" row in the Date/Time Construction table is a worked
+ # example of the already-catalogued generic CAST(...) construct,
+ # not a new one.
+ continue
+ for token in _CODE_SPAN_RE.findall(row[0]) if row else []:
+ identifier_tokens.add(token.strip().upper())
+ syntax_value = _clean(row[syntax_idx])
+ if syntax_value:
+ names.add(syntax_value)
+ elif header_lower[:2] == ["construct", "notes"]:
+ # The top-level "Supported SQL Constructs" table. Its rows range from
+ # genuine one-off operators/keywords (BETWEEN, CASE WHEN, IN / NOT IN)
+ # to category headers elaborated in detail elsewhere (Aggregate
+ # functions, Window functions) - processed once every detailed table
+ # has been seen, so it can tell the two apart (see _extract_summary_rows).
+ for row in data_rows:
+ if row:
+ summary_rows.append((row[0], row[1] if len(row) > 1 else ""))
+ # Any other table shape ("Database Support", "Decomposability Reference",
+ # the working-group roster, a comparison-of-quoting-styles example) names
+ # no new construct and is left unread.
+
+ return names, identifier_tokens, summary_rows
+
+
+def _extract_summary_rows(
+ summary_rows: list[tuple[str, str]], identifier_tokens: set[str]
+) -> set[str]:
+ """Pull genuine constructs out of the top "Supported SQL Constructs" table.
+
+ A row counts only if its Construct cell or its Notes cell carries a
+ backtick-quoted token - the document's own marker for "this cell names a
+ real piece of syntax" - which is how e.g. `BETWEEN`, `` `IN` / `NOT IN` ``
+ and `` `CASE WHEN` `` are told apart from plain category labels like
+ "Column and Metric references" or "Aggregate functions" (elaborated in
+ detailed tables elsewhere, and carrying no backtick markup of their own).
+
+ A token already seen as a detailed table's identifying column (e.g. `LIKE`
+ from the Pattern Matching table) is skipped here to avoid counting the same
+ construct twice under two different spellings.
+ """
+ names: set[str] = set()
+ for construct_cell, notes_cell in summary_rows:
+ tokens = _CODE_SPAN_RE.findall(construct_cell)
+ if not tokens:
+ for part in notes_cell.split(","):
+ tokens.extend(_CODE_SPAN_RE.findall(part))
+ for token in tokens:
+ token = token.strip()
+ if token.upper() in identifier_tokens:
+ continue
+ names.add(token)
+ return names
+
+
+def _extract_extraction_syntax_functions(text: str) -> set[str]:
+ """EXTRACT and DATE_PART: named in a code fence, not a table.
+
+ The "Alternative Extraction Syntax" section is the only place either
+ function is named; the bullet list immediately below it enumerates the
+ date parts they accept (an argument vocabulary, not a construct).
+ That list needs no special exclusion - it is a bullet list, not a table,
+ so `_extract_tables()` never looks at it in the first place.
+ """
+ section = re.search(
+ r"### Alternative Extraction Syntax.*?\n(.*?)\n###", text, re.S
+ )
+ if not section:
+ return set()
+ return set(re.findall(r"\b([A-Z_]+)\(", section.group(1)))
+
+
+def _extract_null_safe_comparison_operators(text: str) -> set[str]:
+ """IS DISTINCT FROM / IS NOT DISTINCT FROM: named only inside a code fence."""
+ section = re.search(r"### Null-Safe Comparison.*?\n(.*?)\n---", text, re.S)
+ if not section:
+ return set()
+ return {m.group(0) for m in re.finditer(r"\bIS (?:NOT )?DISTINCT FROM\b", section.group(1))}
+
+
+def _extract_single_construct_headings(text: str) -> set[str]:
+ """A standalone H3 whose own name (not a table) is the one construct it defines.
+
+ Structural, not name-based: matches any "### {token} ({compliance level})"
+ heading whose section body contains no pipe-table. CAST and TRY_CAST are
+ the only two headings in the current document shaped this way.
+ """
+ names: set[str] = set()
+ for match in _SINGLE_CONSTRUCT_HEADING_RE.finditer(text):
+ token = match.group(1).replace("\\_", "_")
+ start = match.end()
+ next_heading = re.search(r"^#{1,6} ", text[start:], re.M)
+ body = text[start : start + next_heading.start()] if next_heading else text[start:]
+ if "|" not in body:
+ names.add(token)
+ return names
+
+
+def spec_construct_names() -> set[str]:
+ """The construct inventory of the upstream expression-language specification.
+
+ Reads `core-spec/expression_language.md` fresh on every call - the file is
+ small and this is a test-time oracle, not a runtime hot path.
+ """
+ text = _find_spec_path().read_text(encoding="utf-8")
+
+ table_names, identifier_tokens, summary_rows = _extract_tables(text)
+ names = set(table_names)
+ names |= _extract_summary_rows(summary_rows, identifier_tokens)
+ names |= _extract_extraction_syntax_functions(text)
+ names |= _extract_null_safe_comparison_operators(text)
+ names |= _extract_single_construct_headings(text)
+ return names
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/expressions/emit.py b/converters/thoughtspot/src/ossie_thoughtspot/expressions/emit.py
new file mode 100644
index 00000000..d1812282
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/expressions/emit.py
@@ -0,0 +1,274 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+"""Render a catalog `Construct` into an actual ThoughtSpot formula.
+
+Three emitters, one per `Classification` (see `_types.py`):
+
+- `emit_direct` — substitutes `args` into the construct's native ThoughtSpot
+ template positionally. A `direct` row may itself be
+ a composition of native functions, not only a rename — that
+ composition is baked into `construct.template` by the
+ catalog, not by this function.
+- `emit_passthrough` — renders a `sql_*_op` call. The row's `variant`
+ fixes both the emitted function name and, through it, the
+ emitted column's type and measure/attribute role. Every call
+ raises a WARNING issue naming the function and the
+ object, because the body is raw, dialect-specific warehouse
+ SQL, opaque to ThoughtSpot's query planner. A call
+ that would carry a runtime ThoughtSpot parameter is refused
+ outright — it cannot resolve to static SQL, so it is not
+ portable in either direction, and the caller must route it
+ elsewhere (a THOUGHTSPOT-only dialect entry) instead of
+ obtaining a formula string from this function. Pass
+ `partition_column` when the passthrough carries a
+ `PARTITION BY` and the wrapped result guarantees that column
+ reaches ThoughtSpot's GROUP BY regardless of what the user's
+ search selects — enforced by a template/kwarg cross-check in
+ both directions, not left to caller convention.
+- `emit_unmappable` — no formula exists; raises an ERROR issue and returns nothing
+ (the two-bucket rule: never a silent drop — the caller is
+ responsible for preserving the construct in custom_extensions).
+
+Two details that are easy to get subtly wrong (both pinned by tests in test_emit.py):
+
+- `emit_passthrough` does NOT substitute `args` into the SQL body. The body is
+ rendered as a quoted *template string*, followed by the arguments as separate
+ `sql_*_op` positional arguments — ThoughtSpot resolves the `{0}`, `{1}`, ...
+ placeholders itself at formula-evaluation time. Substituting them here would
+ produce a formula that looks right and is wrong.
+- `emit_direct` DOES substitute positionally (via `str.format`), and rejects an
+ argument-count mismatch rather than silently dropping or reusing an argument,
+ which would compute the wrong thing while still importing cleanly.
+"""
+import json
+import re
+
+from ._types import Classification, Construct, VariadicStyle
+from ..issues import IssueLog, Severity
+
+_PLACEHOLDER_RE = re.compile(r"\{(\d+)\}")
+
+
+def _placeholder_count(template: str) -> int:
+ """How many distinct positional `{n}` placeholders a template declares.
+
+ Assumes contiguous 0-based indices (`{0}`, `{1}`, ...), which is the only
+ shape `str.format(*args)` accepts positionally and the only shape any
+ catalog template uses.
+ """
+ indices = {int(m) for m in _PLACEHOLDER_RE.findall(template)}
+ return max(indices) + 1 if indices else 0
+
+
+def _emit_variadic(construct: Construct, args: list[str]) -> str:
+ """Render a construct whose argument count is not fixed.
+
+ JOIN splits the template at its `{*}` marker, formats each side with the
+ positional head arguments, and drops the joined tail between them, so
+ `"{0} in {{ {*} }}"` with four arguments gives `a in { b , c , d }`. The
+ split matters: `str.format` cannot be run over a template still carrying
+ `{*}`, and running it after substitution would re-interpret any brace the
+ arguments themselves contain.
+
+ FOLD applies a BINARY template right-associatively, for a ThoughtSpot
+ function with no n-ary form — `COALESCE(a, b, c)` has to become
+ `ifnull ( a , ifnull ( b , c ) )`, not a flat call.
+ """
+ spec = construct.variadic
+ if len(args) < spec.min_args:
+ plural = "argument" if spec.min_args == 1 else "arguments"
+ raise ValueError(
+ f"{construct.spec_name} takes at least {spec.min_args} {plural}, got {len(args)}"
+ )
+
+ if spec.style is VariadicStyle.FOLD:
+ folded = args[-1]
+ for arg in reversed(args[:-1]):
+ folded = construct.template.format(arg, folded)
+ return folded
+
+ head, _, tail = construct.template.partition("{*}")
+ return head.format(*args) + " , ".join(args[spec.tail_from:]) + tail.format(*args)
+
+
+def emit_direct(construct: Construct, args: list[str]) -> str:
+ """Render a DIRECT construct: substitute `args` into its template positionally.
+
+ Raises ValueError if `construct` is not classified DIRECT, or if `args` does
+ not have exactly the number of positional arguments the template declares —
+ silently dropping or reusing an argument would produce a formula that imports
+ cleanly and computes the wrong thing.
+
+ A construct declaring `variadic` is checked against a MINIMUM instead, and
+ rendered by `_emit_variadic`: CONCAT, GREATEST, LEAST, COALESCE, IN and
+ NOT IN all take any number of arguments, and a fixed placeholder count
+ cannot express that in either direction — it rejects the real arity, or it
+ renders the specification's own `...` notation into the output as if it
+ were syntax.
+ """
+ if construct.classification is not Classification.DIRECT:
+ raise ValueError(
+ f"{construct.spec_name}: emit_direct called on a "
+ f"{construct.classification.value} construct, not direct"
+ )
+ if construct.variadic is not None:
+ return _emit_variadic(construct, args)
+ expected = _placeholder_count(construct.template)
+ if len(args) != expected:
+ plural = "argument" if expected == 1 else "arguments"
+ raise ValueError(
+ f"{construct.spec_name} expects {expected} {plural}, got {len(args)}"
+ )
+ return construct.template.format(*args)
+
+
+def emit_passthrough(
+ construct: Construct,
+ args: list[str],
+ log: IssueLog,
+ *,
+ object_ref: str,
+ has_parameter: bool = False,
+ partition_column: str | None = None,
+) -> str:
+ """Render a PASSTHROUGH construct as a `sql_*_op` call and log a warning.
+
+ `has_parameter=True` refuses the call outright: a `sql_*_op` whose
+ arguments include a ThoughtSpot parameter cannot resolve to static SQL, so it
+ is not portable in either direction. The caller must not obtain a formula
+ string from this function in that case — it routes the construct to a
+ THOUGHTSPOT-only dialect entry instead.
+
+ `partition_column`: when the pass-through's SQL carries a `PARTITION BY`,
+ pass the column it partitions on and the result comes back wrapped in
+ `group_aggregate ( <passthrough> , query_groups ( ) + { <partition_column> } ,
+ query_filters ( ) )`, so the partition column reaches ThoughtSpot's GROUP BY
+ even when the user's search omits it. This is enforced, not left to caller
+ convention: a template that carries `PARTITION BY` (case-insensitive) but no
+ `partition_column` raises, and a `partition_column` supplied for a template
+ with no `PARTITION BY` raises too — a miscopied catalog row fails
+ loudly here instead of silently emitting an unwrapped, only-sometimes-
+ correct pass-through.
+
+ The exemplar convention: not every `construct.template` this function renders is a
+ complete, general-purpose body. Roughly a third of the catalog's PASSTHROUGH rows
+ (`PERCENTILE_CONT`/`DISC`'s `0.75`, `APPROX_PERCENTILE`'s `0.5`, `NTILE`'s `4`,
+ `NTH_VALUE`'s `2`, `LAG`/`LEAD`'s offset `1`, `TO_TIMESTAMP`/`TO_CHAR`'s fixed formats,
+ `DENSE_RANK`/`CUME_DIST`'s fixed `ORDER BY`, typed literals, and window aggregation's
+ `SUM`, among others) bake ONE caller-supplied value into the template as a literal
+ while still declaring a satisfiable arity, rather than exposing that value as its own
+ `{n}` placeholder. This function renders such a row exactly as written — it has no way
+ to tell an exemplar from a genuinely complete template, since both pass the arg-count
+ check the same way. The catalog holds an exemplar for documentation and testing; a
+ caller translating a real occurrence with a different value for that slot must rebuild
+ the template for that occurrence rather than reuse the catalog row's rendering
+ verbatim. Each exemplar row's `note` names the baked-in value.
+ """
+ if construct.classification is not Classification.PASSTHROUGH:
+ raise ValueError(
+ f"{construct.spec_name}: emit_passthrough called on a "
+ f"{construct.classification.value} construct, not passthrough"
+ )
+ if has_parameter:
+ raise ValueError(
+ f"{construct.spec_name}: a passthrough cannot carry a runtime parameter "
+ "— it cannot resolve to static SQL"
+ )
+
+ # Mirrors emit_direct's own arg-count guard: a mismatch means either a caller
+ # passing the wrong number of resolved operands, or a template with a hardcoded
+ # literal (e.g. a fixed date) that declares zero placeholders — either way this
+ # would otherwise render as a call whose args outnumber (or fall short of) what
+ # the template's own {0}, {1}, ... placeholders consume, silently appending an
+ # unused argument or leaving a placeholder unfilled instead of failing loudly.
+ expected = _placeholder_count(construct.template)
+ if len(args) != expected:
+ plural = "argument" if expected == 1 else "arguments"
+ raise ValueError(
+ f"{construct.spec_name} expects {expected} {plural}, got {len(args)}"
+ )
+
+ # Enforced rather than left to caller convention: a passthrough row that needs
+ # the group_aggregate wrap carries the literal string "PARTITION BY" in its SQL
+ # template. Four rows do -- ROW_NUMBER, LAG, LEAD and window aggregation -- and
+ # they are the only ones this check can see. The OVER row's own template is the
+ # dispatch string "per-clause-shape — see note"; CUME_DIST's carries no
+ # PARTITION BY; and RANK and PERCENT_RANK are DIRECT rows, not passthrough rows
+ # at all. Those four fallbacks are described in prose in their `note`, which
+ # this guard never reads, so they are outside it by construction rather than by
+ # oversight. Checking the template against the kwarg in both directions turns
+ # "the catalog author must remember to pass this" into something this function
+ # refuses to get wrong.
+ carries_partition_by = bool(re.search(r"partition\s+by", construct.template, re.IGNORECASE))
+ if carries_partition_by and partition_column is None:
+ raise ValueError(
+ f"{construct.spec_name}: template carries PARTITION BY but no "
+ "partition_column was supplied — the group_aggregate wrapper is required"
+ )
+ if partition_column is not None and not carries_partition_by:
+ raise ValueError(
+ f"{construct.spec_name}: partition_column was supplied but the template "
+ "carries no PARTITION BY — there is nothing to wrap"
+ )
+
+ # variant is guaranteed non-None for a PASSTHROUGH row by Construct.__post_init__.
+ variant = construct.variant
+ quoted_template = json.dumps(construct.template)
+ body = " , ".join([quoted_template, *args])
+ call = f"{variant.value} ( {body} )"
+
+ log.add(
+ code="TS-EXPR-PASSTHROUGH",
+ severity=Severity.WARNING,
+ message=(
+ f"{construct.spec_name} is emitted as a {variant.value} pass-through: "
+ "raw warehouse SQL, opaque to ThoughtSpot's query planner. Review before use."
+ ),
+ object_ref=object_ref,
+ )
+
+ if partition_column is not None:
+ return (
+ f"group_aggregate ( {call} , "
+ f"query_groups ( ) + {{ {partition_column} }} , query_filters ( ) )"
+ )
+ return call
+
+
+def emit_unmappable(construct: Construct, log: IssueLog, *, object_ref: str) -> None:
+ """Raise an ERROR issue for an UNMAPPABLE construct. Never a silent drop.
+
+ Returns nothing — the caller is responsible for preserving the construct in
+ `custom_extensions` for roundtrip; that stash is out of this function's scope.
+ """
+ if construct.classification is not Classification.UNMAPPABLE:
+ raise ValueError(
+ f"{construct.spec_name}: emit_unmappable called on a "
+ f"{construct.classification.value} construct, not unmappable"
+ )
+ log.add(
+ code="TS-EXPR-UNMAPPABLE",
+ severity=Severity.ERROR,
+ message=(
+ f"{construct.spec_name} has no ThoughtSpot representation; "
+ "preserved in custom_extensions for roundtrip."
+ ),
+ object_ref=object_ref,
+ )
+ return None
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/expressions/reverse.py b/converters/thoughtspot/src/ossie_thoughtspot/expressions/reverse.py
new file mode 100644
index 00000000..e988690f
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/expressions/reverse.py
@@ -0,0 +1,1034 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""The reverse-direction inventory: ThoughtSpot functions with no counterpart in the
+Ossie specification.
+
+`CATALOG` (catalog.py) is the specification's own inventory, Ossie construct -> ThoughtSpot
+rendering. This module is the other half of a bidirectional converter: ThoughtSpot's own
+native functions that the 146-row `CATALOG` never targets, sourced from the "Reverse
+direction (ThoughtSpot -> Ossie)" section of
+docs/ossie/ts-ossie-function-mapping.md (thoughtspot-agent-skills repo, not vendored here) —
+its three sub-sections (conditional aggregates and arithmetic helpers; window, LOD and
+semi-additive functions; runtime, display and calendar concepts).
+
+One governing idea shapes the whole module: prefer composition over the stash. Most of ThoughtSpot's
+apparently-proprietary functions are sugar over constructs the specification already has
+(`sum_if` -> `SUM(CASE WHEN ...)`, `safe_divide` -> `COALESCE(a / NULLIF(b, 0), 0)`,
+`group_sum` over a fixed grain -> `SUM(x) OVER (PARTITION BY attr)`). The stash
+(`custom_extensions` + issue) is for what genuinely has no expression — a short
+list dominated by *runtime* concepts (parameters, signed-in-user identity, display markup,
+fiscal calendars), not by missing mathematics.
+
+Four dispositions, not the forward module's three
+------------------------------------------------------------------------------------------
+`Classification` (catalog.py's DIRECT/PASSTHROUGH/UNMAPPABLE) does not fit this direction
+cleanly, so this module defines its own `ReverseDisposition` rather than bending it (as
+instructed): a reverse row can compose fully, compose *partially* (with a real fidelity
+loss that still deserves an issue and a preserved verbatim rendering), resolve to
+the Ossie `dialects[]` mechanism instead of a portable expression at all, or have no
+expression whatsoever.
+
+ COMPOSE a full Ossie expression is produced. May still carry a caveat issue (locale
+ dependence, a week-start-day or DAYOFWEEK-base assumption) without being lossy
+ in the way PARTIAL is — the composition is exact, the caveat is about the
+ *specification's* own portability, not about this construct's translation.
+ PARTIAL a real Ossie expression is produced, but it is provably incomplete (the
+ `moving_*`/`cumulative_*` family: the frame and order translate exactly, the
+ partition does not — a ThoughtSpot window formula cannot declare its own
+ PARTITION BY, and the specification has no way to express that limitation).
+ Always logs a WARNING and the caller should pair the composed expression with a THOUGHTSPOT dialect
+ entry (`thoughtspot_dialect_entry`) carrying the verbatim original — and an
+ ANSI_SQL sibling (`portable_dialect_entry`) for the composed expression itself,
+ since it *is* portable, just incomplete: a consumer that does not implement the
+ THOUGHTSPOT dialect still gets something it can execute.
+ DIALECT the construct's natural home is the Ossie `dialects[]` mechanism, not a
+ portable expression — the eleven `sql_*_op` / `sql_*_aggregate_op` names. No
+ ANSI_SQL sibling is ever emitted for these (the document is explicit: raw
+ warehouse SQL's portability is exactly what is unknown).
+ STASH no Ossie expression exists at all. Always logs an ERROR (mirroring
+ `emit_unmappable`'s severity choice for the same "no representation, preserved
+ only for roundtrip" shape) and the caller should attach a THOUGHTSPOT
+ dialect entry with the verbatim call.
+
+Argument abstraction level
+------------------------------------------------------------------------------------------
+`translate_thoughtspot(name, args, log, *, object_ref, ...)` takes `args` as already-
+extracted operand strings, exactly the abstraction level `emit_direct`/`emit_passthrough`
+take in the forward direction (never raw ThoughtSpot formula text with nested calls or
+brace/quote syntax to parse) — no expression parser exists yet, so there is nothing to parse
+from either direction yet. A caller with a real parsed formula tree supplies the resolved
+operand strings positionally.
+
+`translate_thoughtspot` carries `object_ref` as a required keyword-only parameter beyond the
+plain `(name, args, log)` shape: `IssueLog.add` requires it, the same shape
+`emit_passthrough`/`emit_unmappable` already use for the identical reason. A second keyword-
+only parameter, `connection_dialect`, is added for the DIALECT family only (see
+`_dispatch_sql_op`) — the document itself says resolving these needs the connection's own
+dialect, which is not derivable from a bare function name and argument list.
+
+Two constructs are cross-cutting rather than name-keyed, so they are not `REVERSE` entries
+looked up by `name` at all:
+
+- **The fiscal-calendar argument.** The document describes this as "the rest of the fiscal
+ family" without enumerating every date function it can decorate, so `translate_thoughtspot`
+ checks for a trailing literal `fiscal` argument up front, for any `name` at all, before
+ falling through to a normal lookup.
+- **The runtime-parameter reference.** A bracketed name (`[Discount Threshold]`) is
+ syntactically identical to an ordinary column reference (`[Table::Column]`) — this module
+ has no model metadata to tell the two apart, so there is no way to safely auto-detect it in
+ `translate_thoughtspot`. `stash_runtime_parameter` is exposed separately; the caller (which
+ does have the model's declared parameter list) invokes it directly once it has confirmed
+ the name in hand is a declared parameter and not a column.
+
+Two more are shape-dependent rather than purely name-keyed, and use `ReverseConstruct`'s
+`dispatch_fn` escape hatch (full control over composing vs. stashing, bypassing the
+declarative `template`/`compose_fn` path entirely): `group_aggregate` and its four named
+shorthands (`group_sum`, `group_count`, `group_stddev`, `group_variance`), whose disposition
+depends on the shape of the grouping and filter arguments (see `_compose_grouped`); and the
+eleven `sql_*_op` names, whose disposition depends on whether the caller can supply
+`connection_dialect` (see `_dispatch_sql_op`). `concat` is a third, narrower case: plain
+`concat` has a spec counterpart already in `CATALOG` and is not this module's concern at all
+(returns `None`, no issue) — only the ThoughtSpot hyperlink-markup content pattern inside its
+string arguments (`{caption}` / `{/caption}`) is reverse-inventory territory.
+"""
+import functools
+import re
+from dataclasses import dataclass
+from enum import Enum
+from typing import Callable
+
+from .catalog import CATALOG
+from ..constants import (
+ DIALECT,
+ GROUP_AGGREGATE_CALL_NAMES,
+ GROUP_SHORTHAND_AGGREGATES,
+ PORTABLE_DIALECT,
+)
+from ..issues import IssueLog, Severity
+
+
+class ReverseDisposition(str, Enum):
+ """How a ThoughtSpot-only construct reaches an Ossie document. See the module
+ docstring's "Four dispositions" section for the full reasoning behind each.
+ """
+
+ COMPOSE = "compose"
+ PARTIAL = "partial"
+ DIALECT = "dialect"
+ STASH = "stash"
+
+
+# A dispatch_fn takes (args, log, object_ref, connection_dialect) and returns the composed
+# Ossie expression, or None if it decides — internally, based on argument shape — to stash
+# instead. It owns its own issue logging; disposition on such a row is documentation only.
+DispatchFn = Callable[[list, IssueLog, str, str | None], str | None]
+ComposeFn = Callable[[list], str]
+
+
+@dataclass(frozen=True)
+class ReverseConstruct:
+ """One row of the reverse-direction inventory.
+
+ `thoughtspot_name` the construct as ThoughtSpot spells it, e.g. "sum_if". May be a
+ synthetic, non-callable key for a construct the document describes
+ by content pattern rather than by name (e.g. the hyperlink-markup
+ row) — always documented as such at the registration site.
+ `disposition` see `ReverseDisposition`.
+ `template` for a plain positional COMPOSE/PARTIAL row: the Ossie expression
+ with {0}, {1}, ... placeholders, rendered via `str.format`. Mutually
+ exclusive with `compose_fn` and unused when `dispatch_fn` is set.
+ `compose_fn` for a COMPOSE/PARTIAL row whose composition is not a simple
+ positional substitution (variable arity, a transform on an
+ argument's literal value). Takes precedence over `template`.
+ `dispatch_fn` full override: decides composing vs. stashing itself from argument
+ shape, and does its own issue logging. When set, `disposition`
+ above is documentation only and no other field is validated.
+ `issue_code` `IssueLog.add(code=...)` for this row's issue — never a
+ bare "untranslatable" message.
+ `issue_severity` WARNING for a COMPOSE-with-caveat or PARTIAL row (something usable
+ is still produced); ERROR for STASH (nothing is — mirrors
+ `emit_unmappable`'s choice for the same "no representation" shape).
+ `issue_message` may contain a `{name}` placeholder, filled with the actual
+ ThoughtSpot name/reference at translation time — the same message
+ template serves every case sharing one reason (the five identity
+ functions, the fiscal-calendar family).
+ `note` the row's caveat, traceable to the document, same convention as
+ catalog.py's `Construct.note`.
+ """
+
+ thoughtspot_name: str
+ disposition: ReverseDisposition
+ template: str | None = None
+ compose_fn: ComposeFn | None = None
+ dispatch_fn: DispatchFn | None = None
+ issue_code: str = ""
+ issue_severity: Severity = Severity.WARNING
+ issue_message: str = ""
+ note: str = ""
+
+ def __post_init__(self) -> None:
+ if self.dispatch_fn is not None:
+ # Full custom control - no further shape validation applies (see class docstring).
+ return
+ needs_body = self.disposition in (ReverseDisposition.COMPOSE, ReverseDisposition.PARTIAL)
+ has_body = self.template is not None or self.compose_fn is not None
+ if needs_body and not has_body:
+ raise ValueError(
+ f"{self.thoughtspot_name}: a {self.disposition.value} row needs a "
+ "template or compose_fn"
+ )
+ if not needs_body and has_body:
+ raise ValueError(
+ f"{self.thoughtspot_name}: a {self.disposition.value} row must not carry "
+ "a template or compose_fn"
+ )
+ if self.disposition in (ReverseDisposition.PARTIAL, ReverseDisposition.STASH) and not self.issue_message:
+ raise ValueError(
+ f"{self.thoughtspot_name}: a {self.disposition.value} row must carry an "
+ "issue message — never a bare 'untranslatable'"
+ )
+
+
+REVERSE: dict[str, ReverseConstruct] = {}
+
+_PLACEHOLDER_RE = re.compile(r"\{(\d+)\}")
+
+
+def _placeholder_count(template: str) -> int:
+ indices = {int(m) for m in _PLACEHOLDER_RE.findall(template)}
+ return max(indices) + 1 if indices else 0
+
+
+def _render(construct: ReverseConstruct, args: list[str]) -> str:
+ if construct.compose_fn is not None:
+ return construct.compose_fn(args)
+ expected = _placeholder_count(construct.template)
+ if len(args) != expected:
+ plural = "argument" if expected == 1 else "arguments"
+ raise ValueError(f"{construct.thoughtspot_name} expects {expected} {plural}, got {len(args)}")
+ return construct.template.format(*args)
+
+
+def _apply_stash(construct: ReverseConstruct, name: str, log: IssueLog, *, object_ref: str) -> None:
+ log.add(
+ code=construct.issue_code,
+ severity=construct.issue_severity,
+ message=construct.issue_message.format(name=name),
+ object_ref=object_ref,
+ )
+ return None
+
+
+# --------------------------------------------------------------------------
+# Conditional aggregates and arithmetic helpers — 28 names, all COMPOSE.
+# Source: docs/ossie/ts-ossie-function-mapping.md, "Reverse direction ->
+# Conditional aggregates and arithmetic helpers".
+# --------------------------------------------------------------------------
+
+for _name, _agg in (
+ ("sum_if", "SUM"),
+ ("count_if", "COUNT"),
+ ("average_if", "AVG"),
+ ("min_if", "MIN"),
+ ("max_if", "MAX"),
+ ("stddev_if", "STDDEV"),
+ ("variance_if", "VARIANCE"),
+):
+ REVERSE[_name] = ReverseConstruct(
+ thoughtspot_name=_name,
+ disposition=ReverseDisposition.COMPOSE,
+ template=f"{_agg}(CASE WHEN {{0}} THEN {{1}} END)",
+ note=f"{_name} ( cond , x ) -> {_agg}(CASE WHEN cond THEN x END).",
+ )
+
+REVERSE["unique_count_if"] = ReverseConstruct(
+ thoughtspot_name="unique_count_if",
+ disposition=ReverseDisposition.COMPOSE,
+ template="COUNT(DISTINCT CASE WHEN {0} THEN {1} END)",
+ note="unique_count_if ( cond , x ) -> COUNT(DISTINCT CASE WHEN cond THEN x END).",
+)
+
+REVERSE["unique count"] = ReverseConstruct(
+ thoughtspot_name="unique count",
+ disposition=ReverseDisposition.COMPOSE,
+ template="COUNT(DISTINCT {0})",
+ note="ThoughtSpot's own spelling has a space, not an underscore.",
+)
+
+REVERSE["safe_divide"] = ReverseConstruct(
+ thoughtspot_name="safe_divide",
+ disposition=ReverseDisposition.COMPOSE,
+ template="COALESCE({0} / NULLIF({1}, 0), 0)",
+ note="The zero-not-null result is preserved by the explicit COALESCE.",
+)
+
+REVERSE["pow"] = ReverseConstruct(
+ thoughtspot_name="pow", disposition=ReverseDisposition.COMPOSE, template="POWER({0}, {1})",
+)
+REVERSE["log2"] = ReverseConstruct(
+ thoughtspot_name="log2", disposition=ReverseDisposition.COMPOSE, template="LOG(2, {0})",
+)
+REVERSE["strlen"] = ReverseConstruct(
+ thoughtspot_name="strlen", disposition=ReverseDisposition.COMPOSE, template="LENGTH({0})",
+)
+REVERSE["strpos"] = ReverseConstruct(
+ thoughtspot_name="strpos",
+ disposition=ReverseDisposition.COMPOSE,
+ template="POSITION({1} IN {0})",
+ note="ThoughtSpot strpos(s, sub) -> Ossie POSITION(sub IN s); operand order reverses.",
+)
+REVERSE["substr"] = ReverseConstruct(
+ thoughtspot_name="substr",
+ disposition=ReverseDisposition.COMPOSE,
+ template="SUBSTRING({0}, {1} + 1, {2})",
+ note="ThoughtSpot's substr is 0-based; the +1 is mandatory going this way.",
+)
+REVERSE["left"] = ReverseConstruct(
+ thoughtspot_name="left", disposition=ReverseDisposition.COMPOSE, template="LEFT({0}, {1})",
+)
+REVERSE["right"] = ReverseConstruct(
+ thoughtspot_name="right", disposition=ReverseDisposition.COMPOSE, template="RIGHT({0}, {1})",
+)
+
+for _name in ("sin", "cos", "tan"):
+ REVERSE[_name] = ReverseConstruct(
+ thoughtspot_name=_name,
+ disposition=ReverseDisposition.COMPOSE,
+ template=f"{_name.upper()}(RADIANS({{0}}))",
+ note="ThoughtSpot trigonometry is in degrees; the conversion reverses.",
+ )
+for _name in ("asin", "acos", "atan"):
+ REVERSE[_name] = ReverseConstruct(
+ thoughtspot_name=_name,
+ disposition=ReverseDisposition.COMPOSE,
+ template=f"DEGREES({_name.upper()}({{0}}))",
+ note="ThoughtSpot's inverse trig functions return degrees.",
+ )
+
+REVERSE["to_integer"] = ReverseConstruct(
+ thoughtspot_name="to_integer", disposition=ReverseDisposition.COMPOSE, template="CAST({0} AS INTEGER)",
+)
+REVERSE["to_double"] = ReverseConstruct(
+ thoughtspot_name="to_double", disposition=ReverseDisposition.COMPOSE, template="CAST({0} AS DOUBLE)",
+)
+REVERSE["to_string"] = ReverseConstruct(
+ thoughtspot_name="to_string", disposition=ReverseDisposition.COMPOSE, template="CAST({0} AS VARCHAR)",
+)
+REVERSE["to_date"] = ReverseConstruct(
+ thoughtspot_name="to_date",
+ disposition=ReverseDisposition.COMPOSE,
+ template="TO_DATE({0}, {1})",
+ issue_code="TS-EXPR-FORMAT-TOKENS-PASSTHROUGH",
+ issue_severity=Severity.INFO,
+ issue_message=(
+ "{name}'s format string is passed through verbatim, not mechanically translated "
+ "through the TO_DATE/TO_CHAR format-token table — no expression parser exists yet "
+ "to do that translation. TO_DATE(s, format) is EXPERIMENTAL on the Ossie side."
+ ),
+ note="Judgment call: format-token reversal is deferred until an expression parser exists to do the translation.",
+)
+REVERSE["if"] = ReverseConstruct(
+ thoughtspot_name="if",
+ disposition=ReverseDisposition.COMPOSE,
+ template="CASE WHEN {0} THEN {1} ELSE {2} END",
+ note="if ( c ) then a else b -> CASE WHEN c THEN a ELSE b END, or IF(c, a, b).",
+)
+
+
+# --------------------------------------------------------------------------
+# Window, LOD and semi-additive functions.
+# Source: "Reverse direction -> Window, LOD and semi-additive functions".
+# --------------------------------------------------------------------------
+
+def _direction_keyword(literal: str) -> str:
+ bare = literal.strip().strip("'\"").lower()
+ if not bare.startswith(("asc", "desc")):
+ raise ValueError(f"unrecognised rank direction literal: {literal!r}")
+ return "DESC" if bare.startswith("desc") else "ASC"
+
+
+def _compose_rank(args: list[str]) -> str:
+ if len(args) != 2:
+ raise ValueError(f"rank expects 2 arguments, got {len(args)}")
+ agg, direction = args
+ return f"RANK() OVER (ORDER BY {agg} {_direction_keyword(direction)})"
+
+
+def _compose_rank_percentile(args: list[str]) -> str:
+ if len(args) != 2:
+ raise ValueError(f"rank_percentile expects 2 arguments, got {len(args)}")
+ agg, direction = args
+ return f"(1.0 - PERCENT_RANK() OVER (ORDER BY {agg} {_direction_keyword(direction)})) * 100"
+
+
+REVERSE["rank"] = ReverseConstruct(
+ thoughtspot_name="rank", disposition=ReverseDisposition.COMPOSE, compose_fn=_compose_rank,
+ note="Global, ORDER-BY-only shape only — rank's arity is fixed at exactly two "
+ "(live-confirmed), so there is never a partition to lose in this direction.",
+)
+REVERSE["rank_percentile"] = ReverseConstruct(
+ thoughtspot_name="rank_percentile",
+ disposition=ReverseDisposition.COMPOSE,
+ compose_fn=_compose_rank_percentile,
+ note="Scale (0-100 -> 0-1) and inversion both reverse.",
+)
+
+
+def _frame_start_bound(num_backward: str) -> str:
+ """`moving_*`'s second argument -> the frame's START bound.
+
+ It counts rows BACKWARD, so a positive value is `PRECEDING`. Negative is
+ legal and means the window opens after the current row -- the live-confirmed
+ LEAD idiom, `moving_max(m, -1, 1, ord)`, is exactly one row ahead.
+ """
+ n = int(num_backward.strip())
+ if n > 0:
+ return f"{n} PRECEDING"
+ if n == 0:
+ return "CURRENT ROW"
+ return f"{-n} FOLLOWING"
+
+
+def _frame_end_bound(num_forward: str) -> str:
+ """`moving_*`'s third argument -> the frame's END bound.
+
+ It counts rows FORWARD, so the sign means the OPPOSITE of the start bound's:
+ a positive value is `FOLLOWING`. Both bounds shared one function until now,
+ and every window with a non-zero forward count came out wrong --
+ `moving_average(m, 2, 3, ord)` emitted `BETWEEN 2 PRECEDING AND 3 PRECEDING`,
+ which is valid SQL over the wrong five rows rather than a loud failure, and
+ `(m, 0, 5, ord)` emitted `BETWEEN CURRENT ROW AND 5 PRECEDING`, which no
+ engine will run. Only a forward count of 0 was unaffected, which is why the
+ pinned test case did not show it.
+ """
+ n = int(num_forward.strip())
+ if n > 0:
+ return f"{n} FOLLOWING"
+ if n == 0:
+ return "CURRENT ROW"
+ return f"{-n} PRECEDING"
+
+
+_PARTITION_LOST_ISSUE = (
+ "{name}'s emitted OVER clause has no PARTITION BY: ThoughtSpot completes the partition "
+ "dynamically from the query's own dimensions minus the order columns — a ThoughtSpot "
+ "window formula cannot declare its own PARTITION BY, and a static Ossie window has no "
+ "way to express that limitation. The composed expression is correct "
+ "only when the search returns exactly the grain this formula assumed. "
+ "Pair this with a THOUGHTSPOT dialect entry carrying the verbatim original."
+)
+
+
+def _compose_moving(agg: str) -> ComposeFn:
+ def _compose(args: list[str]) -> str:
+ if len(args) < 4:
+ raise ValueError(
+ f"moving_{agg.lower()} expects at least 4 arguments (m, start, end, order...), "
+ f"got {len(args)}"
+ )
+ m, start, end, *order_cols = args
+ order_clause = ", ".join(order_cols)
+ return (
+ f"{agg}({m}) OVER (ORDER BY {order_clause} "
+ f"ROWS BETWEEN {_frame_start_bound(start)} AND {_frame_end_bound(end)})"
+ )
+ return _compose
+
+
+def _compose_cumulative(agg: str) -> ComposeFn:
+ def _compose(args: list[str]) -> str:
+ if len(args) < 2:
+ raise ValueError(
+ f"cumulative_{agg.lower()} expects at least 2 arguments (m, order...), got {len(args)}"
+ )
+ m, *order_cols = args
+ order_clause = ", ".join(order_cols)
+ return (
+ f"{agg}({m}) OVER (ORDER BY {order_clause} "
+ "ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)"
+ )
+ return _compose
+
+
+for _agg in ("SUM", "AVERAGE", "MAX", "MIN"):
+ _ansi_agg = "AVG" if _agg == "AVERAGE" else _agg
+ REVERSE[f"moving_{_agg.lower()}"] = ReverseConstruct(
+ thoughtspot_name=f"moving_{_agg.lower()}",
+ disposition=ReverseDisposition.PARTIAL,
+ compose_fn=_compose_moving(_ansi_agg),
+ issue_code="TS-EXPR-PARTIAL-PARTITION",
+ issue_severity=Severity.WARNING,
+ issue_message=_PARTITION_LOST_ISSUE,
+ note="Frame and order translate exactly; the partition does not — a ThoughtSpot window formula cannot declare its own PARTITION BY, and the specification has no way to express that limitation.",
+ )
+ REVERSE[f"cumulative_{_agg.lower()}"] = ReverseConstruct(
+ thoughtspot_name=f"cumulative_{_agg.lower()}",
+ disposition=ReverseDisposition.PARTIAL,
+ compose_fn=_compose_cumulative(_ansi_agg),
+ issue_code="TS-EXPR-PARTIAL-PARTITION",
+ issue_severity=Severity.WARNING,
+ issue_message=_PARTITION_LOST_ISSUE,
+ note="Frame and order translate exactly; the partition does not — a ThoughtSpot window formula cannot declare its own PARTITION BY, and the specification has no way to express that limitation.",
+ )
+
+
+def _compose_grouped(
+ agg_call: str,
+ grouping_arg: str,
+ filter_arg: str,
+ log: IssueLog,
+ *,
+ object_ref: str,
+ source_name: str,
+) -> str | None:
+ """Shared shape dispatch for `group_aggregate` and its named shorthands.
+
+ Three dispositions live under one ThoughtSpot spelling, distinguished only by the
+ grouping/filter arguments' shape (all three shapes are live-confirmed real, per the
+ document's "Window rows live-confirmed" section):
+
+ - a `query_filters ( )`-only filter and a fixed `{ ... }` grouping (or `query_groups ( )`
+ alone) composes cleanly — this is the one ThoughtSpot windowing form that is clean in
+ this direction, because its partition is declared in the formula rather than completed
+ from the query;
+ - a `query_groups ( ) ± { attr }` dynamic grouping has no expression (the largest
+ reverse-direction fidelity gap);
+ - any filter argument other than `query_filters ( )` has no expression either (filter
+ scoping is excluded from Ossie expressions).
+ """
+ grouping = grouping_arg.strip()
+ filt = filter_arg.strip()
+ if filt != "query_filters ( )":
+ log.add(
+ code="TS-EXPR-GROUP-FILTER-SCOPE",
+ severity=Severity.ERROR,
+ message=(
+ f"{source_name}'s filter argument ({filter_arg!r}) scopes the aggregate to "
+ "a filtered subset of the query; the specification excludes filter scoping "
+ "from expressions. Preserved verbatim for roundtrip."
+ ),
+ object_ref=object_ref,
+ )
+ return None
+ if "query_groups ( )" in grouping and ("-" in grouping or "+" in grouping):
+ log.add(
+ code="TS-EXPR-GROUP-DYNAMIC-PARTITION",
+ severity=Severity.ERROR,
+ message=(
+ f"{source_name}'s grouping argument ({grouping_arg!r}) completes the "
+ "partition dynamically from the query's own dimensions, which the "
+ "specification cannot express — the largest reverse-direction "
+ "fidelity gap. Preserved verbatim for roundtrip."
+ ),
+ object_ref=object_ref,
+ )
+ return None
+ if grouping == "query_groups ( )":
+ return agg_call
+ if grouping.startswith("{") and grouping.endswith("}"):
+ cols = grouping[1:-1].strip()
+ return f"{agg_call} OVER ()" if not cols else f"{agg_call} OVER (PARTITION BY {cols})"
+ raise ValueError(f"{source_name}: unrecognised grouping argument shape {grouping_arg!r}")
+
+
+def _dispatch_group_aggregate(
+ args: list[str], log: IssueLog, object_ref: str, connection_dialect: str | None
+) -> str | None:
+ if len(args) != 3:
+ raise ValueError(f"group_aggregate expects 3 arguments (agg, grouping, filter), got {len(args)}")
+ agg_call, grouping_arg, filter_arg = args
+ return _compose_grouped(agg_call, grouping_arg, filter_arg, log, object_ref=object_ref, source_name="group_aggregate")
+
+
+REVERSE["group_aggregate"] = ReverseConstruct(
+ thoughtspot_name="group_aggregate",
+ disposition=ReverseDisposition.COMPOSE,
+ dispatch_fn=_dispatch_group_aggregate,
+ note="Shape dispatch on the grouping/filter arguments — see _compose_grouped.",
+)
+
+_GROUP_SHORTHAND_AGGREGATES = GROUP_SHORTHAND_AGGREGATES
+
+
+def _make_group_shorthand_dispatch(agg: str, source_name: str) -> DispatchFn:
+ def _dispatch(args: list[str], log: IssueLog, object_ref: str, connection_dialect: str | None) -> str | None:
+ if len(args) != 3:
+ raise ValueError(f"{source_name} expects 3 arguments (m, grouping, filter), got {len(args)}")
+ m, grouping_arg, filter_arg = args
+ return _compose_grouped(f"{agg}({m})", grouping_arg, filter_arg, log, object_ref=object_ref, source_name=source_name)
+ return _dispatch
+
+
+for _name, _agg in _GROUP_SHORTHAND_AGGREGATES.items():
+ REVERSE[_name] = ReverseConstruct(
+ thoughtspot_name=_name,
+ disposition=ReverseDisposition.COMPOSE,
+ dispatch_fn=_make_group_shorthand_dispatch(_agg, _name),
+ note=(
+ f"Shorthand for group_aggregate({_agg.lower()}(m), ...) — same shape dispatch. "
+ "Judgment call: the (m, grouping, filter) 3-argument shape is assumed by analogy "
+ "with group_aggregate's live-confirmed form; the shorthand family's own arity "
+ "was not independently live-tested."
+ ),
+ )
+
+
+
+_SEMI_ADDITIVE_ISSUE = (
+ "{name} declares a genuine partition and order axis, and that window clause round-trips "
+ "faithfully — but semi-additivity is a roll-up declaration (do not re-sum this measure "
+ "across the axis), not an expression, and the specification has no such declaration. "
+ "Preserved verbatim for roundtrip."
+)
+for _name in ("last_value", "first_value", "last_value_in_period", "first_value_in_period"):
+ REVERSE[_name] = ReverseConstruct(
+ thoughtspot_name=_name,
+ disposition=ReverseDisposition.STASH,
+ issue_code="TS-EXPR-SEMI-ADDITIVE",
+ issue_severity=Severity.ERROR,
+ issue_message=_SEMI_ADDITIVE_ISSUE,
+ note="The window clause itself round-trips; only the roll-up declaration is lost.",
+ )
+
+
+def _dispatch_sql_op(
+ args: list[str], log: IssueLog, object_ref: str, connection_dialect: str | None
+) -> str | None:
+ """The eleven `sql_*_op` / `sql_*_aggregate_op` names -- one MORE than the ten
+ `Variant` members, because `sql_date_op` is reachable in this direction and no
+ catalog row emits it. `args[0]` is the unquoted template
+ body (this module's argument abstraction level — see the module docstring), `args[1:]`
+ are the already-resolved column expressions the template's `{0}`, `{1}`, ... refer to.
+
+ Without a known `connection_dialect` there is nothing to build a `dialects[]` entry
+ for, and — per the document — the converter must not guess a dialect label, so this
+ stashes (ERROR) exactly like any other total loss. With one, the body is rendered as
+ static SQL for that dialect's entry and logged as a WARNING (a pass-through is always
+ reviewable raw SQL) — never paired with an ANSI_SQL sibling, since the document is
+ explicit that this template's portability is exactly what is unknown.
+ """
+ if not args:
+ raise ValueError("a sql_*_op call needs at least its template-body argument")
+ body_template, *cols = args
+ if connection_dialect is None:
+ log.add(
+ code="TS-EXPR-DIALECT-UNKNOWN",
+ severity=Severity.ERROR,
+ message=(
+ "sql_*_op resolves to a dialects[] entry for the connection's own dialect, "
+ "which could not be derived from TML here; the converter must not guess a "
+ "dialect label. Preserved verbatim for roundtrip."
+ ),
+ object_ref=object_ref,
+ )
+ return None
+ try:
+ body = body_template.format(*cols)
+ except (IndexError, KeyError) as exc:
+ raise ValueError(
+ f"sql_*_op template {body_template!r} does not match {len(cols)} argument(s)"
+ ) from exc
+ log.add(
+ code="TS-EXPR-DIALECT-PASSTHROUGH",
+ severity=Severity.WARNING,
+ message=(
+ f"Raw {connection_dialect} SQL, emitted as a dialects[] entry for that dialect; "
+ "opaque to any consumer that does not implement it. No ANSI_SQL sibling is "
+ "emitted — this template's portability is exactly what is unknown. Review "
+ "before use."
+ ),
+ object_ref=object_ref,
+ )
+ return body
+
+
+for _name in (
+ "sql_string_op", "sql_int_op", "sql_number_op", "sql_double_op", "sql_bool_op",
+ "sql_date_op", "sql_date_time_op", "sql_string_aggregate_op",
+ "sql_int_aggregate_op", "sql_number_aggregate_op", "sql_date_time_aggregate_op",
+):
+ REVERSE[_name] = ReverseConstruct(
+ thoughtspot_name=_name,
+ disposition=ReverseDisposition.DIALECT,
+ dispatch_fn=_dispatch_sql_op,
+ note="Resolves to the Ossie dialects[] mechanism for the connection's own dialect, "
+ "not a portable expression — the right home for raw warehouse SQL.",
+ )
+
+
+# --------------------------------------------------------------------------
+# Runtime, display and calendar concepts.
+# Source: "Reverse direction -> Runtime, display and calendar concepts".
+# --------------------------------------------------------------------------
+
+REVERSE["<runtime parameter reference>"] = ReverseConstruct(
+ thoughtspot_name="<runtime parameter reference>",
+ disposition=ReverseDisposition.STASH,
+ issue_code="TS-EXPR-RUNTIME-PARAMETER",
+ issue_severity=Severity.ERROR,
+ issue_message=(
+ "Runtime parameter reference {name} is resolved per-query from user input; the "
+ "definitions are stashed at model level (owned by the construct-mapping document). "
+ "The expression itself stops being portable once it references a parameter. "
+ "Preserved verbatim for roundtrip."
+ ),
+ note="Synthetic key — not a callable name. See stash_runtime_parameter().",
+)
+
+
+def stash_runtime_parameter(parameter_name: str, log: IssueLog, *, object_ref: str) -> None:
+ """The 'Runtime parameter reference' reverse-direction row.
+
+ Not auto-detected inside `translate_thoughtspot`: a bracketed name
+ (`[Discount Threshold]`) is syntactically identical to an ordinary column reference
+ (`[Table::Column]`), and this module has no model metadata to distinguish the two. The
+ caller — which does have the model's declared parameter list — invokes this directly
+ once it has confirmed `parameter_name` names a declared parameter, not a column.
+ """
+ return _apply_stash(REVERSE["<runtime parameter reference>"], parameter_name, log, object_ref=object_ref)
+
+
+_RUNTIME_IDENTITY_ISSUE = (
+ "{name} resolves signed-in-user identity at query time; an interchange document that "
+ "carried it would describe an access-control decision, not semantics. "
+ "Preserved verbatim for roundtrip."
+)
+for _name in ("ts_username", "ts_groups", "ts_groups_int", "ts_org", "ts_email_domain", "ts_var"):
+ REVERSE[_name] = ReverseConstruct(
+ thoughtspot_name=_name,
+ disposition=ReverseDisposition.STASH,
+ issue_code="TS-EXPR-RUNTIME-IDENTITY",
+ issue_severity=Severity.ERROR,
+ issue_message=_RUNTIME_IDENTITY_ISSUE,
+ )
+
+_HYPERLINK_MARKUP_TOKENS = ("{caption}", "{/caption}")
+
+
+def _has_hyperlink_markup(args: list[str]) -> bool:
+ return any(token in a for a in args for token in _HYPERLINK_MARKUP_TOKENS)
+
+
+REVERSE["concat (hyperlink markup)"] = ReverseConstruct(
+ thoughtspot_name="concat (hyperlink markup)",
+ disposition=ReverseDisposition.STASH,
+ issue_code="TS-EXPR-HYPERLINK-MARKUP",
+ issue_severity=Severity.ERROR,
+ issue_message=(
+ "{name}'s string arguments carry ThoughtSpot's {{caption}}/{{/caption}} hyperlink "
+ "display markup; concat itself maps (it has a spec counterpart, CONCAT), but a "
+ "consumer that rendered the tags literally would show them to users. Preserved "
+ "verbatim for roundtrip."
+ ),
+ note="Synthetic key, reached only via the content-pattern check in translate_thoughtspot "
+ "-- plain concat (no markup) is out of this module's scope entirely.",
+)
+
+_FISCAL_MARKERS = {"fiscal", "'fiscal'"}
+
+
+#: Every call name a FORWARD catalog template BEGINS with, where that name is
+#: all lowercase letters and underscores -- read off the templates themselves so
+#: the two halves cannot drift. Deliberately not "every name the catalog
+#: renders": the regex is anchored at the template start and excludes digits, so
+#: it does not see a name behind a leading operator or paren (`rank_percentile`
+#: in `1 - rank_percentile (...)`, `asin`/`acos`/`atan` in `( asin ( {0} ) ...)`),
+#: a name with a digit in it (`log10`), or any nested call (`sum` inside
+#: `rank ( sum ( [m] ) , 'desc' )`). 50 names today. That is sufficient for its
+#: one consumer, `_fiscal_capable_functions`, since no missing name is
+#: date-shaped -- but a second consumer wanting the true set needs a real parse,
+#: not this. A function, not a module-level constant, for the same
+#: evaluation-order reason as the fiscal set below.
+@functools.lru_cache(maxsize=1)
+def _forward_call_names() -> frozenset[str]:
+ return frozenset(
+ match.group(1)
+ for construct in CATALOG.values()
+ if construct.template
+ for match in [re.match(r"\s*([a-z_]+)\s*\(", construct.template)]
+ if match
+ )
+
+
+#: Word-shaped date tokens. `min(?:ute)?` with no boundaries matched the
+#: AGGREGATE `min` -- and `min_if`, `cumulative_min`, `moving_min` with it --
+#: all classed fiscal-capable, which is the very over-inclusion this gate
+#: exists to prevent. `time` likewise matched inside "run-time".
+_DATE_NAME_TOKENS = re.compile(
+ r"(?:^|_)(?:year|quarter|month|week|weekend|day|date|time|hour|minute|second)(?:$|_)"
+)
+
+#: The `diff_*`/`add_*` families take a fiscal argument but appear in neither
+#: inventory under a date-shaped name.
+_FISCAL_EXTRA_FUNCTIONS = frozenset(
+ f"{verb}_{unit}"
+ for verb in ("diff", "add")
+ for unit in ("years", "quarters", "months", "weeks", "days", "hours", "minutes", "seconds")
+)
+
+
+@functools.lru_cache(maxsize=1)
+def _fiscal_capable_functions() -> frozenset[str]:
+ """Names whose trailing `fiscal` argument really selects the fiscal calendar.
+
+ Computed ON FIRST CALL, not at import. The previous version was a
+ module-level constant evaluated partway down this file while `REVERSE` kept
+ growing below it, so twelve date functions registered later were silently
+ excluded -- and a fiscal call on one of them raised an uncaught `ValueError`
+ where it had previously reported a declared loss. Deriving the set was the
+ right instinct; deriving it before its source existed was the bug.
+ """
+ # `start_of_*` / `end_of_*` are date functions whatever unit they name --
+ # `start_of_min` abbreviates "minute" to `min`, which cannot be a token here
+ # without re-admitting the AGGREGATE `min`.
+ return frozenset(
+ name for name in (set(REVERSE) | _forward_call_names())
+ if _DATE_NAME_TOKENS.search(name)
+ or name.startswith(("start_of_", "end_of_"))
+ ) | _FISCAL_EXTRA_FUNCTIONS
+
+
+def _is_fiscal_variant(name: str, args: list[str]) -> bool:
+ return (
+ name.strip().lower() in _fiscal_capable_functions()
+ and bool(args)
+ and args[-1].strip().lower() in _FISCAL_MARKERS
+ )
+
+
+_FISCAL_ISSUE_MESSAGE = (
+ "{name}'s trailing 'fiscal' argument has no expression: the specification has no "
+ "fiscal-calendar concept, and the fiscal year's start month is model-level metadata no "
+ "per-expression rewrite can recover. Emitting the calendar-year composition "
+ "instead would be silently wrong for any organisation whose year does not start in "
+ "January. Preserved verbatim for roundtrip."
+)
+
+
+def _stash_fiscal_variant(name: str, log: IssueLog, *, object_ref: str) -> None:
+ log.add(
+ code="TS-EXPR-FISCAL-CALENDAR",
+ severity=Severity.ERROR,
+ message=_FISCAL_ISSUE_MESSAGE.format(name=name),
+ object_ref=object_ref,
+ )
+ return None
+
+
+_LOCALE_ISSUE_MESSAGE = (
+ "{name} composes via TO_CHAR, which is EXPERIMENTAL on the Ossie side, and its name "
+ "tokens are locale-dependent by the specification's own admission. Review the target "
+ "locale before relying on this column."
+)
+for _name, _fmt in (("month", "MONTH"), ("year_name", "YYYY"), ("day_of_week", "DAY")):
+ REVERSE[_name] = ReverseConstruct(
+ thoughtspot_name=_name,
+ disposition=ReverseDisposition.COMPOSE,
+ template=f"TO_CHAR({{0}}, '{_fmt}')",
+ issue_code="TS-EXPR-LOCALE-DEPENDENT",
+ issue_severity=Severity.WARNING,
+ issue_message=_LOCALE_ISSUE_MESSAGE,
+ note="Name-returning form, distinct from month_number/year/day_number_of_week.",
+ )
+
+REVERSE["month_number_of_quarter"] = ReverseConstruct(
+ thoughtspot_name="month_number_of_quarter",
+ disposition=ReverseDisposition.COMPOSE,
+ template="MOD(MONTH({0}) - 1, 3) + 1",
+)
+REVERSE["day_number_of_quarter"] = ReverseConstruct(
+ thoughtspot_name="day_number_of_quarter",
+ disposition=ReverseDisposition.COMPOSE,
+ template="DATEDIFF(day, DATE_TRUNC('quarter', {0}), {0}) + 1",
+)
+
+_WEEK_START_ISSUE = (
+ "{name} is correct only if the target engine's week start agrees with the "
+ "specification's fixed Monday start; ThoughtSpot's week start is an instance setting. "
+ "Verify alignment before relying on this column."
+)
+REVERSE["week_number_of_month"] = ReverseConstruct(
+ thoughtspot_name="week_number_of_month",
+ disposition=ReverseDisposition.COMPOSE,
+ template="DATEDIFF(week, DATE_TRUNC('month', {0}), {0}) + 1",
+ issue_code="TS-EXPR-WEEK-START-ASSUMED",
+ issue_severity=Severity.WARNING,
+ issue_message=_WEEK_START_ISSUE,
+)
+REVERSE["week_number_of_quarter"] = ReverseConstruct(
+ thoughtspot_name="week_number_of_quarter",
+ disposition=ReverseDisposition.COMPOSE,
+ template="DATEDIFF(week, DATE_TRUNC('quarter', {0}), {0}) + 1",
+ issue_code="TS-EXPR-WEEK-START-ASSUMED",
+ issue_severity=Severity.WARNING,
+ issue_message=_WEEK_START_ISSUE,
+)
+REVERSE["is_weekend"] = ReverseConstruct(
+ thoughtspot_name="is_weekend",
+ disposition=ReverseDisposition.COMPOSE,
+ template="DATE_PART('dayofweek', {0}) IN (6, 7)",
+ issue_code="TS-EXPR-DAYOFWEEK-BASE",
+ issue_severity=Severity.WARNING,
+ issue_message=(
+ "{name}'s member list (6, 7) uses ThoughtSpot's own DAYOFWEEK base (1 = Monday); "
+ "the specification does not fix a base and engines disagree — confirm "
+ "the target engine's base agrees before relying on this column."
+ ),
+)
+REVERSE["start_of_hour"] = ReverseConstruct(
+ thoughtspot_name="start_of_hour", disposition=ReverseDisposition.COMPOSE,
+ template="DATE_TRUNC('hour', {0})",
+)
+REVERSE["start_of_min"] = ReverseConstruct(
+ thoughtspot_name="start_of_min", disposition=ReverseDisposition.COMPOSE,
+ template="DATE_TRUNC('minute', {0})",
+)
+REVERSE["date"] = ReverseConstruct(
+ thoughtspot_name="date", disposition=ReverseDisposition.COMPOSE,
+ template="DATE_TRUNC('day', {0})",
+)
+REVERSE["time"] = ReverseConstruct(
+ thoughtspot_name="time", disposition=ReverseDisposition.COMPOSE,
+ template="CAST({0} AS TIME)",
+)
+
+
+def _compose_variadic(fn: str) -> ComposeFn:
+ def _compose(args: list[str]) -> str:
+ if not args:
+ raise ValueError(f"{fn} expects at least one argument")
+ return f"{fn}({', '.join(args)})"
+ return _compose
+
+
+REVERSE["greatest"] = ReverseConstruct(
+ thoughtspot_name="greatest",
+ disposition=ReverseDisposition.COMPOSE,
+ compose_fn=_compose_variadic("GREATEST"),
+ note="Never MAX — that would turn a row-wise attribute into an aggregate measure.",
+)
+REVERSE["least"] = ReverseConstruct(
+ thoughtspot_name="least",
+ disposition=ReverseDisposition.COMPOSE,
+ compose_fn=_compose_variadic("LEAST"),
+ note="Never MIN, for the same reason.",
+)
+
+
+# --------------------------------------------------------------------------
+# The dispatcher.
+# --------------------------------------------------------------------------
+
+def translate_thoughtspot(
+ name: str,
+ args: list[str],
+ log: IssueLog,
+ *,
+ object_ref: str,
+ connection_dialect: str | None = None,
+) -> str | None:
+ """Translate one ThoughtSpot-only construct call into an Ossie expression, or stash it.
+
+ Returns the composed Ossie expression string, or `None` when the construct stashes (an
+ issue is always logged in that case) or when `name` has no entry in this
+ module's reverse inventory at all (nothing is logged — that name is either a plain
+ column/measure reference or a construct with a spec counterpart already covered by the
+ forward `CATALOG`, neither of which is this module's concern).
+
+ See the module docstring for the two cross-cutting checks below (fiscal-calendar
+ argument, concat hyperlink markup) and for why `object_ref` and `connection_dialect` are
+ keyword-only additions beyond the plain `(name, args, log)` signature.
+ """
+ if _is_fiscal_variant(name, args):
+ _stash_fiscal_variant(name, log, object_ref=object_ref)
+ return None
+
+ if name == "concat" and _has_hyperlink_markup(args):
+ return _apply_stash(REVERSE["concat (hyperlink markup)"], name, log, object_ref=object_ref)
+
+ construct = REVERSE.get(name)
+ if construct is None:
+ return None
+
+ if construct.dispatch_fn is not None:
+ return construct.dispatch_fn(args, log, object_ref, connection_dialect)
+
+ if construct.disposition is ReverseDisposition.STASH:
+ return _apply_stash(construct, name, log, object_ref=object_ref)
+
+ expression = _render(construct, args)
+ if construct.issue_message:
+ log.add(
+ code=construct.issue_code,
+ severity=construct.issue_severity,
+ message=construct.issue_message.format(name=name),
+ object_ref=object_ref,
+ )
+ return expression
+
+
+# --------------------------------------------------------------------------
+# Dialect-entry and custom_extensions helpers.
+#
+# translate_thoughtspot's own return type is `str | None`, so it cannot itself hand back a
+# dialects[] entry or a custom_extensions payload — those are object-level document
+# concerns, one level above a single expression. These three helpers are what a caller
+# operating at the object level combines with translate_thoughtspot's result to guarantee
+# a lossless roundtrip.
+# --------------------------------------------------------------------------
+
+def thoughtspot_dialect_entry(name: str, args: list[str]) -> dict[str, str]:
+ """The verbatim ThoughtSpot call, reconstructed textually (this module never
+ has the original formula's exact whitespace, only the parsed name/args) so a PARTIAL or
+ STASH construct still round-trips losslessly through a THOUGHTSPOT dialect entry even
+ where no full — or no — portable Ossie expression exists.
+ """
+ inner = f" {' , '.join(args)} " if args else " "
+ return {"dialect": DIALECT, "expression": f"{name} ({inner})"}
+
+
+def portable_dialect_entry(expression: str) -> dict[str, str]:
+ """Pair the THOUGHTSPOT dialect entry with a PORTABLE_DIALECT (ANSI_SQL) sibling
+ wherever the expression alongside it is itself portable, so a consumer that does not
+ implement the THOUGHTSPOT dialect still gets something it can execute. Applies to
+ PARTIAL rows (the frame/order composition is genuine, portable ANSI SQL, just an
+ incomplete window) — never to a pure STASH (there is no portable expression to pair)
+ and never to the `sql_*_op` DIALECT family (the document is explicit: no ANSI_SQL
+ sibling is emitted there, because that template's portability is exactly what is
+ unknown).
+ """
+ return {"dialect": PORTABLE_DIALECT, "expression": expression}
+
+
+def custom_extensions_fragment(column: str, name: str, args: list[str]) -> dict[str, dict[str, str]]:
+ """The payload fragment this module contributes toward an object's
+ `custom_extensions[VENDOR_KEY]` entry (`stash.write_stash`) for one construct
+ this module could not fully compose.
+
+ `write_stash(obj, payload)` treats `payload` as the *contents* of the object's
+ THOUGHTSPOT entry, not as `{VENDOR_KEY: contents}` — `write_stash` already owns the
+ vendor-key wrapping. So this fragment must be keyed by `column`, the caller's
+ Ossie metric/column name, not by `VENDOR_KEY`: this module operates at the
+ single-expression level and has no access to the enclosing object, so the caller merges
+ fragments across an object's columns — `{**fragment_for_col_a, **fragment_for_col_b}` —
+ before calling `stash.write_stash` once per object. Keying by `VENDOR_KEY` instead would
+ make that merge lossy (`{**f1, **f2}` collapses to whichever fragment merged last) and
+ would nest the vendor key inside its own entry when passed to `write_stash` directly.
+ """
+ inner = f" {' , '.join(args)} " if args else " "
+ return {column: {"reverse_thoughtspot_call": f"{name} ({inner})"}}
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/formula.py b/converters/thoughtspot/src/ossie_thoughtspot/formula.py
new file mode 100644
index 00000000..5bd7b634
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/formula.py
@@ -0,0 +1,337 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""A shallow tokenizer for ThoughtSpot formulas.
+
+Deliberately not a parser. It answers only what the two conversion directions need: is an
+expression a single outer call and what are its parts, where are its column references, and
+what does it look like with those references rewritten. Building an expression tree would be
+the SQL-parser decision this converter does not take — expressions pass through under a
+tagged dialect rather than being translated across dialects, matching every other converter
+in this repository and the specification's stated default.
+
+Three ThoughtSpot syntax features drive the implementation and are why an off-the-shelf SQL
+tokenizer is not usable here: column references are bracketed and doubly-colon-qualified
+(`[TABLE::Column]`), grouping uses braces (`{ }`), and a bare bracketed name with no `::` is
+a runtime parameter rather than a column.
+
+**Quoting note — doubling works, backslash-escaping is out of scope on purpose.**
+ThoughtSpot's own convention for an embedded quote in a string literal is doubling it
+(`'it''s'`), and `_scan` handles that correctly even though it has no explicit doubling
+case: the character that closes a quote and the character that immediately reopens it are
+both reported as quoted, so nothing in between ever reads as outside the literal. A
+backslash before a quote is *not* an escape in ThoughtSpot's grammar — it is an ordinary
+character — so `'a\'b'` genuinely ends the literal at the escaped quote, and `_scan`
+splitting there is correct behaviour for this language, not a bug to fix.
+"""
+from __future__ import annotations
+
+import re
+from typing import Callable
+
+from . import identifiers
+
+_CALL_HEAD = re.compile(r"^([A-Za-z_][A-Za-z0-9_]*(?:\s+[A-Za-z_][A-Za-z0-9_]*)*)\s*\(")
+#: Same call-head shape as `_CALL_HEAD`, but unanchored (no `^`) so it matches a call
+#: starting anywhere in the text, and guarded on the left by a negative lookbehind so
+#: a match can never start mid-identifier (e.g. inside "ground" when scanning for a
+#: call literally named "round"). Used by `find_call_names` to find every call in an
+#: expression, not just the single outer one `split_call` answers about.
+_CALL_HEAD_ANYWHERE = re.compile(
+ r"(?<![A-Za-z0-9_])([A-Za-z_][A-Za-z0-9_]*(?:\s+[A-Za-z_][A-Za-z0-9_]*)*)\s*\("
+)
+_BRACKETED = re.compile(r"\[([^\]]*)\]")
+_CLOSERS = {"(": ")", "[": "]", "{": "}"}
+_QUOTES = ("'", '"')
+
+# An operator/control-flow keyword can never be part of a function name, so a call head
+# that includes one of these as a space-separated word is not a call — see
+# `test_a_keyword_prefix_is_not_mistaken_for_a_function_name`. This is a blocklist, not a
+# catalog lookup, so this leaf module stays free of an import edge to the function catalog.
+_KEYWORDS = frozenset(
+ {"and", "or", "not", "if", "then", "else", "true", "false", "in", "between", "like"}
+)
+
+
+def _scan(text: str):
+ """Yield `(index, char, depth, in_quote)` with depth counted before the char is applied.
+
+ One pass shared by every function here so that quoting and nesting are treated
+ identically everywhere — a divergence between two hand-rolled scanners is exactly the
+ kind of bug that would surface as a mis-split argument list months later.
+
+ A `[...]` body is an opaque identifier, not code — a quote character inside one (a
+ display name like `Manager's Bonus`) is part of the name, not a string delimiter. So
+ quote-toggling is suppressed for as long as the innermost open bracket is `[`; a bracket
+ stack (not just the depth counter) tracks which opener is innermost so this only applies
+ to `[...]`, not `(...)` or `{...}`, and correctly un-suppresses again once that `[`
+ closes, however deeply it is nested inside calls. The stack pops only when a closer
+ matches the type of its top entry — a mismatched or stray closer leaves the stack
+ untouched rather than popping the wrong entry and desynchronising which bracket type is
+ considered innermost for the rest of the scan.
+ """
+ depth = 0
+ quote: str | None = None
+ bracket_stack: list[str] = []
+ for i, ch in enumerate(text):
+ if quote is not None:
+ yield i, ch, depth, True
+ if ch == quote:
+ quote = None
+ continue
+ in_bracket_body = bool(bracket_stack) and bracket_stack[-1] == "["
+ if not in_bracket_body and ch in _QUOTES:
+ quote = ch
+ yield i, ch, depth, True
+ continue
+ if ch in _CLOSERS:
+ yield i, ch, depth, False
+ bracket_stack.append(ch)
+ depth += 1
+ continue
+ if ch in (")", "]", "}"):
+ depth -= 1
+ if bracket_stack and _CLOSERS[bracket_stack[-1]] == ch:
+ bracket_stack.pop()
+ yield i, ch, depth, False
+ continue
+ yield i, ch, depth, False
+
+
+def _split_top_level_commas(text: str) -> list[str]:
+ parts: list[str] = []
+ start = 0
+ for i, ch, depth, in_quote in _scan(text):
+ if ch == "," and depth == 0 and not in_quote:
+ parts.append(text[start:i].strip())
+ start = i + 1
+ tail = text[start:].strip()
+ if tail or parts:
+ parts.append(tail)
+ return parts
+
+
+def split_call(expression: str) -> tuple[str, list[str]] | None:
+ """`sum ( [A::x] , 2 )` -> `("sum", ["[A::x]", "2"])`; `None` if not a single outer call.
+
+ Returns `None` — never a partial answer — for anything that merely *contains* a call,
+ such as `sum ( [A::x] ) + 1`. A caller that received `("sum", ["[A::x]"])` for that
+ input would silently drop the `+ 1`, which is precisely the class of silent loss this
+ converter exists to prevent.
+ """
+ text = expression.strip()
+ head = _CALL_HEAD.match(text)
+ if head is None:
+ return None
+ name = head.group(1)
+ if any(word.lower() in _KEYWORDS for word in name.split()):
+ # `true and count (...)` is an operator expression whose last operand happens to
+ # look like a call head, not a call named "true and count". `unique count (...)`
+ # is unaffected — none of its words are in the blocklist.
+ return None
+ open_at = head.end() - 1
+ close_at = None
+ for i, ch, depth, in_quote in _scan(text[open_at:]):
+ if ch == ")" and depth == 0 and not in_quote:
+ close_at = open_at + i
+ break
+ if close_at is None:
+ return None
+ if close_at != len(text) - 1:
+ return None
+ inner = text[open_at + 1 : close_at].strip()
+ if not inner:
+ return head.group(1), []
+ return head.group(1), _split_top_level_commas(inner)
+
+
+def find_call_names(expression: str) -> list[str]:
+ """Every function-call name in `expression`, at any nesting depth, that is
+ not also an operator/control-flow keyword — duplicates kept.
+
+ `split_call` deliberately answers only about the single *outer* call — exactly
+ what building a rendering around a whole expression needs. This answers a
+ different question a safety check needs instead: whether a call with a
+ particular name appears *anywhere* inside the expression, however deeply
+ nested — `round ( sum ( [T::x] ) , 2 )` has `round` as its outer call but
+ `sum` buried one level inside it, and a caller checking only the outer call
+ would miss that the expression already aggregates.
+
+ A name is only reported at a genuine call site: immediately followed by `(`,
+ not inside a quoted string literal, and not inside the opaque body of a
+ `[...]` reference — so a display name or string literal that happens to
+ contain text like `sum (` is never mistaken for a real call.
+
+ **What the keyword exclusion actually costs.** A leading run of
+ operator/control-flow keyword words (`and`, `or`, `not`, `if`, ...) is
+ stripped from a matched run before it is reported: `true and count ( ... )`
+ reports `count`, not the bogus "true and count". But `not` and `if` are
+ *also* genuine ThoughtSpot catalog function names — the catalog holds
+ `not ( expr )` and `if ( ... ) then ...` — and the keyword blocklist cannot
+ tell a real call from an operator use of the same word. So a bare
+ `not ( [A::x] )` or `if ( ... )` reports **nothing** here, even though it is
+ a real call. This is deliberate and unfixed: this function's one caller
+ (`_contains_aggregate_call`) only cares about aggregate names, and neither
+ `not` nor `if` is one, so the loss costs that caller nothing. A caller with
+ a different need could not rely on this function to find every real call.
+
+ **Weaker than `split_call`'s own keyword handling.** `split_call` rejects
+ its *whole* candidate the moment *any* word in it is a keyword, wherever
+ that word sits, because there its only job is to say whether the entire
+ expression is one call — being wrong in either direction there is a
+ correctness bug. This function only strips a *leading* run: a keyword
+ appearing after a genuine word is not stripped, and the whole multi-word
+ run — keyword included — is reported as one (bogus) name instead. For
+ example `flag and sum ( x )` reports the single name `"flag and sum"`, not
+ `sum` — silently missing the real call. This shape does not arise from
+ valid ThoughtSpot formula grammar (a bare word cannot precede `and` like
+ that), which is why it is left as is rather than fixed, but it is not the
+ guarantee `split_call` makes, and this docstring says so rather than
+ implying otherwise.
+ """
+ opaque = {i for i, _ch, _d, in_quote in _scan(expression) if in_quote}
+ for start, end, _body in _bracketed_spans(expression):
+ opaque.update(range(start, end))
+
+ names: list[str] = []
+ for match in _CALL_HEAD_ANYWHERE.finditer(expression):
+ if match.start() in opaque:
+ continue
+ words = match.group(1).split()
+ while words and words[0].lower() in _KEYWORDS:
+ words = words[1:]
+ if words:
+ names.append(" ".join(words))
+ return names
+
+
+def _bracketed_spans(expression: str) -> list[tuple[int, int, str]]:
+ """Every `[...]` span that is not inside a quoted literal, as `(start, end, body)`."""
+ quoted = {i for i, _ch, _d, in_quote in _scan(expression) if in_quote}
+ return [
+ (m.start(), m.end(), m.group(1))
+ for m in _BRACKETED.finditer(expression)
+ if m.start() not in quoted
+ ]
+
+
+def find_column_refs(expression: str) -> list[tuple[str, str]]:
+ """Every `[TABLE::Column]` reference, in order, duplicates kept.
+
+ A bracketed name with no `::` is a runtime parameter, not a column — see
+ `find_parameter_refs`. Splitting delegates to `identifiers.split_column_ref` rather
+ than a bare `str.split("::", 1)`, so an ambiguous reference (more than one `::`
+ delimiter) raises `ValueError` instead of silently taking the first one — consistent
+ with every other reader of this reference shape, and because silently misreading one
+ reference in an otherwise-valid expression is worse than failing the whole call.
+ """
+ return [
+ identifiers.split_column_ref(f"[{body}]")
+ for _s, _e, body in _bracketed_spans(expression)
+ if "::" in body
+ ]
+
+
+#: The prefix a bracketed name with no `::` carries when it is a formula
+#: cross-reference (`[formula_Name]`) rather than a genuine
+#: runtime parameter (`[Discount Threshold]`) — the two are the same
+#: textual shape (a bracketed name, no table qualifier) and are told apart
+#: only by this prefix. Shared here because both conversion directions have
+#: to agree on the convention: the Ossie -> TML model builder mints every
+#: formula's id with this prefix and rewrites cross-references that carry
+#: it, and TML -> Ossie's own parameter finder has to recognise the same
+#: prefix or it misclassifies a formula composing another formula as an
+#: expression referencing a nonexistent runtime parameter.
+FORMULA_REFERENCE_PREFIX = "formula_"
+
+
+def is_formula_reference(body: str) -> bool:
+ """Whether a bracketed name with no `::` (see `find_parameter_refs` and
+ `find_formula_refs`) is a formula cross-reference rather than a genuine
+ runtime parameter."""
+ return body.startswith(FORMULA_REFERENCE_PREFIX)
+
+
+def find_parameter_refs(expression: str) -> list[str]:
+ """Every bracketed name with no table qualifier that is **not** a formula
+ cross-reference — a genuine ThoughtSpot runtime parameter.
+
+ Ossie has no equivalent, so an expression carrying one is not portable and the caller
+ raises an issue rather than emitting a portable sibling. A formula cross-reference
+ (`[formula_Name]`) has the same bracketed, unqualified shape but is a different
+ construct entirely — see `find_formula_refs` and `is_formula_reference` — and must
+ not be reported here as a parameter that does not exist.
+ """
+ return [
+ body for _s, _e, body in _bracketed_spans(expression)
+ if "::" not in body and not is_formula_reference(body)
+ ]
+
+
+def find_formula_refs(expression: str) -> list[str]:
+ """Every bracketed name with no table qualifier that **is** a formula
+ cross-reference — the complement of `find_parameter_refs` within the
+ "no `::`" bracket set.
+
+ A formula composing another formula (`sum ( [formula_Margin] )`) is a
+ first-class ThoughtSpot construct, not a runtime parameter — see
+ `FORMULA_REFERENCE_PREFIX`. It is still not portable: a faithful ANSI_SQL
+ sibling would require inlining the referenced formula's own expression,
+ which this converter does not attempt.
+ """
+ return [
+ body for _s, _e, body in _bracketed_spans(expression)
+ if "::" not in body and is_formula_reference(body)
+ ]
+
+
+def is_bare_column_ref(expression: str) -> tuple[str, str] | None:
+ """`(table, column)` when the whole expression is one column reference, else `None`.
+
+ The common case by a wide margin: most fields are physical columns, and this is what
+ lets those fields carry a portable sibling for free.
+ """
+ text = expression.strip()
+ spans = _bracketed_spans(text)
+ if len(spans) != 1:
+ return None
+ start, end, body = spans[0]
+ if start != 0 or end != len(text) or "::" not in body:
+ return None
+ return identifiers.split_column_ref(f"[{body}]")
+
+
+def rewrite_column_refs(
+ expression: str, rename: Callable[[str, str], str]
+) -> str:
+ """Replace each `[TABLE::Column]` with `rename(table, column)`, byte-preserving elsewhere.
+
+ Everything between references — whitespace, literals, operators — is copied verbatim, so
+ an expression whose references are unchanged is returned unchanged. Parameter references
+ and bracketed text inside quoted literals are left alone.
+ """
+ out: list[str] = []
+ cursor = 0
+ for start, end, body in _bracketed_spans(expression):
+ if "::" not in body:
+ continue
+ table, column = identifiers.split_column_ref(f"[{body}]")
+ out.append(expression[cursor:start])
+ out.append(rename(table, column))
+ cursor = end
+ out.append(expression[cursor:])
+ return "".join(out)
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/identifiers.py b/converters/thoughtspot/src/ossie_thoughtspot/identifiers.py
new file mode 100644
index 00000000..c6284da6
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/identifiers.py
@@ -0,0 +1,151 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Identifier derivation and column-reference rewriting.
+
+ThoughtSpot has one `name` per column, serving as display name, search token and
+cross-document key at once. Ossie splits identifier from label, so the
+identifier has to be derived — and derivation collides.
+
+**Known limitation — non-Latin scripts, not diacritics.** An
+earlier revision of this module documented ASCII-only folding as a stated
+boundary rather than fixing it, on the grounds that a transliteration policy
+is a product decision. That reasoning holds for *transliteration* (e.g.
+Japanese -> romaji) but not for Unicode canonical *decomposition*, which is
+stdlib and needs no policy choice. `normalise` now applies NFKD decomposition
+first (`unicodedata.normalize("NFKD", s)`), which separates a base letter from
+its combining diacritical marks, then drops non-ASCII before the existing
+lowercase-and-substitute folding. A character with no ASCII decomposition
+under NFKD (Cyrillic, CJK, and similarly non-Latin scripts) is still dropped,
+not transliterated, exactly as before — and a name with no ASCII
+alphanumerics surviving still raises `ValueError`. The residual limitation is
+narrower than before: `"Café"` -> `"cafe"`, `"Ürün"` -> `"urun"`, and
+`"Zürich"` -> `"zurich"` now fold correctly, while a CJK-only name (e.g.
+`"北京市"`) still raises. There is also a real open question NFKD does not
+settle: some accented Latin folds to a *conventional* ASCII expansion rather
+than the bare decomposed letter — German `"Müller"` decomposes to `"Muller"`
+here, not the conventional `"Mueller"` — and choosing between them is still a
+product decision left to a later change.
+
+`normalise` itself still raises on a name with no surviving ASCII alphanumerics --
+that has not changed. What changed is who is still allowed to let it propagate.
+`tml_to_ossie.py`'s model/field/metric name conversions each catch it and fall
+back to a different, still-usable identifier instead (see that module's
+`_field_or_metric_identifier` and its model-scope counterpart in `convert`) --
+a display name with no ASCII form is common enough for a non-Latin-script
+customer that treating it as fatal dropped their entire model's worth of
+fields and metrics, not just one name. A caller with no such fallback of its
+own is still expected to let the exception propagate.
+"""
+import re
+import unicodedata
+
+_NON_ALNUM = re.compile(r"[^0-9a-z]+")
+_COLUMN_REF = re.compile(r"^\[(?P<table>[^\]]+?)::(?P<column>[^\]]+)\]$")
+
+
+def normalise(display_name: str) -> str:
+ """Fold a ThoughtSpot display name to an Ossie identifier.
+
+ Diacritics are folded via NFKD decomposition before the ASCII
+ lowercase-and-substitute step — see the module docstring's "Known
+ limitation" note. A character with no ASCII decomposition (non-Latin
+ scripts) is dropped, not transliterated; a name with no ASCII
+ alphanumerics surviving still raises.
+ """
+ ascii_form = unicodedata.normalize("NFKD", display_name).encode("ascii", "ignore").decode("ascii")
+ folded = _NON_ALNUM.sub("_", ascii_form.strip().lower()).strip("_")
+ if not folded:
+ raise ValueError(f"{display_name!r} normalises to an empty identifier")
+ if folded[0].isdigit():
+ # A leading digit is not a valid identifier in most consumers' grammars.
+ folded = f"n_{folded}"
+ return folded
+
+
+class Allocator:
+ """Allocates unique identifiers, resolving collisions with a numeric suffix.
+
+ Ossie resolves regular identifiers case-insensitively
+ (`core-spec/expression_language.md:77`) even though `validation/validate.py`
+ only rejects exact-string duplicates, so a case-only difference would
+ validate and still be ambiguous.
+
+ That guarantee is delivered by `normalise`, which lowercases -- not by the
+ `.casefold()` in `allocate` below. Every candidate is built from
+ `normalise(display_name)` and is therefore already lowercase and ASCII by the
+ time it is stored, so that call is a no-op on this path and is kept only as a
+ belt against `normalise` ever ceasing to lowercase. The place a casefold is
+ genuinely load-bearing is `tml_to_ossie._resolve_name_collision`, which
+ compares against sibling names it did not produce.
+ """
+
+ def __init__(self) -> None:
+ self._taken: set[str] = set()
+
+ def allocate(self, display_name: str) -> str:
+ base = normalise(display_name)
+ candidate, suffix = base, 1
+ while candidate.casefold() in self._taken:
+ suffix += 1
+ candidate = f"{base}_{suffix}"
+ self._taken.add(candidate.casefold())
+ return candidate
+
+
+def split_column_ref(ref: str) -> tuple[str, str]:
+ """`[TABLE::Column]` -> `("TABLE", "Column")`.
+
+ Raises if `ref` doesn't match the `[TABLE::Column]` shape at all, and also
+ if it is *ambiguous* — rather than silently taking the first delimiter and
+ mis-splitting a table or column name that itself contains `::` (e.g. one
+ produced by `format_column_ref("A::B", "C")`). Two distinct ambiguity
+ shapes are checked: more than one non-overlapping `::` delimiter in the
+ whole reference (`str.count` is non-overlapping, which correctly catches
+ two separated delimiters), and a captured column that itself starts with
+ `:` — the signature of a *run* of three or more consecutive colons,
+ which `str.count("::") > 1` cannot see because the run has only one
+ non-overlapping match. `format_column_ref("ORDERS:", "Col")` produces
+ `"[ORDERS:::Col]"`, which is exactly as ambiguous as
+ `format_column_ref("ORDERS", ":Col")` (same string, same encoding
+ collision) and must not silently mis-split to `("ORDERS", ":Col")`.
+ Whether the right fix is an escaping scheme or a different delimiter is a
+ real design question against live ThoughtSpot display names, left to a
+ later change; loud failure is the correct interim behaviour.
+
+ A table or column name containing a single `:` round-trips correctly —
+ `split_column_ref(format_column_ref("A:B", "x")) == ("A:B", "x")`. The
+ table group matches lazily up to the *first* `::`, not a character class
+ that excludes colons outright; only a `::` occurring inside either part
+ is the genuinely ambiguous case the checks above catch.
+ """
+ stripped = ref.strip()
+ match = _COLUMN_REF.match(stripped)
+ if match is None:
+ raise ValueError(f"{ref!r} is not a ThoughtSpot column reference")
+ column = match.group("column")
+ if stripped.count("::") > 1 or column.startswith(":"):
+ raise ValueError(
+ f"{ref!r} is an ambiguous ThoughtSpot column reference: "
+ "contains more than one '::' delimiter"
+ )
+ return match.group("table"), column
+
+
+def format_column_ref(table: str, column: str) -> str:
+ """`("TABLE", "Column")` -> `[TABLE::Column]`."""
+ return f"[{table}::{column}]"
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/issues.py b/converters/thoughtspot/src/ossie_thoughtspot/issues.py
new file mode 100644
index 00000000..adb08ab0
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/issues.py
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Structured, never-silent loss reporting.
+
+Discussion apache/ossie#325 treats a silently dropped field as a contract
+violation rather than a documentation gap, so every declared loss produces an
+issue here at conversion time. The same discussion treats a warning storm as a
+defect, which is why severity is first-class and why callers can summarise via
+count_by_severity() instead of printing every line.
+"""
+from collections import Counter
+from dataclasses import dataclass, field
+from enum import Enum
+
+
+class Severity(str, Enum):
+ INFO = "INFO"
+ WARNING = "WARNING"
+ ERROR = "ERROR"
+
+
+@dataclass(frozen=True)
+class ConverterIssue:
+ """One declared loss or degradation, traceable to a specific object.
+
+ object_ref is mandatory: an issue a reader cannot trace to an object cannot
+ be acted on, which is the complaint #325 raises about warning noise.
+ """
+
+ code: str
+ severity: Severity
+ message: str
+ object_ref: str
+ remedy: str | None = None
+
+ def as_dict(self) -> dict[str, str | None]:
+ return {
+ "code": self.code,
+ "severity": self.severity.value,
+ "message": self.message,
+ "object_ref": self.object_ref,
+ "remedy": self.remedy,
+ }
+
+
+@dataclass
+class IssueLog:
+ """Ordered collection of issues raised during one conversion."""
+
+ issues: list[ConverterIssue] = field(default_factory=list)
+
+ def add(
+ self,
+ *,
+ code: str,
+ severity: Severity,
+ message: str,
+ object_ref: str,
+ remedy: str | None = None,
+ ) -> None:
+ self.issues.append(
+ ConverterIssue(
+ code=code,
+ severity=severity,
+ message=message,
+ object_ref=object_ref,
+ remedy=remedy,
+ )
+ )
+
+ def extend(self, other: "IssueLog") -> None:
+ self.issues.extend(other.issues)
+
+ def as_dicts(self) -> list[dict[str, str | None]]:
+ return [i.as_dict() for i in self.issues]
+
+ def has_errors(self) -> bool:
+ return any(i.severity is Severity.ERROR for i in self.issues)
+
+ def count_by_severity(self) -> dict[str, int]:
+ return dict(Counter(i.severity.value for i in self.issues))
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/keys.py b/converters/thoughtspot/src/ossie_thoughtspot/keys.py
new file mode 100644
index 00000000..513452d7
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/keys.py
@@ -0,0 +1,161 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""primary_key / unique_keys derivation.
+
+TML declares no keys, so every key we emit is manufactured from the
+join graph. Upstream PR #330 checks that a relationship's to_columns covers a
+declared key, and converters/databricks turns a declared key into a
+`rely.at_most_one_match` join hint — so a fabricated key becomes another
+vendor's wrong numbers, not just a cosmetic error in ours.
+
+Orientation is re-checked downstream, so do not rely on ours surviving.
+`converters/databricks` (`ossie_to_metric_view.py:446-478`) swaps `from`/`to`
+and their column arrays — via `_warn()` (`ossie_to_metric_view.py:53`), not
+silently — when the *from* side covers a key and the *to* side does not,
+carrying any `custom_extensions` payload onto the reversed relationship.
+"""
+from dataclasses import dataclass
+
+from .issues import IssueLog, Severity
+
+_TO_ONE = frozenset({"MANY_TO_ONE", "ONE_TO_ONE"})
+
+
+@dataclass(frozen=True)
+class Relationship:
+ """The subset of a relationship that key derivation needs.
+
+ `frozen=True` implies hashability, but `to_columns` may hold a `list`
+ (unhashable), so `hash(instance)` is not reliably safe here — do not put
+ a `Relationship` in a set or use it as a dict key without checking the
+ concrete `to_columns` type first.
+ """
+
+ name: str
+ to_dataset: str
+ to_columns: tuple[str, ...] | list[str]
+ cardinality: str
+ has_residual_predicates: bool
+
+
+def _qualifies(rel: Relationship) -> bool:
+ """Key evidence requires a to-one join whose condition is wholly
+ equality, and columns actually present to name as the key."""
+ return (
+ rel.cardinality in _TO_ONE
+ and not rel.has_residual_predicates
+ and bool(rel.to_columns)
+ )
+
+
+def derive_keys(
+ dataset_name: str, relationships: list[Relationship], log: IssueLog
+) -> tuple[list[str] | None, list[list[str]]]:
+ """Return (primary_key, unique_keys) for one dataset.
+
+ primary_key is emitted only when the qualifying relationships agree on a
+ single column set — where they disagree, choosing one is a guess, so the
+ candidates go to unique_keys and no primary key is declared.
+ """
+ inbound = [r for r in relationships if r.to_dataset == dataset_name]
+ qualifying = [r for r in inbound if _qualifies(r)]
+
+ seen: list[list[str]] = []
+ for rel in qualifying:
+ cols = list(rel.to_columns)
+ if cols not in seen:
+ seen.append(cols)
+
+ # Explain every disqualified relationship's non-key status.
+ #
+ # An empty to_columns is a hard schema failure, not a coverage warning:
+ # upstream's schema requires to_columns to be a non-empty list
+ # (minItems: 1), so such a relationship cannot be emitted at all. It is
+ # reported unconditionally, at ERROR severity, with its own remedy.
+ #
+ # For every other disqualification (residual predicates, wrong
+ # cardinality), the base "not a declared key" statement is our own,
+ # unconditional explanation of why we did not derive a key from this
+ # relationship. Upstream's *separate* to_columns coverage check
+ # (`validate_references` in `validation/validate.py` -- named rather than
+ # cited by line, because the line number this comment used to give had
+ # drifted onto an unrelated helper) only warns when a declared key exists
+ # for this dataset AND this relationship's columns fail to cover any of it
+ # — `declared_keys and not any(set(key) <= to_column_set for
+ # key in declared_keys)`. So predicting that warning is only added when
+ # that same condition genuinely holds here: a key was derived (`seen`)
+ # and none of the derived keys is a subset of this relationship's
+ # to_columns. The canonical SCD-2 residual (as-of) join, whose
+ # to_columns exactly covers the derived key, passes upstream's check
+ # clean — predicting a warning for it would be wrong.
+ for rel in inbound:
+ if _qualifies(rel):
+ continue
+
+ if not rel.to_columns:
+ log.add(
+ code="TS_KEY_COVERAGE",
+ severity=Severity.ERROR,
+ message=(
+ f"Relationship {rel.name!r} targets {dataset_name!r} with an empty "
+ f"to_columns. Ossie's schema requires to_columns to be a non-empty "
+ f"list (minItems: 1), so this relationship cannot be emitted as-is."
+ ),
+ object_ref=f"relationship:{rel.name}",
+ remedy=(
+ "Not expected. Populate to_columns with the join columns on the "
+ "'to' dataset, or drop the relationship — an empty to_columns fails "
+ "Ossie schema validation outright; it is not a coverage warning."
+ ),
+ )
+ continue
+
+ if rel.has_residual_predicates:
+ reason = "its condition carries residual (non-equality) predicates"
+ else:
+ reason = f"its cardinality is {rel.cardinality}"
+
+ message = (
+ f"Relationship {rel.name!r} targets {dataset_name!r} on columns that are "
+ f"not a declared key, because {reason}."
+ )
+ to_column_set = set(rel.to_columns)
+ if seen and not any(set(key) <= to_column_set for key in seen):
+ message += (
+ " Ossie validation will report a to_columns coverage warning for it."
+ )
+
+ log.add(
+ code="TS_KEY_COVERAGE",
+ severity=Severity.WARNING,
+ message=message,
+ object_ref=f"relationship:{rel.name}",
+ remedy=(
+ "The relationship is genuinely not a key join; declaring a key to "
+ "silence a coverage warning would assert uniqueness that does not hold."
+ ),
+ )
+
+ # `list(...)`, not `seen[0]`: returning the same list OBJECT in both slots
+ # made PyYAML emit a YAML anchor/alias pair -- `primary_key: &id001` with
+ # `unique_keys: [*id001]` -- into every generated document, and into both
+ # committed fixtures, where the tests pinned it rather than caught it.
+ # Aliases are valid YAML but a common hardening default disables them, and a
+ # non-PyYAML reader that does not resolve them reads null.
+ primary_key = list(seen[0]) if len(seen) == 1 else None
+ return primary_key, seen
diff --git a/converters/thoughtspot/src/ossie_thoughtspot/ossie_to_thoughtspot.py b/converters/thoughtspot/src/ossie_thoughtspot/ossie_to_thoughtspot.py
new file mode 100644
index 00000000..d5794842
--- /dev/null
+++ b/converters/thoughtspot/src/ossie_thoughtspot/ossie_to_thoughtspot.py
@@ -0,0 +1,2563 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Build a ThoughtSpot Table or SQL View TML document from one Ossie dataset.
+
+An Ossie semantic model becomes 1+N TML documents: one Model document plus one
+Table (or SQL View) document per dataset. The Model references each table by
+name, so the tables have to exist first — this module builds the "N" half.
+Building the Model document itself (formulas, surfaced columns, joins) is a
+separate module, because deciding which of a dataset's fields become a
+physical column here versus a Model formula there needs the same test either
+way: a field whose expression is a single, unqualified column reference is
+physical; anything else — a function call, an operator, several references —
+is computed and has no physical column to hold it. Only the first kind is
+handled here.
+
+Two things are unrecoverable *from the field's own expression alone*, and
+both are handled by falling back to a documented default rather than
+guessing, with the fallback always reported:
+
+* **The warehouse column's own physical name, for a field that came from a
+ real ThoughtSpot table.** A round-tripped field's bracketed reference
+ (e.g. ``[ORDERS::Order Date]``) carries the table column's *display* name
+ only, not its own `db_column_name` — a Model's `column_id` is matched by
+ display name, never by warehouse name. When the two genuinely differed,
+ the forward direction now stashes the true warehouse name separately
+ (`FIELD_STASH_DB_COLUMN_NAME`, Table-backed columns only), and that value
+ is used whenever present. Only when it is genuinely absent — a
+ hand-authored bracket, or a document produced before this key existed —
+ does this fall back to assuming the display name and the warehouse name
+ agree, which is correct in the common case and is reported as an
+ assumption otherwise, because a wrong guess here names a column the
+ warehouse may not have. A hand-authored field instead carries a bare,
+ unqualified SQL identifier for its own physical column (e.g.
+ ``order_date``), which genuinely *is* its warehouse name, not a stand-in
+ for one, so no assumption or issue is needed there. Either way,
+ ``db_column_name`` is written — always, even when it is identical to the
+ column's display name, because some ThoughtSpot instances reject an import
+ that omits it.
+* **Whether an Ossie `Time` field's underlying warehouse column is really a
+ full timestamp.** The datatype map gives `Time` a conditional mapping —
+ `VARCHAR` normally, `DATE_TIME` when the column is timestamp-backed — but
+ that condition needs a fact this module has no way to observe. A `Time`
+ value can only ever reach this function from a hand-authored document in
+ the first place: the forward direction never emits `Time` at all (nothing
+ round-trips into it), so there is never a stashed ThoughtSpot column behind
+ it to inspect, and an Ossie `Field` carries no storage-format signal
+ besides `datatype` itself. There is nothing here to condition on, so the
+ unconditional default (`VARCHAR`) is what gets written, and the datatype's
+ declared loss is still reported so the choice is visible rather than silent.
+"""
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from typing import Callable, Sequence
+
+from . import datatypes, formula, identifiers, stash
+from .constants import (
+ FIELD_STASH_FORMULA_ID,
+ MODEL_STASH_OBJ_ID,
+ DATASET_STASH_ALIAS,
+ DATASET_STASH_CONNECTION_NAME,
+ DATASET_STASH_SOURCE_PARTS,
+ DATASET_STASH_SOURCE_PARTS_DB,
+ DATASET_STASH_SOURCE_PARTS_DB_TABLE,
+ DATASET_STASH_SOURCE_PARTS_SCHEMA,
+ DATASET_STASH_SQL_OUTPUT_COLUMNS,
+ DATASET_STASH_TABLE_NAME,
+ DATASET_STASH_TABLE_PROPERTIES,
+ DATASET_STASH_TML_OBJECT,
+ DATASET_STASH_TML_OBJECT_WITNESS,
+ DATASET_STASH_UNSURFACED_COLUMNS,
+ DIALECT,
+ FIELD_STASH_COLUMN_PROPERTIES,
+ FIELD_STASH_DATA_TYPE,
+ FIELD_STASH_DATA_TYPE_WITNESS,
+ FIELD_STASH_DB_COLUMN_NAME,
+ FIELD_STASH_DB_COLUMN_NAME_WITNESS,
+ METRIC_STASH_COLUMN_AGGREGATION,
+ METRIC_SHAPE_COLUMN_AGGREGATION,
+ METRIC_SHAPE_FORMULA,
+ METRIC_SHAPE_SCALAR_FORMULA_PLUS_AGGREGATION,
+ METRIC_STASH_SHAPE,
+ MODEL_STASH_ACTION_OBJECT_ASSOCIATIONS,
+ MODEL_STASH_COLUMN_GROUPS,
+ MODEL_STASH_CONSTRAINTS,
+ MODEL_STASH_FILTERS,
+ MODEL_STASH_LESSON_PLANS,
+ MODEL_STASH_MODEL_JOINS_WITH,
+ MODEL_STASH_MODEL_PROPERTIES,
+ MODEL_STASH_PARAMETERS,
+ MODEL_STASH_UNSURFACED_FORMULAS,
+ MODEL_STASH_UNATTRIBUTED_FORMULAS,
+ MODEL_STASH_UNREPRESENTABLE_JOINS,
+ PORTABLE_DIALECT,
+ RELATIONSHIP_STASH_CARDINALITY,
+ RELATIONSHIP_STASH_ENDPOINTS_SWAPPED,
+ RELATIONSHIP_STASH_ENDPOINTS_SWAPPED_WITNESS,
+ RELATIONSHIP_STASH_JOIN_SHAPE,
+ RELATIONSHIP_STASH_ON_EXPRESSION,
+ RELATIONSHIP_STASH_ON_EXPRESSION_WITNESS,
+ RELATIONSHIP_STASH_REFERENCING_JOIN,
+ RELATIONSHIP_STASH_TYPE,
+ STASH_TML_NAME,
+)
+from .errors import ConversionError
+from .expressions import CATALOG, Classification, emit_direct, emit_passthrough, emit_unmappable
+from .issues import IssueLog, Severity
+from .tml import DocumentSet, TmlDocument, block_scalar
+
+#: A plain ANSI SQL regular identifier (unquoted) or a double-quoted one, per
+#: the specification's own identifier grammar — up to 128 characters, and a
+#: quoted identifier's content is the literal column name with the quotes
+#: stripped. This is what a hand-authored field's own physical-column
+#: expression looks like: no dataset qualifier (a field's expression runs
+#: against its own dataset's source), no operators, no function calls.
+_BARE_IDENTIFIER_RE = re.compile(r'^(?:[A-Za-z_][A-Za-z0-9_]{0,127}|"[^"]{1,128}")$')
+
+#: Any source string containing whitespace outside of a quoted identifier
+#: reads as a query rather than a `db.schema.table` reference — a real
+#: three-part identifier never contains one there, and any genuine SQL query
+#: does (at minimum a `SELECT` and a target). Whitespace *inside* a quoted
+#: identifier (`SALES.PUBLIC."ORDER TABLE"`) is a legitimate table name and
+#: must not trip this — see `_split_three_part_identifier`, which is always
+#: tried first for exactly that reason.
+_WHITESPACE_RE = re.compile(r"\s")
+
+#: A plain, unquoted ANSI SQL identifier segment.
+_PLAIN_IDENTIFIER_SEGMENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
+
+
+def _split_three_part_identifier(source: str) -> list[str] | None:
+ """`source` split on top-level `.` into its parts, or `None` when it does
+ not parse as a dotted identifier sequence at all.
+
+ Each part is either a plain unquoted identifier or a double-quoted one —
+ which may itself contain a `.`, whitespace, or any other character
+ except a literal quote, e.g. `"ORDER TABLE"`. Detecting the three-part
+ shape this way, before ever asking whether `source` merely *contains*
+ whitespace, is what keeps a quoted identifier with a space in it
+ (`SALES.PUBLIC."ORDER TABLE"`) from being misread as a query: the
+ quoted part's own whitespace is never inspected outside the quotes that
+ scope it. A genuine query fails this parse almost immediately -- its
+ first keyword is followed by a space, not a `.` or the end of the
+ string -- and falls through to the whitespace check instead.
+ """
+ parts: list[str] = []
+ i, n = 0, len(source)
+ if n == 0:
+ return None
+ while True:
+ if i >= n:
+ return None # a trailing '.' with nothing after it
+ if source[i] == '"':
+ end = source.find('"', i + 1)
+ if end == -1 or end == i + 1:
+ return None # unterminated or empty quoted identifier
+ parts.append(source[i + 1:end])
+ i = end + 1
+ else:
+ match = _PLAIN_IDENTIFIER_SEGMENT_RE.match(source, i)
+ if match is None:
+ return None
+ parts.append(match.group(0))
+ i = match.end()
+ if i == n:
+ return parts
+ if source[i] != ".":
+ return None
+ i += 1
+
+
+def _bare_sql_identifier(expression: str) -> str | None:
+ """The plain column name `expression` names, or `None` if it is not one
+ single unqualified identifier."""
+ text = expression.strip()
+ if _BARE_IDENTIFIER_RE.match(text) is None:
+ return None
+ if text.startswith('"') and text.endswith('"'):
+ return text[1:-1]
+ return text
+
+
+def _physical_identity(field: dict, log: IssueLog, *, object_ref: str) -> tuple[str, str] | None:
+ """`(display name, warehouse identifier)` for a physical field, or `None`
+ when the field is computed and has no single physical column to become.
+
+ A THOUGHTSPOT-dialect entry, when present, is authoritative and is
+ checked first: it is the verbatim expression a prior TML -> Ossie trip
+ preserved, so a bare `[TABLE::Column]` reference names the table's own
+ column exactly, and anything else in that dialect is unambiguously a
+ formula — no other dialect is worth consulting once a THOUGHTSPOT entry
+ says "computed". Only when there is no THOUGHTSPOT entry at all (a
+ hand-authored document) does a bare, unqualified SQL identifier in any
+ other dialect count as a physical reference instead.
+ """
+ dialects = ((field.get("expression") or {}).get("dialects")) or []
+ if not dialects:
+ log.add(
+ code="TS-FIELD-NO-EXPRESSION",
+ severity=Severity.WARNING,
+ message="field has no expression dialects; it cannot become a table column",
+ object_ref=object_ref,
+ )
+ return None
+
+ ts_entry = next((d for d in dialects if d.get("dialect") == DIALECT), None)
+ if ts_entry is not None:
+ ts_expr = ts_entry.get("expression", "")
+ try:
+ bare = formula.is_bare_column_ref(ts_expr)
+ except ValueError as exc:
+ # split_column_ref raises for two distinct reasons -- the
+ # bracket's table or column part itself contains "::", making the
+ # delimiter genuinely ambiguous, or the text inside the brackets
+ # never matched the [TABLE::Column] shape at all (e.g. an empty
+ # table part) -- and correctly refuses to guess in either case
+ # rather than silently mis-splitting. That refusal must not
+ # propagate as an uncaught exception out of a document
+ # conversion: it is reported and the field is treated the same
+ # as any other THOUGHTSPOT expression that is not a single
+ # column reference (see the `bare is None` case just below).
+ log.add(
+ code="TS-FIELD-COLUMN-REF-MALFORMED",
+ severity=Severity.ERROR,
+ message=(
+ f"expression {ts_expr!r} is not a usable ThoughtSpot column "
+ f"reference ({exc}); it cannot become a table column and is "
+ f"instead carried into the model as a formula, verbatim, "
+ f"which will fail to import until it is fixed"
+ ),
+ object_ref=object_ref,
+ )
+ return None
+ if bare is None:
+ # A THOUGHTSPOT expression that is not a single column reference
+ # is a formula -- computed fields are the Model document's
+ # concern, not the table's.
+ return None
+ _table, column = bare
+ # The bracket's own column part is the table's *display* name (what
+ # a Model column_id must match), not necessarily its warehouse
+ # db_column_name -- a physical column is matched by display name
+ # only. When the forward direction saw the two differ, it stashes
+ # the true warehouse name on the field, witnessed against the
+ # display name it was recorded for: trustworthy only when the
+ # field still names the same physical column, since a user
+ # retargeting the bracket reference to a different column leaves a
+ # stash that now names the WRONG column's warehouse name -- one
+ # that would otherwise be silently applied to this one.
+ field_stash = stash.read_stash(field)
+ was_stashed = FIELD_STASH_DB_COLUMN_NAME in field_stash
+ stashed_db_column_name = stash.restore(
+ field_stash, FIELD_STASH_DB_COLUMN_NAME, None,
+ witness=column, witness_key=FIELD_STASH_DB_COLUMN_NAME_WITNESS,
+ )
+ if isinstance(stashed_db_column_name, str) and stashed_db_column_name:
+ return column, stashed_db_column_name
+ if was_stashed:
+ log.add(
+ code="TS-FIELD-DB-COLUMN-NAME-STALE",
+ severity=Severity.WARNING,
+ message=(
+ f"a stashed warehouse column name was recorded for a "
+ f"different physical column than this field's current "
+ f"{column!r}; the field was retargeted since the stash "
+ f"was written, so the stash is dropped and "
+ f"db_column_name is set equal to the display name "
+ f"instead, which will name a column the warehouse does "
+ f"not have if the two differ"
+ ),
+ object_ref=object_ref,
+ )
+ return column, column
+ # No `db_column_name` stashed. For a field this converter itself
+ # produced that is a RECORDED FACT, not a guess: the forward direction
+ # stashes the key ONLY when the warehouse name differs from the display
+ # name, so its absence means they agreed. Reporting it said "assumed"
+ # about something the document actually establishes -- and it fired 593
+ # times across 30 real models, on the ordinary case, which is precisely
+ # how an issue log stops being read.
+ #
+ # It IS a genuine assumption for a field with no THOUGHTSPOT stash at
+ # all -- hand-authored Ossie, or another vendor's -- because nothing
+ # there ever recorded the relationship. That case still reports.
+ if not stash.read_stash(field):
+ log.add(
+ code="TS-FIELD-DB-COLUMN-NAME-ASSUMED",
+ severity=Severity.WARNING,
+ message=(
+ f"field {column!r} carries no ThoughtSpot stash, so no "
+ f"warehouse column name was ever recorded for it; "
+ f"db_column_name is set equal to the display name, which "
+ f"will name a column the warehouse does not have if the two "
+ f"differ"
+ ),
+ object_ref=object_ref,
+ )
+ return column, column
+
+ display_name = field.get("label") or field.get("name")
+ for entry in dialects:
+ identifier = _bare_sql_identifier(entry.get("expression", ""))
+ if identifier is None:
+ continue
+ if not display_name:
+ log.add(
+ code="TS-FIELD-NO-NAME",
+ severity=Severity.WARNING,
+ message="field has neither a label nor a name; it cannot become a table column",
+ object_ref=object_ref,
+ )
+ return None
+ return display_name, identifier
+ return None
+
+
+def _field_datatype(field: dict, log: IssueLog, *, object_ref: str) -> str:
+ """The `db_column_properties.data_type` for one physical field.
+
+ A field with no declared `datatype` still gets one: ThoughtSpot treats
+ the whole `db_column_properties` block as compulsory, so an absent value
+ is inferred (`datatypes.to_tml(None)`) rather than the key being omitted.
+ """
+ datatype = field.get("datatype")
+ if datatype is not None:
+ loss = datatypes.declared_loss(datatype)
+ if loss is not None:
+ log.add(
+ code="TS-FIELD-DATATYPE-DECLARED-LOSS",
+ severity=Severity.WARNING,
+ message=f"datatype {datatype!r} does not round-trip exactly: {loss}",
+ object_ref=object_ref,
+ )
+
+ field_stash = stash.read_stash(field)
+ was_stashed = FIELD_STASH_DATA_TYPE in field_stash
+ # The exact ThoughtSpot spelling a prior TML -> Ossie trip recorded
+ # (BOOL vs BOOLEAN, FLOAT vs DOUBLE) wins over a freshly derived one only
+ # when the witness -- the Ossie datatype it was recorded against --
+ # still matches this field's current `datatype`. A field whose declared
+ # type was edited since (Boolean -> String, say) makes the stashed
+ # spelling stale: "BOOL" names a warehouse type for the datatype that
+ # *was* there, not the one that is there now.
+ stashed_spelling = stash.restore(
+ field_stash, FIELD_STASH_DATA_TYPE, None,
+ witness=datatype, witness_key=FIELD_STASH_DATA_TYPE_WITNESS,
+ )
+ if isinstance(stashed_spelling, str) and stashed_spelling:
+ return stashed_spelling
+ if was_stashed:
+ log.add(
+ code="TS-FIELD-DATA-TYPE-STASH-STALE",
+ severity=Severity.WARNING,
+ message=(
+ f"a warehouse spelling was stashed for a different datatype "
+ f"than this field's current {datatype!r}; the field was "
+ f"edited since the stash was written, so the stash is "
+ f"dropped and the canonical spelling is derived instead"
+ ),
+ object_ref=object_ref,
+ )
+
+ try:
+ return datatypes.to_tml(datatype)
+ except ValueError:
+ log.add(
+ code="TS-FIELD-DATATYPE-UNKNOWN",
+ severity=Severity.WARNING,
+ message=(
+ f"datatype {datatype!r} is not a recognised Ossie datatype; "
+ f"INT64 is inferred instead"
+ ),
+ object_ref=object_ref,
+ )
+ return datatypes.to_tml(None)
+
+
+def _field_object_ref(field: dict) -> str:
+ return f"field:{field.get('label') or field.get('name') or '<unnamed>'}"
+
+
+def _physical_table_column(field: dict, log: IssueLog) -> dict | None:
+ """One Table `columns[]` entry for `field`, or `None` when it is computed.
+
+ `description` is deliberately never copied here. Every field that
+ reaches this function is, by construction, also surfaced as a Model
+ `columns[]` ATTRIBUTE entry (`_build_field`, which writes the same
+ description there) -- a Table-only physical column never becomes a
+ `field` at all; it survives verbatim through
+ `DATASET_STASH_UNSURFACED_COLUMNS` instead. So the Model entry is the
+ only correct home for a Model-surfaced field's description; writing it
+ here too would assert something on the Table document its own source
+ never carried.
+ """
+ object_ref = _field_object_ref(field)
+ identity = _physical_identity(field, log, object_ref=object_ref)
+ if identity is None:
+ return None
+ name, db_column_name = identity
+ return {
+ "name": name,
+ # Always present, even equal to `name` -- some ThoughtSpot instances
+ # reject an import that omits it.
+ "db_column_name": db_column_name,
+ "db_column_properties": {"data_type": _field_datatype(field, log, object_ref=object_ref)},
+ }
+
+
+def _physical_sql_view_column(field: dict, output_aliases: dict, log: IssueLog) -> dict | None:
+ """One SQL View `sql_view_columns[]` entry for `field`, or `None` when it
+ is computed.
+
+ `output_aliases` is the dataset's stashed `field name -> sql_output_column`
+ map. It wins when present, because a query output alias is not something
+ the field's own expression can be relied on to reconstruct; the bare
+ identifier `_physical_identity` finds is only a fallback for a
+ hand-authored field with no such record.
+ """
+ object_ref = _field_object_ref(field)
+ identity = _physical_identity(field, log, object_ref=object_ref)
+ if identity is None:
+ return None
+ name, fallback_identifier = identity
+ sql_output_column = output_aliases.get(field.get("name")) or fallback_identifier
+ # `description` is never copied here -- see the matching note on
+ # _physical_table_column just above; the same reasoning applies
+ # unchanged to a SQL View's own physical column.
+ return {
+ "name": name,
+ "sql_output_column": sql_output_column,
+ "db_column_properties": {"data_type": _field_datatype(field, log, object_ref=object_ref)},
+ }
+
+
+def _derive_kind(source: str) -> tuple[str, bool]:
+ """`(kind, malformed)` guessed from `source` alone.
+
+ A genuine three-part dotted identifier — quoted parts included, so a
+ quoted identifier's own internal whitespace is never mistaken for a
+ query — reads as a table reference. Failing that, whitespace anywhere
+ else in `source` reads as a query: a real `db.schema.table` reference
+ never contains any outside a quoted part, and a real SQL query always
+ does. Anything else is neither shape clearly enough to guess, so it is
+ reported malformed and a table is still produced -- `_source_parts` is
+ what actually raises the issue for it, so the same root cause is never
+ reported twice.
+ """
+ parts = _split_three_part_identifier(source)
+ if parts is not None and len(parts) == 3 and all(parts):
+ return "table", False
+ if _WHITESPACE_RE.search(source):
+ return "sql_view", False
+ return "table", True
+
+
+def _decide_kind(dataset: dict, payload: dict, log: IssueLog, *, object_ref: str) -> str:
+ """Whether `dataset` becomes a `table:` or `sql_view:` document.
+
+ A stashed `tml_object` (written whenever this dataset came from a prior
+ TML -> Ossie trip) is authoritative -- it also determines which shape
+ `unsurfaced_columns` was captured in, so trusting it keeps that list
+ valid -- but only when its witness (the `source` it was stashed
+ alongside) still matches this dataset's CURRENT `source`. A user who
+ rewrites `source` from a table reference to a query (or back) since the
+ stash was written leaves a `tml_object` that now describes the wrong
+ shape; using it anyway would misread `source` under the old rules (a
+ query parsed as db/schema/table, or vice versa). A hand-authored dataset
+ has no stash at all, and falls through to `_derive_kind` either way.
+ """
+ source = dataset.get("source") or ""
+ was_stashed = DATASET_STASH_TML_OBJECT in payload
+ stashed_kind = stash.restore(
+ payload, DATASET_STASH_TML_OBJECT, None,
+ witness=source, witness_key=DATASET_STASH_TML_OBJECT_WITNESS,
+ )
+ if stashed_kind in ("table", "sql_view"):
+ return stashed_kind
+ if was_stashed:
+ log.add(
+ code="TS-DATASET-TML-OBJECT-STALE",
+ severity=Severity.WARNING,
+ message=(
+ "a stashed document kind (table/sql_view) no longer matches "
+ "this dataset's current source; the source was rewritten "
+ "since the stash was written, so the stash is dropped and "
+ "the kind is re-derived from the current source instead"
+ ),
+ object_ref=object_ref,
+ )
+ kind, _malformed = _derive_kind(source)
+ return kind
+
+
+def _source_parts(dataset: dict, payload: dict, log: IssueLog, *, object_ref: str) -> tuple[str, str, str]:
+ """`(db, schema, db_table)` for a Table document.
+
+ A stashed `source_parts` entry is used only when it still reconstructs
+ the dataset's current `source` exactly -- the dataset may have been
+ hand-edited since the stash was written, and a plain split of the live
+ `source` is the correct behaviour once that has happened, not a stale
+ three-way split nobody asked for any more.
+ """
+ source = dataset.get("source") or ""
+ stashed = payload.get(DATASET_STASH_SOURCE_PARTS)
+ if isinstance(stashed, dict):
+ db, schema, db_table = (
+ stashed.get(DATASET_STASH_SOURCE_PARTS_DB, ""),
+ stashed.get(DATASET_STASH_SOURCE_PARTS_SCHEMA, ""),
+ stashed.get(DATASET_STASH_SOURCE_PARTS_DB_TABLE, ""),
+ )
+ if ".".join((db, schema, db_table)) == source:
+ return db, schema, db_table
+ log.add(
+ code="TS-DATASET-SOURCE-PARTS-STALE",
+ severity=Severity.WARNING,
... 22122 lines suppressed ...