You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ossie.apache.org by "Spissable (via GitHub)" <gi...@apache.org> on 2026/09/04 09:26:15 UTC
[PR] feat: add lightdash converter [ossie]
Spissable opened a new pull request, #360:
URL: https://github.com/apache/ossie/pull/360
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## Summary
Adds the bidirectional apache-ossie-lightdash converter and ossie-lightdash CLI.
- Imports dbt metadata, seeds, catalog column types, and dbt-free Lightdash model files.
- Converts datasets, dimensions, metrics, joins, primary keys, datatypes, and AI hints in both directions.
- Preserves Lightdash-specific metadata in the registered LIGHTDASH extension.
- Reports unsupported or lossy constructs instead of silently producing misleading output.
- Exports standalone Lightdash model files by default, with dbt meta output available via --format dbt-meta.
- Includes mapping documentation and dedicated Python 3.11–3.14 CI.
<!-- Describe what this PR does and why. -->
## Related Issues
Closes #227
Supersedes https://github.com/apache/ossie/pull/253 (with permission from @ota2000
## Checklist
- [x] Spec and Python model register LIGHTDASH as a well-known vendor.
- [x] Converter behavior is covered by focused tests.
- [x] User-facing behavior, examples, and fidelity limitations are documented.
- [x] ASF license headers are present on new source files.
- [x] No ontology or validation-rule changes are required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add lightdash converter [ossie]
Posted by "hamzahc1 (via GitHub)" <gi...@apache.org>.
hamzahc1 commented on PR #360:
URL: https://github.com/apache/ossie/pull/360#issuecomment-6315505150
@jbonofre could we get a 2nd pass on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add lightdash converter [ossie]
Posted by "christianeu-db (via GitHub)" <gi...@apache.org>.
christianeu-db commented on PR #360:
URL: https://github.com/apache/ossie/pull/360#issuecomment-6333994705
closing (I see there is already an update in the ossie-converters)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add lightdash converter [ossie]
Posted by "Spissable (via GitHub)" <gi...@apache.org>.
Spissable commented on code in PR #360:
URL: https://github.com/apache/ossie/pull/360#discussion_r4311606896
##########
converters/lightdash/src/ossie_lightdash/expression_utils.py:
##########
@@ -0,0 +1,292 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Expression helpers shared by both conversion directions.
+
+Lightdash SQL snippets reference the current model as ``${TABLE}.column``,
+sibling fields as ``${column}`` or ``${metric}``, joined models as
+``${other_table.column}`` and project parameters or user attributes as
+``${lightdash.…}`` / ``${ld.…}``. Ossie expressions reference columns as
+``dataset.column``. These helpers translate between the two spellings and
+recognise the aggregation shapes that map onto Lightdash's typed metrics.
+"""
+
+import re
+from dataclasses import dataclass, field
+from typing import Callable, Dict, List, Optional
+
+_REFERENCE_RE = re.compile(r"\$\{([^}]+)\}")
+_NON_PORTABLE_REFERENCE_RE = re.compile(r"\$\{(?:lightdash|ld)\.[^}]*\}")
+# Liquid tags and bare `ld.parameters.x` / `ld.query.filters` / `ld.user.x`
+# references inside them are evaluated by Lightdash at query time.
+_NON_PORTABLE_SYNTAX_RE = re.compile(
+ r"\{%|\{\{|(?<![\w.])(?:lightdash|ld)\.(?:parameters|query|user)\."
+)
+_COLUMN_REFERENCE_RE = re.compile(r"^[A-Za-z_]\w*(?:\.\w+)?$")
+
+# Lightdash metric types that are encoded as an aggregation in the Ossie
+# expression. Anything else is exported as a `number` metric with raw SQL.
+_AGGREGATE_FUNCTIONS = {
+ "sum": "SUM",
+ "min": "MIN",
+ "max": "MAX",
+ "average": "AVG",
+ "median": "MEDIAN",
+ "count": "COUNT",
+}
+_DISTINCT_AGGREGATE_FUNCTIONS = {
+ "count_distinct": "COUNT",
+ "sum_distinct": "SUM",
+ "average_distinct": "AVG",
+}
+AGGREGATE_TYPES = frozenset(
+ {*_AGGREGATE_FUNCTIONS, *_DISTINCT_AGGREGATE_FUNCTIONS, "percentile"}
+)
+
+_FUNCTION_TO_TYPE = {
+ "SUM": "sum",
+ "MIN": "min",
+ "MAX": "max",
+ "AVG": "average",
+ "AVERAGE": "average",
+ "MEDIAN": "median",
+ "COUNT": "count",
+}
+_DISTINCT_FUNCTION_TO_TYPE = {
+ "COUNT": "count_distinct",
+ "SUM": "sum_distinct",
+ "AVG": "average_distinct",
+ "AVERAGE": "average_distinct",
+}
+
+_CALL_RE = re.compile(r"^\s*(?P<func>[A-Za-z_]+)\s*\((?P<body>.*)\)\s*$", re.DOTALL)
+_DISTINCT_RE = re.compile(r"^DISTINCT\s+(?P<inner>.+)$", re.IGNORECASE | re.DOTALL)
+_PERCENTILE_RE = re.compile(
+ r"^\s*PERCENTILE_CONT\s*\(\s*(?P<fraction>[0-9]*\.?[0-9]+)\s*\)\s*"
+ r"WITHIN\s+GROUP\s*\(\s*ORDER\s+BY\s+(?P<inner>.+?)\s*\)\s*$",
+ re.IGNORECASE | re.DOTALL,
+)
+
+
+@dataclass(frozen=True)
+class ParsedAggregation:
+ lightdash_type: str
+ inner: str
+ percentile: Optional[float] = None
+
+
+def _is_single_call_body(body: str) -> bool:
+ """True when the parentheses in ``body`` are balanced, i.e. the outer
+ parentheses around it belong to one function call."""
+ depth = 0
+ for char in body:
+ if char == "(":
+ depth += 1
+ elif char == ")":
+ depth -= 1
+ if depth < 0:
+ return False
+ return depth == 0
+
+
+def parse_aggregation(expression: str) -> Optional[ParsedAggregation]:
+ """Recognise an expression that is exactly one aggregation over an operand.
+
+ ``SUM(x)``, ``COUNT(DISTINCT x)``, ``SUM(DISTINCT x)`` and
+ ``PERCENTILE_CONT(p) WITHIN GROUP (ORDER BY x)`` map onto Lightdash's typed
+ metrics; the operand may be a bare column reference or any expression.
+ Returns None for anything else (``COUNT(*)``, arithmetic between
+ aggregations, unknown functions).
+ """
+ percentile_match = _PERCENTILE_RE.match(expression)
+ if percentile_match:
+ percentile = float(percentile_match.group("fraction")) * 100
+ if percentile.is_integer():
+ percentile = int(percentile)
+ return ParsedAggregation(
+ "percentile", percentile_match.group("inner").strip(), percentile
+ )
+
+ match = _CALL_RE.match(expression)
+ if match is None or not _is_single_call_body(match.group("body")):
+ return None
+ function = match.group("func").upper()
+ inner = match.group("body").strip()
+ distinct = _DISTINCT_RE.match(inner)
+ if distinct:
+ lightdash_type = _DISTINCT_FUNCTION_TO_TYPE.get(function)
+ inner = distinct.group("inner").strip()
+ else:
+ lightdash_type = _FUNCTION_TO_TYPE.get(function)
+ if lightdash_type is None or not inner or inner == "*":
+ return None
+ return ParsedAggregation(lightdash_type, inner)
+
+
+def build_aggregation(
+ lightdash_type: str, inner: str, percentile: Optional[float] = None
+) -> Optional[str]:
+ """Build the Ossie expression for a typed Lightdash metric over ``inner``.
+
+ Returns None for metric types that are not aggregations (``number``,
+ ``string``, ``boolean``, ...).
+ """
+ if lightdash_type == "percentile":
+ fraction = (50 if percentile is None else percentile) / 100
+ return f"PERCENTILE_CONT({fraction:g}) WITHIN GROUP (ORDER BY {inner})"
+ if lightdash_type in _DISTINCT_AGGREGATE_FUNCTIONS:
+ return f"{_DISTINCT_AGGREGATE_FUNCTIONS[lightdash_type]}(DISTINCT {inner})"
+ function = _AGGREGATE_FUNCTIONS.get(lightdash_type)
+ if function is None:
+ return None
+ return f"{function}({inner})"
+
+
+def is_column_reference(expression: str) -> bool:
+ """True for a bare ``column`` or ``qualifier.column`` reference."""
+ return _COLUMN_REFERENCE_RE.match(expression) is not None
+
+
+def strip_qualifier(column_ref: str) -> str:
+ """Return the bare column name of a possibly ``qualifier.column`` reference."""
+ return column_ref.rsplit(".", 1)[-1]
+
+
+def qualifier_of(column_ref: str) -> Optional[str]:
+ """Return the qualifier of a ``qualifier.column`` reference, if present."""
+ if "." in column_ref:
+ return column_ref.rsplit(".", 1)[0]
+ return None
+
+
+def ossie_sql_to_lightdash(
+ expression: str, dataset: str, references: Optional[Dict[str, str]] = None
+) -> str:
+ """Rewrite Ossie column references into Lightdash references.
+
+ ``dataset.column`` becomes ``${TABLE}.column``; a dataset in ``references``
+ (joined from ``dataset``) becomes ``${<join reference>.column}``, where the
+ join reference is the joined model's name or its alias.
+ """
+ names = {dataset: None, **(references or {})}
+
+ def replace(match: "re.Match[str]") -> str:
+ name, column = match.group(1), match.group(2)
+ reference = names[name]
+ if reference is None:
+ return f"${{TABLE}}.{column}"
+ return f"${{{reference}.{column}}}"
+
+ alternatives = "|".join(re.escape(name) for name in names)
+ return re.sub(rf"\b({alternatives})\.(\w+)", replace, expression)
+
+
+def has_non_portable_reference(sql: str) -> bool:
+ """True when the SQL depends on Lightdash query-time evaluation: project
+ parameters, user attributes, or Liquid templating."""
+ return (
+ _NON_PORTABLE_REFERENCE_RE.search(sql) is not None
+ or _NON_PORTABLE_SYNTAX_RE.search(sql) is not None
+ )
+
+
+@dataclass
+class RewriteResult:
+ expression: str
+ inlined_metrics: List[str] = field(default_factory=list)
+ flattened_aliases: List[str] = field(default_factory=list)
+
+
+def lightdash_sql_to_ossie(
+ sql: str,
+ dataset: str,
+ *,
+ aliases: Optional[Dict[str, str]] = None,
+ resolve_metric: Optional[Callable[[str], Optional[str]]] = None,
+) -> RewriteResult:
+ """Rewrite Lightdash references into Ossie ``dataset.column`` references.
+
+ ``${TABLE}.column`` and bare ``${column}`` refer to the current model;
+ ``${other_table.column}`` refers to a joined model and becomes a
+ cross-dataset reference; ``${alias.column}`` is flattened onto the joined
+ model; ``${metric}`` is replaced by that metric's expression when
+ ``resolve_metric`` knows it. Parameter and user-attribute references are
+ left untouched: callers check ``has_non_portable_reference`` first.
+ """
+ alias_map = aliases or {}
+ result = RewriteResult(expression="")
+
+ def replace(match: "re.Match[str]") -> str:
+ reference = match.group(1)
+ if _NON_PORTABLE_REFERENCE_RE.match(match.group(0)):
+ return match.group(0)
+ if reference == "TABLE":
+ return dataset
+ if reference.startswith("TABLE."):
+ return f"{dataset}.{reference[len('TABLE.'):]}"
+ if "." in reference:
+ table, column = reference.split(".", 1)
+ if table in alias_map:
+ result.flattened_aliases.append(table)
+ return f"{alias_map[table]}.{column}"
+ return reference
+ if resolve_metric is not None:
+ resolved = resolve_metric(reference)
+ if resolved is not None:
+ result.inlined_metrics.append(reference)
+ return f"({resolved})"
+ return f"{dataset}.{reference}"
+
+ result.expression = _REFERENCE_RE.sub(replace, sql)
+ return result
+
+
+_STRING_LITERAL_RE = re.compile(r"('(?:[^']|'')*')")
+
+
+def qualify_bare_columns(sql: str, dataset: str, columns) -> str:
Review Comment:
Fixed by replacing the regex with SQLGlot column detection, preserving types, date parts, and qualified references.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add lightdash converter [ossie]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #360:
URL: https://github.com/apache/ossie/pull/360#discussion_r4220931745
##########
converters/lightdash/src/ossie_lightdash/ossie_to_lightdash.py:
##########
@@ -0,0 +1,613 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Convert an Ossie document into Lightdash semantic definitions.
+
+``convert`` produces a dbt ``schema.yml``-shaped dictionary whose ``meta``
+blocks carry Lightdash dimensions, metrics and joins, ready to be merged into
+a dbt project that Lightdash reads. ``convert_models`` produces Lightdash's
+own dbt-free model files (``type: model``, ``sql_from``, typed dimensions),
+which ``lightdash deploy`` reads directly. Lightdash-specific presentation attributes that
+have no Ossie vocabulary round-trip through ``custom_extensions`` entries with
+``vendor_name: LIGHTDASH``; their keys are overlaid onto the generated
+definitions and win for presentation attributes, while structural keys
+(``sql``/``label`` on dimensions, ``sql``/``description`` on metrics,
+``join``/``sql_on`` on joins) are protected so they can never override the
+Ossie-derived definition.
+"""
+
+import json
+from typing import Any, Dict, List, Optional, Set, Tuple
+
+from ossie import (
+ OssieDataset,
+ OssieDialect,
+ OssieDocument,
+ OssieExpression,
+ OssieMetric,
+ OssieSemanticModel,
+)
+
+from ossie_lightdash.converter_issues import (
+ ConverterIssue,
+ ConverterIssueType,
+ ConverterResult,
+)
+from ossie_lightdash.datatype_utils import datatype_to_lightdash_type, is_temporal
+from ossie_lightdash.expression_utils import (
+ is_column_reference,
+ ossie_sql_to_lightdash,
+ parse_aggregation,
+ qualifier_of,
+ referenced_datasets,
+ strip_qualifier,
+)
+
+LIGHTDASH_VENDOR_NAME = "LIGHTDASH"
+
+# Structural keys are owned by Ossie vocabulary (the import direction never puts
+# them into the extension); dropping them here keeps a hand-authored extension
+# from overriding the Ossie-derived definition. ``type`` stays overridable on
+# metrics whose expression is not a recognised aggregation: it is the channel
+# for types an expression cannot express (``boolean``, ``string``, ...).
+_PROTECTED_DIMENSION_KEYS = {"sql", "label", "ai_hint", "hidden", "column_meta"}
+_PROTECTED_METRIC_KEYS = {"sql", "description", "ai_hint", "name", "model"}
+_PROTECTED_AGGREGATION_KEYS = _PROTECTED_METRIC_KEYS | {"type", "percentile"}
+_PROTECTED_JOIN_KEYS = {"join", "sql_on", "alias"}
+# Dataset-level stash keys that map to Ossie vocabulary or are handled apart.
+_PROTECTED_MODEL_KEYS = {"joins", "metrics", "primary_key", "ai_hint"}
+# Order of the top-level keys in a Lightdash model file; anything else
+# (stashed model meta) follows in source order.
+_MODEL_KEY_ORDER = ("type", "name", "label", "description", "sql_from", "primary_key", "ai_hint")
+_DIMENSION_KEY_ORDER = ("name", "type", "label", "description", "sql", "hidden")
+
+# Joins a dataset declares, keyed by the joined dataset: the name Lightdash SQL
+# uses to reference it (the joined model, or its alias). The first join to a
+# dataset wins when it is joined more than once.
+JoinReferences = Dict[str, str]
+
+
+def _lightdash_extension_data(element: Any, issues: List[ConverterIssue]) -> Dict[str, Any]:
+ """Return the ``lightdash`` vendor extension data of an Ossie element, if any."""
+ data: Dict[str, Any] = {}
+ for extension in element.custom_extensions or []:
+ # The registered token is LIGHTDASH; documents written before the
+ # registration used the lowercase name.
+ if extension.vendor_name.upper() == LIGHTDASH_VENDOR_NAME:
+ try:
+ data.update(json.loads(extension.data))
+ except (TypeError, ValueError):
+ issues.append(
+ ConverterIssue(
+ issue_type=ConverterIssueType.EXTENSION_DATA_INVALID,
+ element_name=getattr(element, "name", "<unnamed>"),
+ )
+ )
+ else:
+ issues.append(
+ ConverterIssue(
+ issue_type=ConverterIssueType.FOREIGN_EXTENSION_IGNORED,
+ element_name=getattr(element, "name", "<unnamed>"),
+ )
+ )
+ return data
+
+
+def _ai_hint(ai_context: Any) -> Any:
+ """Ossie `ai_context` as a Lightdash `ai_hint`: one line stays a string, a
+ multi-line instruction becomes a list, and synonyms and examples of the
+ structured form are rendered as extra hints."""
+ if ai_context is None:
+ return None
+ if isinstance(ai_context, str):
+ hints = ai_context.split("\n")
+ else:
+ hints = (ai_context.instructions or "").split("\n")
+ if ai_context.synonyms:
+ hints.append("Also known as: " + ", ".join(ai_context.synonyms))
+ if ai_context.examples:
+ hints.append("Example questions: " + "; ".join(ai_context.examples))
+ hints = [hint for hint in hints if hint]
+ if not hints:
+ return None
+ return hints[0] if len(hints) == 1 else hints
+
+
+def _ordered(entries: Dict[str, Any], order: Tuple[str, ...]) -> Dict[str, Any]:
+ """Return ``entries`` with the keys in ``order`` first, the rest as they came."""
+ return {
+ **{key: entries[key] for key in order if key in entries},
+ **{key: value for key, value in entries.items() if key not in order},
+ }
+
+
+def _to_lightdash_model(
+ dataset: OssieDataset, model: Dict[str, Any], issues: List[ConverterIssue]
+) -> Dict[str, Any]:
+ """Reshape a dbt-flavoured model into a Lightdash model file."""
+ meta = dict(model.get("meta") or {})
+ out: Dict[str, Any] = {"type": "model", "name": model["name"]}
+ if model.get("description"):
+ out["description"] = model["description"]
+ out["sql_from"] = dataset.source
+ joins = meta.pop("joins", None)
+ model_metrics = meta.pop("metrics", None)
+ out.update(meta)
+ if joins:
+ out["joins"] = joins
+ if model_metrics:
+ out["metrics"] = model_metrics
+
+ dimensions: List[Dict[str, Any]] = []
+ for column in model.get("columns") or []:
+ column_meta = column.get("meta") or {}
+ dimension = dict(column_meta.get("dimension") or {})
+ dimension["name"] = column["name"]
+ if column.get("description") and "description" not in dimension:
+ dimension["description"] = column["description"]
+ if not dimension.get("type"):
+ issues.append(
+ ConverterIssue(
+ issue_type=ConverterIssueType.DIMENSION_TYPE_DEFAULTED,
+ element_name=f"{model['name']}.{column['name']}",
+ )
+ )
+ dimension["type"] = "string"
+ if not dimension.get("sql"):
+ dimension["sql"] = f"${{TABLE}}.{column['name']}"
+ if column_meta.get("metrics"):
+ dimension["metrics"] = column_meta["metrics"]
+ if column_meta.get("additional_dimensions"):
+ dimension["additional_dimensions"] = column_meta["additional_dimensions"]
+ if any(key not in ("dimension", "metrics", "additional_dimensions") for key in column_meta):
+ issues.append(
+ ConverterIssue(
+ issue_type=ConverterIssueType.COLUMN_META_NOT_REPRESENTABLE,
+ element_name=f"{model['name']}.{column['name']}",
+ )
+ )
+ dimensions.append(_ordered(dimension, _DIMENSION_KEY_ORDER))
+ out["dimensions"] = dimensions
+ return _ordered(out, _MODEL_KEY_ORDER)
+
+
+def _nest_meta_under_config(node: Dict[str, Any]) -> None:
+ """Move a node's ``meta`` under ``config`` (the dbt 1.10+ placement)."""
+ meta = node.pop("meta", None)
+ if meta:
+ node["config"] = {"meta": meta}
+
+
+def _model_name_for(dataset: OssieDataset) -> str:
Review Comment:
`_model_name_for` takes everything after the last `.` in `dataset.source`, which only works for plain `a.b.c` names:
- A quoted source like `proj.ds.tbl` gives the name `tbl`, and a subquery source gives garbage. The CLI uses the model name directly as the file name (`models/{model['name']}.yml`), so these can produce invalid file names.
- Two datasets with the same table name in different schemas (`sales.orders`, `archive.orders`) get the same model name and the same output file. The second silently overwrites the first.
- Join targets and `sql_on` references use the same names, so they would point at the wrong model.
Could we strip quoting, fall back to the dataset `name` when the source is not a simple identifier path, and detect collisions? Failing with a clear error, or disambiguating the name, would be better than overwriting silently. A test with two same-named tables in different schemas would pin this down.
##########
converters/lightdash/src/ossie_lightdash/expression_utils.py:
##########
@@ -0,0 +1,292 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Expression helpers shared by both conversion directions.
+
+Lightdash SQL snippets reference the current model as ``${TABLE}.column``,
+sibling fields as ``${column}`` or ``${metric}``, joined models as
+``${other_table.column}`` and project parameters or user attributes as
+``${lightdash.…}`` / ``${ld.…}``. Ossie expressions reference columns as
+``dataset.column``. These helpers translate between the two spellings and
+recognise the aggregation shapes that map onto Lightdash's typed metrics.
+"""
+
+import re
+from dataclasses import dataclass, field
+from typing import Callable, Dict, List, Optional
+
+_REFERENCE_RE = re.compile(r"\$\{([^}]+)\}")
+_NON_PORTABLE_REFERENCE_RE = re.compile(r"\$\{(?:lightdash|ld)\.[^}]*\}")
+# Liquid tags and bare `ld.parameters.x` / `ld.query.filters` / `ld.user.x`
+# references inside them are evaluated by Lightdash at query time.
+_NON_PORTABLE_SYNTAX_RE = re.compile(
+ r"\{%|\{\{|(?<![\w.])(?:lightdash|ld)\.(?:parameters|query|user)\."
+)
+_COLUMN_REFERENCE_RE = re.compile(r"^[A-Za-z_]\w*(?:\.\w+)?$")
+
+# Lightdash metric types that are encoded as an aggregation in the Ossie
+# expression. Anything else is exported as a `number` metric with raw SQL.
+_AGGREGATE_FUNCTIONS = {
+ "sum": "SUM",
+ "min": "MIN",
+ "max": "MAX",
+ "average": "AVG",
+ "median": "MEDIAN",
+ "count": "COUNT",
+}
+_DISTINCT_AGGREGATE_FUNCTIONS = {
+ "count_distinct": "COUNT",
+ "sum_distinct": "SUM",
+ "average_distinct": "AVG",
+}
+AGGREGATE_TYPES = frozenset(
+ {*_AGGREGATE_FUNCTIONS, *_DISTINCT_AGGREGATE_FUNCTIONS, "percentile"}
+)
+
+_FUNCTION_TO_TYPE = {
+ "SUM": "sum",
+ "MIN": "min",
+ "MAX": "max",
+ "AVG": "average",
+ "AVERAGE": "average",
+ "MEDIAN": "median",
+ "COUNT": "count",
+}
+_DISTINCT_FUNCTION_TO_TYPE = {
+ "COUNT": "count_distinct",
+ "SUM": "sum_distinct",
+ "AVG": "average_distinct",
+ "AVERAGE": "average_distinct",
+}
+
+_CALL_RE = re.compile(r"^\s*(?P<func>[A-Za-z_]+)\s*\((?P<body>.*)\)\s*$", re.DOTALL)
+_DISTINCT_RE = re.compile(r"^DISTINCT\s+(?P<inner>.+)$", re.IGNORECASE | re.DOTALL)
+_PERCENTILE_RE = re.compile(
+ r"^\s*PERCENTILE_CONT\s*\(\s*(?P<fraction>[0-9]*\.?[0-9]+)\s*\)\s*"
+ r"WITHIN\s+GROUP\s*\(\s*ORDER\s+BY\s+(?P<inner>.+?)\s*\)\s*$",
+ re.IGNORECASE | re.DOTALL,
+)
+
+
+@dataclass(frozen=True)
+class ParsedAggregation:
+ lightdash_type: str
+ inner: str
+ percentile: Optional[float] = None
+
+
+def _is_single_call_body(body: str) -> bool:
+ """True when the parentheses in ``body`` are balanced, i.e. the outer
+ parentheses around it belong to one function call."""
+ depth = 0
+ for char in body:
+ if char == "(":
+ depth += 1
+ elif char == ")":
+ depth -= 1
+ if depth < 0:
+ return False
+ return depth == 0
+
+
+def parse_aggregation(expression: str) -> Optional[ParsedAggregation]:
+ """Recognise an expression that is exactly one aggregation over an operand.
+
+ ``SUM(x)``, ``COUNT(DISTINCT x)``, ``SUM(DISTINCT x)`` and
+ ``PERCENTILE_CONT(p) WITHIN GROUP (ORDER BY x)`` map onto Lightdash's typed
+ metrics; the operand may be a bare column reference or any expression.
+ Returns None for anything else (``COUNT(*)``, arithmetic between
+ aggregations, unknown functions).
+ """
+ percentile_match = _PERCENTILE_RE.match(expression)
+ if percentile_match:
+ percentile = float(percentile_match.group("fraction")) * 100
+ if percentile.is_integer():
+ percentile = int(percentile)
+ return ParsedAggregation(
+ "percentile", percentile_match.group("inner").strip(), percentile
+ )
+
+ match = _CALL_RE.match(expression)
+ if match is None or not _is_single_call_body(match.group("body")):
+ return None
+ function = match.group("func").upper()
+ inner = match.group("body").strip()
+ distinct = _DISTINCT_RE.match(inner)
+ if distinct:
+ lightdash_type = _DISTINCT_FUNCTION_TO_TYPE.get(function)
+ inner = distinct.group("inner").strip()
+ else:
+ lightdash_type = _FUNCTION_TO_TYPE.get(function)
+ if lightdash_type is None or not inner or inner == "*":
+ return None
+ return ParsedAggregation(lightdash_type, inner)
+
+
+def build_aggregation(
+ lightdash_type: str, inner: str, percentile: Optional[float] = None
+) -> Optional[str]:
+ """Build the Ossie expression for a typed Lightdash metric over ``inner``.
+
+ Returns None for metric types that are not aggregations (``number``,
+ ``string``, ``boolean``, ...).
+ """
+ if lightdash_type == "percentile":
+ fraction = (50 if percentile is None else percentile) / 100
+ return f"PERCENTILE_CONT({fraction:g}) WITHIN GROUP (ORDER BY {inner})"
+ if lightdash_type in _DISTINCT_AGGREGATE_FUNCTIONS:
+ return f"{_DISTINCT_AGGREGATE_FUNCTIONS[lightdash_type]}(DISTINCT {inner})"
+ function = _AGGREGATE_FUNCTIONS.get(lightdash_type)
+ if function is None:
+ return None
+ return f"{function}({inner})"
+
+
+def is_column_reference(expression: str) -> bool:
+ """True for a bare ``column`` or ``qualifier.column`` reference."""
+ return _COLUMN_REFERENCE_RE.match(expression) is not None
+
+
+def strip_qualifier(column_ref: str) -> str:
+ """Return the bare column name of a possibly ``qualifier.column`` reference."""
+ return column_ref.rsplit(".", 1)[-1]
+
+
+def qualifier_of(column_ref: str) -> Optional[str]:
+ """Return the qualifier of a ``qualifier.column`` reference, if present."""
+ if "." in column_ref:
+ return column_ref.rsplit(".", 1)[0]
+ return None
+
+
+def ossie_sql_to_lightdash(
+ expression: str, dataset: str, references: Optional[Dict[str, str]] = None
+) -> str:
+ """Rewrite Ossie column references into Lightdash references.
+
+ ``dataset.column`` becomes ``${TABLE}.column``; a dataset in ``references``
+ (joined from ``dataset``) becomes ``${<join reference>.column}``, where the
+ join reference is the joined model's name or its alias.
+ """
+ names = {dataset: None, **(references or {})}
+
+ def replace(match: "re.Match[str]") -> str:
+ name, column = match.group(1), match.group(2)
+ reference = names[name]
+ if reference is None:
+ return f"${{TABLE}}.{column}"
+ return f"${{{reference}.{column}}}"
+
+ alternatives = "|".join(re.escape(name) for name in names)
+ return re.sub(rf"\b({alternatives})\.(\w+)", replace, expression)
+
+
+def has_non_portable_reference(sql: str) -> bool:
+ """True when the SQL depends on Lightdash query-time evaluation: project
+ parameters, user attributes, or Liquid templating."""
+ return (
+ _NON_PORTABLE_REFERENCE_RE.search(sql) is not None
+ or _NON_PORTABLE_SYNTAX_RE.search(sql) is not None
+ )
+
+
+@dataclass
+class RewriteResult:
+ expression: str
+ inlined_metrics: List[str] = field(default_factory=list)
+ flattened_aliases: List[str] = field(default_factory=list)
+
+
+def lightdash_sql_to_ossie(
+ sql: str,
+ dataset: str,
+ *,
+ aliases: Optional[Dict[str, str]] = None,
+ resolve_metric: Optional[Callable[[str], Optional[str]]] = None,
+) -> RewriteResult:
+ """Rewrite Lightdash references into Ossie ``dataset.column`` references.
+
+ ``${TABLE}.column`` and bare ``${column}`` refer to the current model;
+ ``${other_table.column}`` refers to a joined model and becomes a
+ cross-dataset reference; ``${alias.column}`` is flattened onto the joined
+ model; ``${metric}`` is replaced by that metric's expression when
+ ``resolve_metric`` knows it. Parameter and user-attribute references are
+ left untouched: callers check ``has_non_portable_reference`` first.
+ """
+ alias_map = aliases or {}
+ result = RewriteResult(expression="")
+
+ def replace(match: "re.Match[str]") -> str:
+ reference = match.group(1)
+ if _NON_PORTABLE_REFERENCE_RE.match(match.group(0)):
+ return match.group(0)
+ if reference == "TABLE":
+ return dataset
+ if reference.startswith("TABLE."):
+ return f"{dataset}.{reference[len('TABLE.'):]}"
+ if "." in reference:
+ table, column = reference.split(".", 1)
+ if table in alias_map:
+ result.flattened_aliases.append(table)
+ return f"{alias_map[table]}.{column}"
+ return reference
+ if resolve_metric is not None:
+ resolved = resolve_metric(reference)
+ if resolved is not None:
+ result.inlined_metrics.append(reference)
+ return f"({resolved})"
+ return f"{dataset}.{reference}"
+
+ result.expression = _REFERENCE_RE.sub(replace, sql)
+ return result
+
+
+_STRING_LITERAL_RE = re.compile(r"('(?:[^']|'')*')")
+
+
+def qualify_bare_columns(sql: str, dataset: str, columns) -> str:
Review Comment:
Any identifier that matches a column name gets prefixed with the dataset, including SQL keywords and type names outside string literals. With a column named `date` or `year`:
- `CAST(x AS date)` becomes `CAST(x AS ds.date)`
- `EXTRACT(year FROM d)` becomes `EXTRACT(ds.year FROM d)`
The regex skips identifiers preceded by `.` but not ones followed by `.`. So `other.col` becomes `ds.other.col` if the model has a column named `other`. The result is invalid SQL in the imported Ossie expression, and columns like `date` and `year` are common enough that I would expect this to be hit in practice.
Could we also skip identifiers followed by `.` and the positions where a type or keyword is expected (after `AS` in a cast, the first argument of `EXTRACT`/`DATE_TRUNC`)? A tokenizer-based approach would be more robust than a regex. Tests with columns named `date`, `year` and `other` would cover these cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add lightdash converter [ossie]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #360:
URL: https://github.com/apache/ossie/pull/360#issuecomment-6315535531
I will close this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add lightdash converter [ossie]
Posted by "christianeu-db (via GitHub)" <gi...@apache.org>.
christianeu-db closed pull request #360: feat: add lightdash converter
URL: https://github.com/apache/ossie/pull/360
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add lightdash converter [ossie]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #360:
URL: https://github.com/apache/ossie/pull/360#issuecomment-6315529606
@hamzahc1 can you please move this PR to the new Ossie-converters repo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add lightdash converter [ossie]
Posted by "owlas (via GitHub)" <gi...@apache.org>.
owlas commented on PR #360:
URL: https://github.com/apache/ossie/pull/360#issuecomment-6315664520
@jbonofre - i'm on it now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add lightdash converter [ossie]
Posted by "Spissable (via GitHub)" <gi...@apache.org>.
Spissable commented on code in PR #360:
URL: https://github.com/apache/ossie/pull/360#discussion_r4311590174
##########
converters/lightdash/src/ossie_lightdash/ossie_to_lightdash.py:
##########
@@ -0,0 +1,613 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Convert an Ossie document into Lightdash semantic definitions.
+
+``convert`` produces a dbt ``schema.yml``-shaped dictionary whose ``meta``
+blocks carry Lightdash dimensions, metrics and joins, ready to be merged into
+a dbt project that Lightdash reads. ``convert_models`` produces Lightdash's
+own dbt-free model files (``type: model``, ``sql_from``, typed dimensions),
+which ``lightdash deploy`` reads directly. Lightdash-specific presentation attributes that
+have no Ossie vocabulary round-trip through ``custom_extensions`` entries with
+``vendor_name: LIGHTDASH``; their keys are overlaid onto the generated
+definitions and win for presentation attributes, while structural keys
+(``sql``/``label`` on dimensions, ``sql``/``description`` on metrics,
+``join``/``sql_on`` on joins) are protected so they can never override the
+Ossie-derived definition.
+"""
+
+import json
+from typing import Any, Dict, List, Optional, Set, Tuple
+
+from ossie import (
+ OssieDataset,
+ OssieDialect,
+ OssieDocument,
+ OssieExpression,
+ OssieMetric,
+ OssieSemanticModel,
+)
+
+from ossie_lightdash.converter_issues import (
+ ConverterIssue,
+ ConverterIssueType,
+ ConverterResult,
+)
+from ossie_lightdash.datatype_utils import datatype_to_lightdash_type, is_temporal
+from ossie_lightdash.expression_utils import (
+ is_column_reference,
+ ossie_sql_to_lightdash,
+ parse_aggregation,
+ qualifier_of,
+ referenced_datasets,
+ strip_qualifier,
+)
+
+LIGHTDASH_VENDOR_NAME = "LIGHTDASH"
+
+# Structural keys are owned by Ossie vocabulary (the import direction never puts
+# them into the extension); dropping them here keeps a hand-authored extension
+# from overriding the Ossie-derived definition. ``type`` stays overridable on
+# metrics whose expression is not a recognised aggregation: it is the channel
+# for types an expression cannot express (``boolean``, ``string``, ...).
+_PROTECTED_DIMENSION_KEYS = {"sql", "label", "ai_hint", "hidden", "column_meta"}
+_PROTECTED_METRIC_KEYS = {"sql", "description", "ai_hint", "name", "model"}
+_PROTECTED_AGGREGATION_KEYS = _PROTECTED_METRIC_KEYS | {"type", "percentile"}
+_PROTECTED_JOIN_KEYS = {"join", "sql_on", "alias"}
+# Dataset-level stash keys that map to Ossie vocabulary or are handled apart.
+_PROTECTED_MODEL_KEYS = {"joins", "metrics", "primary_key", "ai_hint"}
+# Order of the top-level keys in a Lightdash model file; anything else
+# (stashed model meta) follows in source order.
+_MODEL_KEY_ORDER = ("type", "name", "label", "description", "sql_from", "primary_key", "ai_hint")
+_DIMENSION_KEY_ORDER = ("name", "type", "label", "description", "sql", "hidden")
+
+# Joins a dataset declares, keyed by the joined dataset: the name Lightdash SQL
+# uses to reference it (the joined model, or its alias). The first join to a
+# dataset wins when it is joined more than once.
+JoinReferences = Dict[str, str]
+
+
+def _lightdash_extension_data(element: Any, issues: List[ConverterIssue]) -> Dict[str, Any]:
+ """Return the ``lightdash`` vendor extension data of an Ossie element, if any."""
+ data: Dict[str, Any] = {}
+ for extension in element.custom_extensions or []:
+ # The registered token is LIGHTDASH; documents written before the
+ # registration used the lowercase name.
+ if extension.vendor_name.upper() == LIGHTDASH_VENDOR_NAME:
+ try:
+ data.update(json.loads(extension.data))
+ except (TypeError, ValueError):
+ issues.append(
+ ConverterIssue(
+ issue_type=ConverterIssueType.EXTENSION_DATA_INVALID,
+ element_name=getattr(element, "name", "<unnamed>"),
+ )
+ )
+ else:
+ issues.append(
+ ConverterIssue(
+ issue_type=ConverterIssueType.FOREIGN_EXTENSION_IGNORED,
+ element_name=getattr(element, "name", "<unnamed>"),
+ )
+ )
+ return data
+
+
+def _ai_hint(ai_context: Any) -> Any:
+ """Ossie `ai_context` as a Lightdash `ai_hint`: one line stays a string, a
+ multi-line instruction becomes a list, and synonyms and examples of the
+ structured form are rendered as extra hints."""
+ if ai_context is None:
+ return None
+ if isinstance(ai_context, str):
+ hints = ai_context.split("\n")
+ else:
+ hints = (ai_context.instructions or "").split("\n")
+ if ai_context.synonyms:
+ hints.append("Also known as: " + ", ".join(ai_context.synonyms))
+ if ai_context.examples:
+ hints.append("Example questions: " + "; ".join(ai_context.examples))
+ hints = [hint for hint in hints if hint]
+ if not hints:
+ return None
+ return hints[0] if len(hints) == 1 else hints
+
+
+def _ordered(entries: Dict[str, Any], order: Tuple[str, ...]) -> Dict[str, Any]:
+ """Return ``entries`` with the keys in ``order`` first, the rest as they came."""
+ return {
+ **{key: entries[key] for key in order if key in entries},
+ **{key: value for key, value in entries.items() if key not in order},
+ }
+
+
+def _to_lightdash_model(
+ dataset: OssieDataset, model: Dict[str, Any], issues: List[ConverterIssue]
+) -> Dict[str, Any]:
+ """Reshape a dbt-flavoured model into a Lightdash model file."""
+ meta = dict(model.get("meta") or {})
+ out: Dict[str, Any] = {"type": "model", "name": model["name"]}
+ if model.get("description"):
+ out["description"] = model["description"]
+ out["sql_from"] = dataset.source
+ joins = meta.pop("joins", None)
+ model_metrics = meta.pop("metrics", None)
+ out.update(meta)
+ if joins:
+ out["joins"] = joins
+ if model_metrics:
+ out["metrics"] = model_metrics
+
+ dimensions: List[Dict[str, Any]] = []
+ for column in model.get("columns") or []:
+ column_meta = column.get("meta") or {}
+ dimension = dict(column_meta.get("dimension") or {})
+ dimension["name"] = column["name"]
+ if column.get("description") and "description" not in dimension:
+ dimension["description"] = column["description"]
+ if not dimension.get("type"):
+ issues.append(
+ ConverterIssue(
+ issue_type=ConverterIssueType.DIMENSION_TYPE_DEFAULTED,
+ element_name=f"{model['name']}.{column['name']}",
+ )
+ )
+ dimension["type"] = "string"
+ if not dimension.get("sql"):
+ dimension["sql"] = f"${{TABLE}}.{column['name']}"
+ if column_meta.get("metrics"):
+ dimension["metrics"] = column_meta["metrics"]
+ if column_meta.get("additional_dimensions"):
+ dimension["additional_dimensions"] = column_meta["additional_dimensions"]
+ if any(key not in ("dimension", "metrics", "additional_dimensions") for key in column_meta):
+ issues.append(
+ ConverterIssue(
+ issue_type=ConverterIssueType.COLUMN_META_NOT_REPRESENTABLE,
+ element_name=f"{model['name']}.{column['name']}",
+ )
+ )
+ dimensions.append(_ordered(dimension, _DIMENSION_KEY_ORDER))
+ out["dimensions"] = dimensions
+ return _ordered(out, _MODEL_KEY_ORDER)
+
+
+def _nest_meta_under_config(node: Dict[str, Any]) -> None:
+ """Move a node's ``meta`` under ``config`` (the dbt 1.10+ placement)."""
+ meta = node.pop("meta", None)
+ if meta:
+ node["config"] = {"meta": meta}
+
+
+def _model_name_for(dataset: OssieDataset) -> str:
Review Comment:
Thanks, fixed.
Simple identifier paths now have quoting stripped, and subqueries or other source expressions fall back to the logical dataset name. Model names and joins use the same validated mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org