You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ossie.apache.org by "Haoranli503 (via GitHub)" <gi...@apache.org> on 2026/08/19 17:18:57 UTC
[PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Haoranli503 opened a new pull request, #333:
URL: https://github.com/apache/ossie/pull/333
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## Summary
Adds a **Java implementation** of the Databricks converter alongside the existing Python
reference, and restructures `converters/databricks/` to host both languages. The Java library is
the implementation maintained going forward; the Python converter becomes a reference copy slated
for deprecation.
Conversion is unchanged in spirit: pure YAML text in, YAML text out, reading and writing the two
formats as parsed maps and lists. The Java and Python implementations target the same Databricks
Unity Catalog Metric View YAML v1.1 contract, and the checked-in fixtures pin the expected output
for both directions (`export`: Ossie -> Metric View, `import`: Metric View -> Ossie).
Changes:
- **Restructure** — the existing flat Python converter (`src/`, `tests/`, `pyproject.toml`,
`README.md`, `uv.lock`) moves under `converters/databricks/python/` with no content changes,
making room for the Java tree.
- **Add `java/`** — the bidirectional converter as a Maven module: library, an
`OssieDatabricksConverter` command-line tool, JUnit 5 tests, and YAML fixtures. Package
`org.apache.ossie.converter.databricks`, following the conventions of the existing `polaris` and
`salesforce` Java converters (parent `org.apache:apache:39`; deps SnakeYAML / Jackson / JUnit 5;
`apache-rat-plugin` license audit; shade plugin for the executable CLI jar).
- **Add a root `README.md`** describing the two-language layout and pointing at each
implementation's own README.
- **CI** (`.github/workflows/converter-databricks-ci.yml`) — point the Python job at the new
`python/` working directory, and add a Java job (`mvn -B verify`, Temurin JDK 21) mirroring
`converter-polaris-ci.yml`.
## Related Issues
<!-- Link any related GitHub issues: Closes #123, Fixes #456 -->
## Checklist
### Specification
- [ ] Spec changes are included in `core-spec/` and follow the existing structure
- [ ] Spec changes have been discussed on the mailing list or in a linked issue
- [ ] Breaking changes to the spec are clearly called out in the summary
### Ontology
- [ ] Ontology changes in `ontology/` are consistent with spec changes
- [ ] New or modified terms are defined and documented
### Converters
- [ ] Converter logic in `converters/` is updated to reflect spec or ontology changes
- [x] New converters include tests under the converter's test directory
### Validation
- [ ] Validation rules in `validation/` are updated if the spec changed
- [ ] New validation cases are covered by tests
### Documentation
- [ ] `docs/` is updated to reflect any user-facing changes
- [ ] New features or behaviors are documented with examples where appropriate
- [ ] `CONTRIBUTING.md` is updated if the contribution process changed
### Examples
- [ ] `examples/` are added or updated for any new spec constructs or converter support
### Tests
- [x] All existing tests pass (`pytest` / CI green)
- [x] New functionality is covered by tests
### Compliance
- [x] ASF license headers are present on all new source files
- [ ] No third-party dependencies are added without PMC/IPMC approval
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on PR #333:
URL: https://github.com/apache/ossie/pull/333#issuecomment-5544406572
Hi @jbonofre , all fixes are in. Can you help tal? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on code in PR #333:
URL: https://github.com/apache/ossie/pull/333#discussion_r3983191920
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieToMetricView.java:
##########
@@ -0,0 +1,1167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAX_JOIN_NODES;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SYNONYM_LIMIT;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.findOutsideLiterals;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.foreignVendorExtensions;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.mergeDescription;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.pickExpression;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.readStash;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.replaceOutsideLiterals;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.require;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.synonymsOf;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * EXPORT direction: Apache Ossie semantic model -> Databricks Metric View v1.1 YAML. This is the
+ * harder direction: it reassembles a relationship graph into a join tree. Shared helpers come from
+ * {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class OssieToMetricView {
+
+ private OssieToMetricView() {}
+
+ // -- tree node ------------------------------------------------------------
+ private static final class Node {
+ final String dataset;
+ final Map<String, Object> rel; // empty for the fact/root
+ final boolean parentIsFrom;
+ String alias;
+ boolean isOtm;
+ final List<Node> children = new ArrayList<>();
+
+ Node(String dataset, Map<String, Object> rel, boolean parentIsFrom) {
+ this.dataset = dataset;
+ this.rel = rel;
+ this.parentIsFrom = parentIsFrom;
+ }
+ }
+
+ // -- public entry ---------------------------------------------------------
+ static Result convertOssieToMetricView(String osiYamlStr, String source) {
+ Notices notices = new Notices();
+ if (osiYamlStr == null || osiYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(osiYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> root = asMap(parsed);
+ Object versionValue = get(root, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!OSSIE_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Apache Ossie version '" + version + "'. Supported: " + OSSIE_VERSION,
+ version);
+ }
+ List<Object> models = schemaList(root, "semantic_model", "Apache Ossie YAML");
+ if (models.isEmpty()) {
+ throw ConversionException.invalidInput(
+ "'semantic_model' must be a non-empty list",
+ "its 'semantic_model' must be a non-empty list");
+ }
+ if (!(models.get(0) instanceof Map)) {
+ String reason = "Apache Ossie YAML: 'semantic_model[0]' must be a mapping";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ if (models.size() > 1) {
+ notices.warn("model", "multiple semantic models found; converting only the first");
+ }
+ Map<String, Object> view = convertModel((Map<String, Object>) models.get(0), source, notices);
+ try {
+ return new Result(MAPPER.writeValueAsString(view), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Metric View YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertModel(
+ Map<String, Object> model, String explicitSource, Notices notices) {
+ String name = model.containsKey("name") ? str(get(model, "name")) : "<unnamed>";
+ String modelScope = "Model '" + name + "'";
+ List<Map<String, Object>> datasetList = schemaMapList(model, "datasets", modelScope);
+ if (datasetList.isEmpty()) {
+ throw new ConversionException("Model '" + name + "' has no datasets");
+ }
+ Set<String> seen = new HashSet<>();
+ Map<String, Map<String, Object>> datasets = new LinkedHashMap<>();
+ Map<String, List<Map<String, Object>>> datasetFields = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasetList) {
+ String dsName = requireStr(d, "name", "Model '" + name + "': dataset");
+ if (!seen.add(dsName.trim().toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("Model '" + name + "': duplicate dataset name '" + dsName + "'");
+ }
+ datasets.put(dsName, d);
+ datasetFields.put(dsName, schemaMapList(d, "fields", "Dataset '" + dsName + "'"));
+ }
+ List<Map<String, Object>> relationships = schemaMapList(model, "relationships", modelScope);
+ List<Map<String, Object>> metrics = schemaMapList(model, "metrics", modelScope);
+
+ Map<String, Object> modelStash = readStash(model);
+ // A non-equi/filtered join has no schema-valid Ossie relationship (from/to columns are
+ // required), so MetricViewToOssie stashes it under the model's DATABRICKS custom_extensions
+ // (complex_joins) rather than emitting a stub relationship. Rebuild a columns-less
+ // relationship for each -- carrying its raw `on` (and rely/cardinality) in the stash that
+ // buildJoin restores from -- and merge them in so the join tree includes them.
+ for (Map<String, Object> pj : schemaMapList(modelStash, "complex_joins", modelScope)) {
+ Map<String, Object> rel = new LinkedHashMap<>();
+ rel.put("name", get(pj, "name"));
+ rel.put("from", get(pj, "from"));
+ rel.put("to", get(pj, "to"));
+ Map<String, Object> stash = new LinkedHashMap<>(pj);
+ stash.remove("name");
+ stash.remove("from");
+ stash.remove("to");
+ writeStash(rel, stash);
+ relationships.add(rel);
+ }
+ String factHint = explicitSource != null ? explicitSource : str(get(modelStash, STASH_SOURCE_KEY));
+ Object[] built = buildJoinTree(name, datasets, relationships, factHint, notices);
+ Node root = (Node) built[0];
+ String fact = (String) built[1];
+ Map<String, Integer> counts = assignAliases(root, fact);
+ markOtm(name, root);
+
+ Map<String, Object> factDs = datasets.get(fact);
+ Map<String, Object> view = new LinkedHashMap<>();
+ view.put("version", MV_VERSION);
+ view.put("source", validateSource(get(factDs, "source"), fact));
+
+ String comment = str(get(model, "description"));
+ if (truthy(comment)) {
+ view.put("comment", comment);
+ }
+ if (modelStash.containsKey("filter")) {
+ view.put("filter", modelStash.get("filter"));
+ }
+
+ List<Object> joins = new ArrayList<>();
+ for (Node child : root.children) {
+ joins.add(buildJoin(child, "source", datasets));
+ }
+ if (!joins.isEmpty()) {
+ view.put("joins", joins);
+ }
+
+ Set<String> droppedDims = new HashSet<>();
+ Set<String> droppedMeasures = new HashSet<>();
+ List<Map<String, Object>> dimensions = new ArrayList<>();
+ Set<String> seenDims = new HashSet<>();
+ // Dataset name -> the alias path that addresses its columns from the primary source, collected
+ // from the same walk the dimensions use so measures qualify identically (see qualifyMeasure).
+ Map<String, String> datasetAliasPath = new LinkedHashMap<>();
+ for (Object[] entry : nodeOrder(root)) {
+ Node node = (Node) entry[0];
+ @SuppressWarnings("unchecked")
+ List<String> joinPath = (List<String>) entry[1];
+ boolean isFact = node == root;
+ String prefix = counts.get(node.dataset) > 1 ? node.alias : null;
+ // Fixed per node, like `prefix`: hoisted so it is not rebuilt for every field.
+ String qualifier = String.join(".", joinPath);
+ if (!isFact) {
+ // A dataset reachable by more than one path (diamond) appears once per path; the first
+ // wins, matching the order dimensions are emitted in.
+ datasetAliasPath.putIfAbsent(node.dataset, qualifier);
+ }
+ for (Map<String, Object> field : datasetFields.get(node.dataset)) {
+ String fname = requireStr(field, "name", "dataset '" + node.dataset + "': field");
+ if (node.isOtm) {
+ notices.warn("field '" + fname + "'",
+ "column on a one-to-many-joined table cannot be a dimension "
+ + "(must resolve to one value per source row); dropped");
+ droppedDims.add(fname);
+ continue;
+ }
+ Map<String, Object> dim =
+ convertField(field, fname, qualifier, isFact, prefix, notices);
+ if (dim == null) {
+ droppedDims.add(fname);
+ continue;
+ }
+ String dn = (String) dim.get("name");
+ if (!seenDims.add(dn.toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("dataset '" + node.dataset + "': dimension name '" + dn
+ + "' collides with another dimension/measure; Metric Views require unique "
+ + "dimension/measure names -- rename before use");
+ }
+ dimensions.add(dim);
+ }
+ }
+
+ List<Map<String, Object>> measures = new ArrayList<>();
+ // Depends only on the fact name, so compile it once rather than per metric.
+ Pattern factQualifier = Pattern.compile("\\b" + Pattern.quote(fact) + "\\.");
+ // Likewise fixed once the join tree is walked: one alternation over the joined dataset names.
+ Pattern datasetHead = qualifierChainPattern(datasetAliasPath.keySet());
+ for (Map<String, Object> metric : metrics) {
+ Map<String, Object> measure =
+ convertMetric(metric, factQualifier, datasetHead, datasetAliasPath, seenDims, notices);
+ if (measure == null) {
+ droppedMeasures.add(str(get(metric, "name")));
+ continue;
+ }
+ measures.add(measure);
+ }
+
+ cascadeDrop(dimensions, measures, droppedDims, droppedMeasures, notices);
+
+ // A Metric View must define at least one dimension or measure
+ // (SingleSourceMetricView.validate rejects an empty `select`), so a view with neither is one
+ // the engine refuses at CREATE. Fail here instead, naming the dropped columns: after
+ // cascadeDrop the emptiness is usually a consequence of earlier drops rather than an empty
+ // input, and those names are the actionable part.
+ if (dimensions.isEmpty() && measures.isEmpty()) {
+ StringBuilder msg = new StringBuilder("Model '" + name
+ + "' produced no dimensions or measures; a Metric View requires at least one.");
+ if (!droppedDims.isEmpty() || !droppedMeasures.isEmpty()) {
+ msg.append(" Dropped during conversion:");
+ if (!droppedDims.isEmpty()) {
+ msg.append(" dimensions ").append(sortedNames(droppedDims));
+ }
+ if (!droppedMeasures.isEmpty()) {
+ msg.append(" measures ").append(sortedNames(droppedMeasures));
+ }
+ msg.append(" -- see the warnings for why each was dropped.");
+ }
+ throw new ConversionException(msg.toString());
+ }
+
+ if (!dimensions.isEmpty()) {
+ view.put("dimensions", dimensions);
+ }
+ if (!measures.isEmpty()) {
+ view.put("measures", measures);
+ }
+ if (modelStash.containsKey("parameters")) {
+ view.put("parameters", modelStash.get("parameters"));
+ }
+ if (modelStash.containsKey("materialization")) {
+ view.put("materialization", modelStash.get("materialization"));
+ }
+
+ warnDroppedModel(model, notices);
+ return view;
+ }
+
+ private static List<Map<String, Object>> schemaMapList(
+ Map<String, Object> parent, String key, String scope) {
+ List<Object> values = schemaList(parent, key, scope);
+ List<Map<String, Object>> result = new ArrayList<>();
+ int index = 0;
+ for (Object element : values) {
+ if (!(element instanceof Map)) {
+ String reason = scope + ": '" + key + "[" + index + "]' must be a mapping";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ result.add((Map<String, Object>) element);
+ index++;
+ }
+ return result;
+ }
+
+ private static List<Object> schemaList(
+ Map<String, Object> parent, String key, String scope) {
+ if (!parent.containsKey(key)) {
+ return new ArrayList<>();
+ }
+ Object value = get(parent, key);
+ // A present-but-null value (the bare `key:` YAML idiom) is treated like an absent key, matching
+ // the tolerant base reader (asList) and the sibling customExtensions reader. A scalar or map
+ // value is still rejected below. Required collections stay guarded by their own emptiness
+ // checks, so tolerating null here does not let an empty semantic_model/datasets through.
+ if (value == null) {
+ return new ArrayList<>();
+ }
+ if (!(value instanceof List)) {
+ String reason = scope + ": '" + key + "' must be a list";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ return (List<Object>) value;
+ }
+
+ private static Object[] buildJoinTree(
+ String modelName, Map<String, Map<String, Object>> datasets,
+ List<Map<String, Object>> relationships0, String factHint, Notices notices) {
+ for (Map<String, Object> rel : relationships0) {
+ String scope = "Model '" + modelName + "': relationship '"
+ + (rel.containsKey("name") ? str(get(rel, "name")) : "<unnamed>") + "'";
+ Object f = require(rel, "from", scope);
+ Object t = require(rel, "to", scope);
+ if (!datasets.containsKey(str(f)) || !datasets.containsKey(str(t))) {
+ throw new ConversionException("Model '" + modelName + "': relationship '"
+ + str(get(rel, "name")) + "' references an unknown dataset");
+ }
+ }
+ List<Map<String, Object>> relationships = new ArrayList<>();
+ for (Map<String, Object> rel : relationships0) {
+ relationships.add(orientByKey(rel, datasets, notices));
+ }
+ if (datasets.size() > MAX_JOIN_NODES) {
+ throw new ConversionException("Model '" + modelName + "' has " + datasets.size()
+ + " datasets; at most " + MAX_JOIN_NODES + " are supported.");
+ }
+ String fact = pickFact(modelName, datasets, relationships, factHint);
+ rejectDirectedCycle(modelName, datasets, relationships);
+
+ Map<String, List<String>> adj = new HashMap<>();
+ for (String n : datasets.keySet()) {
+ adj.put(n, new ArrayList<>());
+ }
+ for (Map<String, Object> rel : relationships) {
+ adj.get(str(get(rel, "from"))).add(str(get(rel, "to")));
+ adj.get(str(get(rel, "to"))).add(str(get(rel, "from")));
+ }
+ Map<String, Integer> dist = new HashMap<>();
+ dist.put(fact, 0);
+ Deque<String> queue = new ArrayDeque<>();
+ queue.add(fact);
+ while (!queue.isEmpty()) {
+ String cur = queue.poll();
+ for (String nb : adj.get(cur)) {
+ if (!dist.containsKey(nb)) {
+ dist.put(nb, dist.get(cur) + 1);
+ queue.add(nb);
+ }
+ }
+ }
+ List<String> unreachable = new ArrayList<>();
+ for (String n : datasets.keySet()) {
+ if (!dist.containsKey(n)) {
+ unreachable.add(n);
+ }
+ }
+ if (!unreachable.isEmpty()) {
+ java.util.Collections.sort(unreachable);
+ throw new ConversionException("Model '" + modelName + "': datasets " + unreachable
+ + " are not reachable from fact '" + fact + "' via relationships.");
+ }
+ Map<String, List<Object[]>> childrenOf = new HashMap<>();
+ for (String n : datasets.keySet()) {
+ childrenOf.put(n, new ArrayList<>());
+ }
+ for (Map<String, Object> rel : relationships) {
+ String a = str(get(rel, "from"));
+ String b = str(get(rel, "to"));
+ if (dist.get(a).equals(dist.get(b))) {
+ throw new ConversionException("Model '" + modelName + "': relationship '"
+ + str(get(rel, "name")) + "' joins two datasets equidistant from the fact; "
+ + "the graph is not tree-shaped (it contains a cycle).");
+ }
+ String parent = dist.get(a) < dist.get(b) ? a : b;
+ String child = dist.get(a) < dist.get(b) ? b : a;
+ childrenOf.get(parent).add(new Object[] {child, rel, parent.equals(str(get(rel, "from")))});
+ }
+ int[] counter = {0};
+ Node root = build(modelName, fact, new LinkedHashMap<>(), false, childrenOf, counter);
+ return new Object[] {root, fact};
+ }
+
+ private static Node build(String modelName, String dataset, Map<String, Object> rel,
+ boolean parentIsFrom, Map<String, List<Object[]>> childrenOf, int[] counter) {
+ counter[0]++;
+ if (counter[0] > MAX_JOIN_NODES) {
+ throw new ConversionException("Model '" + modelName + "': join graph fans out to more than "
+ + MAX_JOIN_NODES + " joins; check for an unintended diamond explosion.");
+ }
+ Node node = new Node(dataset, rel, parentIsFrom);
+ for (Object[] c : childrenOf.get(dataset)) {
+ @SuppressWarnings("unchecked")
+ Map<String, Object> crel = (Map<String, Object>) c[1];
+ node.children.add(build(modelName, (String) c[0], crel, (Boolean) c[2], childrenOf, counter));
+ }
+ return node;
+ }
+
+ private static Map<String, Integer> assignAliases(Node root, String fact) {
+ Map<String, Integer> counts = new HashMap<>();
+ countNode(root, counts);
+ Set<String> used = new HashSet<>();
+ used.add("source");
+ assign(root, null, fact, counts, used);
+ return counts;
+ }
+
+ private static void countNode(Node node, Map<String, Integer> counts) {
+ counts.merge(node.dataset, 1, Integer::sum);
+ for (Node c : node.children) {
+ countNode(c, counts);
+ }
+ }
+
+ private static void assign(Node node, String parentAlias, String fact,
+ Map<String, Integer> counts, Set<String> used) {
+ String alias;
+ if (node.dataset.equals(fact)) {
+ alias = "source";
+ } else {
+ String base;
+ if (counts.get(node.dataset) == 1) {
+ base = node.dataset;
+ } else if (parentAlias != null && !parentAlias.equals("source")) {
+ base = parentAlias + "_" + node.dataset;
+ } else {
+ base = node.dataset;
+ }
+ alias = base;
+ int n = 2;
+ while (used.contains(alias)) {
+ alias = base + "_" + n;
+ n++;
+ }
+ }
+ node.alias = alias;
+ used.add(alias);
+ for (Node c : node.children) {
+ assign(c, alias, fact, counts, used);
+ }
+ }
+
+ private static String pickFact(String modelName, Map<String, Map<String, Object>> datasets,
+ List<Map<String, Object>> relationships, String factHint) {
+ if (factHint != null) {
+ if (!datasets.containsKey(factHint)) {
+ throw new ConversionException(
+ "Model '" + modelName + "': requested source '" + factHint + "' is not a dataset");
+ }
+ return factHint;
+ }
+ if (datasets.size() > 1 && relationships.isEmpty()) {
+ throw new ConversionException("Model '" + modelName + "': " + datasets.size()
+ + " datasets but no relationships; cannot determine the fact table.");
+ }
+ Map<String, Integer> incoming = new LinkedHashMap<>();
+ for (String n : datasets.keySet()) {
+ incoming.put(n, 0);
+ }
+ for (Map<String, Object> rel : relationships) {
+ incoming.merge(str(get(rel, "to")), 1, Integer::sum);
+ }
+ List<String> roots = new ArrayList<>();
+ for (Map.Entry<String, Integer> e : incoming.entrySet()) {
+ if (e.getValue() == 0) {
+ roots.add(e.getKey());
+ }
+ }
+ if (roots.isEmpty()) {
+ throw new ConversionException("Model '" + modelName + "': join graph contains a cycle "
+ + "(no root dataset). A Metric View requires an acyclic, tree-shaped graph.");
+ }
+ if (roots.size() > 1) {
+ java.util.Collections.sort(roots);
+ throw new ConversionException("Model '" + modelName + "': multiple candidate fact datasets "
+ + roots + ". Name the grain with --source.");
+ }
+ return roots.get(0);
+ }
+
+ /**
+ * Marks each joined node with its branch's cardinality and rejects a branch that mixes the two.
+ *
+ * <p>A Metric View requires every join within one top-level branch to share a single cardinality:
+ * `Join.validateSubJoinCardinalities` seeds the expected value from the top-level join and fails
+ * any descendant that differs (an absent `cardinality` reads as `many_to_one`). So a mixed branch
+ * is rejected in *either* direction -- a many-to-one nested under one-to-many, and equally a
+ * one-to-many nested under many-to-one. Emitting one would produce a view the engine refuses at
+ * CREATE, so reject it here with a converter-level error instead.
+ */
+ private static void markOtm(String modelName, Node root) {
+ // Each top-level join starts a branch and sets that branch's expected cardinality.
+ for (Node top : root.children) {
+ boolean branchIsOtm = !top.parentIsFrom;
+ top.isOtm = branchIsOtm;
+ markOtmVisit(modelName, top, branchIsOtm);
+ }
+ }
+
+ private static void markOtmVisit(String modelName, Node node, boolean branchIsOtm) {
Review Comment:
This one is intentional. The Java matches the Databricks engine's actual rule. A Metric View validates each top level join branch with Join.validateSubJoinCardinalities, seeded from that branch's top level join cardinality, and fails any descendant whose cardinality differs in either direction (Join.scala, invoked per top level join by SingleSourceMetricView.validateJoins). So your example, many_to_one to customer then one_to_many to customer_notes, is actually rejected by the engine at CREATE: Expected 'customer_notes' to have 'many_to_one' cardinality, but found 'one_to_many'. The Python _mark_otm is the one out of step: it only rejects reverting from one_to_many back to many_to_one, so it would emit that view and hit the engine error at CREATE instead. Its leniency does not actually help, it just moves the same rejection from conversion time to CREATE time. The Python copy is deprecated so we left it as is.
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieConverterCommon.java:
##########
@@ -0,0 +1,673 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.regex.MatchResult;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import com.fasterxml.jackson.dataformat.yaml.YAMLGenerator;
+
+import org.yaml.snakeyaml.DumperOptions;
+import org.yaml.snakeyaml.LoaderOptions;
+import org.yaml.snakeyaml.Yaml;
+import org.yaml.snakeyaml.constructor.SafeConstructor;
+import org.yaml.snakeyaml.nodes.Tag;
+import org.yaml.snakeyaml.representer.Representer;
+import org.yaml.snakeyaml.resolver.Resolver;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+
+/**
+ * Shared constants, YAML I/O, and typed accessors for the Apache Ossie <-> Databricks
+ * Metric View converter. The direction-specific logic lives in {@link OssieToMetricView}
+ * (export) and {@link MetricViewToOssie} (import), which static-import these members. The public
+ * entry points and shared types are re-exported through {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class OssieConverterCommon {
+
+ // -- constants -------------------------------------------------------------
+ static final String OSSIE_VERSION = "0.2.0.dev0";
+ static final String MV_VERSION = "1.1";
+ static final String VENDOR = "DATABRICKS";
+ static final String DIALECT_DATABRICKS = "DATABRICKS";
+ static final String DIALECT_ANSI = "ANSI_SQL";
+ static final int SYNONYM_LIMIT = 10;
+ static final int STASH_VERSION = 1;
+ static final String STASH_SOURCE_KEY = "source_dataset";
+ static final String CARD_ONE_TO_MANY = "one_to_many";
+ static final String CARD_MANY_TO_ONE = "many_to_one";
+ static final int MAX_JOIN_NODES = 200;
+
+ static final Pattern IDENTIFIER_RE = Pattern.compile("^[A-Za-z_][A-Za-z0-9_]*$");
+ static final Pattern SELECT_WITH_RE =
+ Pattern.compile("(?i)^(select|with)\\b");
+
+ static final ObjectMapper MAPPER = buildMapper();
+ private static final ObjectMapper JSON_READER = new ObjectMapper()
+ .enable(JsonParser.Feature.STRICT_DUPLICATE_DETECTION)
+ .enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS);
+ // Writer for the custom_extensions stash blob. The exact byte format is pinned by the
+ // checked-in fixtures: a space after ':' and ', ' between entries, as in
+ // {"_v": 1, "filter": "x"}.
+ static final com.fasterxml.jackson.databind.ObjectWriter JSON_WRITER = buildJsonWriter();
+
+ /** A MinimalPrettyPrinter (no newlines) using the stash blob's separators: ": " / ", ". */
+ private static final class JsonDumpsPrinter
+ extends com.fasterxml.jackson.core.util.MinimalPrettyPrinter {
+ @Override
+ public void writeObjectFieldValueSeparator(com.fasterxml.jackson.core.JsonGenerator g)
+ throws java.io.IOException {
+ g.writeRaw(": ");
+ }
+ @Override
+ public void writeObjectEntrySeparator(com.fasterxml.jackson.core.JsonGenerator g)
+ throws java.io.IOException {
+ g.writeRaw(", ");
+ }
+ @Override
+ public void writeArrayValueSeparator(com.fasterxml.jackson.core.JsonGenerator g)
+ throws java.io.IOException {
+ g.writeRaw(", ");
+ }
+ }
+
+ private static com.fasterxml.jackson.databind.ObjectWriter buildJsonWriter() {
+ // ESCAPE_NON_ASCII keeps the blob pure ASCII (every non-ASCII
+ // char is emitted as a \\uXXXX escape). Set on the JsonFactory so it takes effect on the
+ // generator. Jackson emits the hex in UPPERCASE; writeStash lowercases it so the blob is
+ // byte-identical to the checked-in fixtures.
+ com.fasterxml.jackson.core.JsonFactory jf = new com.fasterxml.jackson.core.JsonFactory();
+ jf.enable(com.fasterxml.jackson.core.JsonGenerator.Feature.ESCAPE_NON_ASCII);
+ return new ObjectMapper(jf).writer(new JsonDumpsPrinter());
+ }
+
+ // Matches a real unicode escape in serialized JSON so writeStash can lowercase its hex digits.
+ //
+ // The escape must be preceded by an EVEN number of backslashes, otherwise the `u` belongs to an
+ // escaped backslash rather than to an escape sequence: a stashed value holding a literal
+ // backslash followed by "uABCD" serializes with a doubled backslash, where the `u` is ordinary
+ // text and must be left alone (lowercasing it there would corrupt the value). Group 1 captures
+ // the (possibly empty) run of escaped backslashes so it can be re-emitted verbatim; group 2 is
+ // the hex to lowercase.
+ private static final Pattern UNICODE_ESCAPE_RE =
+ Pattern.compile("(?<!\\\\)((?:\\\\\\\\)*)\\\\u([0-9A-Fa-f]{4})");
+
+ private static ObjectMapper buildMapper() {
+ // Quote scalars (do NOT minimize quotes) so a string like "1.1" or a bool-like token
+ // is emitted quoted and never re-parsed as a number/boolean. Used for the WRITE path only;
+ // reads go through the YAML-1.2 SnakeYAML loader below.
+ YAMLFactory yf = YAMLFactory.builder()
+ .disable(YAMLGenerator.Feature.WRITE_DOC_START_MARKER)
+ .build();
+ return new ObjectMapper(yf);
+ }
+
+ // A SnakeYAML Resolver with YAML 1.2 boolean semantics: only true/false are booleans, NOT the
+ // YAML 1.1 tokens on/off/yes/no/y/n. A metric view join's bare `on:` key, and any "on"/"off"
+ // string value, must read back as a string rather than silently becoming a boolean (a YAML 1.1
+ // reader would coerce it).
+ private static final class Yaml12Resolver extends Resolver {
+ @Override
+ protected void addImplicitResolvers() {
+ addImplicitResolver(Tag.MERGE, MERGE, "<");
+ addImplicitResolver(Tag.NULL, NULL, "~nN ");
+ addImplicitResolver(Tag.NULL, EMPTY, null);
+ addImplicitResolver(Tag.BOOL, java.util.regex.Pattern.compile(
+ "^(?:true|True|TRUE|false|False|FALSE)$"), "tTfF");
+ addImplicitResolver(Tag.INT, INT, "-+0123456789");
+ addImplicitResolver(Tag.FLOAT, FLOAT, "-+0123456789.");
+ addImplicitResolver(Tag.TIMESTAMP, TIMESTAMP, "0123456789");
+ }
+ }
+
+ // SnakeYAML readers are stateful; create one per conversion instead of retaining thread-local
+ // state on caller threads. Yaml also copies settings from loaderOptions into its constructor, so
+ // the constructor and reader must receive the same configured instance.
+ private static Yaml newYamlReader() {
+ LoaderOptions loaderOptions = new LoaderOptions();
+ loaderOptions.setAllowDuplicateKeys(false);
+ DumperOptions dumperOptions = new DumperOptions();
+ return new Yaml(new SafeConstructor(loaderOptions), new Representer(dumperOptions),
+ dumperOptions, loaderOptions, new Yaml12Resolver());
+ }
+
+ static Object loadYaml(String s) {
+ return newYamlReader().load(s);
+ }
+
+ private OssieConverterCommon() {}
+
+ // -- typed accessors over parsed YAML (Object) ----------------------------
+ @SuppressWarnings("unchecked")
+ static Map<String, Object> asMap(Object x) {
+ if (x instanceof Map) {
+ return (Map<String, Object>) x;
+ }
+ return new LinkedHashMap<>();
+ }
+
+ @SuppressWarnings("unchecked")
+ static List<Object> asList(Object x) {
+ if (x instanceof List) {
+ return (List<Object>) x;
+ }
+ return new ArrayList<>();
+ }
+
+ static Object get(Map<String, Object> m, String k) {
+ return m.get(k);
+ }
+
+ static String str(Object x) {
+ if (x == null) {
+ return null;
+ }
+ return x.toString();
+ }
+
+ static List<String> strList(Object x) {
+ List<String> out = new ArrayList<>();
+ for (Object o : asList(x)) {
+ if (o != null) {
+ out.add(o.toString());
+ }
+ }
+ return out;
+ }
+
+ // -- helpers ---------------------------------------------------------------
+ static boolean isSimpleIdentifier(Object expr) {
+ return expr instanceof String && IDENTIFIER_RE.matcher(((String) expr).trim()).matches();
+ }
+
+ /**
+ * Applies {@code pattern -> replacement} to {@code sql}, but only to the parts of the expression
+ * that are actual SQL code -- spans inside string literals ({@code '...'}, {@code "..."}),
+ * backquoted identifiers, {@code -- line} comments, and {@code /* block *}{@code /} comments are
+ * copied through untouched.
+ *
+ * <p>Measure expressions are rewritten to add or strip a fact qualifier, and a blind
+ * {@code replaceAll} over the raw text also rewrites any occurrence inside a literal: a measure
+ * such as {@code SUM(IF(source.region = 'source.us', amt, 0))} would silently become
+ * {@code ... = 'us'}, changing the predicate and therefore the measure's value. Only code spans
+ * may be rewritten.
+ *
+ * <p>{@code replacement} is treated as a literal string, not as a regex replacement template.
+ */
+ static String replaceOutsideLiterals(String sql, Pattern pattern, String replacement) {
+ return replaceOutsideLiterals(sql, pattern, m -> replacement);
+ }
+
+ /**
+ * As {@link #replaceOutsideLiterals(String, Pattern, String)}, but each match is replaced with
+ * {@code replacer.apply(match)} -- for a rewrite whose result depends on what matched. The
+ * returned text is inserted literally, not as a regex replacement template.
+ */
+ static String replaceOutsideLiterals(
+ String sql, Pattern pattern, Function<MatchResult, String> replacer) {
+ StringBuilder out = new StringBuilder(sql.length());
+ int codeStart = 0;
+ for (int[] span : literalSpans(sql)) {
+ // Rewrite the code span that precedes this literal/comment, then copy the span verbatim.
+ out.append(rewriteLiterally(sql.substring(codeStart, span[0]), pattern, replacer));
+ out.append(sql, span[0], span[1]);
+ codeStart = span[1];
+ }
+ out.append(rewriteLiterally(sql.substring(codeStart), pattern, replacer));
+ return out.toString();
+ }
+
+ /**
+ * True when {@code pattern} matches the SQL code of {@code sql} -- the find-only counterpart of
+ * {@link #replaceOutsideLiterals}, with the same string-literal/comment blindness.
+ *
+ * <p>Needed wherever a decision is made from an expression's text: a bare
+ * {@code pattern.matcher(sql).find()} also sees inside literals, so a column name that only
+ * occurs in a string (as in {@code SUM(IF(region = 'us', amt, 0))}) reads as a reference to a
+ * column named {@code us}.
+ */
+ static boolean findOutsideLiterals(String sql, Pattern pattern) {
+ List<int[]> spans = literalSpans(sql);
+ if (spans.isEmpty()) {
+ return pattern.matcher(sql).find();
+ }
+ // Blank the literal/comment spans rather than removing them, so the surrounding code keeps its
+ // token boundaries (the pattern may rely on \b or a look-around).
+ StringBuilder code = new StringBuilder(sql);
+ for (int[] span : spans) {
+ for (int i = span[0]; i < span[1]; i++) {
+ code.setCharAt(i, ' ');
+ }
+ }
+ return pattern.matcher(code).find();
+ }
+
+ /**
+ * Start (inclusive) and end (exclusive) offsets of every span in {@code sql} that is not SQL
+ * code: string literals ({@code '...'}, {@code "..."}), backquoted identifiers, {@code -- line}
+ * comments, and {@code /* block *}{@code /} comments.
+ */
+ private static List<int[]> literalSpans(String sql) {
+ List<int[]> spans = new ArrayList<>();
+ int i = 0;
+ while (i < sql.length()) {
+ char c = sql.charAt(i);
+ int skipTo = -1;
+ if (c == '\'' || c == '"' || c == '`') {
+ skipTo = endOfQuoted(sql, i, c);
+ } else if (c == '-' && i + 1 < sql.length() && sql.charAt(i + 1) == '-') {
+ int nl = sql.indexOf('\n', i);
+ skipTo = nl < 0 ? sql.length() : nl;
+ } else if (c == '/' && i + 1 < sql.length() && sql.charAt(i + 1) == '*') {
+ int end = sql.indexOf("*/", i + 2);
+ skipTo = end < 0 ? sql.length() : end + 2;
+ }
+ if (skipTo < 0) {
+ i++;
+ continue;
+ }
+ spans.add(new int[] {i, skipTo});
+ i = skipTo;
+ }
+ return spans;
+ }
+
+ /** Index just past the quoted span starting at {@code start}; handles doubled-quote escapes. */
+ private static int endOfQuoted(String sql, int start, char quote) {
+ int i = start + 1;
+ while (i < sql.length()) {
+ char c = sql.charAt(i);
+ if (c == '\\' && quote != '`' && i + 1 < sql.length()) {
+ i += 2;
+ continue;
+ }
+ if (c == quote) {
+ // A doubled quote is an escaped quote, not the end of the span.
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i += 2;
+ continue;
+ }
+ return i + 1;
+ }
+ i++;
+ }
+ // Unterminated literal: treat the remainder as part of the span rather than rewriting it.
+ return sql.length();
+ }
+
+ private static String rewriteLiterally(
+ String code, Pattern pattern, Function<MatchResult, String> replacer) {
+ return pattern.matcher(code)
+ .replaceAll(m -> Matcher.quoteReplacement(replacer.apply(m)));
+ }
+
+ /**
+ * A pattern matching a maximal dotted run of {@code names} that qualifies a column reference --
+ * {@code parent.child.} -- with group 1 holding the run without its trailing dot. Null when
+ * {@code names} is empty.
+ *
+ * <p>The whole run is matched at once, rather than one name at a time, so a rewrite driven by
+ * this pattern can never re-match a qualifier it has just produced: rewriting
+ * {@code customer.region.} name-by-name would see the {@code region.} inside its own output and
+ * qualify it a second time. The required trailing dot keeps a column that merely shares a name
+ * with a dataset ({@code SUM(customer.region)}) from being read as a qualifier.
+ */
+ static Pattern qualifierChainPattern(Collection<String> names) {
+ if (names.isEmpty()) {
+ return null;
+ }
+ // Longest name first: regex alternation is ordered, so at a given position the longest name
+ // wins (`nation_x` must never match as `nation`).
+ List<String> sorted = new ArrayList<>(names);
+ sorted.sort((a, b) -> b.length() - a.length());
+ StringBuilder alternation = new StringBuilder();
+ for (String name : sorted) {
+ if (alternation.length() > 0) {
+ alternation.append('|');
+ }
+ alternation.append(Pattern.quote(name));
+ }
+ String one = "(?:" + alternation + ")";
+ return Pattern.compile("\\b(" + one + "(?:\\." + one + ")*)\\.");
+ }
+
+ /**
+ * Rewrites each qualifier run {@code chain} matches by mapping the run's LAST name through
+ * {@code resolve} -- the name closest to the column is the one being addressed, as
+ * {@code resolveColumn} treats a dimension's alias path. A run whose leaf {@code resolve} does
+ * not know is left alone. Literals and comments are skipped, and the pass is single, so the
+ * result is independent of the order the names happen to be in.
+ */
+ static String rewriteQualifiers(String expr, Pattern chain, Function<String, String> resolve) {
+ if (chain == null) {
+ return expr;
+ }
+ return replaceOutsideLiterals(expr, chain, m -> {
+ String leaf = lastIdentifier(m.group(1));
+ String resolved = leaf == null ? null : resolve.apply(leaf);
+ return resolved == null ? m.group() : resolved + ".";
+ });
+ }
+
+ // Presence is tested by key, so a legitimately falsy non-string value (0, false) is returned;
+ // a missing key, a null, or an empty/whitespace-only string is rejected.
+ static Object require(Map<String, Object> obj, String key, String what) {
+ if (!obj.containsKey(key) || obj.get(key) == null) {
+ throw new ConversionException(what + " is missing required '" + key + "'");
+ }
+ Object value = obj.get(key);
+ if (value instanceof String && ((String) value).trim().isEmpty()) {
+ throw new ConversionException(what + " has an empty '" + key + "'");
+ }
+ return value;
+ }
+
+ // Like require(), but the value must be a string.
+ static String requireStr(Map<String, Object> obj, String key, String what) {
+ Object v = require(obj, key, what);
+ if (v instanceof String) {
+ return (String) v;
+ }
+ throw new ConversionException(
+ what + ": '" + key + "' must be a string, got " + v.getClass().getSimpleName());
+ }
+
+ static String validateSource(Object source, String datasetName) {
+ String s = source == null ? "" : source.toString().trim();
+ if (s.isEmpty()) {
+ throw new ConversionException("Dataset '" + datasetName + "': missing/empty 'source'");
+ }
+ if (SELECT_WITH_RE.matcher(s).find()) {
+ return s;
+ }
+ String[] parts = s.split("\\.", -1);
+ boolean ok = parts.length == 3;
+ if (ok) {
+ for (String p : parts) {
+ if (p.isEmpty() || containsWhitespace(p)) {
+ ok = false;
+ break;
+ }
+ }
+ }
+ if (ok) {
+ return s;
+ }
+ throw new ConversionException("Dataset '" + datasetName + "': source '" + source
+ + "' must be a 3-part catalog.schema.table identifier or a SELECT/WITH subquery");
+ }
+
+ private static boolean containsWhitespace(String p) {
+ for (int i = 0; i < p.length(); i++) {
+ if (Character.isWhitespace(p.charAt(i))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ static String pickExpression(Object osiExpression, String scope) {
+ // Keep the raw (possibly non-string) values so
+ // the type check below can fire; select DATABRICKS-or-ANSI by truthiness (`or`), so a
+ // null/empty DATABRICKS expr falls through to ANSI; and raise on a non-string chosen
+ // value rather than silently coercing it.
+ if (osiExpression == null) {
+ return null;
+ }
+ if (!(osiExpression instanceof Map)) {
+ String reason = scope + ": 'expression' must be a mapping";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ Map<String, Object> expression = asMap(osiExpression);
+ Object dialectValues = get(expression, "dialects");
+ if (dialectValues == null) {
+ return null;
+ }
+ if (!(dialectValues instanceof List)) {
+ String reason = scope + ": 'expression.dialects' must be a list";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ Map<String, Object> dialects = new LinkedHashMap<>();
+ int index = 0;
+ for (Object d : (List<Object>) dialectValues) {
+ if (!(d instanceof Map)) {
+ String reason = scope + ": 'expression.dialects[" + index + "]' must be a mapping";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ Map<String, Object> dm = (Map<String, Object>) d;
+ dialects.put(str(get(dm, "dialect")), get(dm, "expression"));
+ index++;
+ }
+ Object chosen = truthy(dialects.get(DIALECT_DATABRICKS))
+ ? dialects.get(DIALECT_DATABRICKS) : dialects.get(DIALECT_ANSI);
+ if (chosen != null && !(chosen instanceof String)) {
+ throw new ConversionException(
+ "expression must be a string, got " + chosen.getClass().getSimpleName());
+ }
+ return (String) chosen; // null when neither dialect present -> caller warns and skips
+ }
+
+ /**
+ * Emptiness test used throughout the converter for optional YAML values: null, the empty string,
+ * an empty list/map, numeric zero, and boolean false are all treated as absent; everything else
+ * is present. Used for the DATABRICKS-or-ANSI expression fallthrough (so an empty or absent
+ * DATABRICKS expr falls through to ANSI) and for optional-field mapping (so an empty
+ * `comment`/`synonyms` is dropped rather than emitted as an empty value).
+ */
+ static boolean truthy(Object v) {
+ if (v == null) {
+ return false;
+ }
+ if (v instanceof String) {
+ return !((String) v).isEmpty();
+ }
+ if (v instanceof java.util.Collection) {
+ return !((java.util.Collection<?>) v).isEmpty();
+ }
+ if (v instanceof Map) {
+ return !((Map<?, ?>) v).isEmpty();
+ }
+ if (v instanceof Number) {
+ return ((Number) v).doubleValue() != 0.0;
+ }
+ if (v instanceof Boolean) {
+ return (Boolean) v;
+ }
+ return true;
+ }
+
+ static List<String> synonymsOf(Object aiContext) {
+ if (aiContext instanceof Map) {
+ return strList(get(asMap(aiContext), "synonyms"));
+ }
+ return new ArrayList<>();
+ }
+
+ static String mergeDescription(Object description, Object aiContext) {
+ String desc = str(description);
+ if (aiContext instanceof String && !((String) aiContext).trim().isEmpty()) {
+ String s = (String) aiContext;
+ // When both are present they are joined with a newline; otherwise the
+ // `if description` is a truthiness test, so an empty (or null) description returns the
+ // ai_context alone rather than prepending a stray newline.
+ return (desc != null && !desc.isEmpty()) ? desc + "\n" + s : s;
+ }
+ return desc;
+ }
+
+ private static List<Map<String, Object>> customExtensions(Map<String, Object> obj) {
+ Object value = get(obj, "custom_extensions");
+ if (value == null) {
+ return new ArrayList<>();
+ }
+ if (!(value instanceof List)) {
+ String reason = "'custom_extensions' must be a list";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ List<Map<String, Object>> extensions = new ArrayList<>();
+ int index = 0;
+ for (Object extension : (List<Object>) value) {
+ if (!(extension instanceof Map)) {
+ String reason = "'custom_extensions[" + index + "]' must be a mapping";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ extensions.add((Map<String, Object>) extension);
+ index++;
+ }
+ return extensions;
+ }
+
+ static Map<String, Object> readStash(Map<String, Object> obj) {
+ Map<String, Object> stash = null;
+ for (Map<String, Object> ext : customExtensions(obj)) {
+ if (VENDOR.equals(str(get(ext, "vendor_name")))) {
+ if (stash != null) {
Review Comment:
Good point, changed it. readStash no longer throws on a duplicate DATABRICKS custom_extensions entry. It keeps the first, ignores the rest to match first wins, and emits a converter notice so the dropped entry is not lost silently rather than vanishing the way a plain first wins would. So a model with two entries converts instead of failing, and the user still sees that one was ignored. Added a test duplicateDatabricksStashKeepsFirstAndWarns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre merged PR #333:
URL: https://github.com/apache/ossie/pull/333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on code in PR #333:
URL: https://github.com/apache/ossie/pull/333#discussion_r3983194108
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java:
##########
@@ -0,0 +1,581 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.DIALECT_DATABRICKS;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SELECT_WITH_RE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.lastIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * IMPORT direction: Metric View v1.1 YAML -> Apache Ossie semantic model. Shared helpers come
+ * from {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class MetricViewToOssie {
+
+ private static final String[] MODEL_STASH_KEYS = {"filter", "parameters", "materialization"};
+ private static final String[] JOIN_STASH_KEYS = {"rely", "cardinality"};
+ // Metric-View-only column fields with no Apache Ossie representation, preserved verbatim in the
+ // DATABRICKS stash so an import can restore them. `window` and `partition` are measure-only
+ // (they live on MeasureExpression); a dimension never carries them, so the shared key list is
+ // simply never hit for those on the dimension path.
+ private static final String[] COLUMN_STASH_KEYS = {"format", "window", "partition"};
+ // Measure-only fields with no Apache Ossie Metric representation. A dimension's display_name
+ // maps to the Field `label`, but the Ossie Metric schema has no `label`, so a measure's
+ // display_name is preserved in the DATABRICKS stash instead. Kept separate from
+ // COLUMN_STASH_KEYS so the dimension path (which already maps display_name to `label`) does not
+ // also stash it.
+ private static final String[] MEASURE_STASH_KEYS = {"display_name"};
+ private static final Pattern NON_EQUI_RE = Pattern.compile("[<>!]=|<>|[<>]");
+ private static final Pattern AND_SPLIT_RE = Pattern.compile("\\s+AND\\s+", Pattern.CASE_INSENSITIVE);
+ private static final Pattern EQ_CLAUSE_RE = Pattern.compile("^\\s*(.+?)\\s*=\\s*(.+?)\\s*$");
+
+ private MetricViewToOssie() {}
+
+ static Result convertMetricViewToOssie(String mvYamlStr, String modelName) {
+ Notices notices = new Notices();
+ if (mvYamlStr == null || mvYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(mvYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> view = asMap(parsed);
+ Object versionValue = get(view, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!MV_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Metric View version '" + version
+ + "'. This converter targets v" + MV_VERSION + " only.",
+ version);
+ }
+ Map<String, Object> model = convertView(view, modelName, notices);
+ Map<String, Object> out = new LinkedHashMap<>();
+ out.put("version", OSSIE_VERSION);
+ List<Object> models = new ArrayList<>();
+ models.add(model);
+ out.put("semantic_model", models);
+ return serializeOssie(out, notices);
+ }
+
+ static Result serializeOssie(Map<String, Object> out, Notices notices) {
+ try {
+ return new Result(MAPPER.writeValueAsString(out), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Apache Ossie YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertView(
+ Map<String, Object> view, String modelName, Notices notices) {
+ Object source = get(view, "source");
+ if (source == null || source.toString().isEmpty()) {
+ throw new ConversionException("Metric View is missing required 'source'");
+ }
+ // Same subquery test validateSource applies below, so the two cannot drift.
+ boolean isSql = SELECT_WITH_RE.matcher(source.toString().trim()).find();
+ String lastId = lastIdentifier(source);
+ String factName;
+ if (modelName != null) {
+ factName = modelName;
+ } else if (!isSql && lastId != null && isSimpleIdentifier(lastId)) {
+ factName = lastId;
+ } else {
+ factName = "metric_view";
+ }
+ validateSource(source, factName);
+
+ List<Map<String, Object>> datasets = new ArrayList<>();
+ Map<String, Object> factDs = new LinkedHashMap<>();
+ factDs.put("name", factName);
+ factDs.put("source", source);
+ datasets.add(factDs);
+ List<Map<String, Object>> relationships = new ArrayList<>();
+ // Joins with no schema-valid Ossie relationship (a non-decomposable `on`) are collected here
+ // and stashed under the model's DATABRICKS custom_extensions instead of `relationships`.
+ List<Map<String, Object>> complexJoins = new ArrayList<>();
+ Map<String, String> aliasToDataset = new HashMap<>();
+ aliasToDataset.put("source", factName);
+ aliasToDataset.put(factName, factName);
+ Set<String> seenNames = new HashSet<>();
+ seenNames.add(factName.trim().toLowerCase(Locale.ROOT));
+
+ walk(factName, "source", asList(get(view, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+
+ // `fields` is a v1.1 alias for `dimensions`: an empty `dimensions: []` falls through to
+ // `fields`, and the "both set" warning fires only when BOTH are non-empty (not merely
+ // present).
+ List<Object> dimList = asList(get(view, "dimensions"));
+ List<Object> fieldList = asList(get(view, "fields"));
+ if (!dimList.isEmpty() && !fieldList.isEmpty()) {
+ notices.warn("view", "both 'dimensions' and 'fields' are set; 'fields' is a v1.1 alias "
+ + "for 'dimensions', so the 'fields' list is ignored");
+ }
+ Map<String, List<Object>> fieldsByDataset = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasets) {
+ fieldsByDataset.put((String) d.get("name"), new ArrayList<>());
+ }
+ List<Object> dims = !dimList.isEmpty() ? dimList : fieldList;
+ for (Object dimObj : dims) {
+ Map<String, Object> dim = asMap(dimObj);
+ if (isWildcard(dim)) {
+ notices.warn("dimension", "wildcard column '" + str(get(dim, "expr"))
+ + "' has no Apache Ossie field representation; skipped");
+ continue;
+ }
+ Object[] converted = convertDimension(dim, aliasToDataset, factName);
+ fieldsByDataset.get((String) converted[0]).add(converted[1]);
+ }
+ for (Map<String, Object> d : datasets) {
+ List<Object> flds = fieldsByDataset.get((String) d.get("name"));
+ if (!flds.isEmpty()) {
+ d.put("fields", flds);
+ }
+ }
+
+ List<Object> metrics = new ArrayList<>();
+ // Fixed once the join tree is walked, so compile it once for all measures.
+ Pattern aliasHead = qualifierChainPattern(aliasToDataset.keySet());
+ for (Object mObj : asList(get(view, "measures"))) {
+ Map<String, Object> m = asMap(mObj);
+ if (isWildcard(m)) {
+ notices.warn("measure", "wildcard measure '" + str(get(m, "expr"))
+ + "' has no Apache Ossie metric representation; skipped");
+ continue;
+ }
+ metrics.add(convertMeasure(m, aliasHead, aliasToDataset));
+ }
+
+ Map<String, Object> model = new LinkedHashMap<>();
+ model.put("name", factName);
+ if (truthy(get(view, "comment"))) {
+ model.put("description", get(view, "comment"));
+ }
+ model.put("datasets", datasets);
+ if (!relationships.isEmpty()) {
+ model.put("relationships", relationships);
+ }
+ if (!metrics.isEmpty()) {
+ model.put("metrics", metrics);
+ }
+
+ Map<String, Object> modelStash = new LinkedHashMap<>();
+ for (String k : MODEL_STASH_KEYS) {
+ if (view.containsKey(k)) {
+ modelStash.put(k, view.get(k));
+ }
+ }
+ if (hasOtm(asList(get(view, "joins")))) {
+ modelStash.put(STASH_SOURCE_KEY, factName);
+ }
+ if (!complexJoins.isEmpty()) {
+ modelStash.put("complex_joins", complexJoins);
+ }
+ writeStash(model, modelStash);
+ return model;
+ }
+
+ private static void walk(String parentName, String parentAlias, List<Object> joins,
+ List<Map<String, Object>> datasets, List<Map<String, Object>> relationships,
+ List<Map<String, Object>> complexJoins, Map<String, String> aliasToDataset,
+ Set<String> seenNames, Notices notices) {
+ for (Object joinObj : joins) {
+ Map<String, Object> join = asMap(joinObj);
+ String child = requireStr(join, "name", "join");
+ if (child.trim().equalsIgnoreCase("source")) {
+ throw new ConversionException(
+ "Join name 'source' is reserved for the fact source; rename the join.");
+ }
+ if (!seenNames.add(child.trim().toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("Duplicate dataset/join name '" + child
+ + "'; Metric View join names and the source must be distinct (case-insensitively).");
+ }
+ Map<String, Object> childDs = new LinkedHashMap<>();
+ childDs.put("name", child);
+ // Validated with the same rule the export applies to a dataset source, so a Metric View that
+ // imports cleanly is always exportable again (a 2-part join source used to import fine and
+ // then fail on the way back out).
+ String childSource = requireStr(join, "source", "join '" + child + "'");
+ validateSource(childSource, child);
+ childDs.put("source", childSource);
+ datasets.add(childDs);
+ aliasToDataset.put(child, child);
+ Map<String, Object> rel = convertJoin(join, parentName, parentAlias, child, notices);
+ // convertJoin omits from/to columns for a non-decomposable `on`; those entries have no valid
+ // Ossie relationship form, so they go to the model-level stash instead of `relationships`.
+ if (rel.containsKey("from_columns")) {
+ relationships.add(rel);
+ } else {
+ complexJoins.add(rel);
+ }
+ // rely.at_most_one_match on a many_to_one join -> recover a unique_key on the child.
+ Map<String, Object> rely = asMap(get(join, "rely"));
+ List<String> toCols = strList(get(rel, "to_columns"));
+ if (child.equals(str(get(rel, "to"))) && !toCols.isEmpty()
+ && Boolean.TRUE.equals(rely.get("at_most_one_match"))) {
+ List<Object> uk = new ArrayList<>();
+ uk.add(new ArrayList<>(toCols));
+ childDs.put("unique_keys", uk);
+ }
+ walk(child, child, asList(get(join, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+ }
+ }
+
+ private static boolean hasOtm(List<Object> joins) {
+ for (Object jObj : joins) {
+ Map<String, Object> j = asMap(jObj);
+ if (CARD_ONE_TO_MANY.equalsIgnoreCase(str(get(j, "cardinality")))) {
+ return true;
+ }
+ if (hasOtm(asList(get(j, "joins")))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static Map<String, Object> convertJoin(
+ Map<String, Object> join, String parentName, String parentAlias, String child,
+ Notices notices) {
+ boolean hasUsing = get(join, "using") != null && !asList(get(join, "using")).isEmpty();
+ boolean hasOn = str(get(join, "on")) != null && !str(get(join, "on")).isEmpty();
+ if (!hasUsing && !hasOn) {
+ throw new ConversionException("Join '" + child + "' has no join condition (empty or "
+ + "absent 'on'/'using'); condition-less (cross) joins have no Apache Ossie "
+ + "relationship representation.");
+ }
+ Object[] decomposed = decomposeOn(join, parentAlias, parentName, child);
+ List<String> parentCols = (List<String>) decomposed[0];
+ List<String> childCols = (List<String>) decomposed[1];
+ String rawOn = (String) decomposed[2];
+ if (rawOn != null) {
+ // The `on` is not an equi-join of simple `alias.column` pairs (it has a non-equi operator, a
+ // SQL-function-wrapped key, or an extra filter predicate), so an Apache Ossie relationship
+ // cannot represent it (from/to columns are required). Rather than abort the whole
+ // conversion, warn here; the columns-less entry built below is stashed under the model's
+ // custom_extensions (complex_joins) so it round-trips, instead of an invalid relationship.
+ notices.warn("join '" + child + "'", "non-equi or unsupported join condition ('on: " + rawOn
Review Comment:
Intentional and tracked. A join condition Apache Ossie cannot represent as a relationship (a non equi comparison, a function wrapped key, or an extra filter predicate) has no schema valid relationship form, since an Ossie relationship requires from_columns and to_columns. Rather than abort the whole export, the Java preserves the raw clause in a model level complex_joins stash inside custom_extensions, and the reverse converter rebuilds it. That keeps the round trip lossless for a join that carries real business logic, and the emitted model stays schema valid with no misleading columns less stub relationship, so a non Databricks consumer simply does not see the stashed join rather than seeing a wrong one. The Python reference still raises because it is the deprecated copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on code in PR #333:
URL: https://github.com/apache/ossie/pull/333#discussion_r4084995681
##########
converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterSuite.java:
##########
@@ -0,0 +1,1639 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.junit.jupiter.api.Test;
+
+/**
+ * Example-based tests for the Apache Ossie <-> Metric View converter, plus the fixture
+ * comparisons that pin the expected output of both directions.
+ */
+public class OssieConverterSuite {
+
+ private static Object export(String osi, String source) {
+ return OssieConverter.parseYaml(
+ OssieConverter.convertOssieToMetricView(osi, source).yaml);
+ }
+
+ private static final class ThrowingBean {
+ public String getValue() {
+ throw new IllegalStateException("expected serialization failure");
+ }
+ }
+
+ private static Map<String, Object> valueThatFailsSerialization() {
+ Map<String, Object> value = new HashMap<>();
+ value.put("bad", new ThrowingBean());
+ return value;
+ }
+
+ private static String ossieModelWithBody(String body) {
+ // Flat model at the document root. Callers pass a body indented two spaces (it used to sit
+ // under a `- name: m` list item); de-indent it so it sits at the root next to `name`.
+ StringBuilder root = new StringBuilder(
+ "version: '" + OssieConverter.OSSIE_VERSION + "'\nname: m\n");
+ for (String line : body.split("\n", -1)) {
+ root.append(line.startsWith(" ") ? line.substring(2) : line).append("\n");
+ }
+ root.setLength(root.length() - 1);
+ return root.toString();
+ }
+
+ private static String cascadeNotice(String kind, String name, String reference) {
+ return "[" + kind + " '" + name + "'] references dropped '" + reference
+ + "'; dropping (downstream of a dropped field/metric)";
+ }
+
+ private static void assertInvalidOssieInput(String yaml, String expectedReason) {
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(yaml, null));
+ assertEquals(OssieConverter.ConversionException.Kind.INVALID_INPUT, e.getKind());
+ assertEquals(expectedReason, e.getReason());
+ }
+
+ @Test
+ public void fixtureAStarSchemaExportsToExpectedMetricView() {
+ String osi =
+ "version: \"0.2.0.dev0\"\n"
+ + "name: sales\n"
+ + "description: Sales orders with customer attributes\n"
+ + "datasets:\n"
+ + " - name: orders\n"
+ + " source: samples.tpch.orders\n"
+ + " primary_key: [o_orderkey]\n"
+ + " description: One row per order\n"
+ + " fields:\n"
+ + " - name: o_orderkey\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - dialect: DATABRICKS\n"
+ + " expression: o_orderkey\n"
+ + " description: Order identifier\n"
+ + " - name: o_orderdate\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - dialect: DATABRICKS\n"
+ + " expression: o_orderdate\n"
+ + " label: Order Date\n"
+ + " ai_context:\n"
+ + " synonyms: [order date, date]\n"
+ + " - name: customer\n"
+ + " source: samples.tpch.customer\n"
+ + " primary_key: [c_custkey]\n"
+ + " fields:\n"
+ + " - name: c_name\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - dialect: DATABRICKS\n"
+ + " expression: c_name\n"
+ + " description: Customer name\n"
+ + "relationships:\n"
+ + " - name: orders_to_customer\n"
+ + " from: orders\n"
+ + " to: customer\n"
+ + " from_columns: [o_custkey]\n"
+ + " to_columns: [c_custkey]\n"
+ + "metrics:\n"
+ + " - name: total_revenue\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - dialect: DATABRICKS\n"
+ + " expression: SUM(o_totalprice)\n"
+ + " description: Total order revenue\n"
+ + " ai_context:\n"
+ + " synonyms: [revenue, total revenue, sales]\n"
+ + " - name: order_count\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - dialect: DATABRICKS\n"
+ + " expression: COUNT(*)\n"
+ + " description: Number of orders\n";
+
+ String expected =
+ "version: '1.1'\n"
+ + "source: samples.tpch.orders\n"
+ + "comment: Sales orders with customer attributes\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: samples.tpch.customer\n"
+ + " on: source.o_custkey = customer.c_custkey\n"
+ + " rely:\n"
+ + " at_most_one_match: true\n"
+ + "dimensions:\n"
+ + "- name: o_orderkey\n"
+ + " expr: o_orderkey\n"
+ + " comment: Order identifier\n"
+ + "- name: o_orderdate\n"
+ + " expr: o_orderdate\n"
+ + " display_name: Order Date\n"
+ + " synonyms:\n"
+ + " - order date\n"
+ + " - date\n"
+ + "- name: c_name\n"
+ + " expr: customer.c_name\n"
+ + " comment: Customer name\n"
+ + "measures:\n"
+ + "- name: total_revenue\n"
+ + " expr: SUM(o_totalprice)\n"
+ + " comment: Total order revenue\n"
+ + " synonyms:\n"
+ + " - revenue\n"
+ + " - total revenue\n"
+ + " - sales\n"
+ + "- name: order_count\n"
+ + " expr: COUNT(*)\n"
+ + " comment: Number of orders\n";
+
+ assertEquals(OssieConverter.parseYaml(expected), export(osi, null));
+ }
+
+ @Test
+ public void measureDisplayNameSurvivesRoundTripViaStash() {
+ // A dimension's display_name maps to the Ossie Field `label`, but the Ossie Metric schema has
+ // no `label`, so a measure's display_name is preserved in the DATABRICKS custom_extensions
+ // stash (like format/window) and restored on the way back.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.fact\n"
+ + "dimensions:\n"
+ + "- name: region\n"
+ + " expr: region\n"
+ + "measures:\n"
+ + "- name: total_revenue\n"
+ + " expr: SUM(amount)\n"
+ + " display_name: Total Revenue\n";
+
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ // The only display_name in the input is on the measure; it rides in the metric's stash, since
+ // there is no native Ossie field for it.
+ assertTrue(ossie.contains("display_name"),
+ "expected the measure display_name preserved in the stash, got: " + ossie);
+
+ String mv2 = OssieConverter.convertOssieToMetricView(ossie, null).yaml;
+ assertEquals(OssieConverter.parseYaml(mv), OssieConverter.parseYaml(mv2));
+ }
+
+ @Test
+ public void unsupportedVersionIsRejected() {
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView("version: '9.9'\nname: m\n", null));
+ assertTrue(e.getMessage().contains("Unsupported Apache Ossie version"));
+ }
+
+ @Test
+ public void multipleCandidateFactsWithoutSourceIsRejected() {
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- {name: orders, source: c.s.orders}\n"
+ + "- {name: returns, source: c.s.returns}\n"
+ + "- {name: customer, source: c.s.customer, primary_key: [c_custkey]}\n"
+ + "relationships:\n"
+ + "- {name: oc, from: orders, to: customer, from_columns: [o_custkey], to_columns: [c_custkey]}\n"
+ + "- {name: rc, from: returns, to: customer, from_columns: [re_custkey], to_columns: [c_custkey]}\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, null));
+ assertTrue(e.getMessage().contains("multiple candidate fact datasets"));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void oneToManyEmitsCardinality() {
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: orders\n"
+ + " source: c.s.orders\n"
+ + " primary_key: [o_orderkey]\n"
+ + " fields:\n"
+ + " - {name: o_orderstatus, expression: {dialects: [{dialect: DATABRICKS, expression: o_orderstatus}]}}\n"
+ + "- name: lineitem\n"
+ + " source: c.s.lineitem\n"
+ + "relationships:\n"
+ + "- {name: lio, from: lineitem, to: orders, from_columns: [l_orderkey], to_columns: [o_orderkey]}\n"
+ + "metrics:\n"
+ + "- {name: qty, expression: {dialects: [{dialect: DATABRICKS, expression: SUM(lineitem.l_quantity)}]}}\n";
+ Map<String, Object> view = (Map<String, Object>) export(osi, "orders");
+ List<Object> joins = (List<Object>) view.get("joins");
+ Map<String, Object> join = (Map<String, Object>) joins.get(0);
+ assertEquals("one_to_many", join.get("cardinality"));
+ }
+
+ @Test
+ public void oneToManyNestedUnderManyToOneIsRejected() {
+ // A Metric View requires one cardinality per top-level branch, so a one-to-many join nested
+ // under a many-to-one parent is rejected by Databricks just as the reverse nesting is. Fact
+ // `d`; `f -> d` points at d (many-to-one from d's perspective), and `g -> f` points away from
+ // f (one-to-many), which would nest one_to_many inside the many-to-one branch.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: d\n"
+ + " source: c.s.d\n"
+ + " fields:\n"
+ + " - {name: dcol, expression: {dialects: [{dialect: DATABRICKS, expression: dcol}]}}\n"
+ + "- name: f\n"
+ + " source: c.s.f\n"
+ + " primary_key: [fk]\n"
+ + "- name: g\n"
+ + " source: c.s.g\n"
+ + "relationships:\n"
+ + "- {name: df, from: d, to: f, from_columns: [fk], to_columns: [fk]}\n"
+ + "- {name: gf, from: g, to: f, from_columns: [fk], to_columns: [fk]}\n"
+ + "metrics:\n"
+ + "- {name: c, expression: {dialects: [{dialect: DATABRICKS, expression: COUNT(1)}]}}\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, "d"));
+ assertTrue(e.getMessage().contains("share the same cardinality"),
+ "expected a mixed-cardinality rejection, got: " + e.getMessage());
+ }
+
+ @Test
+ public void directedCycleWithNoEquidistantEdgeIsRejected() {
+ // a -> b -> c -> d -> e -> b with fact `a`: every edge spans adjacent BFS levels, so the
+ // equidistance heuristic sees nothing, and `a` has zero incoming edges so pickFact finds a
+ // root. Without a real acyclicity check this expanded into duplicate join paths (`d` under
+ // both `c` and `e`), fabricating a tree from a cyclic model.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: a\n"
+ + " source: c.s.a\n"
+ + " fields:\n"
+ + " - {name: acol, expression: {dialects: [{dialect: DATABRICKS, expression: acol}]}}\n"
+ + "- name: b\n"
+ + " source: c.s.b\n"
+ + "- name: c\n"
+ + " source: c.s.c\n"
+ + "- name: d\n"
+ + " source: c.s.d\n"
+ + "- name: e\n"
+ + " source: c.s.e\n"
+ + "relationships:\n"
+ + "- {name: ab, from: a, to: b, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: bc, from: b, to: c, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: cd, from: c, to: d, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: de, from: d, to: e, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: eb, from: e, to: b, from_columns: [k], to_columns: [k]}\n";
+ OssieConverter.ConversionException ex = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, "a"));
+ assertTrue(ex.getMessage().contains("directed cycle"),
+ "expected a directed-cycle rejection, got: " + ex.getMessage());
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void diamondIsStillAcceptedByTheCycleCheck() {
+ // A diamond (`a -> b -> d` plus `a -> c -> d`) is an UNDIRECTED cycle but directed-acyclic, and
+ // is supported via the fan-out aliases. The cycle check must not reject it.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: a\n"
+ + " source: c.s.a\n"
+ + "- name: b\n"
+ + " source: c.s.b\n"
+ + "- name: c\n"
+ + " source: c.s.c\n"
+ + "- name: d\n"
+ + " source: c.s.d\n"
+ + " fields:\n"
+ + " - {name: dcol, expression: {dialects: [{dialect: DATABRICKS, expression: dcol}]}}\n"
+ + "relationships:\n"
+ + "- {name: ab, from: a, to: b, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: ac, from: a, to: c, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: bd, from: b, to: d, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: cd, from: c, to: d, from_columns: [k], to_columns: [k]}\n";
+ Map<String, Object> view = (Map<String, Object>) export(osi, "a");
+ assertEquals("c.s.a", view.get("source"));
+ // `d` is reached by two paths, so its column is emitted once per fan-out alias.
+ List<Object> dims = (List<Object>) view.get("dimensions");
+ assertEquals(2, dims.size(), "diamond should fan out to one dimension per path, got: " + dims);
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void measureNamingADiamondDatasetIsDroppedWithNotice() {
+ // Same diamond as above (`d` reached via `a -> b -> d` and `a -> c -> d`), plus a measure that
+ // names `d` by its bare dataset name. A measure addresses a dataset by name, so it cannot be
+ // split per fan-out path the way the dimension on `d` is; it is dropped with a notice rather
+ // than silently bound to one arbitrary branch (as the dimension complex-expression path does).
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: a\n"
+ + " source: c.s.a\n"
+ + "- name: b\n"
+ + " source: c.s.b\n"
+ + "- name: c\n"
+ + " source: c.s.c\n"
+ + "- name: d\n"
+ + " source: c.s.d\n"
+ + " fields:\n"
+ + " - {name: dcol, expression: {dialects: [{dialect: DATABRICKS, expression: dcol}]}}\n"
+ + "relationships:\n"
+ + "- {name: ab, from: a, to: b, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: ac, from: a, to: c, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: bd, from: b, to: d, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: cd, from: c, to: d, from_columns: [k], to_columns: [k]}\n"
+ + "metrics:\n"
+ + "- {name: dsum, expression: {dialects: [{dialect: DATABRICKS, expression: SUM(d.dcol)}]}}\n";
+ OssieConverter.Result result = OssieConverter.convertOssieToMetricView(osi, "a");
+ Map<String, Object> view = (Map<String, Object>) OssieConverter.parseYaml(result.yaml);
+ // The diamond dimension on `d` still fans out to one dimension per path.
+ assertEquals(2, ((List<Object>) view.get("dimensions")).size());
+ // The measure that names `d` is dropped rather than bound to an arbitrary branch.
+ assertFalse(view.containsKey("measures"),
+ "a measure naming a diamond dataset must be dropped, got: " + view.get("measures"));
+ String expected = "expression references dataset 'd', reached by more than one join path "
+ + "(diamond); cannot be unambiguously qualified; dropped";
+ assertTrue(result.notices.stream().anyMatch(n -> n.contains(expected)),
+ result.notices.toString());
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void nestedJoinColumnInMeasureGetsFullAliasPath() {
+ // orders -> customer -> nation: `nation` nests under `customer`, so its columns are addressed
+ // as `customer.nation.col`. A bare `nation.` head would be read as struct access on a
+ // parameter, so the measure must be re-qualified -- and identically to the dimension path,
+ // which already emits `customer.nation.population` for the same column.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: orders\n"
+ + " source: c.s.orders\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " primary_key: [c_custkey]\n"
+ + "- name: nation\n"
+ + " source: c.s.nation\n"
+ + " primary_key: [n_nationkey]\n"
+ + " fields:\n"
+ + " - {name: population, expression: {dialects: [{dialect: DATABRICKS, expression: population}]}}\n"
+ + "relationships:\n"
+ + "- {name: oc, from: orders, to: customer, from_columns: [c_custkey], to_columns: [c_custkey]}\n"
+ + "- {name: cn, from: customer, to: nation, from_columns: [n_nationkey], to_columns: [n_nationkey]}\n"
+ + "metrics:\n"
+ + "- {name: pop, expression: {dialects: [{dialect: DATABRICKS, expression: SUM(nation.population)}]}}\n";
+ Map<String, Object> view = (Map<String, Object>) export(osi, "orders");
+
+ List<Object> dims = (List<Object>) view.get("dimensions");
+ Map<String, Object> dim = (Map<String, Object>) dims.get(0);
+ assertEquals("customer.nation.population", dim.get("expr"),
+ "dimension path should qualify with the full alias path");
+
+ List<Object> measures = (List<Object>) view.get("measures");
+ Map<String, Object> measure = (Map<String, Object>) measures.get(0);
+ assertEquals("SUM(customer.nation.population)", measure.get("expr"),
+ "measure must use the same full alias path as the dimension, not a bare nested alias");
+ }
+
+ @Test
+ public void modelWithNoFieldsOrMetricsIsRejected() {
+ // A Metric View requires at least one dimension or measure, so emitting a version+source-only
+ // view would just fail at CREATE. Fail at conversion time instead.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: d\n"
+ + " source: c.s.d\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, null));
+ assertTrue(e.getMessage().contains("no dimensions or measures"),
+ "expected an empty-view rejection, got: " + e.getMessage());
+ }
+
+ @Test
+ public void emptyAfterDropsNamesTheDroppedColumns() {
+ // Here the input is non-empty but everything drops: the only metric has no DATABRICKS/ANSI_SQL
+ // dialect. The error must name the dropped column so the cause is actionable, since after the
+ // cascade the emptiness is a consequence of the drop rather than an empty input.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: d\n"
+ + " source: c.s.d\n"
+ + "metrics:\n"
+ + "- {name: only_metric, expression: {dialects: [{dialect: SNOWFLAKE, expression: SUM(x)}]}}\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, null));
+ assertTrue(e.getMessage().contains("no dimensions or measures"),
+ "expected an empty-view rejection, got: " + e.getMessage());
+ assertTrue(e.getMessage().contains("only_metric"),
+ "the message must name the dropped column, got: " + e.getMessage());
+ }
+
+ @Test
+ public void duplicateDimensionNameIsRejected() {
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: orders\n"
+ + " source: c.s.orders\n"
+ + " fields:\n"
+ + " - {name: id, expression: {dialects: [{dialect: DATABRICKS, expression: id}]}}\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " fields:\n"
+ + " - {name: id, expression: {dialects: [{dialect: DATABRICKS, expression: id}]}}\n"
+ + "relationships:\n"
+ + "- {name: r, from: orders, to: customer, from_columns: [cid], to_columns: [id]}\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, null));
+ assertTrue(e.getMessage().contains("collides"));
+ }
+
+ @Test
+ public void foreignVendorExtensionDroppedWithNotice() {
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "custom_extensions:\n"
+ + "- {vendor_name: SNOWFLAKE, data: '{}'}\n"
+ + "datasets:\n"
+ + "- name: orders\n"
+ + " source: c.s.orders\n"
+ + " fields:\n"
+ + " - {name: s, expression: {dialects: [{dialect: DATABRICKS, expression: s}]}}\n"
+ + "metrics:\n"
+ + "- {name: n, expression: {dialects: [{dialect: DATABRICKS, expression: COUNT(*)}]}}\n";
+ OssieConverter.Result r = OssieConverter.convertOssieToMetricView(osi, null);
+ assertTrue(r.notices.stream().anyMatch(m -> m.contains("foreign-vendor custom_extensions dropped")));
+ }
+
+ // -- import direction (Metric View -> Apache Ossie) -----------------------
+
+ private static Object importMv(String mv) {
+ return OssieConverter.parseYaml(
+ OssieConverter.convertMetricViewToOssie(mv, null).yaml);
+ }
+
+ /** A Metric View YAML with {@code joinCount} sibling equi-joins on the fact source. */
+ private static String metricViewWithJoins(int joinCount) {
+ StringBuilder mv = new StringBuilder("version: '1.1'\nsource: c.s.fact\njoins:\n");
+ for (int i = 1; i <= joinCount; i++) {
+ mv.append("- name: j").append(i).append("\n")
+ .append(" source: c.s.j").append(i).append("\n")
+ .append(" on: source.k = j").append(i).append(".k\n");
+ }
+ return mv.toString();
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void importDecomposesJoinIntoRelationship() {
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey = customer.c_custkey\n"
+ + " rely: {at_most_one_match: true}\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus}\n"
+ + "- {name: c_name, expr: customer.c_name}\n"
+ + "measures:\n"
+ + "- {name: revenue, expr: SUM(o_totalprice)}\n";
+ Map<String, Object> out = (Map<String, Object>) importMv(mv);
+ Map<String, Object> model = out;
+ List<Object> rels = (List<Object>) model.get("relationships");
+ Map<String, Object> rel = (Map<String, Object>) rels.get(0);
+ assertEquals("orders", rel.get("from"));
+ assertEquals("customer", rel.get("to"));
+ assertEquals(List.of("o_custkey"), rel.get("from_columns"));
+ assertEquals(List.of("c_custkey"), rel.get("to_columns"));
+ }
+
+ @Test
+ public void exportRejectsMoreJoinsThanImportCanRebuild() {
+ // MV -> Ossie has no join bound of its own, but Ossie -> MV rejects a model with more than
+ // MAX_JOIN_NODES datasets. Without a matching bound here, a Metric View with too many joins
+ // would export to a model that could never be imported again; reject it at export instead.
+ int max = OssieConverterCommon.MAX_JOIN_NODES;
+ String mv = metricViewWithJoins(max);
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertMetricViewToOssie(mv, null));
+ assertTrue(e.getMessage().contains("at most " + max + " are supported"),
+ "expected a join-count rejection, got: " + e.getMessage());
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void exportAtTheJoinLimitStillSucceeds() {
+ // The fact plus (MAX_JOIN_NODES - 1) joins is exactly MAX_JOIN_NODES datasets, the boundary the
+ // reverse conversion accepts, so export must still succeed here.
+ int max = OssieConverterCommon.MAX_JOIN_NODES;
+ Map<String, Object> out = (Map<String, Object>) importMv(metricViewWithJoins(max - 1));
+ Map<String, Object> model = out;
+ List<Object> datasets = (List<Object>) model.get("datasets");
+ assertEquals(max, datasets.size(), "fact plus joins should be exactly the limit");
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void importPreservesNonEquiJoinInStash() {
+ // A non-equi `on` has no schema-valid Ossie relationship (from/to columns are required), so the
+ // converter warns and stashes the whole join under the model's DATABRICKS custom_extensions
+ // rather than emitting an invalid stub relationship (issue #321).
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey >= customer.c_custkey\n";
+ OssieConverter.Result result = OssieConverter.convertMetricViewToOssie(mv, null);
+ assertTrue(result.notices.stream().anyMatch(n -> n.contains("non-equi or unsupported")),
+ result.notices.toString());
+ Map<String, Object> out = (Map<String, Object>) OssieConverter.parseYaml(result.yaml);
+ Map<String, Object> model = out;
+ // No stub relationship is emitted; the join lives only in the model's custom_extensions.
+ assertFalse(model.containsKey("relationships"), result.yaml);
+ assertTrue(model.containsKey("custom_extensions"), result.yaml);
+ assertTrue(result.yaml.contains("source.o_custkey >= customer.c_custkey"), result.yaml);
+ }
+
+ @Test
+ public void nonEquiJoinSurvivesRoundTripViaStash() {
+ // The preserved raw `on` is restored verbatim on the way back, so a metric view whose join
+ // carries business logic round-trips instead of being rejected (issue #321).
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey >= customer.c_custkey\n"
+ + "dimensions:\n"
+ + "- name: cust\n"
+ + " expr: customer.c_name\n"
+ + "measures:\n"
+ + "- name: revenue\n"
+ + " expr: SUM(o_totalprice)\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ String mv2 = OssieConverter.convertOssieToMetricView(ossie, null).yaml;
+ assertTrue(mv2.contains("source.o_custkey >= customer.c_custkey"), mv2);
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void nestedNonEquiJoinKeepsAliasesOnRoundTrip() {
+ // A nested join's non-equi `on` references the parent join's alias. On import the alias is
+ // assigned from the (unique) dataset name our exporter emits, so the verbatim-restored `on`
+ // still names the aliases the importer actually assigned -- no stale-qualifier rewrite needed.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey = customer.c_custkey\n"
+ + " joins:\n"
+ + " - name: nation\n"
+ + " source: c.s.nation\n"
+ + " on: customer.c_nationkey <> nation.n_nationkey\n"
+ + "measures:\n"
+ + "- name: revenue\n"
+ + " expr: SUM(o_totalprice)\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ Map<String, Object> mv2 = (Map<String, Object>) OssieConverter.parseYaml(
+ OssieConverter.convertOssieToMetricView(ossie, null).yaml);
+ Map<String, Object> customer = (Map<String, Object>) ((List<Object>) mv2.get("joins")).get(0);
+ assertEquals("customer", customer.get("name"));
+ Map<String, Object> nation =
+ (Map<String, Object>) ((List<Object>) customer.get("joins")).get(0);
+ // The nested join keeps its alias, and the restored `on` references it and the parent alias
+ // `customer` -- exactly the aliases the importer assigned, so the clause resolves.
+ assertEquals("nation", nation.get("name"));
+ assertEquals("customer.c_nationkey <> nation.n_nationkey", nation.get("on"));
+ }
+
+ @Test
+ public void equiJoinOnPreservedVerbatimWhenNonCanonical() {
+ // An equi-join is a valid Ossie relationship (from/to columns), but rebuilding its `on` from
+ // those columns canonicalizes the fact qualifier to `source`. When the original used the source
+ // table name instead, the raw `on` is stashed so the round trip reproduces it exactly.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: orders.o_custkey = customer.c_custkey\n"
+ + "measures:\n"
+ + "- name: revenue\n"
+ + " expr: SUM(o_totalprice)\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ String mv2 = OssieConverter.convertOssieToMetricView(ossie, null).yaml;
+ assertTrue(mv2.contains("orders.o_custkey = customer.c_custkey"), mv2);
+ }
+
+ @Test
+ public void complexJoinPreservesCardinalityOnRoundTrip() {
+ // A complex (non-decomposable) join on the one_to_many branch stashes `on` and `cardinality`
+ // as flat keys on the columns-less entry; both are restored verbatim on the way back.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: lineitem\n"
+ + " source: c.s.lineitem\n"
+ + " on: source.o_orderkey <> lineitem.l_orderkey\n"
+ + " cardinality: one_to_many\n"
+ + "measures:\n"
+ + "- name: cnt\n"
+ + " expr: COUNT(1)\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ String mv2 = OssieConverter.convertOssieToMetricView(ossie, null).yaml;
+ assertTrue(mv2.contains("source.o_orderkey <> lineitem.l_orderkey"), mv2);
+ assertTrue(mv2.contains("one_to_many"), mv2);
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void complexJoinNestedUnderComplexJoinRoundTrips() {
+ // A complex join nested under another complex join: both edges live in the model stash, and the
+ // reverse merge rebuilds the tree so each raw `on` and the nesting are restored.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey <> customer.c_custkey\n"
+ + " joins:\n"
+ + " - name: nation\n"
+ + " source: c.s.nation\n"
+ + " on: customer.c_nationkey <> nation.n_nationkey\n"
+ + "measures:\n"
+ + "- name: cnt\n"
+ + " expr: COUNT(1)\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ Map<String, Object> mv2 = (Map<String, Object>) OssieConverter.parseYaml(
+ OssieConverter.convertOssieToMetricView(ossie, null).yaml);
+ Map<String, Object> customer = (Map<String, Object>) ((List<Object>) mv2.get("joins")).get(0);
+ assertEquals("source.o_custkey <> customer.c_custkey", customer.get("on"));
+ Map<String, Object> nation =
+ (Map<String, Object>) ((List<Object>) customer.get("joins")).get(0);
+ assertEquals("nation", nation.get("name"));
+ assertEquals("customer.c_nationkey <> nation.n_nationkey", nation.get("on"));
+ }
+
+ @Test
+ public void equiJoinOnOverEqualColumnsStaysOnNotUsing() {
+ // An equi-join `on` over equal column names rebuilds from columns as `using`, so the original
+ // `on` is stashed and restored -- the round trip keeps `on`, it does not collapse to `using`.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.custkey = customer.custkey\n"
+ + "measures:\n"
+ + "- name: cnt\n"
+ + " expr: COUNT(1)\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ String mv2 = OssieConverter.convertOssieToMetricView(ossie, null).yaml;
+ assertTrue(mv2.contains("source.custkey = customer.custkey"), mv2);
+ assertFalse(mv2.contains("using"), mv2);
+ }
+
+ @Test
+ public void importRejectsCrossJoin() {
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertMetricViewToOssie(mv, null));
+ assertTrue(e.getMessage().contains("no join condition"));
+ }
+
+ @Test
+ public void importRejectsUnsupportedVersion() {
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertMetricViewToOssie("version: '0.1'\nsource: c.s.t\n", null));
+ assertTrue(e.getMessage().contains("Unsupported Metric View version"));
+ }
+
+ @Test
+ public void mvToOssieToMvRoundTripsStash() {
+ // A view with MV-only features (filter/rely/format/window) must survive
+ // MV -> Ossie -> MV unchanged (the custom_extensions stash carries them).
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "filter: o_orderstatus = 'F'\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey = customer.c_custkey\n"
+ + " rely:\n"
+ + " at_most_one_match: true\n"
+ + "dimensions:\n"
+ + "- name: net\n"
+ + " expr: o_totalprice\n"
+ + " format:\n"
+ + " type: currency\n"
+ + " currency_code: USD\n"
+ + "measures:\n"
+ + "- name: running\n"
+ + " expr: SUM(o_totalprice)\n"
+ + " window:\n"
+ + " - order: net\n"
+ + " semiadditive: last\n"
+ + " range: cumulative\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ String back = OssieConverter.convertOssieToMetricView(ossie, null).yaml;
+ assertEquals(OssieConverter.parseYaml(mv), OssieConverter.parseYaml(back));
+ }
+
+ // -- fixture comparisons ---------------------------------------------------
+ // The fixtures under src/test/resources/ossie_*.yaml pin the expected outputs
+ // checked into apache/ossie. Asserting the Java output parses equal to them (structure
+ // + stash blob STRINGS) is the real "one behavior, two implementations" guarantee --
+ // it catches divergences like stash JSON spacing that a Java->Java round-trip misses.
+
+ private static String loadFixture(String name) {
+ try (InputStream in =
+ OssieConverterSuite.class.getClassLoader().getResourceAsStream("ossie_" + name)) {
+ if (in == null) {
+ throw new IllegalStateException("fixture not found: ossie_" + name);
+ }
+ return new String(in.readAllBytes(), StandardCharsets.UTF_8);
+ } catch (java.io.IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ public void fixtureAExportMatchesFixture() {
+ String out = OssieConverter.convertOssieToMetricView(loadFixture("fixtureA_ossie.yaml"), null).yaml;
+ assertEquals(OssieConverter.parseYaml(loadFixture("fixtureA_metric_view.yaml")),
+ OssieConverter.parseYaml(out));
+ }
+
+ @Test
+ public void fixtureBImportMatchesFixture() {
+ // fixtureB exercises the custom_extensions stash (format/rely/filter) -- the parse
+ // includes the blob strings, so this is what pins the stash blob's exact spacing.
+ String out = OssieConverter.convertMetricViewToOssie(loadFixture("fixtureB_metric_view.yaml"), null).yaml;
+ assertEquals(OssieConverter.parseYaml(loadFixture("fixtureB_ossie.yaml")),
+ OssieConverter.parseYaml(out));
+ }
+
+ @Test
+ public void tpcdsExportMatchesFixture() {
+ String out = OssieConverter.convertOssieToMetricView(loadFixture("tpcds_ossie.yaml"), null).yaml;
+ assertEquals(OssieConverter.parseYaml(loadFixture("tpcds_metric_view.yaml")),
+ OssieConverter.parseYaml(out));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void stashBlobUsesExpectedSpacing() {
+ // The strongest byte-level check: the emitted stash blob string (the `data` value of a
+ // custom_extensions entry) must use the stash format's separators
+ // (", " / ": "). Pull the blob out of the parsed model rather than substring-matching
+ // the outer YAML (where it appears escaped).
+ Object out = OssieConverter.parseYaml(
+ OssieConverter.convertMetricViewToOssie(loadFixture("fixtureB_metric_view.yaml"), null).yaml);
+ Map<String, Object> model = (Map<String, Object>) out;
+ List<Object> exts = (List<Object>) model.get("custom_extensions");
+ String blob = (String) ((Map<String, Object>) exts.get(0)).get("data");
+ assertTrue(blob.startsWith("{\"_v\": 1, "),
+ "stash blob must use the spacing '{\"_v\": 1, ...}', got: " + blob);
+ }
+
+ // -- parity edge cases found in review round 3 ----------------------------
+
+ @Test
+ public void pickExpressionFallsThroughEmptyDatabricksToAnsi() {
+ // An empty DATABRICKS dialect must fall through to ANSI_SQL,
+ // not be selected as the (empty) expression.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: f\n"
+ + " source: c.s.f\n"
+ + " fields:\n"
+ + " - name: d\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - {dialect: DATABRICKS, expression: ''}\n"
+ + " - {dialect: ANSI_SQL, expression: ansi_col}\n"
+ + "metrics:\n"
+ + "- {name: n, expression: {dialects: [{dialect: DATABRICKS, expression: COUNT(*)}]}}\n";
+ Object out = export(osi, null);
+ @SuppressWarnings("unchecked")
+ List<Object> dims = (List<Object>) ((Map<String, Object>) out).get("dimensions");
+ @SuppressWarnings("unchecked")
+ Map<String, Object> dim = (Map<String, Object>) dims.get(0);
+ assertEquals("ansi_col", dim.get("expr"));
+ }
+
+ @Test
+ public void pickExpressionRejectsNonStringExpression() {
+ // A non-string dialect expression (e.g. a YAML number) must raise, not be coerced.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: f\n"
+ + " source: c.s.f\n"
+ + " fields:\n"
+ + " - name: d\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - {dialect: DATABRICKS, expression: 123}\n"
+ + "metrics:\n"
+ + "- {name: n, expression: {dialects: [{dialect: DATABRICKS, expression: COUNT(*)}]}}\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, null));
+ assertTrue(e.getMessage().contains("expression must be a string"));
+ }
+
+ @Test
+ public void bareOnOffValuesStayStringsNotBooleans() {
+ // YAML 1.1 would read a bare `on`/`off`/`yes`/`no` value as a boolean, silently losing
+ // a join condition or turning a synonym into `true`. Confirm the converter's parser
+ // keeps them as strings (the reader uses YAML 1.2 boolean semantics).
+ Object parsed = OssieConverter.parseYaml("a: on\nb: off\nc: yes\nd: no\n");
+ @SuppressWarnings("unchecked")
+ Map<String, Object> m = (Map<String, Object>) parsed;
+ assertEquals("on", m.get("a"));
+ assertEquals("off", m.get("b"));
+ assertEquals("yes", m.get("c"));
+ assertEquals("no", m.get("d"));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void importDropsEmptyOptionalFields() {
+ // Optional fields are mapped only when non-empty, so an empty
+ // comment / empty synonyms list are omitted, not emitted as `description: ""` or an
+ // empty ai_context. The Java port must match (empty string / empty list are falsy).
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "comment: ''\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus, comment: '', display_name: '', synonyms: []}\n"
+ + "measures:\n"
+ + "- {name: revenue, expr: SUM(o_totalprice), comment: '', synonyms: []}\n";
+ Map<String, Object> out = (Map<String, Object>) importMv(mv);
+ Map<String, Object> model = out;
+ assertFalse(model.containsKey("description"), "empty comment must not become a description");
+ Map<String, Object> ds = (Map<String, Object>) ((List<Object>) model.get("datasets")).get(0);
+ Map<String, Object> field = (Map<String, Object>) ((List<Object>) ds.get("fields")).get(0);
+ assertFalse(field.containsKey("description"), "empty comment must not map to description");
+ assertFalse(field.containsKey("label"), "empty display_name must not map to label");
+ assertFalse(field.containsKey("ai_context"), "empty synonyms must not map to ai_context");
+ Map<String, Object> metric = (Map<String, Object>) ((List<Object>) model.get("metrics")).get(0);
+ assertFalse(metric.containsKey("description"), "empty comment must not map to description");
+ assertFalse(metric.containsKey("ai_context"), "empty synonyms must not map to ai_context");
+ }
+
+ @Test
+ public void stashEscapesNonAsciiAsLowercaseHex() {
+ // The stash blob is pure ASCII: non-ASCII is escaped to \\uXXXX. The
+ // stash blob must be byte-identical, so a non-ASCII stashed value (here a `filter`
+ // literal) escapes rather than emitting raw UTF-8.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "filter: \"region = 'café'\"\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus}\n";
+ String ossieYaml = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ // Extract the stash blob and assert the exact expected bytes:
+ // the non-ASCII char is escaped as lowercase \\u00e9 (a single backslash + 5 chars),
+ // not emitted raw. (Comparing the parsed `data` string sidesteps YAML's own quoting.)
+ @SuppressWarnings("unchecked")
+ Map<String, Object> out = (Map<String, Object>) OssieConverter.parseYaml(ossieYaml);
+ @SuppressWarnings("unchecked")
+ Map<String, Object> model = out;
+ @SuppressWarnings("unchecked")
+ List<Object> exts = (List<Object>) model.get("custom_extensions");
+ @SuppressWarnings("unchecked")
+ String blob = (String) ((Map<String, Object>) exts.get(0)).get("data");
+ assertEquals("{\"_v\": 1, \"filter\": \"region = 'caf\\u00e9'\"}", blob,
+ "stash blob must use a lowercase \\u escape");
+ // And it still round-trips back to the original view.
+ String back = OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml;
+ assertEquals(OssieConverter.parseYaml(mv), OssieConverter.parseYaml(back));
+ }
+
+ @Test
+ public void stashPreservesAValueContainingALiteralUnicodeEscape() {
+ // A stashed value may itself contain the text of a unicode escape. Serialized, that is a
+ // DOUBLED backslash, so the hex-lowercasing pass must not treat it as a real escape --
+ // doing so silently lowercases the value's own characters.
+ // Single-quoted YAML: a backslash is an ordinary character there, so `filter` really holds
+ // the six characters \ u A B C D rather than the character U+ABCD.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "filter: 'tag = \\uABCD'\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus}\n";
+ @SuppressWarnings("unchecked")
+ Map<String, Object> parsedIn = (Map<String, Object>) OssieConverter.parseYaml(mv);
+ String original = (String) parsedIn.get("filter");
+ // Guard the fixture itself: the value must contain a real backslash for this to be a test.
+ assertTrue(original.indexOf('\\') >= 0,
+ "test setup: filter must hold a literal backslash, got: " + original);
+
+ String ossieYaml = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ String back = OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml;
+ @SuppressWarnings("unchecked")
+ Map<String, Object> restored = (Map<String, Object>) OssieConverter.parseYaml(back);
+ assertEquals(original, restored.get("filter"),
+ "a literal unicode-escape sequence in a stashed value must survive unchanged");
+ }
+
+ @Test
+ public void joinOnTakesPrecedenceOverUsing() {
+ // Metric View validation requires only that one of `on`/`using` is present, so both may be
+ // set. Databricks resolves the criteria from `on` when it is present, so the converter must
+ // decompose `on` and ignore `using` -- otherwise the relationship joins on other columns.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: cust\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey = cust.c_custkey\n"
+ + " using: [nation_key]\n"
+ + "dimensions:\n"
+ + "- {name: c_name, expr: cust.c_name}\n"
+ + "measures:\n"
+ + "- {name: cnt, expr: COUNT(1)}\n";
+ @SuppressWarnings("unchecked")
+ Map<String, Object> out = (Map<String, Object>) OssieConverter.parseYaml(
+ OssieConverter.convertMetricViewToOssie(mv, null).yaml);
+ Map<String, Object> model = out;
+ @SuppressWarnings("unchecked")
+ List<Object> rels = (List<Object>) model.get("relationships");
+ @SuppressWarnings("unchecked")
+ Map<String, Object> rel = (Map<String, Object>) rels.get(0);
+ assertEquals(List.of("o_custkey"), rel.get("from_columns"),
+ "`on` must win over `using`: expected the o_custkey/c_custkey pair");
+ assertEquals(List.of("c_custkey"), rel.get("to_columns"),
+ "`on` must win over `using`: expected the o_custkey/c_custkey pair");
+ }
+
+ @Test
+ public void measureRewriteLeavesStringLiteralsAlone() {
+ // The fact qualifier is added/stripped by rewriting the measure expression. That rewrite must
+ // skip string literals: rewriting inside one changes the predicate and therefore the value.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus}\n"
+ + "measures:\n"
+ + "- name: tagged\n"
+ + " expr: \"SUM(IF(source.region = 'source.us', 1, 0))\"\n";
+ String ossieYaml = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ assertTrue(ossieYaml.contains("'source.us'"),
+ "a literal mentioning the qualifier must not be rewritten, got:\n" + ossieYaml);
+ // The literal also survives the trip back. Note the *code* qualifier is normalized on the
+ // way through (`source.region` -> bare `region`, the Metric View idiom for fact columns);
+ // only the literal is required to come back byte-identical.
+ String back = OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml;
+ assertTrue(back.contains("'source.us'"),
+ "the literal must survive the round trip, got:\n" + back);
+ }
+
+ /** A fact with `customer` joined to it and `region` nested under `customer`, plus one metric. */
+ private static String nestedJoinModel(String metricExpr) {
+ return "version: \"0.2.0.dev0\"\n"
+ + "name: sales\n"
+ + "datasets:\n"
+ + " - name: lineitem\n"
+ + " source: cat.sch.lineitem\n"
+ + " fields:\n"
+ + " - name: l_key\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: l_key}]\n"
+ + " - name: customer\n"
+ + " source: cat.sch.customer\n"
+ + " primary_key: [c_key]\n"
+ + " - name: region\n"
+ + " source: cat.sch.region\n"
+ + " primary_key: [r_key]\n"
+ + "relationships:\n"
+ + " - name: l_to_c\n"
+ + " from: lineitem\n"
+ + " to: customer\n"
+ + " from_columns: [c_key]\n"
+ + " to_columns: [c_key]\n"
+ + " - name: c_to_r\n"
+ + " from: customer\n"
+ + " to: region\n"
+ + " from_columns: [r_key]\n"
+ + " to_columns: [r_key]\n"
+ + "metrics:\n"
+ + " - name: pop\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: \"" + metricExpr + "\"}]\n";
+ }
+
+ @SuppressWarnings("unchecked")
+ private static String firstMeasureExpr(Object view) {
+ Map<String, Object> mv = (Map<String, Object>) view;
+ List<Object> measures = (List<Object>) mv.get("measures");
+ assertFalse(measures == null || measures.isEmpty(), "expected one measure, got: " + mv);
+ return (String) ((Map<String, Object>) measures.get(0)).get("expr");
+ }
+
+ @Test
+ public void exportQualifiesANestedMeasureWithTheFullJoinPath() {
+ // A Metric View addresses a nested join column by its full path, so a metric naming the
+ // dataset must come out as `customer.region.population`.
+ assertEquals("SUM(customer.region.population)",
+ firstMeasureExpr(export(nestedJoinModel("SUM(region.population)"), null)));
+ }
+
+ @Test
+ public void exportDoesNotQualifyAMeasurePathTwice() {
+ // The rewrite used to run one dataset name at a time, so it matched the `region.` inside the
+ // path it had just written and produced `SUM(customer.customer.region.population)` -- silently,
+ // with no notice. An expression that already carries the full path (what an import leaves
+ // behind) must come out unchanged.
+ assertEquals("SUM(customer.region.population)",
+ firstMeasureExpr(export(nestedJoinModel("SUM(customer.region.population)"), null)));
+ }
+
+ @Test
+ public void nestedQualifiedMeasureRoundTripsBothWays() {
+ String mv =
+ "version: '1.1'\n"
+ + "source: cat.sch.lineitem\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: cat.sch.customer\n"
+ + " using: [c_key]\n"
+ + " joins:\n"
+ + " - name: region\n"
+ + " source: cat.sch.region\n"
+ + " using: [r_key]\n"
+ + "dimensions:\n"
+ + "- {name: l_key, expr: l_key}\n"
+ + "measures:\n"
+ + "- {name: pop, expr: SUM(customer.region.population)}\n";
+ String ossieYaml = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ // The import de-aliases the path down to the dataset that owns the column...
+ assertTrue(ossieYaml.contains("SUM(region.population)"),
+ "import should de-alias the join path to the dataset name, got:\n" + ossieYaml);
+ // ...and the export puts it back, so the measure survives MV -> Ossie -> MV.
+ assertEquals("SUM(customer.region.population)",
+ firstMeasureExpr(OssieConverter.parseYaml(
+ OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml)));
+ }
+
+ @Test
+ public void cascadeDropIgnoresADroppedNameInsideAStringLiteral() {
+ // A field dropped for lacking a usable dialect must not take an unrelated measure with it just
+ // because its name appears in a string literal: `'us'` is not a reference to a column `us`.
+ String osi =
+ "version: \"0.2.0.dev0\"\n"
+ + "name: sales\n"
+ + "datasets:\n"
+ + " - name: orders\n"
+ + " source: cat.sch.orders\n"
+ + " fields:\n"
+ + " - name: amount\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: amount}]\n"
+ + " - name: us\n"
+ + " expression:\n"
+ + " dialects: [{dialect: SNOWFLAKE, expression: us_col}]\n"
+ + "metrics:\n"
+ + " - name: amt_us\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - dialect: DATABRICKS\n"
+ + " expression: \"SUM(IF(region = 'us', amount, 0))\"\n";
+ OssieConverter.Result result = OssieConverter.convertOssieToMetricView(osi, null);
+ assertEquals("SUM(IF(region = 'us', amount, 0))",
+ firstMeasureExpr(OssieConverter.parseYaml(result.yaml)));
+ for (String notice : result.notices) {
+ assertFalse(notice.contains("amt_us") && notice.contains("references dropped"),
+ "the measure only mentions 'us' in a literal, got notice: " + notice);
+ }
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void cascadeDropMatchesADroppedNameCaseInsensitively() {
+ // Databricks SQL identifiers are case-insensitive. A metric that references a dropped field in a
+ // different case (COUNT(DISTINCT REGION_NAME) over a dropped region_name) must cascade-drop
+ // rather than survive as a dangling reference.
+ String osi =
+ "version: \"0.2.0.dev0\"\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + " - name: d\n"
+ + " source: cat.sch.t\n"
+ + " fields:\n"
+ + " - name: id\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: id}]\n"
+ + " - name: region_name\n"
+ + " expression:\n"
+ + " dialects: [{dialect: T_SQL, expression: region_name}]\n"
+ + "metrics:\n"
+ + " - name: region_count\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: COUNT(DISTINCT REGION_NAME)}]\n";
+ OssieConverter.Result result = OssieConverter.convertOssieToMetricView(osi, null);
+ Map<String, Object> view = (Map<String, Object>) OssieConverter.parseYaml(result.yaml);
+ // The measure references the dropped field in upper case, so it is cascade-dropped.
+ assertFalse(view.containsKey("measures"),
+ "region_count must cascade-drop, got: " + view.get("measures"));
+ // The unrelated dimension survives.
+ assertEquals(1, ((List<Object>) view.get("dimensions")).size());
+ assertTrue(result.notices.contains(cascadeNotice("measure", "region_count", "region_name")),
+ result.notices.toString());
+ }
+
+ @Test
+ public void stashPreservesABackslashBeforeAUnicodeEscape() {
+ // The stash lowercases the hex of Jackson's \\uXXXX escapes. That rewrite must re-emit any
+ // escaped-backslash run in front of the escape verbatim; it used to halve it, so a value
+ // holding a literal backslash came back as the six characters "u00e9".
+ String value = "tag = \\" + "é";
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "filter: '" + value + "'\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus}\n";
+ String ossieYaml = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ @SuppressWarnings("unchecked")
+ Map<String, Object> back = (Map<String, Object>) OssieConverter.parseYaml(
+ OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml);
+ assertEquals(value, back.get("filter"),
+ "the stashed filter must survive MV -> Ossie -> MV byte-for-byte");
+ }
+
+ @Test
+ public void exportWarnsWhenAColumnAiContextObjectIsDropped() {
+ // `ai_context` is `string | object`; only the object's `synonyms` has a Metric View slot, so
+ // the other members are dropped -- and the README promises a notice when that happens. The
+ // same goes for foreign-vendor extensions on a metric.
+ String osi =
+ "version: \"0.2.0.dev0\"\n"
+ + "name: sales\n"
+ + "datasets:\n"
+ + " - name: orders\n"
+ + " source: cat.sch.orders\n"
+ + " fields:\n"
+ + " - name: amount\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: amount}]\n"
+ + " ai_context:\n"
+ + " instructions: do not use this column\n"
+ + " synonyms: [amt]\n"
+ + "metrics:\n"
+ + " - name: total\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: SUM(amount)}]\n"
+ + " ai_context:\n"
+ + " examples: [how much did we sell]\n"
+ + " custom_extensions:\n"
+ + " - vendor_name: SNOWFLAKE\n"
+ + " data: \"{}\"\n";
+ List<String> notices = OssieConverter.convertOssieToMetricView(osi, null).notices;
+ assertTrue(notices.contains(
+ "[field 'amount'] ai_context [instructions] dropped "
+ + "(only 'synonyms' has a Metric View counterpart)"),
+ "expected a notice for the field's dropped ai_context members, got: " + notices);
+ assertTrue(notices.contains(
+ "[metric 'total'] ai_context [examples] dropped "
+ + "(only 'synonyms' has a Metric View counterpart)"),
+ "expected a notice for the metric's dropped ai_context members, got: " + notices);
+ assertTrue(notices.contains("[metric 'total'] foreign-vendor custom_extensions dropped"),
+ "expected a notice for the metric's foreign-vendor extensions, got: " + notices);
+ }
+
+ @Test
+ public void importValidatesAJoinSourceTheWayTheExportDoes() {
+ // A 2-part join source used to import cleanly and then fail on the way back out, so a view
+ // that imported could not be exported. Reject it at the same point either direction would.
+ String mv =
+ "version: '1.1'\n"
+ + "source: cat.sch.orders\n"
+ + "joins:\n"
+ + "- name: d\n"
+ + " source: sch.d\n"
+ + " using: [k]\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus}\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertMetricViewToOssie(mv, null));
+ assertTrue(e.getMessage().contains("3-part"),
+ "expected the same source-shape error the export raises, got: " + e.getMessage());
+ }
+
+ // -- hardening: strict schema/stash parsing and structured failures -------
+
+ @Test
+ public void bareNullOptionalCollectionsConvertLikeAbsent() {
+ // A present-but-null optional collection (the bare `relationships:` YAML idiom) must convert
+ // like an absent key, as the base reader (asList) did. Only a scalar/map there is rejected.
+ String yaml = "version: '" + OssieConverter.OSSIE_VERSION + "'\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: d\n"
+ + " source: c.s.t\n"
+ + " fields:\n"
+ + " - name: region\n"
+ + " expression: {dialects: [{dialect: DATABRICKS, expression: region}]}\n"
+ + "relationships:\n"
+ + "metrics:\n"
+ + "- name: cnt\n"
+ + " expression: {dialects: [{dialect: DATABRICKS, expression: 'count(*)'}]}\n";
+ String mv = OssieConverter.convertOssieToMetricView(yaml, null).yaml;
+ assertTrue(mv.contains("cnt"), "expected the model to convert, got: " + mv);
+ }
+
+ @Test
+ public void invalidOssieInputHasStructuredFailureDetails() {
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView("just a scalar", null));
+ assertEquals(OssieConverter.ConversionException.Kind.INVALID_INPUT, e.getKind());
+ assertEquals("it is not a mapping at the root", e.getReason());
+ }
+
+ @Test
+ public void malformedSchemaCollectionsAreRejected() {
+ assertInvalidOssieInput(
+ "version: '" + OssieConverter.OSSIE_VERSION + "'\nsemantic_model:\n- name: m\n",
+ "Legacy 'semantic_model' wrappers are not supported; place the model properties "
+ + "directly at the document root");
+ assertInvalidOssieInput(
+ "version: '" + OssieConverter.OSSIE_VERSION + "'\ndatasets: []\n",
+ "Apache Ossie model requires a string 'name' at the document root");
+ assertInvalidOssieInput(
+ ossieModelWithBody(" datasets: nope\n"),
+ "Model 'm': 'datasets' must be a list");
+ assertInvalidOssieInput(
+ ossieModelWithBody(" datasets:\n - nope\n"),
+ "Model 'm': 'datasets[0]' must be a mapping");
+
+ String dataset =
+ " datasets:\n"
+ + " - name: d\n"
+ + " source: c.s.t\n";
+ assertInvalidOssieInput(
+ ossieModelWithBody(dataset + " relationships: nope\n"),
+ "Model 'm': 'relationships' must be a list");
+ assertInvalidOssieInput(
+ ossieModelWithBody(dataset + " relationships:\n - nope\n"),
+ "Model 'm': 'relationships[0]' must be a mapping");
+ assertInvalidOssieInput(
+ ossieModelWithBody(dataset + " fields: nope\n"),
+ "Dataset 'd': 'fields' must be a list");
+ assertInvalidOssieInput(
+ ossieModelWithBody(dataset + " fields:\n - nope\n"),
+ "Dataset 'd': 'fields[0]' must be a mapping");
+ assertInvalidOssieInput(
+ ossieModelWithBody(dataset + " metrics: nope\n"),
+ "Model 'm': 'metrics' must be a list");
+ assertInvalidOssieInput(
+ ossieModelWithBody(dataset + " metrics:\n - nope\n"),
+ "Model 'm': 'metrics[0]' must be a mapping");
+ }
+
+ @Test
+ public void malformedExpressionCollectionsAreRejected() {
+ String datasetPrefix =
+ " datasets:\n"
+ + " - name: d\n"
+ + " source: c.s.t\n"
+ + " fields:\n"
+ + " - name: id\n";
+ assertInvalidOssieInput(
+ ossieModelWithBody(datasetPrefix + " expression: nope\n"),
+ "field 'id': 'expression' must be a mapping");
+ assertInvalidOssieInput(
+ ossieModelWithBody(datasetPrefix + " expression: {dialects: nope}\n"),
+ "field 'id': 'expression.dialects' must be a list");
+ assertInvalidOssieInput(
+ ossieModelWithBody(datasetPrefix + " expression: {dialects: [nope]}\n"),
+ "field 'id': 'expression.dialects[0]' must be a mapping");
+
+ String validField = datasetPrefix
+ + " expression:\n"
+ + " dialects:\n"
+ + " - {dialect: DATABRICKS, expression: id}\n";
+ assertInvalidOssieInput(
+ ossieModelWithBody(validField
+ + " metrics:\n"
+ + " - name: count_id\n"
+ + " expression: {dialects: nope}\n"),
+ "metric 'count_id': 'expression.dialects' must be a list");
+ }
+
+ @Test
+ public void malformedCustomExtensionsAreRejected() {
+ String validModelBody =
+ " datasets:\n"
+ + " - name: d\n"
+ + " source: c.s.t\n"
+ + " fields:\n"
+ + " - name: id\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - {dialect: DATABRICKS, expression: id}\n";
+ assertInvalidOssieInput(
+ ossieModelWithBody(
+ " custom_extensions: {vendor_name: DATABRICKS, data: '{}'}\n" + validModelBody),
+ "'custom_extensions' must be a list");
+ assertInvalidOssieInput(
+ ossieModelWithBody(" custom_extensions: [nope]\n" + validModelBody),
+ "'custom_extensions[0]' must be a mapping");
+ assertInvalidOssieInput(
+ ossieModelWithBody(
+ " custom_extensions:\n"
+ + " - {vendor_name: DATABRICKS, data: 123}\n"
+ + validModelBody),
+ "DATABRICKS custom_extensions data must be a string");
+ assertInvalidOssieInput(
+ ossieModelWithBody(
+ " custom_extensions:\n"
+ + " - {vendor_name: DATABRICKS, data: '[]'}\n"
+ + validModelBody),
+ "DATABRICKS custom_extensions data must be a JSON object");
+
+ for (String data : List.of(
+ "filter: x = 1",
+ "{\"filter\": \"x\", \"filter\": \"y\"}",
+ "{\"filter\": \"x\"} true")) {
+ String yaml = ossieModelWithBody(
+ " custom_extensions:\n"
+ + " - vendor_name: DATABRICKS\n"
+ + " data: '" + data + "'\n"
+ + validModelBody);
+ OssieConverter.ConversionException e =
+ assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(yaml, null));
+ assertEquals(OssieConverter.ConversionException.Kind.INVALID_INPUT, e.getKind());
+ assertTrue(e.getReason().startsWith(
+ "DATABRICKS custom_extensions data is not valid JSON:"), e.getReason());
+ }
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void duplicateDatabricksStashKeepsFirstAndWarns() {
+ // Two DATABRICKS custom_extensions entries on one object is malformed, but rather than fail we
+ // keep the first, ignore the rest, and warn so the dropped entry is not lost silently.
+ String osi = ossieModelWithBody(
+ " custom_extensions:\n"
+ + " - {vendor_name: DATABRICKS, data: '{\"filter\": \"a = 1\"}'}\n"
+ + " - {vendor_name: DATABRICKS, data: '{\"filter\": \"b = 2\"}'}\n"
+ + " datasets:\n"
+ + " - name: d\n"
+ + " source: c.s.t\n"
+ + " fields:\n"
+ + " - {name: id, expression: {dialects: [{dialect: DATABRICKS, expression: id}]}}\n");
+ OssieConverter.Result result = OssieConverter.convertOssieToMetricView(osi, null);
+ Map<String, Object> view = (Map<String, Object>) OssieConverter.parseYaml(result.yaml);
+ assertEquals("a = 1", view.get("filter"), "the first DATABRICKS entry must win");
+ assertTrue(
+ result.notices.stream().anyMatch(n ->
+ n.contains("more than one DATABRICKS custom_extensions entry")),
+ result.notices.toString());
+ }
+
+ @Test
+ public void oversizedDatasetGraphIsRejectedBeforeRecursiveTraversal() {
+ StringBuilder body = new StringBuilder(" datasets:\n");
+ for (int i = 0; i <= OssieConverterCommon.MAX_JOIN_NODES; i++) {
+ body.append(" - {name: d").append(i).append(", source: c.s.t").append(i).append("}\n");
+ }
+ body.append(" relationships:\n");
+ for (int i = 0; i < OssieConverterCommon.MAX_JOIN_NODES; i++) {
+ body.append(" - {name: r").append(i)
+ .append(", from: d").append(i)
+ .append(", to: d").append(i + 1)
+ .append(", from_columns: [id], to_columns: [id]}\n");
+ }
+
+ OssieConverter.ConversionException e =
+ assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(
+ ossieModelWithBody(body.toString()), null));
+
+ assertTrue(e.getMessage().contains("at most 200 are supported"), e.getMessage());
+ }
+
+ @Test
+ public void duplicateYamlKeysAreRejectedAtEveryDepth() {
+ String topLevel =
+ "version: '" + OssieConverter.OSSIE_VERSION + "'\n"
+ + "version: '" + OssieConverter.OSSIE_VERSION + "'\n"
+ + "name: m\n";
+ String nested = ossieModelWithBody(
+ " datasets:\n"
+ + " - name: d\n"
+ + " source: c.s.first\n"
+ + " source: c.s.second\n");
+ for (String yaml : List.of(topLevel, nested)) {
+ OssieConverter.ConversionException e =
+ assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(yaml, null));
+ assertEquals(OssieConverter.ConversionException.Kind.INVALID_INPUT, e.getKind());
+ assertTrue(e.getReason().contains("duplicate key"), e.getReason());
+ }
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void reverseOrderedDropChainIsPropagatedWithoutRepeatedFullScans() {
+ // Every field depends on the preceding field, but reverse source order means the old repeated
+ // pass implementation could discover only one new drop per pass.
+ int chainLength = 400;
+ StringBuilder body = new StringBuilder(
+ " datasets:\n"
+ + " - name: d\n"
+ + " source: c.s.d\n"
+ + " fields:\n");
+ for (int index = chainLength; index >= 1; index--) {
+ body.append(" - name: d").append(index).append("\n")
+ .append(" expression:\n")
+ .append(" dialects:\n")
+ .append(" - {dialect: DATABRICKS, expression: d")
+ .append(index - 1).append("}\n");
+ }
+ body.append(" - name: d0\n")
+ .append(" expression:\n")
+ .append(" dialects:\n")
+ .append(" - {dialect: SNOWFLAKE, expression: d0}\n")
+ .append(" metrics:\n")
+ .append(" - name: row_count\n")
+ .append(" expression:\n")
+ .append(" dialects:\n")
+ .append(" - {dialect: DATABRICKS, expression: count(*)}\n");
+
+ OssieConverter.Result result =
+ OssieConverter.convertOssieToMetricView(ossieModelWithBody(body.toString()), null);
+ Map<String, Object> view = (Map<String, Object>) OssieConverter.parseYaml(result.yaml);
+
+ assertFalse(view.containsKey("dimensions"), result.yaml);
+ assertEquals(1, ((List<Object>) view.get("measures")).size(), result.yaml);
+ assertEquals(chainLength + 1, result.notices.size(), result.notices.toString());
+ List<String> cascadeNotices = result.notices.stream()
+ .filter(notice -> notice.contains("downstream of a dropped field/metric"))
+ .toList();
+ assertEquals(chainLength, cascadeNotices.size(), cascadeNotices.toString());
+ for (int index = 1; index <= chainLength; index++) {
+ assertEquals(
+ cascadeNotice("dimension", "d" + index, "d" + (index - 1)),
+ cascadeNotices.get(index - 1));
+ }
+ }
+
+ @Test
+ public void cascadeDropPreservesDimensionThenMeasurePhaseOrder() {
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: d\n"
+ + " source: c.s.d\n"
+ + " fields:\n"
+ + " - {name: d1, expression: {dialects: "
+ + "[{dialect: DATABRICKS, expression: bad_dim}]}}\n"
+ + " - {name: d0, expression: {dialects: "
+ + "[{dialect: DATABRICKS, expression: 'measure(m1)'}]}}\n"
+ + " - {name: bad_dim, expression: {dialects: "
+ + "[{dialect: SNOWFLAKE, expression: bad_dim}]}}\n"
+ + " metrics:\n"
Review Comment:
Fixed. The `metrics:` block was nested under the dataset by mistake; moved it to the model root so the cascade test sees all four notices again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on PR #333:
URL: https://github.com/apache/ossie/pull/333#issuecomment-5485539130
Hi @jbonofre , let's pause this a bit. I want to add a few more commits to address https://github.com/apache/ossie/issues/321 and https://github.com/apache/ossie/issues/326 and some findings from our sql function bug bash. Will notify you when these are in. Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on code in PR #333:
URL: https://github.com/apache/ossie/pull/333#discussion_r4075414007
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java:
##########
@@ -0,0 +1,588 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.DIALECT_DATABRICKS;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAX_JOIN_NODES;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SELECT_WITH_RE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.lastIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * IMPORT direction: Metric View v1.1 YAML -> Apache Ossie semantic model. Shared helpers come
+ * from {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class MetricViewToOssie {
+
+ private static final String[] MODEL_STASH_KEYS = {"filter", "parameters", "materialization"};
+ private static final String[] JOIN_STASH_KEYS = {"rely", "cardinality"};
+ // Metric-View-only column fields with no Apache Ossie representation, preserved verbatim in the
+ // DATABRICKS stash so an import can restore them. `window` and `partition` are measure-only
+ // (they live on MeasureExpression); a dimension never carries them, so the shared key list is
+ // simply never hit for those on the dimension path.
+ private static final String[] COLUMN_STASH_KEYS = {"format", "window", "partition"};
+ // Measure-only fields with no Apache Ossie Metric representation. A dimension's display_name
+ // maps to the Field `label`, but the Ossie Metric schema has no `label`, so a measure's
+ // display_name is preserved in the DATABRICKS stash instead. Kept separate from
+ // COLUMN_STASH_KEYS so the dimension path (which already maps display_name to `label`) does not
+ // also stash it.
+ private static final String[] MEASURE_STASH_KEYS = {"display_name"};
+ private static final Pattern NON_EQUI_RE = Pattern.compile("[<>!]=|<>|[<>]");
+ private static final Pattern AND_SPLIT_RE = Pattern.compile("\\s+AND\\s+", Pattern.CASE_INSENSITIVE);
+ private static final Pattern EQ_CLAUSE_RE = Pattern.compile("^\\s*(.+?)\\s*=\\s*(.+?)\\s*$");
+
+ private MetricViewToOssie() {}
+
+ static Result convertMetricViewToOssie(String mvYamlStr, String modelName) {
+ Notices notices = new Notices();
+ if (mvYamlStr == null || mvYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(mvYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> view = asMap(parsed);
+ Object versionValue = get(view, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!MV_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Metric View version '" + version
+ + "'. This converter targets v" + MV_VERSION + " only.",
+ version);
+ }
+ Map<String, Object> model = convertView(view, modelName, notices);
+ Map<String, Object> out = new LinkedHashMap<>();
+ out.put("version", OSSIE_VERSION);
+ out.putAll(model);
+ return serializeOssie(out, notices);
+ }
+
+ static Result serializeOssie(Map<String, Object> out, Notices notices) {
+ try {
+ return new Result(MAPPER.writeValueAsString(out), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Apache Ossie YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertView(
+ Map<String, Object> view, String modelName, Notices notices) {
+ Object source = get(view, "source");
+ if (source == null || source.toString().isEmpty()) {
+ throw new ConversionException("Metric View is missing required 'source'");
+ }
+ // Same subquery test validateSource applies below, so the two cannot drift.
+ boolean isSql = SELECT_WITH_RE.matcher(source.toString().trim()).find();
+ String lastId = lastIdentifier(source);
+ String factName;
+ if (modelName != null) {
+ factName = modelName;
+ } else if (!isSql && lastId != null && isSimpleIdentifier(lastId)) {
+ factName = lastId;
+ } else {
+ factName = "metric_view";
+ }
+ validateSource(source, factName);
+
+ List<Map<String, Object>> datasets = new ArrayList<>();
+ Map<String, Object> factDs = new LinkedHashMap<>();
+ factDs.put("name", factName);
+ factDs.put("source", source);
+ datasets.add(factDs);
+ List<Map<String, Object>> relationships = new ArrayList<>();
+ // Joins with no schema-valid Ossie relationship (a non-decomposable `on`) are collected here
+ // and stashed under the model's DATABRICKS custom_extensions instead of `relationships`.
+ List<Map<String, Object>> complexJoins = new ArrayList<>();
+ Map<String, String> aliasToDataset = new HashMap<>();
+ aliasToDataset.put("source", factName);
+ aliasToDataset.put(factName, factName);
+ Set<String> seenNames = new HashSet<>();
+ seenNames.add(factName.trim().toLowerCase(Locale.ROOT));
+
+ walk(factName, "source", asList(get(view, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+
+ // The reverse conversion bounds how many datasets it will build (see buildJoinTree), so a
+ // Metric View that expands past that bound could not be imported again. Reject it here to keep
+ // MV -> Ossie -> MV symmetric rather than emitting a model that fails on the way back in.
+ if (datasets.size() > MAX_JOIN_NODES) {
+ throw new ConversionException("Metric View '" + factName + "' has " + datasets.size()
+ + " datasets; at most " + MAX_JOIN_NODES + " are supported.");
+ }
+
+ // `fields` is a v1.1 alias for `dimensions`: an empty `dimensions: []` falls through to
+ // `fields`, and the "both set" warning fires only when BOTH are non-empty (not merely
+ // present).
+ List<Object> dimList = asList(get(view, "dimensions"));
+ List<Object> fieldList = asList(get(view, "fields"));
+ if (!dimList.isEmpty() && !fieldList.isEmpty()) {
+ notices.warn("view", "both 'dimensions' and 'fields' are set; 'fields' is a v1.1 alias "
+ + "for 'dimensions', so the 'fields' list is ignored");
+ }
+ Map<String, List<Object>> fieldsByDataset = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasets) {
+ fieldsByDataset.put((String) d.get("name"), new ArrayList<>());
+ }
+ List<Object> dims = !dimList.isEmpty() ? dimList : fieldList;
+ for (Object dimObj : dims) {
+ Map<String, Object> dim = asMap(dimObj);
+ if (isWildcard(dim)) {
+ notices.warn("dimension", "wildcard column '" + str(get(dim, "expr"))
+ + "' has no Apache Ossie field representation; skipped");
+ continue;
+ }
+ Object[] converted = convertDimension(dim, aliasToDataset, factName);
+ fieldsByDataset.get((String) converted[0]).add(converted[1]);
+ }
+ for (Map<String, Object> d : datasets) {
+ List<Object> flds = fieldsByDataset.get((String) d.get("name"));
+ if (!flds.isEmpty()) {
+ d.put("fields", flds);
+ }
+ }
+
+ List<Object> metrics = new ArrayList<>();
+ // Fixed once the join tree is walked, so compile it once for all measures.
+ Pattern aliasHead = qualifierChainPattern(aliasToDataset.keySet());
+ for (Object mObj : asList(get(view, "measures"))) {
+ Map<String, Object> m = asMap(mObj);
+ if (isWildcard(m)) {
+ notices.warn("measure", "wildcard measure '" + str(get(m, "expr"))
+ + "' has no Apache Ossie metric representation; skipped");
+ continue;
+ }
+ metrics.add(convertMeasure(m, aliasHead, aliasToDataset));
+ }
+
+ Map<String, Object> model = new LinkedHashMap<>();
+ model.put("name", factName);
+ if (truthy(get(view, "comment"))) {
+ model.put("description", get(view, "comment"));
+ }
+ model.put("datasets", datasets);
+ if (!relationships.isEmpty()) {
+ model.put("relationships", relationships);
+ }
+ if (!metrics.isEmpty()) {
+ model.put("metrics", metrics);
+ }
+
+ Map<String, Object> modelStash = new LinkedHashMap<>();
+ for (String k : MODEL_STASH_KEYS) {
+ if (view.containsKey(k)) {
+ modelStash.put(k, view.get(k));
+ }
+ }
+ if (hasOtm(asList(get(view, "joins")))) {
+ modelStash.put(STASH_SOURCE_KEY, factName);
+ }
+ if (!complexJoins.isEmpty()) {
+ modelStash.put("complex_joins", complexJoins);
+ }
+ writeStash(model, modelStash);
+ return model;
+ }
+
+ private static void walk(String parentName, String parentAlias, List<Object> joins,
+ List<Map<String, Object>> datasets, List<Map<String, Object>> relationships,
+ List<Map<String, Object>> complexJoins, Map<String, String> aliasToDataset,
+ Set<String> seenNames, Notices notices) {
+ for (Object joinObj : joins) {
+ Map<String, Object> join = asMap(joinObj);
+ String child = requireStr(join, "name", "join");
+ if (child.trim().equalsIgnoreCase("source")) {
+ throw new ConversionException(
+ "Join name 'source' is reserved for the fact source; rename the join.");
+ }
+ if (!seenNames.add(child.trim().toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("Duplicate dataset/join name '" + child
+ + "'; Metric View join names and the source must be distinct (case-insensitively).");
+ }
+ Map<String, Object> childDs = new LinkedHashMap<>();
+ childDs.put("name", child);
+ // Validated with the same rule the export applies to a dataset source, so a Metric View that
+ // imports cleanly is always exportable again (a 2-part join source used to import fine and
+ // then fail on the way back out).
+ String childSource = requireStr(join, "source", "join '" + child + "'");
+ validateSource(childSource, child);
+ childDs.put("source", childSource);
+ datasets.add(childDs);
+ aliasToDataset.put(child, child);
+ Map<String, Object> rel = convertJoin(join, parentName, parentAlias, child, notices);
+ // convertJoin omits from/to columns for a non-decomposable `on`; those entries have no valid
+ // Ossie relationship form, so they go to the model-level stash instead of `relationships`.
+ if (rel.containsKey("from_columns")) {
+ relationships.add(rel);
+ } else {
+ complexJoins.add(rel);
+ }
+ // rely.at_most_one_match on a many_to_one join -> recover a unique_key on the child.
+ Map<String, Object> rely = asMap(get(join, "rely"));
+ List<String> toCols = strList(get(rel, "to_columns"));
+ if (child.equals(str(get(rel, "to"))) && !toCols.isEmpty()
+ && Boolean.TRUE.equals(rely.get("at_most_one_match"))) {
+ List<Object> uk = new ArrayList<>();
+ uk.add(new ArrayList<>(toCols));
+ childDs.put("unique_keys", uk);
+ }
+ walk(child, child, asList(get(join, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+ }
+ }
+
+ private static boolean hasOtm(List<Object> joins) {
+ for (Object jObj : joins) {
+ Map<String, Object> j = asMap(jObj);
+ if (CARD_ONE_TO_MANY.equalsIgnoreCase(str(get(j, "cardinality")))) {
+ return true;
+ }
+ if (hasOtm(asList(get(j, "joins")))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static Map<String, Object> convertJoin(
+ Map<String, Object> join, String parentName, String parentAlias, String child,
+ Notices notices) {
+ boolean hasUsing = get(join, "using") != null && !asList(get(join, "using")).isEmpty();
+ boolean hasOn = str(get(join, "on")) != null && !str(get(join, "on")).isEmpty();
+ if (!hasUsing && !hasOn) {
+ throw new ConversionException("Join '" + child + "' has no join condition (empty or "
+ + "absent 'on'/'using'); condition-less (cross) joins have no Apache Ossie "
+ + "relationship representation.");
+ }
+ Object[] decomposed = decomposeOn(join, parentAlias, parentName, child);
+ List<String> parentCols = (List<String>) decomposed[0];
+ List<String> childCols = (List<String>) decomposed[1];
+ String rawOn = (String) decomposed[2];
+ if (rawOn != null) {
Review Comment:
hmmm, @christianeu-db can you tal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on PR #333:
URL: https://github.com/apache/ossie/pull/333#issuecomment-5625003155
Hi @jbonofre
Thanks for the careful pass, these are good catches. Some context that ties several together: the Java converter is now the maintained, on track implementation. It tracks the current Metric View engine rules, and it carries the newer decisions (for example the complex_joins stash). The Python converter is the original reference and is on its way to deprecation, so we are intentionally letting it keep some stale behavior rather than chasing parity into a copy we are retiring.
So where Java and Python differ, Java is the intended source of truth:
- #3 (cardinality nesting): Java matches the engine, which rejects a mixed cardinality branch in either direction. Python is looser and would emit a view the engine refuses at CREATE, so we are not matching Python here.
- #5 (non equi joins): Java stashes them via complex_joins. Python still rejects. This divergence is intentional and tracked.
The other three are improvements we applied:
- #1: drop a measure that names a diamond dataset, with a notice.
- #2: bound the export join count by MAX_JOIN_NODES so the round trip stays symmetric.
- #4: keep the first DATABRICKS stash entry and warn instead of failing on a duplicate.
Please help re-verify, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on PR #333:
URL: https://github.com/apache/ossie/pull/333#issuecomment-5798992904
Also ported apache/ossie#446 (OSSIE_SQL_2026) to the Java converter: pickExpression now falls back `DATABRICKS`, then `ANSI_SQL`, then `OSSIE_SQL_2026`, with the two drop warnings and four tests updated to match. Merged latest main in, so the python/ copy picks up #446 too and stays in sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on code in PR #333:
URL: https://github.com/apache/ossie/pull/333#discussion_r3983186371
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieToMetricView.java:
##########
@@ -0,0 +1,1167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAX_JOIN_NODES;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SYNONYM_LIMIT;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.findOutsideLiterals;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.foreignVendorExtensions;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.mergeDescription;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.pickExpression;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.readStash;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.replaceOutsideLiterals;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.require;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.synonymsOf;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * EXPORT direction: Apache Ossie semantic model -> Databricks Metric View v1.1 YAML. This is the
+ * harder direction: it reassembles a relationship graph into a join tree. Shared helpers come from
+ * {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class OssieToMetricView {
+
+ private OssieToMetricView() {}
+
+ // -- tree node ------------------------------------------------------------
+ private static final class Node {
+ final String dataset;
+ final Map<String, Object> rel; // empty for the fact/root
+ final boolean parentIsFrom;
+ String alias;
+ boolean isOtm;
+ final List<Node> children = new ArrayList<>();
+
+ Node(String dataset, Map<String, Object> rel, boolean parentIsFrom) {
+ this.dataset = dataset;
+ this.rel = rel;
+ this.parentIsFrom = parentIsFrom;
+ }
+ }
+
+ // -- public entry ---------------------------------------------------------
+ static Result convertOssieToMetricView(String osiYamlStr, String source) {
+ Notices notices = new Notices();
+ if (osiYamlStr == null || osiYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(osiYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> root = asMap(parsed);
+ Object versionValue = get(root, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!OSSIE_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Apache Ossie version '" + version + "'. Supported: " + OSSIE_VERSION,
+ version);
+ }
+ List<Object> models = schemaList(root, "semantic_model", "Apache Ossie YAML");
+ if (models.isEmpty()) {
+ throw ConversionException.invalidInput(
+ "'semantic_model' must be a non-empty list",
+ "its 'semantic_model' must be a non-empty list");
+ }
+ if (!(models.get(0) instanceof Map)) {
+ String reason = "Apache Ossie YAML: 'semantic_model[0]' must be a mapping";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ if (models.size() > 1) {
+ notices.warn("model", "multiple semantic models found; converting only the first");
+ }
+ Map<String, Object> view = convertModel((Map<String, Object>) models.get(0), source, notices);
+ try {
+ return new Result(MAPPER.writeValueAsString(view), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Metric View YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertModel(
+ Map<String, Object> model, String explicitSource, Notices notices) {
+ String name = model.containsKey("name") ? str(get(model, "name")) : "<unnamed>";
+ String modelScope = "Model '" + name + "'";
+ List<Map<String, Object>> datasetList = schemaMapList(model, "datasets", modelScope);
+ if (datasetList.isEmpty()) {
+ throw new ConversionException("Model '" + name + "' has no datasets");
+ }
+ Set<String> seen = new HashSet<>();
+ Map<String, Map<String, Object>> datasets = new LinkedHashMap<>();
+ Map<String, List<Map<String, Object>>> datasetFields = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasetList) {
+ String dsName = requireStr(d, "name", "Model '" + name + "': dataset");
+ if (!seen.add(dsName.trim().toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("Model '" + name + "': duplicate dataset name '" + dsName + "'");
+ }
+ datasets.put(dsName, d);
+ datasetFields.put(dsName, schemaMapList(d, "fields", "Dataset '" + dsName + "'"));
+ }
+ List<Map<String, Object>> relationships = schemaMapList(model, "relationships", modelScope);
+ List<Map<String, Object>> metrics = schemaMapList(model, "metrics", modelScope);
+
+ Map<String, Object> modelStash = readStash(model);
+ // A non-equi/filtered join has no schema-valid Ossie relationship (from/to columns are
+ // required), so MetricViewToOssie stashes it under the model's DATABRICKS custom_extensions
+ // (complex_joins) rather than emitting a stub relationship. Rebuild a columns-less
+ // relationship for each -- carrying its raw `on` (and rely/cardinality) in the stash that
+ // buildJoin restores from -- and merge them in so the join tree includes them.
+ for (Map<String, Object> pj : schemaMapList(modelStash, "complex_joins", modelScope)) {
+ Map<String, Object> rel = new LinkedHashMap<>();
+ rel.put("name", get(pj, "name"));
+ rel.put("from", get(pj, "from"));
+ rel.put("to", get(pj, "to"));
+ Map<String, Object> stash = new LinkedHashMap<>(pj);
+ stash.remove("name");
+ stash.remove("from");
+ stash.remove("to");
+ writeStash(rel, stash);
+ relationships.add(rel);
+ }
+ String factHint = explicitSource != null ? explicitSource : str(get(modelStash, STASH_SOURCE_KEY));
+ Object[] built = buildJoinTree(name, datasets, relationships, factHint, notices);
+ Node root = (Node) built[0];
+ String fact = (String) built[1];
+ Map<String, Integer> counts = assignAliases(root, fact);
+ markOtm(name, root);
+
+ Map<String, Object> factDs = datasets.get(fact);
+ Map<String, Object> view = new LinkedHashMap<>();
+ view.put("version", MV_VERSION);
+ view.put("source", validateSource(get(factDs, "source"), fact));
+
+ String comment = str(get(model, "description"));
+ if (truthy(comment)) {
+ view.put("comment", comment);
+ }
+ if (modelStash.containsKey("filter")) {
+ view.put("filter", modelStash.get("filter"));
+ }
+
+ List<Object> joins = new ArrayList<>();
+ for (Node child : root.children) {
+ joins.add(buildJoin(child, "source", datasets));
+ }
+ if (!joins.isEmpty()) {
+ view.put("joins", joins);
+ }
+
+ Set<String> droppedDims = new HashSet<>();
+ Set<String> droppedMeasures = new HashSet<>();
+ List<Map<String, Object>> dimensions = new ArrayList<>();
+ Set<String> seenDims = new HashSet<>();
+ // Dataset name -> the alias path that addresses its columns from the primary source, collected
+ // from the same walk the dimensions use so measures qualify identically (see qualifyMeasure).
+ Map<String, String> datasetAliasPath = new LinkedHashMap<>();
+ for (Object[] entry : nodeOrder(root)) {
+ Node node = (Node) entry[0];
+ @SuppressWarnings("unchecked")
+ List<String> joinPath = (List<String>) entry[1];
+ boolean isFact = node == root;
+ String prefix = counts.get(node.dataset) > 1 ? node.alias : null;
+ // Fixed per node, like `prefix`: hoisted so it is not rebuilt for every field.
+ String qualifier = String.join(".", joinPath);
+ if (!isFact) {
+ // A dataset reachable by more than one path (diamond) appears once per path; the first
+ // wins, matching the order dimensions are emitted in.
+ datasetAliasPath.putIfAbsent(node.dataset, qualifier);
Review Comment:
Good catch. Fixed: a measure that names a diamond dataset by its bare name is now dropped with a notice, the same way the dimension path drops a complex expression it cannot unambiguously qualify. A measure addresses a dataset by name, so it cannot be split per branch the way a dimension column is, which makes warn and drop the consistent choice. Added a test measureNamingADiamondDatasetIsDroppedWithNotice covering the orders to dim1/dim2 to shared example.
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java:
##########
@@ -0,0 +1,581 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.DIALECT_DATABRICKS;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SELECT_WITH_RE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.lastIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * IMPORT direction: Metric View v1.1 YAML -> Apache Ossie semantic model. Shared helpers come
+ * from {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class MetricViewToOssie {
+
+ private static final String[] MODEL_STASH_KEYS = {"filter", "parameters", "materialization"};
+ private static final String[] JOIN_STASH_KEYS = {"rely", "cardinality"};
+ // Metric-View-only column fields with no Apache Ossie representation, preserved verbatim in the
+ // DATABRICKS stash so an import can restore them. `window` and `partition` are measure-only
+ // (they live on MeasureExpression); a dimension never carries them, so the shared key list is
+ // simply never hit for those on the dimension path.
+ private static final String[] COLUMN_STASH_KEYS = {"format", "window", "partition"};
+ // Measure-only fields with no Apache Ossie Metric representation. A dimension's display_name
+ // maps to the Field `label`, but the Ossie Metric schema has no `label`, so a measure's
+ // display_name is preserved in the DATABRICKS stash instead. Kept separate from
+ // COLUMN_STASH_KEYS so the dimension path (which already maps display_name to `label`) does not
+ // also stash it.
+ private static final String[] MEASURE_STASH_KEYS = {"display_name"};
+ private static final Pattern NON_EQUI_RE = Pattern.compile("[<>!]=|<>|[<>]");
+ private static final Pattern AND_SPLIT_RE = Pattern.compile("\\s+AND\\s+", Pattern.CASE_INSENSITIVE);
+ private static final Pattern EQ_CLAUSE_RE = Pattern.compile("^\\s*(.+?)\\s*=\\s*(.+?)\\s*$");
+
+ private MetricViewToOssie() {}
+
+ static Result convertMetricViewToOssie(String mvYamlStr, String modelName) {
+ Notices notices = new Notices();
+ if (mvYamlStr == null || mvYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(mvYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> view = asMap(parsed);
+ Object versionValue = get(view, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!MV_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Metric View version '" + version
+ + "'. This converter targets v" + MV_VERSION + " only.",
+ version);
+ }
+ Map<String, Object> model = convertView(view, modelName, notices);
+ Map<String, Object> out = new LinkedHashMap<>();
+ out.put("version", OSSIE_VERSION);
+ List<Object> models = new ArrayList<>();
+ models.add(model);
+ out.put("semantic_model", models);
+ return serializeOssie(out, notices);
+ }
+
+ static Result serializeOssie(Map<String, Object> out, Notices notices) {
+ try {
+ return new Result(MAPPER.writeValueAsString(out), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Apache Ossie YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertView(
+ Map<String, Object> view, String modelName, Notices notices) {
+ Object source = get(view, "source");
+ if (source == null || source.toString().isEmpty()) {
+ throw new ConversionException("Metric View is missing required 'source'");
+ }
+ // Same subquery test validateSource applies below, so the two cannot drift.
+ boolean isSql = SELECT_WITH_RE.matcher(source.toString().trim()).find();
+ String lastId = lastIdentifier(source);
+ String factName;
+ if (modelName != null) {
+ factName = modelName;
+ } else if (!isSql && lastId != null && isSimpleIdentifier(lastId)) {
+ factName = lastId;
+ } else {
+ factName = "metric_view";
+ }
+ validateSource(source, factName);
+
+ List<Map<String, Object>> datasets = new ArrayList<>();
+ Map<String, Object> factDs = new LinkedHashMap<>();
+ factDs.put("name", factName);
+ factDs.put("source", source);
+ datasets.add(factDs);
+ List<Map<String, Object>> relationships = new ArrayList<>();
+ // Joins with no schema-valid Ossie relationship (a non-decomposable `on`) are collected here
+ // and stashed under the model's DATABRICKS custom_extensions instead of `relationships`.
+ List<Map<String, Object>> complexJoins = new ArrayList<>();
+ Map<String, String> aliasToDataset = new HashMap<>();
+ aliasToDataset.put("source", factName);
+ aliasToDataset.put(factName, factName);
+ Set<String> seenNames = new HashSet<>();
+ seenNames.add(factName.trim().toLowerCase(Locale.ROOT));
+
+ walk(factName, "source", asList(get(view, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+
+ // `fields` is a v1.1 alias for `dimensions`: an empty `dimensions: []` falls through to
+ // `fields`, and the "both set" warning fires only when BOTH are non-empty (not merely
+ // present).
+ List<Object> dimList = asList(get(view, "dimensions"));
+ List<Object> fieldList = asList(get(view, "fields"));
+ if (!dimList.isEmpty() && !fieldList.isEmpty()) {
+ notices.warn("view", "both 'dimensions' and 'fields' are set; 'fields' is a v1.1 alias "
+ + "for 'dimensions', so the 'fields' list is ignored");
+ }
+ Map<String, List<Object>> fieldsByDataset = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasets) {
+ fieldsByDataset.put((String) d.get("name"), new ArrayList<>());
+ }
+ List<Object> dims = !dimList.isEmpty() ? dimList : fieldList;
+ for (Object dimObj : dims) {
+ Map<String, Object> dim = asMap(dimObj);
+ if (isWildcard(dim)) {
+ notices.warn("dimension", "wildcard column '" + str(get(dim, "expr"))
+ + "' has no Apache Ossie field representation; skipped");
+ continue;
+ }
+ Object[] converted = convertDimension(dim, aliasToDataset, factName);
+ fieldsByDataset.get((String) converted[0]).add(converted[1]);
+ }
+ for (Map<String, Object> d : datasets) {
+ List<Object> flds = fieldsByDataset.get((String) d.get("name"));
+ if (!flds.isEmpty()) {
+ d.put("fields", flds);
+ }
+ }
+
+ List<Object> metrics = new ArrayList<>();
+ // Fixed once the join tree is walked, so compile it once for all measures.
+ Pattern aliasHead = qualifierChainPattern(aliasToDataset.keySet());
+ for (Object mObj : asList(get(view, "measures"))) {
+ Map<String, Object> m = asMap(mObj);
+ if (isWildcard(m)) {
+ notices.warn("measure", "wildcard measure '" + str(get(m, "expr"))
+ + "' has no Apache Ossie metric representation; skipped");
+ continue;
+ }
+ metrics.add(convertMeasure(m, aliasHead, aliasToDataset));
+ }
+
+ Map<String, Object> model = new LinkedHashMap<>();
+ model.put("name", factName);
+ if (truthy(get(view, "comment"))) {
+ model.put("description", get(view, "comment"));
+ }
+ model.put("datasets", datasets);
+ if (!relationships.isEmpty()) {
+ model.put("relationships", relationships);
+ }
+ if (!metrics.isEmpty()) {
+ model.put("metrics", metrics);
+ }
+
+ Map<String, Object> modelStash = new LinkedHashMap<>();
+ for (String k : MODEL_STASH_KEYS) {
+ if (view.containsKey(k)) {
+ modelStash.put(k, view.get(k));
+ }
+ }
+ if (hasOtm(asList(get(view, "joins")))) {
+ modelStash.put(STASH_SOURCE_KEY, factName);
+ }
+ if (!complexJoins.isEmpty()) {
+ modelStash.put("complex_joins", complexJoins);
+ }
+ writeStash(model, modelStash);
+ return model;
+ }
+
+ private static void walk(String parentName, String parentAlias, List<Object> joins,
Review Comment:
Good catch, added the bound. MetricViewToOssie (MV to Ossie) now rejects a Metric View whose fact plus joins exceed MAX_JOIN_NODES, the same limit OssieToMetricView already enforces, so the two directions stay symmetric and the round trip holds. One note on the framing: the failure is actually the reverse. A Metric View with 200 or more joins exports cleanly today, but the emitted model then fails to import again, since the MAX_JOIN_NODES checks live on the Ossie to MV side. The fix rejects it at export instead. Added tests exportRejectsMoreJoinsThanImportCanRebuild and exportAtTheJoinLimitStillSucceeds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on code in PR #333:
URL: https://github.com/apache/ossie/pull/333#discussion_r4084993511
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java:
##########
@@ -0,0 +1,588 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.DIALECT_DATABRICKS;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAX_JOIN_NODES;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SELECT_WITH_RE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.lastIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * IMPORT direction: Metric View v1.1 YAML -> Apache Ossie semantic model. Shared helpers come
+ * from {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class MetricViewToOssie {
+
+ private static final String[] MODEL_STASH_KEYS = {"filter", "parameters", "materialization"};
+ private static final String[] JOIN_STASH_KEYS = {"rely", "cardinality"};
+ // Metric-View-only column fields with no Apache Ossie representation, preserved verbatim in the
+ // DATABRICKS stash so an import can restore them. `window` and `partition` are measure-only
+ // (they live on MeasureExpression); a dimension never carries them, so the shared key list is
+ // simply never hit for those on the dimension path.
+ private static final String[] COLUMN_STASH_KEYS = {"format", "window", "partition"};
+ // Measure-only fields with no Apache Ossie Metric representation. A dimension's display_name
+ // maps to the Field `label`, but the Ossie Metric schema has no `label`, so a measure's
+ // display_name is preserved in the DATABRICKS stash instead. Kept separate from
+ // COLUMN_STASH_KEYS so the dimension path (which already maps display_name to `label`) does not
+ // also stash it.
+ private static final String[] MEASURE_STASH_KEYS = {"display_name"};
+ private static final Pattern NON_EQUI_RE = Pattern.compile("[<>!]=|<>|[<>]");
+ private static final Pattern AND_SPLIT_RE = Pattern.compile("\\s+AND\\s+", Pattern.CASE_INSENSITIVE);
+ private static final Pattern EQ_CLAUSE_RE = Pattern.compile("^\\s*(.+?)\\s*=\\s*(.+?)\\s*$");
+
+ private MetricViewToOssie() {}
+
+ static Result convertMetricViewToOssie(String mvYamlStr, String modelName) {
+ Notices notices = new Notices();
+ if (mvYamlStr == null || mvYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(mvYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> view = asMap(parsed);
+ Object versionValue = get(view, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!MV_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Metric View version '" + version
+ + "'. This converter targets v" + MV_VERSION + " only.",
+ version);
+ }
+ Map<String, Object> model = convertView(view, modelName, notices);
+ Map<String, Object> out = new LinkedHashMap<>();
+ out.put("version", OSSIE_VERSION);
+ out.putAll(model);
+ return serializeOssie(out, notices);
+ }
+
+ static Result serializeOssie(Map<String, Object> out, Notices notices) {
+ try {
+ return new Result(MAPPER.writeValueAsString(out), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Apache Ossie YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertView(
+ Map<String, Object> view, String modelName, Notices notices) {
+ Object source = get(view, "source");
+ if (source == null || source.toString().isEmpty()) {
+ throw new ConversionException("Metric View is missing required 'source'");
+ }
+ // Same subquery test validateSource applies below, so the two cannot drift.
+ boolean isSql = SELECT_WITH_RE.matcher(source.toString().trim()).find();
+ String lastId = lastIdentifier(source);
+ String factName;
+ if (modelName != null) {
+ factName = modelName;
+ } else if (!isSql && lastId != null && isSimpleIdentifier(lastId)) {
+ factName = lastId;
+ } else {
+ factName = "metric_view";
+ }
+ validateSource(source, factName);
+
+ List<Map<String, Object>> datasets = new ArrayList<>();
+ Map<String, Object> factDs = new LinkedHashMap<>();
+ factDs.put("name", factName);
+ factDs.put("source", source);
+ datasets.add(factDs);
+ List<Map<String, Object>> relationships = new ArrayList<>();
+ // Joins with no schema-valid Ossie relationship (a non-decomposable `on`) are collected here
+ // and stashed under the model's DATABRICKS custom_extensions instead of `relationships`.
+ List<Map<String, Object>> complexJoins = new ArrayList<>();
+ Map<String, String> aliasToDataset = new HashMap<>();
+ aliasToDataset.put("source", factName);
+ aliasToDataset.put(factName, factName);
+ Set<String> seenNames = new HashSet<>();
+ seenNames.add(factName.trim().toLowerCase(Locale.ROOT));
+
+ walk(factName, "source", asList(get(view, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+
+ // The reverse conversion bounds how many datasets it will build (see buildJoinTree), so a
+ // Metric View that expands past that bound could not be imported again. Reject it here to keep
+ // MV -> Ossie -> MV symmetric rather than emitting a model that fails on the way back in.
+ if (datasets.size() > MAX_JOIN_NODES) {
+ throw new ConversionException("Metric View '" + factName + "' has " + datasets.size()
+ + " datasets; at most " + MAX_JOIN_NODES + " are supported.");
+ }
+
+ // `fields` is a v1.1 alias for `dimensions`: an empty `dimensions: []` falls through to
+ // `fields`, and the "both set" warning fires only when BOTH are non-empty (not merely
+ // present).
+ List<Object> dimList = asList(get(view, "dimensions"));
+ List<Object> fieldList = asList(get(view, "fields"));
+ if (!dimList.isEmpty() && !fieldList.isEmpty()) {
+ notices.warn("view", "both 'dimensions' and 'fields' are set; 'fields' is a v1.1 alias "
+ + "for 'dimensions', so the 'fields' list is ignored");
+ }
+ Map<String, List<Object>> fieldsByDataset = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasets) {
+ fieldsByDataset.put((String) d.get("name"), new ArrayList<>());
+ }
+ List<Object> dims = !dimList.isEmpty() ? dimList : fieldList;
+ for (Object dimObj : dims) {
+ Map<String, Object> dim = asMap(dimObj);
+ if (isWildcard(dim)) {
+ notices.warn("dimension", "wildcard column '" + str(get(dim, "expr"))
+ + "' has no Apache Ossie field representation; skipped");
+ continue;
+ }
+ Object[] converted = convertDimension(dim, aliasToDataset, factName);
+ fieldsByDataset.get((String) converted[0]).add(converted[1]);
+ }
+ for (Map<String, Object> d : datasets) {
+ List<Object> flds = fieldsByDataset.get((String) d.get("name"));
+ if (!flds.isEmpty()) {
+ d.put("fields", flds);
+ }
+ }
+
+ List<Object> metrics = new ArrayList<>();
+ // Fixed once the join tree is walked, so compile it once for all measures.
+ Pattern aliasHead = qualifierChainPattern(aliasToDataset.keySet());
+ for (Object mObj : asList(get(view, "measures"))) {
+ Map<String, Object> m = asMap(mObj);
+ if (isWildcard(m)) {
+ notices.warn("measure", "wildcard measure '" + str(get(m, "expr"))
+ + "' has no Apache Ossie metric representation; skipped");
+ continue;
+ }
+ metrics.add(convertMeasure(m, aliasHead, aliasToDataset));
+ }
+
+ Map<String, Object> model = new LinkedHashMap<>();
+ model.put("name", factName);
+ if (truthy(get(view, "comment"))) {
+ model.put("description", get(view, "comment"));
+ }
+ model.put("datasets", datasets);
+ if (!relationships.isEmpty()) {
+ model.put("relationships", relationships);
+ }
+ if (!metrics.isEmpty()) {
+ model.put("metrics", metrics);
+ }
+
+ Map<String, Object> modelStash = new LinkedHashMap<>();
+ for (String k : MODEL_STASH_KEYS) {
+ if (view.containsKey(k)) {
+ modelStash.put(k, view.get(k));
+ }
+ }
+ if (hasOtm(asList(get(view, "joins")))) {
+ modelStash.put(STASH_SOURCE_KEY, factName);
+ }
+ if (!complexJoins.isEmpty()) {
+ modelStash.put("complex_joins", complexJoins);
+ }
+ writeStash(model, modelStash);
+ return model;
+ }
+
+ private static void walk(String parentName, String parentAlias, List<Object> joins,
+ List<Map<String, Object>> datasets, List<Map<String, Object>> relationships,
+ List<Map<String, Object>> complexJoins, Map<String, String> aliasToDataset,
+ Set<String> seenNames, Notices notices) {
+ for (Object joinObj : joins) {
+ Map<String, Object> join = asMap(joinObj);
+ String child = requireStr(join, "name", "join");
+ if (child.trim().equalsIgnoreCase("source")) {
+ throw new ConversionException(
+ "Join name 'source' is reserved for the fact source; rename the join.");
+ }
+ if (!seenNames.add(child.trim().toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("Duplicate dataset/join name '" + child
+ + "'; Metric View join names and the source must be distinct (case-insensitively).");
+ }
+ Map<String, Object> childDs = new LinkedHashMap<>();
+ childDs.put("name", child);
+ // Validated with the same rule the export applies to a dataset source, so a Metric View that
+ // imports cleanly is always exportable again (a 2-part join source used to import fine and
+ // then fail on the way back out).
+ String childSource = requireStr(join, "source", "join '" + child + "'");
+ validateSource(childSource, child);
+ childDs.put("source", childSource);
+ datasets.add(childDs);
+ aliasToDataset.put(child, child);
+ Map<String, Object> rel = convertJoin(join, parentName, parentAlias, child, notices);
+ // convertJoin omits from/to columns for a non-decomposable `on`; those entries have no valid
+ // Ossie relationship form, so they go to the model-level stash instead of `relationships`.
+ if (rel.containsKey("from_columns")) {
+ relationships.add(rel);
+ } else {
+ complexJoins.add(rel);
+ }
+ // rely.at_most_one_match on a many_to_one join -> recover a unique_key on the child.
+ Map<String, Object> rely = asMap(get(join, "rely"));
+ List<String> toCols = strList(get(rel, "to_columns"));
+ if (child.equals(str(get(rel, "to"))) && !toCols.isEmpty()
+ && Boolean.TRUE.equals(rely.get("at_most_one_match"))) {
+ List<Object> uk = new ArrayList<>();
+ uk.add(new ArrayList<>(toCols));
+ childDs.put("unique_keys", uk);
+ }
+ walk(child, child, asList(get(join, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+ }
+ }
+
+ private static boolean hasOtm(List<Object> joins) {
+ for (Object jObj : joins) {
+ Map<String, Object> j = asMap(jObj);
+ if (CARD_ONE_TO_MANY.equalsIgnoreCase(str(get(j, "cardinality")))) {
+ return true;
+ }
+ if (hasOtm(asList(get(j, "joins")))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static Map<String, Object> convertJoin(
+ Map<String, Object> join, String parentName, String parentAlias, String child,
+ Notices notices) {
+ boolean hasUsing = get(join, "using") != null && !asList(get(join, "using")).isEmpty();
+ boolean hasOn = str(get(join, "on")) != null && !str(get(join, "on")).isEmpty();
+ if (!hasUsing && !hasOn) {
+ throw new ConversionException("Join '" + child + "' has no join condition (empty or "
+ + "absent 'on'/'using'); condition-less (cross) joins have no Apache Ossie "
+ + "relationship representation.");
+ }
+ Object[] decomposed = decomposeOn(join, parentAlias, parentName, child);
+ List<String> parentCols = (List<String>) decomposed[0];
+ List<String> childCols = (List<String>) decomposed[1];
+ String rawOn = (String) decomposed[2];
+ if (rawOn != null) {
Review Comment:
Confirmed offline with Chris, went with rejecting, like Python. Fixed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on code in PR #333:
URL: https://github.com/apache/ossie/pull/333#discussion_r3983194108
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java:
##########
@@ -0,0 +1,581 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.DIALECT_DATABRICKS;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SELECT_WITH_RE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.lastIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * IMPORT direction: Metric View v1.1 YAML -> Apache Ossie semantic model. Shared helpers come
+ * from {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class MetricViewToOssie {
+
+ private static final String[] MODEL_STASH_KEYS = {"filter", "parameters", "materialization"};
+ private static final String[] JOIN_STASH_KEYS = {"rely", "cardinality"};
+ // Metric-View-only column fields with no Apache Ossie representation, preserved verbatim in the
+ // DATABRICKS stash so an import can restore them. `window` and `partition` are measure-only
+ // (they live on MeasureExpression); a dimension never carries them, so the shared key list is
+ // simply never hit for those on the dimension path.
+ private static final String[] COLUMN_STASH_KEYS = {"format", "window", "partition"};
+ // Measure-only fields with no Apache Ossie Metric representation. A dimension's display_name
+ // maps to the Field `label`, but the Ossie Metric schema has no `label`, so a measure's
+ // display_name is preserved in the DATABRICKS stash instead. Kept separate from
+ // COLUMN_STASH_KEYS so the dimension path (which already maps display_name to `label`) does not
+ // also stash it.
+ private static final String[] MEASURE_STASH_KEYS = {"display_name"};
+ private static final Pattern NON_EQUI_RE = Pattern.compile("[<>!]=|<>|[<>]");
+ private static final Pattern AND_SPLIT_RE = Pattern.compile("\\s+AND\\s+", Pattern.CASE_INSENSITIVE);
+ private static final Pattern EQ_CLAUSE_RE = Pattern.compile("^\\s*(.+?)\\s*=\\s*(.+?)\\s*$");
+
+ private MetricViewToOssie() {}
+
+ static Result convertMetricViewToOssie(String mvYamlStr, String modelName) {
+ Notices notices = new Notices();
+ if (mvYamlStr == null || mvYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(mvYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> view = asMap(parsed);
+ Object versionValue = get(view, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!MV_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Metric View version '" + version
+ + "'. This converter targets v" + MV_VERSION + " only.",
+ version);
+ }
+ Map<String, Object> model = convertView(view, modelName, notices);
+ Map<String, Object> out = new LinkedHashMap<>();
+ out.put("version", OSSIE_VERSION);
+ List<Object> models = new ArrayList<>();
+ models.add(model);
+ out.put("semantic_model", models);
+ return serializeOssie(out, notices);
+ }
+
+ static Result serializeOssie(Map<String, Object> out, Notices notices) {
+ try {
+ return new Result(MAPPER.writeValueAsString(out), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Apache Ossie YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertView(
+ Map<String, Object> view, String modelName, Notices notices) {
+ Object source = get(view, "source");
+ if (source == null || source.toString().isEmpty()) {
+ throw new ConversionException("Metric View is missing required 'source'");
+ }
+ // Same subquery test validateSource applies below, so the two cannot drift.
+ boolean isSql = SELECT_WITH_RE.matcher(source.toString().trim()).find();
+ String lastId = lastIdentifier(source);
+ String factName;
+ if (modelName != null) {
+ factName = modelName;
+ } else if (!isSql && lastId != null && isSimpleIdentifier(lastId)) {
+ factName = lastId;
+ } else {
+ factName = "metric_view";
+ }
+ validateSource(source, factName);
+
+ List<Map<String, Object>> datasets = new ArrayList<>();
+ Map<String, Object> factDs = new LinkedHashMap<>();
+ factDs.put("name", factName);
+ factDs.put("source", source);
+ datasets.add(factDs);
+ List<Map<String, Object>> relationships = new ArrayList<>();
+ // Joins with no schema-valid Ossie relationship (a non-decomposable `on`) are collected here
+ // and stashed under the model's DATABRICKS custom_extensions instead of `relationships`.
+ List<Map<String, Object>> complexJoins = new ArrayList<>();
+ Map<String, String> aliasToDataset = new HashMap<>();
+ aliasToDataset.put("source", factName);
+ aliasToDataset.put(factName, factName);
+ Set<String> seenNames = new HashSet<>();
+ seenNames.add(factName.trim().toLowerCase(Locale.ROOT));
+
+ walk(factName, "source", asList(get(view, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+
+ // `fields` is a v1.1 alias for `dimensions`: an empty `dimensions: []` falls through to
+ // `fields`, and the "both set" warning fires only when BOTH are non-empty (not merely
+ // present).
+ List<Object> dimList = asList(get(view, "dimensions"));
+ List<Object> fieldList = asList(get(view, "fields"));
+ if (!dimList.isEmpty() && !fieldList.isEmpty()) {
+ notices.warn("view", "both 'dimensions' and 'fields' are set; 'fields' is a v1.1 alias "
+ + "for 'dimensions', so the 'fields' list is ignored");
+ }
+ Map<String, List<Object>> fieldsByDataset = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasets) {
+ fieldsByDataset.put((String) d.get("name"), new ArrayList<>());
+ }
+ List<Object> dims = !dimList.isEmpty() ? dimList : fieldList;
+ for (Object dimObj : dims) {
+ Map<String, Object> dim = asMap(dimObj);
+ if (isWildcard(dim)) {
+ notices.warn("dimension", "wildcard column '" + str(get(dim, "expr"))
+ + "' has no Apache Ossie field representation; skipped");
+ continue;
+ }
+ Object[] converted = convertDimension(dim, aliasToDataset, factName);
+ fieldsByDataset.get((String) converted[0]).add(converted[1]);
+ }
+ for (Map<String, Object> d : datasets) {
+ List<Object> flds = fieldsByDataset.get((String) d.get("name"));
+ if (!flds.isEmpty()) {
+ d.put("fields", flds);
+ }
+ }
+
+ List<Object> metrics = new ArrayList<>();
+ // Fixed once the join tree is walked, so compile it once for all measures.
+ Pattern aliasHead = qualifierChainPattern(aliasToDataset.keySet());
+ for (Object mObj : asList(get(view, "measures"))) {
+ Map<String, Object> m = asMap(mObj);
+ if (isWildcard(m)) {
+ notices.warn("measure", "wildcard measure '" + str(get(m, "expr"))
+ + "' has no Apache Ossie metric representation; skipped");
+ continue;
+ }
+ metrics.add(convertMeasure(m, aliasHead, aliasToDataset));
+ }
+
+ Map<String, Object> model = new LinkedHashMap<>();
+ model.put("name", factName);
+ if (truthy(get(view, "comment"))) {
+ model.put("description", get(view, "comment"));
+ }
+ model.put("datasets", datasets);
+ if (!relationships.isEmpty()) {
+ model.put("relationships", relationships);
+ }
+ if (!metrics.isEmpty()) {
+ model.put("metrics", metrics);
+ }
+
+ Map<String, Object> modelStash = new LinkedHashMap<>();
+ for (String k : MODEL_STASH_KEYS) {
+ if (view.containsKey(k)) {
+ modelStash.put(k, view.get(k));
+ }
+ }
+ if (hasOtm(asList(get(view, "joins")))) {
+ modelStash.put(STASH_SOURCE_KEY, factName);
+ }
+ if (!complexJoins.isEmpty()) {
+ modelStash.put("complex_joins", complexJoins);
+ }
+ writeStash(model, modelStash);
+ return model;
+ }
+
+ private static void walk(String parentName, String parentAlias, List<Object> joins,
+ List<Map<String, Object>> datasets, List<Map<String, Object>> relationships,
+ List<Map<String, Object>> complexJoins, Map<String, String> aliasToDataset,
+ Set<String> seenNames, Notices notices) {
+ for (Object joinObj : joins) {
+ Map<String, Object> join = asMap(joinObj);
+ String child = requireStr(join, "name", "join");
+ if (child.trim().equalsIgnoreCase("source")) {
+ throw new ConversionException(
+ "Join name 'source' is reserved for the fact source; rename the join.");
+ }
+ if (!seenNames.add(child.trim().toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("Duplicate dataset/join name '" + child
+ + "'; Metric View join names and the source must be distinct (case-insensitively).");
+ }
+ Map<String, Object> childDs = new LinkedHashMap<>();
+ childDs.put("name", child);
+ // Validated with the same rule the export applies to a dataset source, so a Metric View that
+ // imports cleanly is always exportable again (a 2-part join source used to import fine and
+ // then fail on the way back out).
+ String childSource = requireStr(join, "source", "join '" + child + "'");
+ validateSource(childSource, child);
+ childDs.put("source", childSource);
+ datasets.add(childDs);
+ aliasToDataset.put(child, child);
+ Map<String, Object> rel = convertJoin(join, parentName, parentAlias, child, notices);
+ // convertJoin omits from/to columns for a non-decomposable `on`; those entries have no valid
+ // Ossie relationship form, so they go to the model-level stash instead of `relationships`.
+ if (rel.containsKey("from_columns")) {
+ relationships.add(rel);
+ } else {
+ complexJoins.add(rel);
+ }
+ // rely.at_most_one_match on a many_to_one join -> recover a unique_key on the child.
+ Map<String, Object> rely = asMap(get(join, "rely"));
+ List<String> toCols = strList(get(rel, "to_columns"));
+ if (child.equals(str(get(rel, "to"))) && !toCols.isEmpty()
+ && Boolean.TRUE.equals(rely.get("at_most_one_match"))) {
+ List<Object> uk = new ArrayList<>();
+ uk.add(new ArrayList<>(toCols));
+ childDs.put("unique_keys", uk);
+ }
+ walk(child, child, asList(get(join, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+ }
+ }
+
+ private static boolean hasOtm(List<Object> joins) {
+ for (Object jObj : joins) {
+ Map<String, Object> j = asMap(jObj);
+ if (CARD_ONE_TO_MANY.equalsIgnoreCase(str(get(j, "cardinality")))) {
+ return true;
+ }
+ if (hasOtm(asList(get(j, "joins")))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static Map<String, Object> convertJoin(
+ Map<String, Object> join, String parentName, String parentAlias, String child,
+ Notices notices) {
+ boolean hasUsing = get(join, "using") != null && !asList(get(join, "using")).isEmpty();
+ boolean hasOn = str(get(join, "on")) != null && !str(get(join, "on")).isEmpty();
+ if (!hasUsing && !hasOn) {
+ throw new ConversionException("Join '" + child + "' has no join condition (empty or "
+ + "absent 'on'/'using'); condition-less (cross) joins have no Apache Ossie "
+ + "relationship representation.");
+ }
+ Object[] decomposed = decomposeOn(join, parentAlias, parentName, child);
+ List<String> parentCols = (List<String>) decomposed[0];
+ List<String> childCols = (List<String>) decomposed[1];
+ String rawOn = (String) decomposed[2];
+ if (rawOn != null) {
+ // The `on` is not an equi-join of simple `alias.column` pairs (it has a non-equi operator, a
+ // SQL-function-wrapped key, or an extra filter predicate), so an Apache Ossie relationship
+ // cannot represent it (from/to columns are required). Rather than abort the whole
+ // conversion, warn here; the columns-less entry built below is stashed under the model's
+ // custom_extensions (complex_joins) so it round-trips, instead of an invalid relationship.
+ notices.warn("join '" + child + "'", "non-equi or unsupported join condition ('on: " + rawOn
Review Comment:
Intentional and tracked. A join condition Apache Ossie cannot represent as a relationship (a non equi comparison, a function wrapped key, or an extra filter predicate) has no schema valid relationship form, since an Ossie relationship requires from_columns and to_columns. Rather than abort the whole export, the Java preserves the raw clause in a model level complex_joins stash inside custom_extensions, and the reverse converter rebuilds it. That keeps the round trip lossless for a join that carries real business logic, and the emitted model stays schema valid with no misleading columns less stub relationship, so a non Databricks consumer simply does not see the stashed join rather than seeing a wrong one. The Python reference still raises because it has not been updated for #321 and is the deprecated copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #333:
URL: https://github.com/apache/ossie/pull/333#issuecomment-5448708226
@Haoranli503 I pushed some fixes here. Can you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on code in PR #333:
URL: https://github.com/apache/ossie/pull/333#discussion_r4084991503
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieToMetricView.java:
##########
@@ -0,0 +1,1210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAX_JOIN_NODES;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SYNONYM_LIMIT;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.findOutsideLiterals;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.foreignVendorExtensions;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.mergeDescription;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.pickExpression;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.readStash;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.replaceOutsideLiterals;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.require;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.synonymsOf;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * EXPORT direction: Apache Ossie semantic model -> Databricks Metric View v1.1 YAML. This is the
+ * harder direction: it reassembles a relationship graph into a join tree. Shared helpers come from
+ * {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class OssieToMetricView {
+
+ private OssieToMetricView() {}
+
+ // -- tree node ------------------------------------------------------------
+ private static final class Node {
+ final String dataset;
+ final Map<String, Object> rel; // empty for the fact/root
+ final boolean parentIsFrom;
+ String alias;
+ boolean isOtm;
+ final List<Node> children = new ArrayList<>();
+
+ Node(String dataset, Map<String, Object> rel, boolean parentIsFrom) {
+ this.dataset = dataset;
+ this.rel = rel;
+ this.parentIsFrom = parentIsFrom;
+ }
+ }
+
+ // -- public entry ---------------------------------------------------------
+ static Result convertOssieToMetricView(String osiYamlStr, String source) {
+ Notices notices = new Notices();
+ if (osiYamlStr == null || osiYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(osiYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> root = asMap(parsed);
+ Object versionValue = get(root, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!OSSIE_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Apache Ossie version '" + version + "'. Supported: " + OSSIE_VERSION,
+ version);
+ }
+ if (root.containsKey("semantic_model")) {
+ String reason = "Legacy 'semantic_model' wrappers are not supported; place the model "
+ + "properties directly at the document root";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ if (root.containsKey("dialects") || root.containsKey("vendors")) {
+ String reason = "Root 'dialects' and 'vendors' are not supported by the Apache Ossie spec";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ if (!(get(root, "name") instanceof String)) {
+ String reason = "Apache Ossie model requires a string 'name' at the document root";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ Map<String, Object> view = convertModel(root, source, notices);
+ try {
+ return new Result(MAPPER.writeValueAsString(view), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Metric View YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertModel(
+ Map<String, Object> model, String explicitSource, Notices notices) {
+ String name = model.containsKey("name") ? str(get(model, "name")) : "<unnamed>";
+ String modelScope = "Model '" + name + "'";
+ List<Map<String, Object>> datasetList = schemaMapList(model, "datasets", modelScope);
+ if (datasetList.isEmpty()) {
+ throw new ConversionException("Model '" + name + "' has no datasets");
+ }
+ Set<String> seen = new HashSet<>();
+ Map<String, Map<String, Object>> datasets = new LinkedHashMap<>();
+ Map<String, List<Map<String, Object>>> datasetFields = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasetList) {
+ String dsName = requireStr(d, "name", "Model '" + name + "': dataset");
+ if (!seen.add(dsName.trim().toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("Model '" + name + "': duplicate dataset name '" + dsName + "'");
+ }
+ datasets.put(dsName, d);
+ datasetFields.put(dsName, schemaMapList(d, "fields", "Dataset '" + dsName + "'"));
+ }
+ List<Map<String, Object>> relationships = schemaMapList(model, "relationships", modelScope);
+ List<Map<String, Object>> metrics = schemaMapList(model, "metrics", modelScope);
+
+ Map<String, Object> modelStash = readStash(model, notices, modelScope);
+ // A non-equi/filtered join has no schema-valid Ossie relationship (from/to columns are
+ // required), so MetricViewToOssie stashes it under the model's DATABRICKS custom_extensions
+ // (complex_joins) rather than emitting a stub relationship. Rebuild a columns-less
+ // relationship for each -- carrying its raw `on` (and rely/cardinality) in the stash that
+ // buildJoin restores from -- and merge them in so the join tree includes them.
+ for (Map<String, Object> pj : schemaMapList(modelStash, "complex_joins", modelScope)) {
+ Map<String, Object> rel = new LinkedHashMap<>();
+ rel.put("name", get(pj, "name"));
+ rel.put("from", get(pj, "from"));
+ rel.put("to", get(pj, "to"));
+ Map<String, Object> stash = new LinkedHashMap<>(pj);
+ stash.remove("name");
+ stash.remove("from");
+ stash.remove("to");
+ writeStash(rel, stash);
+ relationships.add(rel);
+ }
+ String factHint = explicitSource != null ? explicitSource : str(get(modelStash, STASH_SOURCE_KEY));
Review Comment:
Good catch, fixed. `explicitSource` now uses a truthy check (`explicitSource != null && !explicitSource.isEmpty()`), matching Python's `explicit_source` or ..., so an empty `--source` falls back to the model hint instead of failing as `requested source '' is not a dataset`. Added a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #333:
URL: https://github.com/apache/ossie/pull/333#discussion_r4073085708
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java:
##########
@@ -0,0 +1,588 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.DIALECT_DATABRICKS;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAX_JOIN_NODES;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SELECT_WITH_RE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.lastIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * IMPORT direction: Metric View v1.1 YAML -> Apache Ossie semantic model. Shared helpers come
+ * from {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class MetricViewToOssie {
+
+ private static final String[] MODEL_STASH_KEYS = {"filter", "parameters", "materialization"};
+ private static final String[] JOIN_STASH_KEYS = {"rely", "cardinality"};
+ // Metric-View-only column fields with no Apache Ossie representation, preserved verbatim in the
+ // DATABRICKS stash so an import can restore them. `window` and `partition` are measure-only
+ // (they live on MeasureExpression); a dimension never carries them, so the shared key list is
+ // simply never hit for those on the dimension path.
+ private static final String[] COLUMN_STASH_KEYS = {"format", "window", "partition"};
+ // Measure-only fields with no Apache Ossie Metric representation. A dimension's display_name
+ // maps to the Field `label`, but the Ossie Metric schema has no `label`, so a measure's
+ // display_name is preserved in the DATABRICKS stash instead. Kept separate from
+ // COLUMN_STASH_KEYS so the dimension path (which already maps display_name to `label`) does not
+ // also stash it.
+ private static final String[] MEASURE_STASH_KEYS = {"display_name"};
+ private static final Pattern NON_EQUI_RE = Pattern.compile("[<>!]=|<>|[<>]");
+ private static final Pattern AND_SPLIT_RE = Pattern.compile("\\s+AND\\s+", Pattern.CASE_INSENSITIVE);
+ private static final Pattern EQ_CLAUSE_RE = Pattern.compile("^\\s*(.+?)\\s*=\\s*(.+?)\\s*$");
+
+ private MetricViewToOssie() {}
+
+ static Result convertMetricViewToOssie(String mvYamlStr, String modelName) {
+ Notices notices = new Notices();
+ if (mvYamlStr == null || mvYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(mvYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> view = asMap(parsed);
+ Object versionValue = get(view, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!MV_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Metric View version '" + version
+ + "'. This converter targets v" + MV_VERSION + " only.",
+ version);
+ }
+ Map<String, Object> model = convertView(view, modelName, notices);
+ Map<String, Object> out = new LinkedHashMap<>();
+ out.put("version", OSSIE_VERSION);
+ out.putAll(model);
+ return serializeOssie(out, notices);
+ }
+
+ static Result serializeOssie(Map<String, Object> out, Notices notices) {
+ try {
+ return new Result(MAPPER.writeValueAsString(out), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Apache Ossie YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertView(
+ Map<String, Object> view, String modelName, Notices notices) {
+ Object source = get(view, "source");
+ if (source == null || source.toString().isEmpty()) {
+ throw new ConversionException("Metric View is missing required 'source'");
+ }
+ // Same subquery test validateSource applies below, so the two cannot drift.
+ boolean isSql = SELECT_WITH_RE.matcher(source.toString().trim()).find();
+ String lastId = lastIdentifier(source);
+ String factName;
+ if (modelName != null) {
Review Comment:
Same pattern as `OssieToMetricView`: `modelName` is checked with `if (modelName != null)` instead of a truthy check, so `--name ""` produces `name: ""` in the output instead of falling back to deriving the name from the source table.
Python falls back for the same input: this should too I guess.
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieToMetricView.java:
##########
@@ -0,0 +1,1210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAX_JOIN_NODES;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SYNONYM_LIMIT;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.findOutsideLiterals;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.foreignVendorExtensions;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.mergeDescription;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.pickExpression;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.readStash;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.replaceOutsideLiterals;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.require;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.synonymsOf;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * EXPORT direction: Apache Ossie semantic model -> Databricks Metric View v1.1 YAML. This is the
+ * harder direction: it reassembles a relationship graph into a join tree. Shared helpers come from
+ * {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class OssieToMetricView {
+
+ private OssieToMetricView() {}
+
+ // -- tree node ------------------------------------------------------------
+ private static final class Node {
+ final String dataset;
+ final Map<String, Object> rel; // empty for the fact/root
+ final boolean parentIsFrom;
+ String alias;
+ boolean isOtm;
+ final List<Node> children = new ArrayList<>();
+
+ Node(String dataset, Map<String, Object> rel, boolean parentIsFrom) {
+ this.dataset = dataset;
+ this.rel = rel;
+ this.parentIsFrom = parentIsFrom;
+ }
+ }
+
+ // -- public entry ---------------------------------------------------------
+ static Result convertOssieToMetricView(String osiYamlStr, String source) {
+ Notices notices = new Notices();
+ if (osiYamlStr == null || osiYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(osiYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> root = asMap(parsed);
+ Object versionValue = get(root, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!OSSIE_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Apache Ossie version '" + version + "'. Supported: " + OSSIE_VERSION,
+ version);
+ }
+ if (root.containsKey("semantic_model")) {
+ String reason = "Legacy 'semantic_model' wrappers are not supported; place the model "
+ + "properties directly at the document root";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ if (root.containsKey("dialects") || root.containsKey("vendors")) {
+ String reason = "Root 'dialects' and 'vendors' are not supported by the Apache Ossie spec";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ if (!(get(root, "name") instanceof String)) {
+ String reason = "Apache Ossie model requires a string 'name' at the document root";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ Map<String, Object> view = convertModel(root, source, notices);
+ try {
+ return new Result(MAPPER.writeValueAsString(view), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Metric View YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertModel(
+ Map<String, Object> model, String explicitSource, Notices notices) {
+ String name = model.containsKey("name") ? str(get(model, "name")) : "<unnamed>";
+ String modelScope = "Model '" + name + "'";
+ List<Map<String, Object>> datasetList = schemaMapList(model, "datasets", modelScope);
+ if (datasetList.isEmpty()) {
+ throw new ConversionException("Model '" + name + "' has no datasets");
+ }
+ Set<String> seen = new HashSet<>();
+ Map<String, Map<String, Object>> datasets = new LinkedHashMap<>();
+ Map<String, List<Map<String, Object>>> datasetFields = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasetList) {
+ String dsName = requireStr(d, "name", "Model '" + name + "': dataset");
+ if (!seen.add(dsName.trim().toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("Model '" + name + "': duplicate dataset name '" + dsName + "'");
+ }
+ datasets.put(dsName, d);
+ datasetFields.put(dsName, schemaMapList(d, "fields", "Dataset '" + dsName + "'"));
+ }
+ List<Map<String, Object>> relationships = schemaMapList(model, "relationships", modelScope);
+ List<Map<String, Object>> metrics = schemaMapList(model, "metrics", modelScope);
+
+ Map<String, Object> modelStash = readStash(model, notices, modelScope);
+ // A non-equi/filtered join has no schema-valid Ossie relationship (from/to columns are
+ // required), so MetricViewToOssie stashes it under the model's DATABRICKS custom_extensions
+ // (complex_joins) rather than emitting a stub relationship. Rebuild a columns-less
+ // relationship for each -- carrying its raw `on` (and rely/cardinality) in the stash that
+ // buildJoin restores from -- and merge them in so the join tree includes them.
+ for (Map<String, Object> pj : schemaMapList(modelStash, "complex_joins", modelScope)) {
+ Map<String, Object> rel = new LinkedHashMap<>();
+ rel.put("name", get(pj, "name"));
+ rel.put("from", get(pj, "from"));
+ rel.put("to", get(pj, "to"));
+ Map<String, Object> stash = new LinkedHashMap<>(pj);
+ stash.remove("name");
+ stash.remove("from");
+ stash.remove("to");
+ writeStash(rel, stash);
+ relationships.add(rel);
+ }
+ String factHint = explicitSource != null ? explicitSource : str(get(modelStash, STASH_SOURCE_KEY));
Review Comment:
`explicitSource` is checked with a plain null-check, so an explicit empty string (e.g. `--source "$SRC"` with `$SRC` unset) is treated as a real override instead of "absent".
That throws `ConversionException("requested source '' is not a dataset")` instead of falling back to the model `source_dataset` hint.
Python uses a truthy check here: this should match (`explicitSource != null && !explicitSource.isEmpty()` or similar).
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java:
##########
@@ -0,0 +1,588 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.DIALECT_DATABRICKS;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAX_JOIN_NODES;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SELECT_WITH_RE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.lastIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * IMPORT direction: Metric View v1.1 YAML -> Apache Ossie semantic model. Shared helpers come
+ * from {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class MetricViewToOssie {
+
+ private static final String[] MODEL_STASH_KEYS = {"filter", "parameters", "materialization"};
+ private static final String[] JOIN_STASH_KEYS = {"rely", "cardinality"};
+ // Metric-View-only column fields with no Apache Ossie representation, preserved verbatim in the
+ // DATABRICKS stash so an import can restore them. `window` and `partition` are measure-only
+ // (they live on MeasureExpression); a dimension never carries them, so the shared key list is
+ // simply never hit for those on the dimension path.
+ private static final String[] COLUMN_STASH_KEYS = {"format", "window", "partition"};
+ // Measure-only fields with no Apache Ossie Metric representation. A dimension's display_name
+ // maps to the Field `label`, but the Ossie Metric schema has no `label`, so a measure's
+ // display_name is preserved in the DATABRICKS stash instead. Kept separate from
+ // COLUMN_STASH_KEYS so the dimension path (which already maps display_name to `label`) does not
+ // also stash it.
+ private static final String[] MEASURE_STASH_KEYS = {"display_name"};
+ private static final Pattern NON_EQUI_RE = Pattern.compile("[<>!]=|<>|[<>]");
+ private static final Pattern AND_SPLIT_RE = Pattern.compile("\\s+AND\\s+", Pattern.CASE_INSENSITIVE);
+ private static final Pattern EQ_CLAUSE_RE = Pattern.compile("^\\s*(.+?)\\s*=\\s*(.+?)\\s*$");
+
+ private MetricViewToOssie() {}
+
+ static Result convertMetricViewToOssie(String mvYamlStr, String modelName) {
+ Notices notices = new Notices();
+ if (mvYamlStr == null || mvYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(mvYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> view = asMap(parsed);
+ Object versionValue = get(view, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!MV_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Metric View version '" + version
+ + "'. This converter targets v" + MV_VERSION + " only.",
+ version);
+ }
+ Map<String, Object> model = convertView(view, modelName, notices);
+ Map<String, Object> out = new LinkedHashMap<>();
+ out.put("version", OSSIE_VERSION);
+ out.putAll(model);
+ return serializeOssie(out, notices);
+ }
+
+ static Result serializeOssie(Map<String, Object> out, Notices notices) {
+ try {
+ return new Result(MAPPER.writeValueAsString(out), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Apache Ossie YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertView(
+ Map<String, Object> view, String modelName, Notices notices) {
+ Object source = get(view, "source");
+ if (source == null || source.toString().isEmpty()) {
+ throw new ConversionException("Metric View is missing required 'source'");
+ }
+ // Same subquery test validateSource applies below, so the two cannot drift.
+ boolean isSql = SELECT_WITH_RE.matcher(source.toString().trim()).find();
+ String lastId = lastIdentifier(source);
+ String factName;
+ if (modelName != null) {
+ factName = modelName;
+ } else if (!isSql && lastId != null && isSimpleIdentifier(lastId)) {
+ factName = lastId;
+ } else {
+ factName = "metric_view";
+ }
+ validateSource(source, factName);
+
+ List<Map<String, Object>> datasets = new ArrayList<>();
+ Map<String, Object> factDs = new LinkedHashMap<>();
+ factDs.put("name", factName);
+ factDs.put("source", source);
+ datasets.add(factDs);
+ List<Map<String, Object>> relationships = new ArrayList<>();
+ // Joins with no schema-valid Ossie relationship (a non-decomposable `on`) are collected here
+ // and stashed under the model's DATABRICKS custom_extensions instead of `relationships`.
+ List<Map<String, Object>> complexJoins = new ArrayList<>();
+ Map<String, String> aliasToDataset = new HashMap<>();
+ aliasToDataset.put("source", factName);
+ aliasToDataset.put(factName, factName);
+ Set<String> seenNames = new HashSet<>();
+ seenNames.add(factName.trim().toLowerCase(Locale.ROOT));
+
+ walk(factName, "source", asList(get(view, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+
+ // The reverse conversion bounds how many datasets it will build (see buildJoinTree), so a
+ // Metric View that expands past that bound could not be imported again. Reject it here to keep
+ // MV -> Ossie -> MV symmetric rather than emitting a model that fails on the way back in.
+ if (datasets.size() > MAX_JOIN_NODES) {
+ throw new ConversionException("Metric View '" + factName + "' has " + datasets.size()
+ + " datasets; at most " + MAX_JOIN_NODES + " are supported.");
+ }
+
+ // `fields` is a v1.1 alias for `dimensions`: an empty `dimensions: []` falls through to
+ // `fields`, and the "both set" warning fires only when BOTH are non-empty (not merely
+ // present).
+ List<Object> dimList = asList(get(view, "dimensions"));
+ List<Object> fieldList = asList(get(view, "fields"));
+ if (!dimList.isEmpty() && !fieldList.isEmpty()) {
+ notices.warn("view", "both 'dimensions' and 'fields' are set; 'fields' is a v1.1 alias "
+ + "for 'dimensions', so the 'fields' list is ignored");
+ }
+ Map<String, List<Object>> fieldsByDataset = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasets) {
+ fieldsByDataset.put((String) d.get("name"), new ArrayList<>());
+ }
+ List<Object> dims = !dimList.isEmpty() ? dimList : fieldList;
+ for (Object dimObj : dims) {
+ Map<String, Object> dim = asMap(dimObj);
+ if (isWildcard(dim)) {
+ notices.warn("dimension", "wildcard column '" + str(get(dim, "expr"))
+ + "' has no Apache Ossie field representation; skipped");
+ continue;
+ }
+ Object[] converted = convertDimension(dim, aliasToDataset, factName);
+ fieldsByDataset.get((String) converted[0]).add(converted[1]);
+ }
+ for (Map<String, Object> d : datasets) {
+ List<Object> flds = fieldsByDataset.get((String) d.get("name"));
+ if (!flds.isEmpty()) {
+ d.put("fields", flds);
+ }
+ }
+
+ List<Object> metrics = new ArrayList<>();
+ // Fixed once the join tree is walked, so compile it once for all measures.
+ Pattern aliasHead = qualifierChainPattern(aliasToDataset.keySet());
+ for (Object mObj : asList(get(view, "measures"))) {
+ Map<String, Object> m = asMap(mObj);
+ if (isWildcard(m)) {
+ notices.warn("measure", "wildcard measure '" + str(get(m, "expr"))
+ + "' has no Apache Ossie metric representation; skipped");
+ continue;
+ }
+ metrics.add(convertMeasure(m, aliasHead, aliasToDataset));
+ }
+
+ Map<String, Object> model = new LinkedHashMap<>();
+ model.put("name", factName);
+ if (truthy(get(view, "comment"))) {
+ model.put("description", get(view, "comment"));
+ }
+ model.put("datasets", datasets);
+ if (!relationships.isEmpty()) {
+ model.put("relationships", relationships);
+ }
+ if (!metrics.isEmpty()) {
+ model.put("metrics", metrics);
+ }
+
+ Map<String, Object> modelStash = new LinkedHashMap<>();
+ for (String k : MODEL_STASH_KEYS) {
+ if (view.containsKey(k)) {
+ modelStash.put(k, view.get(k));
+ }
+ }
+ if (hasOtm(asList(get(view, "joins")))) {
+ modelStash.put(STASH_SOURCE_KEY, factName);
+ }
+ if (!complexJoins.isEmpty()) {
+ modelStash.put("complex_joins", complexJoins);
+ }
+ writeStash(model, modelStash);
+ return model;
+ }
+
+ private static void walk(String parentName, String parentAlias, List<Object> joins,
+ List<Map<String, Object>> datasets, List<Map<String, Object>> relationships,
+ List<Map<String, Object>> complexJoins, Map<String, String> aliasToDataset,
+ Set<String> seenNames, Notices notices) {
+ for (Object joinObj : joins) {
+ Map<String, Object> join = asMap(joinObj);
+ String child = requireStr(join, "name", "join");
+ if (child.trim().equalsIgnoreCase("source")) {
+ throw new ConversionException(
+ "Join name 'source' is reserved for the fact source; rename the join.");
+ }
+ if (!seenNames.add(child.trim().toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("Duplicate dataset/join name '" + child
+ + "'; Metric View join names and the source must be distinct (case-insensitively).");
+ }
+ Map<String, Object> childDs = new LinkedHashMap<>();
+ childDs.put("name", child);
+ // Validated with the same rule the export applies to a dataset source, so a Metric View that
+ // imports cleanly is always exportable again (a 2-part join source used to import fine and
+ // then fail on the way back out).
+ String childSource = requireStr(join, "source", "join '" + child + "'");
+ validateSource(childSource, child);
+ childDs.put("source", childSource);
+ datasets.add(childDs);
+ aliasToDataset.put(child, child);
+ Map<String, Object> rel = convertJoin(join, parentName, parentAlias, child, notices);
+ // convertJoin omits from/to columns for a non-decomposable `on`; those entries have no valid
+ // Ossie relationship form, so they go to the model-level stash instead of `relationships`.
+ if (rel.containsKey("from_columns")) {
+ relationships.add(rel);
+ } else {
+ complexJoins.add(rel);
+ }
+ // rely.at_most_one_match on a many_to_one join -> recover a unique_key on the child.
+ Map<String, Object> rely = asMap(get(join, "rely"));
+ List<String> toCols = strList(get(rel, "to_columns"));
+ if (child.equals(str(get(rel, "to"))) && !toCols.isEmpty()
+ && Boolean.TRUE.equals(rely.get("at_most_one_match"))) {
+ List<Object> uk = new ArrayList<>();
+ uk.add(new ArrayList<>(toCols));
+ childDs.put("unique_keys", uk);
+ }
+ walk(child, child, asList(get(join, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+ }
+ }
+
+ private static boolean hasOtm(List<Object> joins) {
+ for (Object jObj : joins) {
+ Map<String, Object> j = asMap(jObj);
+ if (CARD_ONE_TO_MANY.equalsIgnoreCase(str(get(j, "cardinality")))) {
+ return true;
+ }
+ if (hasOtm(asList(get(j, "joins")))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static Map<String, Object> convertJoin(
+ Map<String, Object> join, String parentName, String parentAlias, String child,
+ Notices notices) {
+ boolean hasUsing = get(join, "using") != null && !asList(get(join, "using")).isEmpty();
+ boolean hasOn = str(get(join, "on")) != null && !str(get(join, "on")).isEmpty();
+ if (!hasUsing && !hasOn) {
+ throw new ConversionException("Join '" + child + "' has no join condition (empty or "
+ + "absent 'on'/'using'); condition-less (cross) joins have no Apache Ossie "
+ + "relationship representation.");
+ }
+ Object[] decomposed = decomposeOn(join, parentAlias, parentName, child);
+ List<String> parentCols = (List<String>) decomposed[0];
+ List<String> childCols = (List<String>) decomposed[1];
+ String rawOn = (String) decomposed[2];
+ if (rawOn != null) {
Review Comment:
This unconditionally warns-and-stashes a non-equi/unsupported join condition rather than rejecting it.
I see two issues:
1. it diverges from the Python reference, which raises `ConversionError` for the same input (`test_non_equi_on_rejected`)
2. I believe issue #321, cited as justification, asked for this to be opt-in, not unconditional.
Should this be gated behind a flag, or should it reject like Python does?
##########
converters/databricks/java/src/test/java/org/apache/ossie/converter/databricks/OssieConverterSuite.java:
##########
@@ -0,0 +1,1639 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.junit.jupiter.api.Test;
+
+/**
+ * Example-based tests for the Apache Ossie <-> Metric View converter, plus the fixture
+ * comparisons that pin the expected output of both directions.
+ */
+public class OssieConverterSuite {
+
+ private static Object export(String osi, String source) {
+ return OssieConverter.parseYaml(
+ OssieConverter.convertOssieToMetricView(osi, source).yaml);
+ }
+
+ private static final class ThrowingBean {
+ public String getValue() {
+ throw new IllegalStateException("expected serialization failure");
+ }
+ }
+
+ private static Map<String, Object> valueThatFailsSerialization() {
+ Map<String, Object> value = new HashMap<>();
+ value.put("bad", new ThrowingBean());
+ return value;
+ }
+
+ private static String ossieModelWithBody(String body) {
+ // Flat model at the document root. Callers pass a body indented two spaces (it used to sit
+ // under a `- name: m` list item); de-indent it so it sits at the root next to `name`.
+ StringBuilder root = new StringBuilder(
+ "version: '" + OssieConverter.OSSIE_VERSION + "'\nname: m\n");
+ for (String line : body.split("\n", -1)) {
+ root.append(line.startsWith(" ") ? line.substring(2) : line).append("\n");
+ }
+ root.setLength(root.length() - 1);
+ return root.toString();
+ }
+
+ private static String cascadeNotice(String kind, String name, String reference) {
+ return "[" + kind + " '" + name + "'] references dropped '" + reference
+ + "'; dropping (downstream of a dropped field/metric)";
+ }
+
+ private static void assertInvalidOssieInput(String yaml, String expectedReason) {
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(yaml, null));
+ assertEquals(OssieConverter.ConversionException.Kind.INVALID_INPUT, e.getKind());
+ assertEquals(expectedReason, e.getReason());
+ }
+
+ @Test
+ public void fixtureAStarSchemaExportsToExpectedMetricView() {
+ String osi =
+ "version: \"0.2.0.dev0\"\n"
+ + "name: sales\n"
+ + "description: Sales orders with customer attributes\n"
+ + "datasets:\n"
+ + " - name: orders\n"
+ + " source: samples.tpch.orders\n"
+ + " primary_key: [o_orderkey]\n"
+ + " description: One row per order\n"
+ + " fields:\n"
+ + " - name: o_orderkey\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - dialect: DATABRICKS\n"
+ + " expression: o_orderkey\n"
+ + " description: Order identifier\n"
+ + " - name: o_orderdate\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - dialect: DATABRICKS\n"
+ + " expression: o_orderdate\n"
+ + " label: Order Date\n"
+ + " ai_context:\n"
+ + " synonyms: [order date, date]\n"
+ + " - name: customer\n"
+ + " source: samples.tpch.customer\n"
+ + " primary_key: [c_custkey]\n"
+ + " fields:\n"
+ + " - name: c_name\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - dialect: DATABRICKS\n"
+ + " expression: c_name\n"
+ + " description: Customer name\n"
+ + "relationships:\n"
+ + " - name: orders_to_customer\n"
+ + " from: orders\n"
+ + " to: customer\n"
+ + " from_columns: [o_custkey]\n"
+ + " to_columns: [c_custkey]\n"
+ + "metrics:\n"
+ + " - name: total_revenue\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - dialect: DATABRICKS\n"
+ + " expression: SUM(o_totalprice)\n"
+ + " description: Total order revenue\n"
+ + " ai_context:\n"
+ + " synonyms: [revenue, total revenue, sales]\n"
+ + " - name: order_count\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - dialect: DATABRICKS\n"
+ + " expression: COUNT(*)\n"
+ + " description: Number of orders\n";
+
+ String expected =
+ "version: '1.1'\n"
+ + "source: samples.tpch.orders\n"
+ + "comment: Sales orders with customer attributes\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: samples.tpch.customer\n"
+ + " on: source.o_custkey = customer.c_custkey\n"
+ + " rely:\n"
+ + " at_most_one_match: true\n"
+ + "dimensions:\n"
+ + "- name: o_orderkey\n"
+ + " expr: o_orderkey\n"
+ + " comment: Order identifier\n"
+ + "- name: o_orderdate\n"
+ + " expr: o_orderdate\n"
+ + " display_name: Order Date\n"
+ + " synonyms:\n"
+ + " - order date\n"
+ + " - date\n"
+ + "- name: c_name\n"
+ + " expr: customer.c_name\n"
+ + " comment: Customer name\n"
+ + "measures:\n"
+ + "- name: total_revenue\n"
+ + " expr: SUM(o_totalprice)\n"
+ + " comment: Total order revenue\n"
+ + " synonyms:\n"
+ + " - revenue\n"
+ + " - total revenue\n"
+ + " - sales\n"
+ + "- name: order_count\n"
+ + " expr: COUNT(*)\n"
+ + " comment: Number of orders\n";
+
+ assertEquals(OssieConverter.parseYaml(expected), export(osi, null));
+ }
+
+ @Test
+ public void measureDisplayNameSurvivesRoundTripViaStash() {
+ // A dimension's display_name maps to the Ossie Field `label`, but the Ossie Metric schema has
+ // no `label`, so a measure's display_name is preserved in the DATABRICKS custom_extensions
+ // stash (like format/window) and restored on the way back.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.fact\n"
+ + "dimensions:\n"
+ + "- name: region\n"
+ + " expr: region\n"
+ + "measures:\n"
+ + "- name: total_revenue\n"
+ + " expr: SUM(amount)\n"
+ + " display_name: Total Revenue\n";
+
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ // The only display_name in the input is on the measure; it rides in the metric's stash, since
+ // there is no native Ossie field for it.
+ assertTrue(ossie.contains("display_name"),
+ "expected the measure display_name preserved in the stash, got: " + ossie);
+
+ String mv2 = OssieConverter.convertOssieToMetricView(ossie, null).yaml;
+ assertEquals(OssieConverter.parseYaml(mv), OssieConverter.parseYaml(mv2));
+ }
+
+ @Test
+ public void unsupportedVersionIsRejected() {
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView("version: '9.9'\nname: m\n", null));
+ assertTrue(e.getMessage().contains("Unsupported Apache Ossie version"));
+ }
+
+ @Test
+ public void multipleCandidateFactsWithoutSourceIsRejected() {
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- {name: orders, source: c.s.orders}\n"
+ + "- {name: returns, source: c.s.returns}\n"
+ + "- {name: customer, source: c.s.customer, primary_key: [c_custkey]}\n"
+ + "relationships:\n"
+ + "- {name: oc, from: orders, to: customer, from_columns: [o_custkey], to_columns: [c_custkey]}\n"
+ + "- {name: rc, from: returns, to: customer, from_columns: [re_custkey], to_columns: [c_custkey]}\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, null));
+ assertTrue(e.getMessage().contains("multiple candidate fact datasets"));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void oneToManyEmitsCardinality() {
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: orders\n"
+ + " source: c.s.orders\n"
+ + " primary_key: [o_orderkey]\n"
+ + " fields:\n"
+ + " - {name: o_orderstatus, expression: {dialects: [{dialect: DATABRICKS, expression: o_orderstatus}]}}\n"
+ + "- name: lineitem\n"
+ + " source: c.s.lineitem\n"
+ + "relationships:\n"
+ + "- {name: lio, from: lineitem, to: orders, from_columns: [l_orderkey], to_columns: [o_orderkey]}\n"
+ + "metrics:\n"
+ + "- {name: qty, expression: {dialects: [{dialect: DATABRICKS, expression: SUM(lineitem.l_quantity)}]}}\n";
+ Map<String, Object> view = (Map<String, Object>) export(osi, "orders");
+ List<Object> joins = (List<Object>) view.get("joins");
+ Map<String, Object> join = (Map<String, Object>) joins.get(0);
+ assertEquals("one_to_many", join.get("cardinality"));
+ }
+
+ @Test
+ public void oneToManyNestedUnderManyToOneIsRejected() {
+ // A Metric View requires one cardinality per top-level branch, so a one-to-many join nested
+ // under a many-to-one parent is rejected by Databricks just as the reverse nesting is. Fact
+ // `d`; `f -> d` points at d (many-to-one from d's perspective), and `g -> f` points away from
+ // f (one-to-many), which would nest one_to_many inside the many-to-one branch.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: d\n"
+ + " source: c.s.d\n"
+ + " fields:\n"
+ + " - {name: dcol, expression: {dialects: [{dialect: DATABRICKS, expression: dcol}]}}\n"
+ + "- name: f\n"
+ + " source: c.s.f\n"
+ + " primary_key: [fk]\n"
+ + "- name: g\n"
+ + " source: c.s.g\n"
+ + "relationships:\n"
+ + "- {name: df, from: d, to: f, from_columns: [fk], to_columns: [fk]}\n"
+ + "- {name: gf, from: g, to: f, from_columns: [fk], to_columns: [fk]}\n"
+ + "metrics:\n"
+ + "- {name: c, expression: {dialects: [{dialect: DATABRICKS, expression: COUNT(1)}]}}\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, "d"));
+ assertTrue(e.getMessage().contains("share the same cardinality"),
+ "expected a mixed-cardinality rejection, got: " + e.getMessage());
+ }
+
+ @Test
+ public void directedCycleWithNoEquidistantEdgeIsRejected() {
+ // a -> b -> c -> d -> e -> b with fact `a`: every edge spans adjacent BFS levels, so the
+ // equidistance heuristic sees nothing, and `a` has zero incoming edges so pickFact finds a
+ // root. Without a real acyclicity check this expanded into duplicate join paths (`d` under
+ // both `c` and `e`), fabricating a tree from a cyclic model.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: a\n"
+ + " source: c.s.a\n"
+ + " fields:\n"
+ + " - {name: acol, expression: {dialects: [{dialect: DATABRICKS, expression: acol}]}}\n"
+ + "- name: b\n"
+ + " source: c.s.b\n"
+ + "- name: c\n"
+ + " source: c.s.c\n"
+ + "- name: d\n"
+ + " source: c.s.d\n"
+ + "- name: e\n"
+ + " source: c.s.e\n"
+ + "relationships:\n"
+ + "- {name: ab, from: a, to: b, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: bc, from: b, to: c, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: cd, from: c, to: d, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: de, from: d, to: e, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: eb, from: e, to: b, from_columns: [k], to_columns: [k]}\n";
+ OssieConverter.ConversionException ex = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, "a"));
+ assertTrue(ex.getMessage().contains("directed cycle"),
+ "expected a directed-cycle rejection, got: " + ex.getMessage());
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void diamondIsStillAcceptedByTheCycleCheck() {
+ // A diamond (`a -> b -> d` plus `a -> c -> d`) is an UNDIRECTED cycle but directed-acyclic, and
+ // is supported via the fan-out aliases. The cycle check must not reject it.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: a\n"
+ + " source: c.s.a\n"
+ + "- name: b\n"
+ + " source: c.s.b\n"
+ + "- name: c\n"
+ + " source: c.s.c\n"
+ + "- name: d\n"
+ + " source: c.s.d\n"
+ + " fields:\n"
+ + " - {name: dcol, expression: {dialects: [{dialect: DATABRICKS, expression: dcol}]}}\n"
+ + "relationships:\n"
+ + "- {name: ab, from: a, to: b, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: ac, from: a, to: c, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: bd, from: b, to: d, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: cd, from: c, to: d, from_columns: [k], to_columns: [k]}\n";
+ Map<String, Object> view = (Map<String, Object>) export(osi, "a");
+ assertEquals("c.s.a", view.get("source"));
+ // `d` is reached by two paths, so its column is emitted once per fan-out alias.
+ List<Object> dims = (List<Object>) view.get("dimensions");
+ assertEquals(2, dims.size(), "diamond should fan out to one dimension per path, got: " + dims);
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void measureNamingADiamondDatasetIsDroppedWithNotice() {
+ // Same diamond as above (`d` reached via `a -> b -> d` and `a -> c -> d`), plus a measure that
+ // names `d` by its bare dataset name. A measure addresses a dataset by name, so it cannot be
+ // split per fan-out path the way the dimension on `d` is; it is dropped with a notice rather
+ // than silently bound to one arbitrary branch (as the dimension complex-expression path does).
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: a\n"
+ + " source: c.s.a\n"
+ + "- name: b\n"
+ + " source: c.s.b\n"
+ + "- name: c\n"
+ + " source: c.s.c\n"
+ + "- name: d\n"
+ + " source: c.s.d\n"
+ + " fields:\n"
+ + " - {name: dcol, expression: {dialects: [{dialect: DATABRICKS, expression: dcol}]}}\n"
+ + "relationships:\n"
+ + "- {name: ab, from: a, to: b, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: ac, from: a, to: c, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: bd, from: b, to: d, from_columns: [k], to_columns: [k]}\n"
+ + "- {name: cd, from: c, to: d, from_columns: [k], to_columns: [k]}\n"
+ + "metrics:\n"
+ + "- {name: dsum, expression: {dialects: [{dialect: DATABRICKS, expression: SUM(d.dcol)}]}}\n";
+ OssieConverter.Result result = OssieConverter.convertOssieToMetricView(osi, "a");
+ Map<String, Object> view = (Map<String, Object>) OssieConverter.parseYaml(result.yaml);
+ // The diamond dimension on `d` still fans out to one dimension per path.
+ assertEquals(2, ((List<Object>) view.get("dimensions")).size());
+ // The measure that names `d` is dropped rather than bound to an arbitrary branch.
+ assertFalse(view.containsKey("measures"),
+ "a measure naming a diamond dataset must be dropped, got: " + view.get("measures"));
+ String expected = "expression references dataset 'd', reached by more than one join path "
+ + "(diamond); cannot be unambiguously qualified; dropped";
+ assertTrue(result.notices.stream().anyMatch(n -> n.contains(expected)),
+ result.notices.toString());
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void nestedJoinColumnInMeasureGetsFullAliasPath() {
+ // orders -> customer -> nation: `nation` nests under `customer`, so its columns are addressed
+ // as `customer.nation.col`. A bare `nation.` head would be read as struct access on a
+ // parameter, so the measure must be re-qualified -- and identically to the dimension path,
+ // which already emits `customer.nation.population` for the same column.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: orders\n"
+ + " source: c.s.orders\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " primary_key: [c_custkey]\n"
+ + "- name: nation\n"
+ + " source: c.s.nation\n"
+ + " primary_key: [n_nationkey]\n"
+ + " fields:\n"
+ + " - {name: population, expression: {dialects: [{dialect: DATABRICKS, expression: population}]}}\n"
+ + "relationships:\n"
+ + "- {name: oc, from: orders, to: customer, from_columns: [c_custkey], to_columns: [c_custkey]}\n"
+ + "- {name: cn, from: customer, to: nation, from_columns: [n_nationkey], to_columns: [n_nationkey]}\n"
+ + "metrics:\n"
+ + "- {name: pop, expression: {dialects: [{dialect: DATABRICKS, expression: SUM(nation.population)}]}}\n";
+ Map<String, Object> view = (Map<String, Object>) export(osi, "orders");
+
+ List<Object> dims = (List<Object>) view.get("dimensions");
+ Map<String, Object> dim = (Map<String, Object>) dims.get(0);
+ assertEquals("customer.nation.population", dim.get("expr"),
+ "dimension path should qualify with the full alias path");
+
+ List<Object> measures = (List<Object>) view.get("measures");
+ Map<String, Object> measure = (Map<String, Object>) measures.get(0);
+ assertEquals("SUM(customer.nation.population)", measure.get("expr"),
+ "measure must use the same full alias path as the dimension, not a bare nested alias");
+ }
+
+ @Test
+ public void modelWithNoFieldsOrMetricsIsRejected() {
+ // A Metric View requires at least one dimension or measure, so emitting a version+source-only
+ // view would just fail at CREATE. Fail at conversion time instead.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: d\n"
+ + " source: c.s.d\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, null));
+ assertTrue(e.getMessage().contains("no dimensions or measures"),
+ "expected an empty-view rejection, got: " + e.getMessage());
+ }
+
+ @Test
+ public void emptyAfterDropsNamesTheDroppedColumns() {
+ // Here the input is non-empty but everything drops: the only metric has no DATABRICKS/ANSI_SQL
+ // dialect. The error must name the dropped column so the cause is actionable, since after the
+ // cascade the emptiness is a consequence of the drop rather than an empty input.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: d\n"
+ + " source: c.s.d\n"
+ + "metrics:\n"
+ + "- {name: only_metric, expression: {dialects: [{dialect: SNOWFLAKE, expression: SUM(x)}]}}\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, null));
+ assertTrue(e.getMessage().contains("no dimensions or measures"),
+ "expected an empty-view rejection, got: " + e.getMessage());
+ assertTrue(e.getMessage().contains("only_metric"),
+ "the message must name the dropped column, got: " + e.getMessage());
+ }
+
+ @Test
+ public void duplicateDimensionNameIsRejected() {
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: orders\n"
+ + " source: c.s.orders\n"
+ + " fields:\n"
+ + " - {name: id, expression: {dialects: [{dialect: DATABRICKS, expression: id}]}}\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " fields:\n"
+ + " - {name: id, expression: {dialects: [{dialect: DATABRICKS, expression: id}]}}\n"
+ + "relationships:\n"
+ + "- {name: r, from: orders, to: customer, from_columns: [cid], to_columns: [id]}\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, null));
+ assertTrue(e.getMessage().contains("collides"));
+ }
+
+ @Test
+ public void foreignVendorExtensionDroppedWithNotice() {
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "custom_extensions:\n"
+ + "- {vendor_name: SNOWFLAKE, data: '{}'}\n"
+ + "datasets:\n"
+ + "- name: orders\n"
+ + " source: c.s.orders\n"
+ + " fields:\n"
+ + " - {name: s, expression: {dialects: [{dialect: DATABRICKS, expression: s}]}}\n"
+ + "metrics:\n"
+ + "- {name: n, expression: {dialects: [{dialect: DATABRICKS, expression: COUNT(*)}]}}\n";
+ OssieConverter.Result r = OssieConverter.convertOssieToMetricView(osi, null);
+ assertTrue(r.notices.stream().anyMatch(m -> m.contains("foreign-vendor custom_extensions dropped")));
+ }
+
+ // -- import direction (Metric View -> Apache Ossie) -----------------------
+
+ private static Object importMv(String mv) {
+ return OssieConverter.parseYaml(
+ OssieConverter.convertMetricViewToOssie(mv, null).yaml);
+ }
+
+ /** A Metric View YAML with {@code joinCount} sibling equi-joins on the fact source. */
+ private static String metricViewWithJoins(int joinCount) {
+ StringBuilder mv = new StringBuilder("version: '1.1'\nsource: c.s.fact\njoins:\n");
+ for (int i = 1; i <= joinCount; i++) {
+ mv.append("- name: j").append(i).append("\n")
+ .append(" source: c.s.j").append(i).append("\n")
+ .append(" on: source.k = j").append(i).append(".k\n");
+ }
+ return mv.toString();
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void importDecomposesJoinIntoRelationship() {
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey = customer.c_custkey\n"
+ + " rely: {at_most_one_match: true}\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus}\n"
+ + "- {name: c_name, expr: customer.c_name}\n"
+ + "measures:\n"
+ + "- {name: revenue, expr: SUM(o_totalprice)}\n";
+ Map<String, Object> out = (Map<String, Object>) importMv(mv);
+ Map<String, Object> model = out;
+ List<Object> rels = (List<Object>) model.get("relationships");
+ Map<String, Object> rel = (Map<String, Object>) rels.get(0);
+ assertEquals("orders", rel.get("from"));
+ assertEquals("customer", rel.get("to"));
+ assertEquals(List.of("o_custkey"), rel.get("from_columns"));
+ assertEquals(List.of("c_custkey"), rel.get("to_columns"));
+ }
+
+ @Test
+ public void exportRejectsMoreJoinsThanImportCanRebuild() {
+ // MV -> Ossie has no join bound of its own, but Ossie -> MV rejects a model with more than
+ // MAX_JOIN_NODES datasets. Without a matching bound here, a Metric View with too many joins
+ // would export to a model that could never be imported again; reject it at export instead.
+ int max = OssieConverterCommon.MAX_JOIN_NODES;
+ String mv = metricViewWithJoins(max);
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertMetricViewToOssie(mv, null));
+ assertTrue(e.getMessage().contains("at most " + max + " are supported"),
+ "expected a join-count rejection, got: " + e.getMessage());
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void exportAtTheJoinLimitStillSucceeds() {
+ // The fact plus (MAX_JOIN_NODES - 1) joins is exactly MAX_JOIN_NODES datasets, the boundary the
+ // reverse conversion accepts, so export must still succeed here.
+ int max = OssieConverterCommon.MAX_JOIN_NODES;
+ Map<String, Object> out = (Map<String, Object>) importMv(metricViewWithJoins(max - 1));
+ Map<String, Object> model = out;
+ List<Object> datasets = (List<Object>) model.get("datasets");
+ assertEquals(max, datasets.size(), "fact plus joins should be exactly the limit");
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void importPreservesNonEquiJoinInStash() {
+ // A non-equi `on` has no schema-valid Ossie relationship (from/to columns are required), so the
+ // converter warns and stashes the whole join under the model's DATABRICKS custom_extensions
+ // rather than emitting an invalid stub relationship (issue #321).
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey >= customer.c_custkey\n";
+ OssieConverter.Result result = OssieConverter.convertMetricViewToOssie(mv, null);
+ assertTrue(result.notices.stream().anyMatch(n -> n.contains("non-equi or unsupported")),
+ result.notices.toString());
+ Map<String, Object> out = (Map<String, Object>) OssieConverter.parseYaml(result.yaml);
+ Map<String, Object> model = out;
+ // No stub relationship is emitted; the join lives only in the model's custom_extensions.
+ assertFalse(model.containsKey("relationships"), result.yaml);
+ assertTrue(model.containsKey("custom_extensions"), result.yaml);
+ assertTrue(result.yaml.contains("source.o_custkey >= customer.c_custkey"), result.yaml);
+ }
+
+ @Test
+ public void nonEquiJoinSurvivesRoundTripViaStash() {
+ // The preserved raw `on` is restored verbatim on the way back, so a metric view whose join
+ // carries business logic round-trips instead of being rejected (issue #321).
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey >= customer.c_custkey\n"
+ + "dimensions:\n"
+ + "- name: cust\n"
+ + " expr: customer.c_name\n"
+ + "measures:\n"
+ + "- name: revenue\n"
+ + " expr: SUM(o_totalprice)\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ String mv2 = OssieConverter.convertOssieToMetricView(ossie, null).yaml;
+ assertTrue(mv2.contains("source.o_custkey >= customer.c_custkey"), mv2);
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void nestedNonEquiJoinKeepsAliasesOnRoundTrip() {
+ // A nested join's non-equi `on` references the parent join's alias. On import the alias is
+ // assigned from the (unique) dataset name our exporter emits, so the verbatim-restored `on`
+ // still names the aliases the importer actually assigned -- no stale-qualifier rewrite needed.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey = customer.c_custkey\n"
+ + " joins:\n"
+ + " - name: nation\n"
+ + " source: c.s.nation\n"
+ + " on: customer.c_nationkey <> nation.n_nationkey\n"
+ + "measures:\n"
+ + "- name: revenue\n"
+ + " expr: SUM(o_totalprice)\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ Map<String, Object> mv2 = (Map<String, Object>) OssieConverter.parseYaml(
+ OssieConverter.convertOssieToMetricView(ossie, null).yaml);
+ Map<String, Object> customer = (Map<String, Object>) ((List<Object>) mv2.get("joins")).get(0);
+ assertEquals("customer", customer.get("name"));
+ Map<String, Object> nation =
+ (Map<String, Object>) ((List<Object>) customer.get("joins")).get(0);
+ // The nested join keeps its alias, and the restored `on` references it and the parent alias
+ // `customer` -- exactly the aliases the importer assigned, so the clause resolves.
+ assertEquals("nation", nation.get("name"));
+ assertEquals("customer.c_nationkey <> nation.n_nationkey", nation.get("on"));
+ }
+
+ @Test
+ public void equiJoinOnPreservedVerbatimWhenNonCanonical() {
+ // An equi-join is a valid Ossie relationship (from/to columns), but rebuilding its `on` from
+ // those columns canonicalizes the fact qualifier to `source`. When the original used the source
+ // table name instead, the raw `on` is stashed so the round trip reproduces it exactly.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: orders.o_custkey = customer.c_custkey\n"
+ + "measures:\n"
+ + "- name: revenue\n"
+ + " expr: SUM(o_totalprice)\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ String mv2 = OssieConverter.convertOssieToMetricView(ossie, null).yaml;
+ assertTrue(mv2.contains("orders.o_custkey = customer.c_custkey"), mv2);
+ }
+
+ @Test
+ public void complexJoinPreservesCardinalityOnRoundTrip() {
+ // A complex (non-decomposable) join on the one_to_many branch stashes `on` and `cardinality`
+ // as flat keys on the columns-less entry; both are restored verbatim on the way back.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: lineitem\n"
+ + " source: c.s.lineitem\n"
+ + " on: source.o_orderkey <> lineitem.l_orderkey\n"
+ + " cardinality: one_to_many\n"
+ + "measures:\n"
+ + "- name: cnt\n"
+ + " expr: COUNT(1)\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ String mv2 = OssieConverter.convertOssieToMetricView(ossie, null).yaml;
+ assertTrue(mv2.contains("source.o_orderkey <> lineitem.l_orderkey"), mv2);
+ assertTrue(mv2.contains("one_to_many"), mv2);
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void complexJoinNestedUnderComplexJoinRoundTrips() {
+ // A complex join nested under another complex join: both edges live in the model stash, and the
+ // reverse merge rebuilds the tree so each raw `on` and the nesting are restored.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey <> customer.c_custkey\n"
+ + " joins:\n"
+ + " - name: nation\n"
+ + " source: c.s.nation\n"
+ + " on: customer.c_nationkey <> nation.n_nationkey\n"
+ + "measures:\n"
+ + "- name: cnt\n"
+ + " expr: COUNT(1)\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ Map<String, Object> mv2 = (Map<String, Object>) OssieConverter.parseYaml(
+ OssieConverter.convertOssieToMetricView(ossie, null).yaml);
+ Map<String, Object> customer = (Map<String, Object>) ((List<Object>) mv2.get("joins")).get(0);
+ assertEquals("source.o_custkey <> customer.c_custkey", customer.get("on"));
+ Map<String, Object> nation =
+ (Map<String, Object>) ((List<Object>) customer.get("joins")).get(0);
+ assertEquals("nation", nation.get("name"));
+ assertEquals("customer.c_nationkey <> nation.n_nationkey", nation.get("on"));
+ }
+
+ @Test
+ public void equiJoinOnOverEqualColumnsStaysOnNotUsing() {
+ // An equi-join `on` over equal column names rebuilds from columns as `using`, so the original
+ // `on` is stashed and restored -- the round trip keeps `on`, it does not collapse to `using`.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.custkey = customer.custkey\n"
+ + "measures:\n"
+ + "- name: cnt\n"
+ + " expr: COUNT(1)\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ String mv2 = OssieConverter.convertOssieToMetricView(ossie, null).yaml;
+ assertTrue(mv2.contains("source.custkey = customer.custkey"), mv2);
+ assertFalse(mv2.contains("using"), mv2);
+ }
+
+ @Test
+ public void importRejectsCrossJoin() {
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertMetricViewToOssie(mv, null));
+ assertTrue(e.getMessage().contains("no join condition"));
+ }
+
+ @Test
+ public void importRejectsUnsupportedVersion() {
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertMetricViewToOssie("version: '0.1'\nsource: c.s.t\n", null));
+ assertTrue(e.getMessage().contains("Unsupported Metric View version"));
+ }
+
+ @Test
+ public void mvToOssieToMvRoundTripsStash() {
+ // A view with MV-only features (filter/rely/format/window) must survive
+ // MV -> Ossie -> MV unchanged (the custom_extensions stash carries them).
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "filter: o_orderstatus = 'F'\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey = customer.c_custkey\n"
+ + " rely:\n"
+ + " at_most_one_match: true\n"
+ + "dimensions:\n"
+ + "- name: net\n"
+ + " expr: o_totalprice\n"
+ + " format:\n"
+ + " type: currency\n"
+ + " currency_code: USD\n"
+ + "measures:\n"
+ + "- name: running\n"
+ + " expr: SUM(o_totalprice)\n"
+ + " window:\n"
+ + " - order: net\n"
+ + " semiadditive: last\n"
+ + " range: cumulative\n";
+ String ossie = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ String back = OssieConverter.convertOssieToMetricView(ossie, null).yaml;
+ assertEquals(OssieConverter.parseYaml(mv), OssieConverter.parseYaml(back));
+ }
+
+ // -- fixture comparisons ---------------------------------------------------
+ // The fixtures under src/test/resources/ossie_*.yaml pin the expected outputs
+ // checked into apache/ossie. Asserting the Java output parses equal to them (structure
+ // + stash blob STRINGS) is the real "one behavior, two implementations" guarantee --
+ // it catches divergences like stash JSON spacing that a Java->Java round-trip misses.
+
+ private static String loadFixture(String name) {
+ try (InputStream in =
+ OssieConverterSuite.class.getClassLoader().getResourceAsStream("ossie_" + name)) {
+ if (in == null) {
+ throw new IllegalStateException("fixture not found: ossie_" + name);
+ }
+ return new String(in.readAllBytes(), StandardCharsets.UTF_8);
+ } catch (java.io.IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ public void fixtureAExportMatchesFixture() {
+ String out = OssieConverter.convertOssieToMetricView(loadFixture("fixtureA_ossie.yaml"), null).yaml;
+ assertEquals(OssieConverter.parseYaml(loadFixture("fixtureA_metric_view.yaml")),
+ OssieConverter.parseYaml(out));
+ }
+
+ @Test
+ public void fixtureBImportMatchesFixture() {
+ // fixtureB exercises the custom_extensions stash (format/rely/filter) -- the parse
+ // includes the blob strings, so this is what pins the stash blob's exact spacing.
+ String out = OssieConverter.convertMetricViewToOssie(loadFixture("fixtureB_metric_view.yaml"), null).yaml;
+ assertEquals(OssieConverter.parseYaml(loadFixture("fixtureB_ossie.yaml")),
+ OssieConverter.parseYaml(out));
+ }
+
+ @Test
+ public void tpcdsExportMatchesFixture() {
+ String out = OssieConverter.convertOssieToMetricView(loadFixture("tpcds_ossie.yaml"), null).yaml;
+ assertEquals(OssieConverter.parseYaml(loadFixture("tpcds_metric_view.yaml")),
+ OssieConverter.parseYaml(out));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void stashBlobUsesExpectedSpacing() {
+ // The strongest byte-level check: the emitted stash blob string (the `data` value of a
+ // custom_extensions entry) must use the stash format's separators
+ // (", " / ": "). Pull the blob out of the parsed model rather than substring-matching
+ // the outer YAML (where it appears escaped).
+ Object out = OssieConverter.parseYaml(
+ OssieConverter.convertMetricViewToOssie(loadFixture("fixtureB_metric_view.yaml"), null).yaml);
+ Map<String, Object> model = (Map<String, Object>) out;
+ List<Object> exts = (List<Object>) model.get("custom_extensions");
+ String blob = (String) ((Map<String, Object>) exts.get(0)).get("data");
+ assertTrue(blob.startsWith("{\"_v\": 1, "),
+ "stash blob must use the spacing '{\"_v\": 1, ...}', got: " + blob);
+ }
+
+ // -- parity edge cases found in review round 3 ----------------------------
+
+ @Test
+ public void pickExpressionFallsThroughEmptyDatabricksToAnsi() {
+ // An empty DATABRICKS dialect must fall through to ANSI_SQL,
+ // not be selected as the (empty) expression.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: f\n"
+ + " source: c.s.f\n"
+ + " fields:\n"
+ + " - name: d\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - {dialect: DATABRICKS, expression: ''}\n"
+ + " - {dialect: ANSI_SQL, expression: ansi_col}\n"
+ + "metrics:\n"
+ + "- {name: n, expression: {dialects: [{dialect: DATABRICKS, expression: COUNT(*)}]}}\n";
+ Object out = export(osi, null);
+ @SuppressWarnings("unchecked")
+ List<Object> dims = (List<Object>) ((Map<String, Object>) out).get("dimensions");
+ @SuppressWarnings("unchecked")
+ Map<String, Object> dim = (Map<String, Object>) dims.get(0);
+ assertEquals("ansi_col", dim.get("expr"));
+ }
+
+ @Test
+ public void pickExpressionRejectsNonStringExpression() {
+ // A non-string dialect expression (e.g. a YAML number) must raise, not be coerced.
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: f\n"
+ + " source: c.s.f\n"
+ + " fields:\n"
+ + " - name: d\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - {dialect: DATABRICKS, expression: 123}\n"
+ + "metrics:\n"
+ + "- {name: n, expression: {dialects: [{dialect: DATABRICKS, expression: COUNT(*)}]}}\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(osi, null));
+ assertTrue(e.getMessage().contains("expression must be a string"));
+ }
+
+ @Test
+ public void bareOnOffValuesStayStringsNotBooleans() {
+ // YAML 1.1 would read a bare `on`/`off`/`yes`/`no` value as a boolean, silently losing
+ // a join condition or turning a synonym into `true`. Confirm the converter's parser
+ // keeps them as strings (the reader uses YAML 1.2 boolean semantics).
+ Object parsed = OssieConverter.parseYaml("a: on\nb: off\nc: yes\nd: no\n");
+ @SuppressWarnings("unchecked")
+ Map<String, Object> m = (Map<String, Object>) parsed;
+ assertEquals("on", m.get("a"));
+ assertEquals("off", m.get("b"));
+ assertEquals("yes", m.get("c"));
+ assertEquals("no", m.get("d"));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void importDropsEmptyOptionalFields() {
+ // Optional fields are mapped only when non-empty, so an empty
+ // comment / empty synonyms list are omitted, not emitted as `description: ""` or an
+ // empty ai_context. The Java port must match (empty string / empty list are falsy).
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "comment: ''\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus, comment: '', display_name: '', synonyms: []}\n"
+ + "measures:\n"
+ + "- {name: revenue, expr: SUM(o_totalprice), comment: '', synonyms: []}\n";
+ Map<String, Object> out = (Map<String, Object>) importMv(mv);
+ Map<String, Object> model = out;
+ assertFalse(model.containsKey("description"), "empty comment must not become a description");
+ Map<String, Object> ds = (Map<String, Object>) ((List<Object>) model.get("datasets")).get(0);
+ Map<String, Object> field = (Map<String, Object>) ((List<Object>) ds.get("fields")).get(0);
+ assertFalse(field.containsKey("description"), "empty comment must not map to description");
+ assertFalse(field.containsKey("label"), "empty display_name must not map to label");
+ assertFalse(field.containsKey("ai_context"), "empty synonyms must not map to ai_context");
+ Map<String, Object> metric = (Map<String, Object>) ((List<Object>) model.get("metrics")).get(0);
+ assertFalse(metric.containsKey("description"), "empty comment must not map to description");
+ assertFalse(metric.containsKey("ai_context"), "empty synonyms must not map to ai_context");
+ }
+
+ @Test
+ public void stashEscapesNonAsciiAsLowercaseHex() {
+ // The stash blob is pure ASCII: non-ASCII is escaped to \\uXXXX. The
+ // stash blob must be byte-identical, so a non-ASCII stashed value (here a `filter`
+ // literal) escapes rather than emitting raw UTF-8.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "filter: \"region = 'café'\"\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus}\n";
+ String ossieYaml = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ // Extract the stash blob and assert the exact expected bytes:
+ // the non-ASCII char is escaped as lowercase \\u00e9 (a single backslash + 5 chars),
+ // not emitted raw. (Comparing the parsed `data` string sidesteps YAML's own quoting.)
+ @SuppressWarnings("unchecked")
+ Map<String, Object> out = (Map<String, Object>) OssieConverter.parseYaml(ossieYaml);
+ @SuppressWarnings("unchecked")
+ Map<String, Object> model = out;
+ @SuppressWarnings("unchecked")
+ List<Object> exts = (List<Object>) model.get("custom_extensions");
+ @SuppressWarnings("unchecked")
+ String blob = (String) ((Map<String, Object>) exts.get(0)).get("data");
+ assertEquals("{\"_v\": 1, \"filter\": \"region = 'caf\\u00e9'\"}", blob,
+ "stash blob must use a lowercase \\u escape");
+ // And it still round-trips back to the original view.
+ String back = OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml;
+ assertEquals(OssieConverter.parseYaml(mv), OssieConverter.parseYaml(back));
+ }
+
+ @Test
+ public void stashPreservesAValueContainingALiteralUnicodeEscape() {
+ // A stashed value may itself contain the text of a unicode escape. Serialized, that is a
+ // DOUBLED backslash, so the hex-lowercasing pass must not treat it as a real escape --
+ // doing so silently lowercases the value's own characters.
+ // Single-quoted YAML: a backslash is an ordinary character there, so `filter` really holds
+ // the six characters \ u A B C D rather than the character U+ABCD.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "filter: 'tag = \\uABCD'\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus}\n";
+ @SuppressWarnings("unchecked")
+ Map<String, Object> parsedIn = (Map<String, Object>) OssieConverter.parseYaml(mv);
+ String original = (String) parsedIn.get("filter");
+ // Guard the fixture itself: the value must contain a real backslash for this to be a test.
+ assertTrue(original.indexOf('\\') >= 0,
+ "test setup: filter must hold a literal backslash, got: " + original);
+
+ String ossieYaml = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ String back = OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml;
+ @SuppressWarnings("unchecked")
+ Map<String, Object> restored = (Map<String, Object>) OssieConverter.parseYaml(back);
+ assertEquals(original, restored.get("filter"),
+ "a literal unicode-escape sequence in a stashed value must survive unchanged");
+ }
+
+ @Test
+ public void joinOnTakesPrecedenceOverUsing() {
+ // Metric View validation requires only that one of `on`/`using` is present, so both may be
+ // set. Databricks resolves the criteria from `on` when it is present, so the converter must
+ // decompose `on` and ignore `using` -- otherwise the relationship joins on other columns.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "joins:\n"
+ + "- name: cust\n"
+ + " source: c.s.customer\n"
+ + " on: source.o_custkey = cust.c_custkey\n"
+ + " using: [nation_key]\n"
+ + "dimensions:\n"
+ + "- {name: c_name, expr: cust.c_name}\n"
+ + "measures:\n"
+ + "- {name: cnt, expr: COUNT(1)}\n";
+ @SuppressWarnings("unchecked")
+ Map<String, Object> out = (Map<String, Object>) OssieConverter.parseYaml(
+ OssieConverter.convertMetricViewToOssie(mv, null).yaml);
+ Map<String, Object> model = out;
+ @SuppressWarnings("unchecked")
+ List<Object> rels = (List<Object>) model.get("relationships");
+ @SuppressWarnings("unchecked")
+ Map<String, Object> rel = (Map<String, Object>) rels.get(0);
+ assertEquals(List.of("o_custkey"), rel.get("from_columns"),
+ "`on` must win over `using`: expected the o_custkey/c_custkey pair");
+ assertEquals(List.of("c_custkey"), rel.get("to_columns"),
+ "`on` must win over `using`: expected the o_custkey/c_custkey pair");
+ }
+
+ @Test
+ public void measureRewriteLeavesStringLiteralsAlone() {
+ // The fact qualifier is added/stripped by rewriting the measure expression. That rewrite must
+ // skip string literals: rewriting inside one changes the predicate and therefore the value.
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus}\n"
+ + "measures:\n"
+ + "- name: tagged\n"
+ + " expr: \"SUM(IF(source.region = 'source.us', 1, 0))\"\n";
+ String ossieYaml = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ assertTrue(ossieYaml.contains("'source.us'"),
+ "a literal mentioning the qualifier must not be rewritten, got:\n" + ossieYaml);
+ // The literal also survives the trip back. Note the *code* qualifier is normalized on the
+ // way through (`source.region` -> bare `region`, the Metric View idiom for fact columns);
+ // only the literal is required to come back byte-identical.
+ String back = OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml;
+ assertTrue(back.contains("'source.us'"),
+ "the literal must survive the round trip, got:\n" + back);
+ }
+
+ /** A fact with `customer` joined to it and `region` nested under `customer`, plus one metric. */
+ private static String nestedJoinModel(String metricExpr) {
+ return "version: \"0.2.0.dev0\"\n"
+ + "name: sales\n"
+ + "datasets:\n"
+ + " - name: lineitem\n"
+ + " source: cat.sch.lineitem\n"
+ + " fields:\n"
+ + " - name: l_key\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: l_key}]\n"
+ + " - name: customer\n"
+ + " source: cat.sch.customer\n"
+ + " primary_key: [c_key]\n"
+ + " - name: region\n"
+ + " source: cat.sch.region\n"
+ + " primary_key: [r_key]\n"
+ + "relationships:\n"
+ + " - name: l_to_c\n"
+ + " from: lineitem\n"
+ + " to: customer\n"
+ + " from_columns: [c_key]\n"
+ + " to_columns: [c_key]\n"
+ + " - name: c_to_r\n"
+ + " from: customer\n"
+ + " to: region\n"
+ + " from_columns: [r_key]\n"
+ + " to_columns: [r_key]\n"
+ + "metrics:\n"
+ + " - name: pop\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: \"" + metricExpr + "\"}]\n";
+ }
+
+ @SuppressWarnings("unchecked")
+ private static String firstMeasureExpr(Object view) {
+ Map<String, Object> mv = (Map<String, Object>) view;
+ List<Object> measures = (List<Object>) mv.get("measures");
+ assertFalse(measures == null || measures.isEmpty(), "expected one measure, got: " + mv);
+ return (String) ((Map<String, Object>) measures.get(0)).get("expr");
+ }
+
+ @Test
+ public void exportQualifiesANestedMeasureWithTheFullJoinPath() {
+ // A Metric View addresses a nested join column by its full path, so a metric naming the
+ // dataset must come out as `customer.region.population`.
+ assertEquals("SUM(customer.region.population)",
+ firstMeasureExpr(export(nestedJoinModel("SUM(region.population)"), null)));
+ }
+
+ @Test
+ public void exportDoesNotQualifyAMeasurePathTwice() {
+ // The rewrite used to run one dataset name at a time, so it matched the `region.` inside the
+ // path it had just written and produced `SUM(customer.customer.region.population)` -- silently,
+ // with no notice. An expression that already carries the full path (what an import leaves
+ // behind) must come out unchanged.
+ assertEquals("SUM(customer.region.population)",
+ firstMeasureExpr(export(nestedJoinModel("SUM(customer.region.population)"), null)));
+ }
+
+ @Test
+ public void nestedQualifiedMeasureRoundTripsBothWays() {
+ String mv =
+ "version: '1.1'\n"
+ + "source: cat.sch.lineitem\n"
+ + "joins:\n"
+ + "- name: customer\n"
+ + " source: cat.sch.customer\n"
+ + " using: [c_key]\n"
+ + " joins:\n"
+ + " - name: region\n"
+ + " source: cat.sch.region\n"
+ + " using: [r_key]\n"
+ + "dimensions:\n"
+ + "- {name: l_key, expr: l_key}\n"
+ + "measures:\n"
+ + "- {name: pop, expr: SUM(customer.region.population)}\n";
+ String ossieYaml = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ // The import de-aliases the path down to the dataset that owns the column...
+ assertTrue(ossieYaml.contains("SUM(region.population)"),
+ "import should de-alias the join path to the dataset name, got:\n" + ossieYaml);
+ // ...and the export puts it back, so the measure survives MV -> Ossie -> MV.
+ assertEquals("SUM(customer.region.population)",
+ firstMeasureExpr(OssieConverter.parseYaml(
+ OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml)));
+ }
+
+ @Test
+ public void cascadeDropIgnoresADroppedNameInsideAStringLiteral() {
+ // A field dropped for lacking a usable dialect must not take an unrelated measure with it just
+ // because its name appears in a string literal: `'us'` is not a reference to a column `us`.
+ String osi =
+ "version: \"0.2.0.dev0\"\n"
+ + "name: sales\n"
+ + "datasets:\n"
+ + " - name: orders\n"
+ + " source: cat.sch.orders\n"
+ + " fields:\n"
+ + " - name: amount\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: amount}]\n"
+ + " - name: us\n"
+ + " expression:\n"
+ + " dialects: [{dialect: SNOWFLAKE, expression: us_col}]\n"
+ + "metrics:\n"
+ + " - name: amt_us\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - dialect: DATABRICKS\n"
+ + " expression: \"SUM(IF(region = 'us', amount, 0))\"\n";
+ OssieConverter.Result result = OssieConverter.convertOssieToMetricView(osi, null);
+ assertEquals("SUM(IF(region = 'us', amount, 0))",
+ firstMeasureExpr(OssieConverter.parseYaml(result.yaml)));
+ for (String notice : result.notices) {
+ assertFalse(notice.contains("amt_us") && notice.contains("references dropped"),
+ "the measure only mentions 'us' in a literal, got notice: " + notice);
+ }
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void cascadeDropMatchesADroppedNameCaseInsensitively() {
+ // Databricks SQL identifiers are case-insensitive. A metric that references a dropped field in a
+ // different case (COUNT(DISTINCT REGION_NAME) over a dropped region_name) must cascade-drop
+ // rather than survive as a dangling reference.
+ String osi =
+ "version: \"0.2.0.dev0\"\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + " - name: d\n"
+ + " source: cat.sch.t\n"
+ + " fields:\n"
+ + " - name: id\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: id}]\n"
+ + " - name: region_name\n"
+ + " expression:\n"
+ + " dialects: [{dialect: T_SQL, expression: region_name}]\n"
+ + "metrics:\n"
+ + " - name: region_count\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: COUNT(DISTINCT REGION_NAME)}]\n";
+ OssieConverter.Result result = OssieConverter.convertOssieToMetricView(osi, null);
+ Map<String, Object> view = (Map<String, Object>) OssieConverter.parseYaml(result.yaml);
+ // The measure references the dropped field in upper case, so it is cascade-dropped.
+ assertFalse(view.containsKey("measures"),
+ "region_count must cascade-drop, got: " + view.get("measures"));
+ // The unrelated dimension survives.
+ assertEquals(1, ((List<Object>) view.get("dimensions")).size());
+ assertTrue(result.notices.contains(cascadeNotice("measure", "region_count", "region_name")),
+ result.notices.toString());
+ }
+
+ @Test
+ public void stashPreservesABackslashBeforeAUnicodeEscape() {
+ // The stash lowercases the hex of Jackson's \\uXXXX escapes. That rewrite must re-emit any
+ // escaped-backslash run in front of the escape verbatim; it used to halve it, so a value
+ // holding a literal backslash came back as the six characters "u00e9".
+ String value = "tag = \\" + "é";
+ String mv =
+ "version: '1.1'\n"
+ + "source: c.s.orders\n"
+ + "filter: '" + value + "'\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus}\n";
+ String ossieYaml = OssieConverter.convertMetricViewToOssie(mv, null).yaml;
+ @SuppressWarnings("unchecked")
+ Map<String, Object> back = (Map<String, Object>) OssieConverter.parseYaml(
+ OssieConverter.convertOssieToMetricView(ossieYaml, null).yaml);
+ assertEquals(value, back.get("filter"),
+ "the stashed filter must survive MV -> Ossie -> MV byte-for-byte");
+ }
+
+ @Test
+ public void exportWarnsWhenAColumnAiContextObjectIsDropped() {
+ // `ai_context` is `string | object`; only the object's `synonyms` has a Metric View slot, so
+ // the other members are dropped -- and the README promises a notice when that happens. The
+ // same goes for foreign-vendor extensions on a metric.
+ String osi =
+ "version: \"0.2.0.dev0\"\n"
+ + "name: sales\n"
+ + "datasets:\n"
+ + " - name: orders\n"
+ + " source: cat.sch.orders\n"
+ + " fields:\n"
+ + " - name: amount\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: amount}]\n"
+ + " ai_context:\n"
+ + " instructions: do not use this column\n"
+ + " synonyms: [amt]\n"
+ + "metrics:\n"
+ + " - name: total\n"
+ + " expression:\n"
+ + " dialects: [{dialect: DATABRICKS, expression: SUM(amount)}]\n"
+ + " ai_context:\n"
+ + " examples: [how much did we sell]\n"
+ + " custom_extensions:\n"
+ + " - vendor_name: SNOWFLAKE\n"
+ + " data: \"{}\"\n";
+ List<String> notices = OssieConverter.convertOssieToMetricView(osi, null).notices;
+ assertTrue(notices.contains(
+ "[field 'amount'] ai_context [instructions] dropped "
+ + "(only 'synonyms' has a Metric View counterpart)"),
+ "expected a notice for the field's dropped ai_context members, got: " + notices);
+ assertTrue(notices.contains(
+ "[metric 'total'] ai_context [examples] dropped "
+ + "(only 'synonyms' has a Metric View counterpart)"),
+ "expected a notice for the metric's dropped ai_context members, got: " + notices);
+ assertTrue(notices.contains("[metric 'total'] foreign-vendor custom_extensions dropped"),
+ "expected a notice for the metric's foreign-vendor extensions, got: " + notices);
+ }
+
+ @Test
+ public void importValidatesAJoinSourceTheWayTheExportDoes() {
+ // A 2-part join source used to import cleanly and then fail on the way back out, so a view
+ // that imported could not be exported. Reject it at the same point either direction would.
+ String mv =
+ "version: '1.1'\n"
+ + "source: cat.sch.orders\n"
+ + "joins:\n"
+ + "- name: d\n"
+ + " source: sch.d\n"
+ + " using: [k]\n"
+ + "dimensions:\n"
+ + "- {name: o_status, expr: o_orderstatus}\n";
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertMetricViewToOssie(mv, null));
+ assertTrue(e.getMessage().contains("3-part"),
+ "expected the same source-shape error the export raises, got: " + e.getMessage());
+ }
+
+ // -- hardening: strict schema/stash parsing and structured failures -------
+
+ @Test
+ public void bareNullOptionalCollectionsConvertLikeAbsent() {
+ // A present-but-null optional collection (the bare `relationships:` YAML idiom) must convert
+ // like an absent key, as the base reader (asList) did. Only a scalar/map there is rejected.
+ String yaml = "version: '" + OssieConverter.OSSIE_VERSION + "'\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: d\n"
+ + " source: c.s.t\n"
+ + " fields:\n"
+ + " - name: region\n"
+ + " expression: {dialects: [{dialect: DATABRICKS, expression: region}]}\n"
+ + "relationships:\n"
+ + "metrics:\n"
+ + "- name: cnt\n"
+ + " expression: {dialects: [{dialect: DATABRICKS, expression: 'count(*)'}]}\n";
+ String mv = OssieConverter.convertOssieToMetricView(yaml, null).yaml;
+ assertTrue(mv.contains("cnt"), "expected the model to convert, got: " + mv);
+ }
+
+ @Test
+ public void invalidOssieInputHasStructuredFailureDetails() {
+ OssieConverter.ConversionException e = assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView("just a scalar", null));
+ assertEquals(OssieConverter.ConversionException.Kind.INVALID_INPUT, e.getKind());
+ assertEquals("it is not a mapping at the root", e.getReason());
+ }
+
+ @Test
+ public void malformedSchemaCollectionsAreRejected() {
+ assertInvalidOssieInput(
+ "version: '" + OssieConverter.OSSIE_VERSION + "'\nsemantic_model:\n- name: m\n",
+ "Legacy 'semantic_model' wrappers are not supported; place the model properties "
+ + "directly at the document root");
+ assertInvalidOssieInput(
+ "version: '" + OssieConverter.OSSIE_VERSION + "'\ndatasets: []\n",
+ "Apache Ossie model requires a string 'name' at the document root");
+ assertInvalidOssieInput(
+ ossieModelWithBody(" datasets: nope\n"),
+ "Model 'm': 'datasets' must be a list");
+ assertInvalidOssieInput(
+ ossieModelWithBody(" datasets:\n - nope\n"),
+ "Model 'm': 'datasets[0]' must be a mapping");
+
+ String dataset =
+ " datasets:\n"
+ + " - name: d\n"
+ + " source: c.s.t\n";
+ assertInvalidOssieInput(
+ ossieModelWithBody(dataset + " relationships: nope\n"),
+ "Model 'm': 'relationships' must be a list");
+ assertInvalidOssieInput(
+ ossieModelWithBody(dataset + " relationships:\n - nope\n"),
+ "Model 'm': 'relationships[0]' must be a mapping");
+ assertInvalidOssieInput(
+ ossieModelWithBody(dataset + " fields: nope\n"),
+ "Dataset 'd': 'fields' must be a list");
+ assertInvalidOssieInput(
+ ossieModelWithBody(dataset + " fields:\n - nope\n"),
+ "Dataset 'd': 'fields[0]' must be a mapping");
+ assertInvalidOssieInput(
+ ossieModelWithBody(dataset + " metrics: nope\n"),
+ "Model 'm': 'metrics' must be a list");
+ assertInvalidOssieInput(
+ ossieModelWithBody(dataset + " metrics:\n - nope\n"),
+ "Model 'm': 'metrics[0]' must be a mapping");
+ }
+
+ @Test
+ public void malformedExpressionCollectionsAreRejected() {
+ String datasetPrefix =
+ " datasets:\n"
+ + " - name: d\n"
+ + " source: c.s.t\n"
+ + " fields:\n"
+ + " - name: id\n";
+ assertInvalidOssieInput(
+ ossieModelWithBody(datasetPrefix + " expression: nope\n"),
+ "field 'id': 'expression' must be a mapping");
+ assertInvalidOssieInput(
+ ossieModelWithBody(datasetPrefix + " expression: {dialects: nope}\n"),
+ "field 'id': 'expression.dialects' must be a list");
+ assertInvalidOssieInput(
+ ossieModelWithBody(datasetPrefix + " expression: {dialects: [nope]}\n"),
+ "field 'id': 'expression.dialects[0]' must be a mapping");
+
+ String validField = datasetPrefix
+ + " expression:\n"
+ + " dialects:\n"
+ + " - {dialect: DATABRICKS, expression: id}\n";
+ assertInvalidOssieInput(
+ ossieModelWithBody(validField
+ + " metrics:\n"
+ + " - name: count_id\n"
+ + " expression: {dialects: nope}\n"),
+ "metric 'count_id': 'expression.dialects' must be a list");
+ }
+
+ @Test
+ public void malformedCustomExtensionsAreRejected() {
+ String validModelBody =
+ " datasets:\n"
+ + " - name: d\n"
+ + " source: c.s.t\n"
+ + " fields:\n"
+ + " - name: id\n"
+ + " expression:\n"
+ + " dialects:\n"
+ + " - {dialect: DATABRICKS, expression: id}\n";
+ assertInvalidOssieInput(
+ ossieModelWithBody(
+ " custom_extensions: {vendor_name: DATABRICKS, data: '{}'}\n" + validModelBody),
+ "'custom_extensions' must be a list");
+ assertInvalidOssieInput(
+ ossieModelWithBody(" custom_extensions: [nope]\n" + validModelBody),
+ "'custom_extensions[0]' must be a mapping");
+ assertInvalidOssieInput(
+ ossieModelWithBody(
+ " custom_extensions:\n"
+ + " - {vendor_name: DATABRICKS, data: 123}\n"
+ + validModelBody),
+ "DATABRICKS custom_extensions data must be a string");
+ assertInvalidOssieInput(
+ ossieModelWithBody(
+ " custom_extensions:\n"
+ + " - {vendor_name: DATABRICKS, data: '[]'}\n"
+ + validModelBody),
+ "DATABRICKS custom_extensions data must be a JSON object");
+
+ for (String data : List.of(
+ "filter: x = 1",
+ "{\"filter\": \"x\", \"filter\": \"y\"}",
+ "{\"filter\": \"x\"} true")) {
+ String yaml = ossieModelWithBody(
+ " custom_extensions:\n"
+ + " - vendor_name: DATABRICKS\n"
+ + " data: '" + data + "'\n"
+ + validModelBody);
+ OssieConverter.ConversionException e =
+ assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(yaml, null));
+ assertEquals(OssieConverter.ConversionException.Kind.INVALID_INPUT, e.getKind());
+ assertTrue(e.getReason().startsWith(
+ "DATABRICKS custom_extensions data is not valid JSON:"), e.getReason());
+ }
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void duplicateDatabricksStashKeepsFirstAndWarns() {
+ // Two DATABRICKS custom_extensions entries on one object is malformed, but rather than fail we
+ // keep the first, ignore the rest, and warn so the dropped entry is not lost silently.
+ String osi = ossieModelWithBody(
+ " custom_extensions:\n"
+ + " - {vendor_name: DATABRICKS, data: '{\"filter\": \"a = 1\"}'}\n"
+ + " - {vendor_name: DATABRICKS, data: '{\"filter\": \"b = 2\"}'}\n"
+ + " datasets:\n"
+ + " - name: d\n"
+ + " source: c.s.t\n"
+ + " fields:\n"
+ + " - {name: id, expression: {dialects: [{dialect: DATABRICKS, expression: id}]}}\n");
+ OssieConverter.Result result = OssieConverter.convertOssieToMetricView(osi, null);
+ Map<String, Object> view = (Map<String, Object>) OssieConverter.parseYaml(result.yaml);
+ assertEquals("a = 1", view.get("filter"), "the first DATABRICKS entry must win");
+ assertTrue(
+ result.notices.stream().anyMatch(n ->
+ n.contains("more than one DATABRICKS custom_extensions entry")),
+ result.notices.toString());
+ }
+
+ @Test
+ public void oversizedDatasetGraphIsRejectedBeforeRecursiveTraversal() {
+ StringBuilder body = new StringBuilder(" datasets:\n");
+ for (int i = 0; i <= OssieConverterCommon.MAX_JOIN_NODES; i++) {
+ body.append(" - {name: d").append(i).append(", source: c.s.t").append(i).append("}\n");
+ }
+ body.append(" relationships:\n");
+ for (int i = 0; i < OssieConverterCommon.MAX_JOIN_NODES; i++) {
+ body.append(" - {name: r").append(i)
+ .append(", from: d").append(i)
+ .append(", to: d").append(i + 1)
+ .append(", from_columns: [id], to_columns: [id]}\n");
+ }
+
+ OssieConverter.ConversionException e =
+ assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(
+ ossieModelWithBody(body.toString()), null));
+
+ assertTrue(e.getMessage().contains("at most 200 are supported"), e.getMessage());
+ }
+
+ @Test
+ public void duplicateYamlKeysAreRejectedAtEveryDepth() {
+ String topLevel =
+ "version: '" + OssieConverter.OSSIE_VERSION + "'\n"
+ + "version: '" + OssieConverter.OSSIE_VERSION + "'\n"
+ + "name: m\n";
+ String nested = ossieModelWithBody(
+ " datasets:\n"
+ + " - name: d\n"
+ + " source: c.s.first\n"
+ + " source: c.s.second\n");
+ for (String yaml : List.of(topLevel, nested)) {
+ OssieConverter.ConversionException e =
+ assertThrows(OssieConverter.ConversionException.class,
+ () -> OssieConverter.convertOssieToMetricView(yaml, null));
+ assertEquals(OssieConverter.ConversionException.Kind.INVALID_INPUT, e.getKind());
+ assertTrue(e.getReason().contains("duplicate key"), e.getReason());
+ }
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void reverseOrderedDropChainIsPropagatedWithoutRepeatedFullScans() {
+ // Every field depends on the preceding field, but reverse source order means the old repeated
+ // pass implementation could discover only one new drop per pass.
+ int chainLength = 400;
+ StringBuilder body = new StringBuilder(
+ " datasets:\n"
+ + " - name: d\n"
+ + " source: c.s.d\n"
+ + " fields:\n");
+ for (int index = chainLength; index >= 1; index--) {
+ body.append(" - name: d").append(index).append("\n")
+ .append(" expression:\n")
+ .append(" dialects:\n")
+ .append(" - {dialect: DATABRICKS, expression: d")
+ .append(index - 1).append("}\n");
+ }
+ body.append(" - name: d0\n")
+ .append(" expression:\n")
+ .append(" dialects:\n")
+ .append(" - {dialect: SNOWFLAKE, expression: d0}\n")
+ .append(" metrics:\n")
+ .append(" - name: row_count\n")
+ .append(" expression:\n")
+ .append(" dialects:\n")
+ .append(" - {dialect: DATABRICKS, expression: count(*)}\n");
+
+ OssieConverter.Result result =
+ OssieConverter.convertOssieToMetricView(ossieModelWithBody(body.toString()), null);
+ Map<String, Object> view = (Map<String, Object>) OssieConverter.parseYaml(result.yaml);
+
+ assertFalse(view.containsKey("dimensions"), result.yaml);
+ assertEquals(1, ((List<Object>) view.get("measures")).size(), result.yaml);
+ assertEquals(chainLength + 1, result.notices.size(), result.notices.toString());
+ List<String> cascadeNotices = result.notices.stream()
+ .filter(notice -> notice.contains("downstream of a dropped field/metric"))
+ .toList();
+ assertEquals(chainLength, cascadeNotices.size(), cascadeNotices.toString());
+ for (int index = 1; index <= chainLength; index++) {
+ assertEquals(
+ cascadeNotice("dimension", "d" + index, "d" + (index - 1)),
+ cascadeNotices.get(index - 1));
+ }
+ }
+
+ @Test
+ public void cascadeDropPreservesDimensionThenMeasurePhaseOrder() {
+ String osi =
+ "version: 0.2.0.dev0\n"
+ + "name: m\n"
+ + "datasets:\n"
+ + "- name: d\n"
+ + " source: c.s.d\n"
+ + " fields:\n"
+ + " - {name: d1, expression: {dialects: "
+ + "[{dialect: DATABRICKS, expression: bad_dim}]}}\n"
+ + " - {name: d0, expression: {dialects: "
+ + "[{dialect: DATABRICKS, expression: 'measure(m1)'}]}}\n"
+ + " - {name: bad_dim, expression: {dialects: "
+ + "[{dialect: SNOWFLAKE, expression: bad_dim}]}}\n"
+ + " metrics:\n"
Review Comment:
This YAML literal indents `metrics:` at 2 spaces, which nests it under the dataset item instead of putting it at the model root.
`schemaMapList(model, "metrics", ...)` returns empty, so `m1`/`m0`/`bad_measure`/`keep` never get parsed.
Running `mvn test` on this PR fails as a result (1 cascade notice observed vs 4 expected).
`metrics:` needs re-indented to column 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #333:
URL: https://github.com/apache/ossie/pull/333#discussion_r3980561133
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieToMetricView.java:
##########
@@ -0,0 +1,1167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAX_JOIN_NODES;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SYNONYM_LIMIT;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.findOutsideLiterals;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.foreignVendorExtensions;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.mergeDescription;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.pickExpression;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.readStash;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.replaceOutsideLiterals;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.require;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.synonymsOf;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * EXPORT direction: Apache Ossie semantic model -> Databricks Metric View v1.1 YAML. This is the
+ * harder direction: it reassembles a relationship graph into a join tree. Shared helpers come from
+ * {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class OssieToMetricView {
+
+ private OssieToMetricView() {}
+
+ // -- tree node ------------------------------------------------------------
+ private static final class Node {
+ final String dataset;
+ final Map<String, Object> rel; // empty for the fact/root
+ final boolean parentIsFrom;
+ String alias;
+ boolean isOtm;
+ final List<Node> children = new ArrayList<>();
+
+ Node(String dataset, Map<String, Object> rel, boolean parentIsFrom) {
+ this.dataset = dataset;
+ this.rel = rel;
+ this.parentIsFrom = parentIsFrom;
+ }
+ }
+
+ // -- public entry ---------------------------------------------------------
+ static Result convertOssieToMetricView(String osiYamlStr, String source) {
+ Notices notices = new Notices();
+ if (osiYamlStr == null || osiYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(osiYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> root = asMap(parsed);
+ Object versionValue = get(root, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!OSSIE_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Apache Ossie version '" + version + "'. Supported: " + OSSIE_VERSION,
+ version);
+ }
+ List<Object> models = schemaList(root, "semantic_model", "Apache Ossie YAML");
+ if (models.isEmpty()) {
+ throw ConversionException.invalidInput(
+ "'semantic_model' must be a non-empty list",
+ "its 'semantic_model' must be a non-empty list");
+ }
+ if (!(models.get(0) instanceof Map)) {
+ String reason = "Apache Ossie YAML: 'semantic_model[0]' must be a mapping";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ if (models.size() > 1) {
+ notices.warn("model", "multiple semantic models found; converting only the first");
+ }
+ Map<String, Object> view = convertModel((Map<String, Object>) models.get(0), source, notices);
+ try {
+ return new Result(MAPPER.writeValueAsString(view), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Metric View YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertModel(
+ Map<String, Object> model, String explicitSource, Notices notices) {
+ String name = model.containsKey("name") ? str(get(model, "name")) : "<unnamed>";
+ String modelScope = "Model '" + name + "'";
+ List<Map<String, Object>> datasetList = schemaMapList(model, "datasets", modelScope);
+ if (datasetList.isEmpty()) {
+ throw new ConversionException("Model '" + name + "' has no datasets");
+ }
+ Set<String> seen = new HashSet<>();
+ Map<String, Map<String, Object>> datasets = new LinkedHashMap<>();
+ Map<String, List<Map<String, Object>>> datasetFields = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasetList) {
+ String dsName = requireStr(d, "name", "Model '" + name + "': dataset");
+ if (!seen.add(dsName.trim().toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("Model '" + name + "': duplicate dataset name '" + dsName + "'");
+ }
+ datasets.put(dsName, d);
+ datasetFields.put(dsName, schemaMapList(d, "fields", "Dataset '" + dsName + "'"));
+ }
+ List<Map<String, Object>> relationships = schemaMapList(model, "relationships", modelScope);
+ List<Map<String, Object>> metrics = schemaMapList(model, "metrics", modelScope);
+
+ Map<String, Object> modelStash = readStash(model);
+ // A non-equi/filtered join has no schema-valid Ossie relationship (from/to columns are
+ // required), so MetricViewToOssie stashes it under the model's DATABRICKS custom_extensions
+ // (complex_joins) rather than emitting a stub relationship. Rebuild a columns-less
+ // relationship for each -- carrying its raw `on` (and rely/cardinality) in the stash that
+ // buildJoin restores from -- and merge them in so the join tree includes them.
+ for (Map<String, Object> pj : schemaMapList(modelStash, "complex_joins", modelScope)) {
+ Map<String, Object> rel = new LinkedHashMap<>();
+ rel.put("name", get(pj, "name"));
+ rel.put("from", get(pj, "from"));
+ rel.put("to", get(pj, "to"));
+ Map<String, Object> stash = new LinkedHashMap<>(pj);
+ stash.remove("name");
+ stash.remove("from");
+ stash.remove("to");
+ writeStash(rel, stash);
+ relationships.add(rel);
+ }
+ String factHint = explicitSource != null ? explicitSource : str(get(modelStash, STASH_SOURCE_KEY));
+ Object[] built = buildJoinTree(name, datasets, relationships, factHint, notices);
+ Node root = (Node) built[0];
+ String fact = (String) built[1];
+ Map<String, Integer> counts = assignAliases(root, fact);
+ markOtm(name, root);
+
+ Map<String, Object> factDs = datasets.get(fact);
+ Map<String, Object> view = new LinkedHashMap<>();
+ view.put("version", MV_VERSION);
+ view.put("source", validateSource(get(factDs, "source"), fact));
+
+ String comment = str(get(model, "description"));
+ if (truthy(comment)) {
+ view.put("comment", comment);
+ }
+ if (modelStash.containsKey("filter")) {
+ view.put("filter", modelStash.get("filter"));
+ }
+
+ List<Object> joins = new ArrayList<>();
+ for (Node child : root.children) {
+ joins.add(buildJoin(child, "source", datasets));
+ }
+ if (!joins.isEmpty()) {
+ view.put("joins", joins);
+ }
+
+ Set<String> droppedDims = new HashSet<>();
+ Set<String> droppedMeasures = new HashSet<>();
+ List<Map<String, Object>> dimensions = new ArrayList<>();
+ Set<String> seenDims = new HashSet<>();
+ // Dataset name -> the alias path that addresses its columns from the primary source, collected
+ // from the same walk the dimensions use so measures qualify identically (see qualifyMeasure).
+ Map<String, String> datasetAliasPath = new LinkedHashMap<>();
+ for (Object[] entry : nodeOrder(root)) {
+ Node node = (Node) entry[0];
+ @SuppressWarnings("unchecked")
+ List<String> joinPath = (List<String>) entry[1];
+ boolean isFact = node == root;
+ String prefix = counts.get(node.dataset) > 1 ? node.alias : null;
+ // Fixed per node, like `prefix`: hoisted so it is not rebuilt for every field.
+ String qualifier = String.join(".", joinPath);
+ if (!isFact) {
+ // A dataset reachable by more than one path (diamond) appears once per path; the first
+ // wins, matching the order dimensions are emitted in.
+ datasetAliasPath.putIfAbsent(node.dataset, qualifier);
Review Comment:
A dataset reached via two distinct join paths (a diamond) is silently mapped to only its first-seen alias path via `putIfAbsent`. A measure expression qualified by this dataset's bare name (`SUM(shared.amount)`) will always resolve to one arbitrary branch with no warning (unlike the dimension-path handling of the same ambiguity, which explicitily warns and drops).
For instance, fact `orders` -> `dim1` -> `shared` and `orders` -> `dim2` -> `shared`, plus `SUM(shared.amount)`, silently emits `SUM(dim1.dim1_shared.amount)` and drops the `dim2` branch.
Can we warn-and-drop here the same way the dimension path does?
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieToMetricView.java:
##########
@@ -0,0 +1,1167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAX_JOIN_NODES;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SYNONYM_LIMIT;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.findOutsideLiterals;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.foreignVendorExtensions;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.mergeDescription;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.pickExpression;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.readStash;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.replaceOutsideLiterals;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.require;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.synonymsOf;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * EXPORT direction: Apache Ossie semantic model -> Databricks Metric View v1.1 YAML. This is the
+ * harder direction: it reassembles a relationship graph into a join tree. Shared helpers come from
+ * {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class OssieToMetricView {
+
+ private OssieToMetricView() {}
+
+ // -- tree node ------------------------------------------------------------
+ private static final class Node {
+ final String dataset;
+ final Map<String, Object> rel; // empty for the fact/root
+ final boolean parentIsFrom;
+ String alias;
+ boolean isOtm;
+ final List<Node> children = new ArrayList<>();
+
+ Node(String dataset, Map<String, Object> rel, boolean parentIsFrom) {
+ this.dataset = dataset;
+ this.rel = rel;
+ this.parentIsFrom = parentIsFrom;
+ }
+ }
+
+ // -- public entry ---------------------------------------------------------
+ static Result convertOssieToMetricView(String osiYamlStr, String source) {
+ Notices notices = new Notices();
+ if (osiYamlStr == null || osiYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(osiYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> root = asMap(parsed);
+ Object versionValue = get(root, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Apache Ossie YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!OSSIE_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Apache Ossie version '" + version + "'. Supported: " + OSSIE_VERSION,
+ version);
+ }
+ List<Object> models = schemaList(root, "semantic_model", "Apache Ossie YAML");
+ if (models.isEmpty()) {
+ throw ConversionException.invalidInput(
+ "'semantic_model' must be a non-empty list",
+ "its 'semantic_model' must be a non-empty list");
+ }
+ if (!(models.get(0) instanceof Map)) {
+ String reason = "Apache Ossie YAML: 'semantic_model[0]' must be a mapping";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ if (models.size() > 1) {
+ notices.warn("model", "multiple semantic models found; converting only the first");
+ }
+ Map<String, Object> view = convertModel((Map<String, Object>) models.get(0), source, notices);
+ try {
+ return new Result(MAPPER.writeValueAsString(view), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Metric View YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertModel(
+ Map<String, Object> model, String explicitSource, Notices notices) {
+ String name = model.containsKey("name") ? str(get(model, "name")) : "<unnamed>";
+ String modelScope = "Model '" + name + "'";
+ List<Map<String, Object>> datasetList = schemaMapList(model, "datasets", modelScope);
+ if (datasetList.isEmpty()) {
+ throw new ConversionException("Model '" + name + "' has no datasets");
+ }
+ Set<String> seen = new HashSet<>();
+ Map<String, Map<String, Object>> datasets = new LinkedHashMap<>();
+ Map<String, List<Map<String, Object>>> datasetFields = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasetList) {
+ String dsName = requireStr(d, "name", "Model '" + name + "': dataset");
+ if (!seen.add(dsName.trim().toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("Model '" + name + "': duplicate dataset name '" + dsName + "'");
+ }
+ datasets.put(dsName, d);
+ datasetFields.put(dsName, schemaMapList(d, "fields", "Dataset '" + dsName + "'"));
+ }
+ List<Map<String, Object>> relationships = schemaMapList(model, "relationships", modelScope);
+ List<Map<String, Object>> metrics = schemaMapList(model, "metrics", modelScope);
+
+ Map<String, Object> modelStash = readStash(model);
+ // A non-equi/filtered join has no schema-valid Ossie relationship (from/to columns are
+ // required), so MetricViewToOssie stashes it under the model's DATABRICKS custom_extensions
+ // (complex_joins) rather than emitting a stub relationship. Rebuild a columns-less
+ // relationship for each -- carrying its raw `on` (and rely/cardinality) in the stash that
+ // buildJoin restores from -- and merge them in so the join tree includes them.
+ for (Map<String, Object> pj : schemaMapList(modelStash, "complex_joins", modelScope)) {
+ Map<String, Object> rel = new LinkedHashMap<>();
+ rel.put("name", get(pj, "name"));
+ rel.put("from", get(pj, "from"));
+ rel.put("to", get(pj, "to"));
+ Map<String, Object> stash = new LinkedHashMap<>(pj);
+ stash.remove("name");
+ stash.remove("from");
+ stash.remove("to");
+ writeStash(rel, stash);
+ relationships.add(rel);
+ }
+ String factHint = explicitSource != null ? explicitSource : str(get(modelStash, STASH_SOURCE_KEY));
+ Object[] built = buildJoinTree(name, datasets, relationships, factHint, notices);
+ Node root = (Node) built[0];
+ String fact = (String) built[1];
+ Map<String, Integer> counts = assignAliases(root, fact);
+ markOtm(name, root);
+
+ Map<String, Object> factDs = datasets.get(fact);
+ Map<String, Object> view = new LinkedHashMap<>();
+ view.put("version", MV_VERSION);
+ view.put("source", validateSource(get(factDs, "source"), fact));
+
+ String comment = str(get(model, "description"));
+ if (truthy(comment)) {
+ view.put("comment", comment);
+ }
+ if (modelStash.containsKey("filter")) {
+ view.put("filter", modelStash.get("filter"));
+ }
+
+ List<Object> joins = new ArrayList<>();
+ for (Node child : root.children) {
+ joins.add(buildJoin(child, "source", datasets));
+ }
+ if (!joins.isEmpty()) {
+ view.put("joins", joins);
+ }
+
+ Set<String> droppedDims = new HashSet<>();
+ Set<String> droppedMeasures = new HashSet<>();
+ List<Map<String, Object>> dimensions = new ArrayList<>();
+ Set<String> seenDims = new HashSet<>();
+ // Dataset name -> the alias path that addresses its columns from the primary source, collected
+ // from the same walk the dimensions use so measures qualify identically (see qualifyMeasure).
+ Map<String, String> datasetAliasPath = new LinkedHashMap<>();
+ for (Object[] entry : nodeOrder(root)) {
+ Node node = (Node) entry[0];
+ @SuppressWarnings("unchecked")
+ List<String> joinPath = (List<String>) entry[1];
+ boolean isFact = node == root;
+ String prefix = counts.get(node.dataset) > 1 ? node.alias : null;
+ // Fixed per node, like `prefix`: hoisted so it is not rebuilt for every field.
+ String qualifier = String.join(".", joinPath);
+ if (!isFact) {
+ // A dataset reachable by more than one path (diamond) appears once per path; the first
+ // wins, matching the order dimensions are emitted in.
+ datasetAliasPath.putIfAbsent(node.dataset, qualifier);
+ }
+ for (Map<String, Object> field : datasetFields.get(node.dataset)) {
+ String fname = requireStr(field, "name", "dataset '" + node.dataset + "': field");
+ if (node.isOtm) {
+ notices.warn("field '" + fname + "'",
+ "column on a one-to-many-joined table cannot be a dimension "
+ + "(must resolve to one value per source row); dropped");
+ droppedDims.add(fname);
+ continue;
+ }
+ Map<String, Object> dim =
+ convertField(field, fname, qualifier, isFact, prefix, notices);
+ if (dim == null) {
+ droppedDims.add(fname);
+ continue;
+ }
+ String dn = (String) dim.get("name");
+ if (!seenDims.add(dn.toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("dataset '" + node.dataset + "': dimension name '" + dn
+ + "' collides with another dimension/measure; Metric Views require unique "
+ + "dimension/measure names -- rename before use");
+ }
+ dimensions.add(dim);
+ }
+ }
+
+ List<Map<String, Object>> measures = new ArrayList<>();
+ // Depends only on the fact name, so compile it once rather than per metric.
+ Pattern factQualifier = Pattern.compile("\\b" + Pattern.quote(fact) + "\\.");
+ // Likewise fixed once the join tree is walked: one alternation over the joined dataset names.
+ Pattern datasetHead = qualifierChainPattern(datasetAliasPath.keySet());
+ for (Map<String, Object> metric : metrics) {
+ Map<String, Object> measure =
+ convertMetric(metric, factQualifier, datasetHead, datasetAliasPath, seenDims, notices);
+ if (measure == null) {
+ droppedMeasures.add(str(get(metric, "name")));
+ continue;
+ }
+ measures.add(measure);
+ }
+
+ cascadeDrop(dimensions, measures, droppedDims, droppedMeasures, notices);
+
+ // A Metric View must define at least one dimension or measure
+ // (SingleSourceMetricView.validate rejects an empty `select`), so a view with neither is one
+ // the engine refuses at CREATE. Fail here instead, naming the dropped columns: after
+ // cascadeDrop the emptiness is usually a consequence of earlier drops rather than an empty
+ // input, and those names are the actionable part.
+ if (dimensions.isEmpty() && measures.isEmpty()) {
+ StringBuilder msg = new StringBuilder("Model '" + name
+ + "' produced no dimensions or measures; a Metric View requires at least one.");
+ if (!droppedDims.isEmpty() || !droppedMeasures.isEmpty()) {
+ msg.append(" Dropped during conversion:");
+ if (!droppedDims.isEmpty()) {
+ msg.append(" dimensions ").append(sortedNames(droppedDims));
+ }
+ if (!droppedMeasures.isEmpty()) {
+ msg.append(" measures ").append(sortedNames(droppedMeasures));
+ }
+ msg.append(" -- see the warnings for why each was dropped.");
+ }
+ throw new ConversionException(msg.toString());
+ }
+
+ if (!dimensions.isEmpty()) {
+ view.put("dimensions", dimensions);
+ }
+ if (!measures.isEmpty()) {
+ view.put("measures", measures);
+ }
+ if (modelStash.containsKey("parameters")) {
+ view.put("parameters", modelStash.get("parameters"));
+ }
+ if (modelStash.containsKey("materialization")) {
+ view.put("materialization", modelStash.get("materialization"));
+ }
+
+ warnDroppedModel(model, notices);
+ return view;
+ }
+
+ private static List<Map<String, Object>> schemaMapList(
+ Map<String, Object> parent, String key, String scope) {
+ List<Object> values = schemaList(parent, key, scope);
+ List<Map<String, Object>> result = new ArrayList<>();
+ int index = 0;
+ for (Object element : values) {
+ if (!(element instanceof Map)) {
+ String reason = scope + ": '" + key + "[" + index + "]' must be a mapping";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ result.add((Map<String, Object>) element);
+ index++;
+ }
+ return result;
+ }
+
+ private static List<Object> schemaList(
+ Map<String, Object> parent, String key, String scope) {
+ if (!parent.containsKey(key)) {
+ return new ArrayList<>();
+ }
+ Object value = get(parent, key);
+ // A present-but-null value (the bare `key:` YAML idiom) is treated like an absent key, matching
+ // the tolerant base reader (asList) and the sibling customExtensions reader. A scalar or map
+ // value is still rejected below. Required collections stay guarded by their own emptiness
+ // checks, so tolerating null here does not let an empty semantic_model/datasets through.
+ if (value == null) {
+ return new ArrayList<>();
+ }
+ if (!(value instanceof List)) {
+ String reason = scope + ": '" + key + "' must be a list";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ return (List<Object>) value;
+ }
+
+ private static Object[] buildJoinTree(
+ String modelName, Map<String, Map<String, Object>> datasets,
+ List<Map<String, Object>> relationships0, String factHint, Notices notices) {
+ for (Map<String, Object> rel : relationships0) {
+ String scope = "Model '" + modelName + "': relationship '"
+ + (rel.containsKey("name") ? str(get(rel, "name")) : "<unnamed>") + "'";
+ Object f = require(rel, "from", scope);
+ Object t = require(rel, "to", scope);
+ if (!datasets.containsKey(str(f)) || !datasets.containsKey(str(t))) {
+ throw new ConversionException("Model '" + modelName + "': relationship '"
+ + str(get(rel, "name")) + "' references an unknown dataset");
+ }
+ }
+ List<Map<String, Object>> relationships = new ArrayList<>();
+ for (Map<String, Object> rel : relationships0) {
+ relationships.add(orientByKey(rel, datasets, notices));
+ }
+ if (datasets.size() > MAX_JOIN_NODES) {
+ throw new ConversionException("Model '" + modelName + "' has " + datasets.size()
+ + " datasets; at most " + MAX_JOIN_NODES + " are supported.");
+ }
+ String fact = pickFact(modelName, datasets, relationships, factHint);
+ rejectDirectedCycle(modelName, datasets, relationships);
+
+ Map<String, List<String>> adj = new HashMap<>();
+ for (String n : datasets.keySet()) {
+ adj.put(n, new ArrayList<>());
+ }
+ for (Map<String, Object> rel : relationships) {
+ adj.get(str(get(rel, "from"))).add(str(get(rel, "to")));
+ adj.get(str(get(rel, "to"))).add(str(get(rel, "from")));
+ }
+ Map<String, Integer> dist = new HashMap<>();
+ dist.put(fact, 0);
+ Deque<String> queue = new ArrayDeque<>();
+ queue.add(fact);
+ while (!queue.isEmpty()) {
+ String cur = queue.poll();
+ for (String nb : adj.get(cur)) {
+ if (!dist.containsKey(nb)) {
+ dist.put(nb, dist.get(cur) + 1);
+ queue.add(nb);
+ }
+ }
+ }
+ List<String> unreachable = new ArrayList<>();
+ for (String n : datasets.keySet()) {
+ if (!dist.containsKey(n)) {
+ unreachable.add(n);
+ }
+ }
+ if (!unreachable.isEmpty()) {
+ java.util.Collections.sort(unreachable);
+ throw new ConversionException("Model '" + modelName + "': datasets " + unreachable
+ + " are not reachable from fact '" + fact + "' via relationships.");
+ }
+ Map<String, List<Object[]>> childrenOf = new HashMap<>();
+ for (String n : datasets.keySet()) {
+ childrenOf.put(n, new ArrayList<>());
+ }
+ for (Map<String, Object> rel : relationships) {
+ String a = str(get(rel, "from"));
+ String b = str(get(rel, "to"));
+ if (dist.get(a).equals(dist.get(b))) {
+ throw new ConversionException("Model '" + modelName + "': relationship '"
+ + str(get(rel, "name")) + "' joins two datasets equidistant from the fact; "
+ + "the graph is not tree-shaped (it contains a cycle).");
+ }
+ String parent = dist.get(a) < dist.get(b) ? a : b;
+ String child = dist.get(a) < dist.get(b) ? b : a;
+ childrenOf.get(parent).add(new Object[] {child, rel, parent.equals(str(get(rel, "from")))});
+ }
+ int[] counter = {0};
+ Node root = build(modelName, fact, new LinkedHashMap<>(), false, childrenOf, counter);
+ return new Object[] {root, fact};
+ }
+
+ private static Node build(String modelName, String dataset, Map<String, Object> rel,
+ boolean parentIsFrom, Map<String, List<Object[]>> childrenOf, int[] counter) {
+ counter[0]++;
+ if (counter[0] > MAX_JOIN_NODES) {
+ throw new ConversionException("Model '" + modelName + "': join graph fans out to more than "
+ + MAX_JOIN_NODES + " joins; check for an unintended diamond explosion.");
+ }
+ Node node = new Node(dataset, rel, parentIsFrom);
+ for (Object[] c : childrenOf.get(dataset)) {
+ @SuppressWarnings("unchecked")
+ Map<String, Object> crel = (Map<String, Object>) c[1];
+ node.children.add(build(modelName, (String) c[0], crel, (Boolean) c[2], childrenOf, counter));
+ }
+ return node;
+ }
+
+ private static Map<String, Integer> assignAliases(Node root, String fact) {
+ Map<String, Integer> counts = new HashMap<>();
+ countNode(root, counts);
+ Set<String> used = new HashSet<>();
+ used.add("source");
+ assign(root, null, fact, counts, used);
+ return counts;
+ }
+
+ private static void countNode(Node node, Map<String, Integer> counts) {
+ counts.merge(node.dataset, 1, Integer::sum);
+ for (Node c : node.children) {
+ countNode(c, counts);
+ }
+ }
+
+ private static void assign(Node node, String parentAlias, String fact,
+ Map<String, Integer> counts, Set<String> used) {
+ String alias;
+ if (node.dataset.equals(fact)) {
+ alias = "source";
+ } else {
+ String base;
+ if (counts.get(node.dataset) == 1) {
+ base = node.dataset;
+ } else if (parentAlias != null && !parentAlias.equals("source")) {
+ base = parentAlias + "_" + node.dataset;
+ } else {
+ base = node.dataset;
+ }
+ alias = base;
+ int n = 2;
+ while (used.contains(alias)) {
+ alias = base + "_" + n;
+ n++;
+ }
+ }
+ node.alias = alias;
+ used.add(alias);
+ for (Node c : node.children) {
+ assign(c, alias, fact, counts, used);
+ }
+ }
+
+ private static String pickFact(String modelName, Map<String, Map<String, Object>> datasets,
+ List<Map<String, Object>> relationships, String factHint) {
+ if (factHint != null) {
+ if (!datasets.containsKey(factHint)) {
+ throw new ConversionException(
+ "Model '" + modelName + "': requested source '" + factHint + "' is not a dataset");
+ }
+ return factHint;
+ }
+ if (datasets.size() > 1 && relationships.isEmpty()) {
+ throw new ConversionException("Model '" + modelName + "': " + datasets.size()
+ + " datasets but no relationships; cannot determine the fact table.");
+ }
+ Map<String, Integer> incoming = new LinkedHashMap<>();
+ for (String n : datasets.keySet()) {
+ incoming.put(n, 0);
+ }
+ for (Map<String, Object> rel : relationships) {
+ incoming.merge(str(get(rel, "to")), 1, Integer::sum);
+ }
+ List<String> roots = new ArrayList<>();
+ for (Map.Entry<String, Integer> e : incoming.entrySet()) {
+ if (e.getValue() == 0) {
+ roots.add(e.getKey());
+ }
+ }
+ if (roots.isEmpty()) {
+ throw new ConversionException("Model '" + modelName + "': join graph contains a cycle "
+ + "(no root dataset). A Metric View requires an acyclic, tree-shaped graph.");
+ }
+ if (roots.size() > 1) {
+ java.util.Collections.sort(roots);
+ throw new ConversionException("Model '" + modelName + "': multiple candidate fact datasets "
+ + roots + ". Name the grain with --source.");
+ }
+ return roots.get(0);
+ }
+
+ /**
+ * Marks each joined node with its branch's cardinality and rejects a branch that mixes the two.
+ *
+ * <p>A Metric View requires every join within one top-level branch to share a single cardinality:
+ * `Join.validateSubJoinCardinalities` seeds the expected value from the top-level join and fails
+ * any descendant that differs (an absent `cardinality` reads as `many_to_one`). So a mixed branch
+ * is rejected in *either* direction -- a many-to-one nested under one-to-many, and equally a
+ * one-to-many nested under many-to-one. Emitting one would produce a view the engine refuses at
+ * CREATE, so reject it here with a converter-level error instead.
+ */
+ private static void markOtm(String modelName, Node root) {
+ // Each top-level join starts a branch and sets that branch's expected cardinality.
+ for (Node top : root.children) {
+ boolean branchIsOtm = !top.parentIsFrom;
+ top.isOtm = branchIsOtm;
+ markOtmVisit(modelName, top, branchIsOtm);
+ }
+ }
+
+ private static void markOtmVisit(String modelName, Node node, boolean branchIsOtm) {
Review Comment:
`markOtmVisit` rejects a join whose cardinality differs from its top-level branch's cardinality in either direction.
The Python reference (`_mark_otm`) only rejects reverting from one-to-many back to many-to-one: it explicitly allows a one-to-many join nested under a many-to-one branch. As written, a normal star-schema-with-fanout model (many-to-one to `customer`, then one-to-many to `customer_notes`) that Python accepts gets rejected here with "is one_to_many but descends from a many_to_one join".
Is the stricter check intentional, or should this match the Python behavior?
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/OssieConverterCommon.java:
##########
@@ -0,0 +1,673 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.regex.MatchResult;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.DeserializationFeature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import com.fasterxml.jackson.dataformat.yaml.YAMLGenerator;
+
+import org.yaml.snakeyaml.DumperOptions;
+import org.yaml.snakeyaml.LoaderOptions;
+import org.yaml.snakeyaml.Yaml;
+import org.yaml.snakeyaml.constructor.SafeConstructor;
+import org.yaml.snakeyaml.nodes.Tag;
+import org.yaml.snakeyaml.representer.Representer;
+import org.yaml.snakeyaml.resolver.Resolver;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+
+/**
+ * Shared constants, YAML I/O, and typed accessors for the Apache Ossie <-> Databricks
+ * Metric View converter. The direction-specific logic lives in {@link OssieToMetricView}
+ * (export) and {@link MetricViewToOssie} (import), which static-import these members. The public
+ * entry points and shared types are re-exported through {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class OssieConverterCommon {
+
+ // -- constants -------------------------------------------------------------
+ static final String OSSIE_VERSION = "0.2.0.dev0";
+ static final String MV_VERSION = "1.1";
+ static final String VENDOR = "DATABRICKS";
+ static final String DIALECT_DATABRICKS = "DATABRICKS";
+ static final String DIALECT_ANSI = "ANSI_SQL";
+ static final int SYNONYM_LIMIT = 10;
+ static final int STASH_VERSION = 1;
+ static final String STASH_SOURCE_KEY = "source_dataset";
+ static final String CARD_ONE_TO_MANY = "one_to_many";
+ static final String CARD_MANY_TO_ONE = "many_to_one";
+ static final int MAX_JOIN_NODES = 200;
+
+ static final Pattern IDENTIFIER_RE = Pattern.compile("^[A-Za-z_][A-Za-z0-9_]*$");
+ static final Pattern SELECT_WITH_RE =
+ Pattern.compile("(?i)^(select|with)\\b");
+
+ static final ObjectMapper MAPPER = buildMapper();
+ private static final ObjectMapper JSON_READER = new ObjectMapper()
+ .enable(JsonParser.Feature.STRICT_DUPLICATE_DETECTION)
+ .enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS);
+ // Writer for the custom_extensions stash blob. The exact byte format is pinned by the
+ // checked-in fixtures: a space after ':' and ', ' between entries, as in
+ // {"_v": 1, "filter": "x"}.
+ static final com.fasterxml.jackson.databind.ObjectWriter JSON_WRITER = buildJsonWriter();
+
+ /** A MinimalPrettyPrinter (no newlines) using the stash blob's separators: ": " / ", ". */
+ private static final class JsonDumpsPrinter
+ extends com.fasterxml.jackson.core.util.MinimalPrettyPrinter {
+ @Override
+ public void writeObjectFieldValueSeparator(com.fasterxml.jackson.core.JsonGenerator g)
+ throws java.io.IOException {
+ g.writeRaw(": ");
+ }
+ @Override
+ public void writeObjectEntrySeparator(com.fasterxml.jackson.core.JsonGenerator g)
+ throws java.io.IOException {
+ g.writeRaw(", ");
+ }
+ @Override
+ public void writeArrayValueSeparator(com.fasterxml.jackson.core.JsonGenerator g)
+ throws java.io.IOException {
+ g.writeRaw(", ");
+ }
+ }
+
+ private static com.fasterxml.jackson.databind.ObjectWriter buildJsonWriter() {
+ // ESCAPE_NON_ASCII keeps the blob pure ASCII (every non-ASCII
+ // char is emitted as a \\uXXXX escape). Set on the JsonFactory so it takes effect on the
+ // generator. Jackson emits the hex in UPPERCASE; writeStash lowercases it so the blob is
+ // byte-identical to the checked-in fixtures.
+ com.fasterxml.jackson.core.JsonFactory jf = new com.fasterxml.jackson.core.JsonFactory();
+ jf.enable(com.fasterxml.jackson.core.JsonGenerator.Feature.ESCAPE_NON_ASCII);
+ return new ObjectMapper(jf).writer(new JsonDumpsPrinter());
+ }
+
+ // Matches a real unicode escape in serialized JSON so writeStash can lowercase its hex digits.
+ //
+ // The escape must be preceded by an EVEN number of backslashes, otherwise the `u` belongs to an
+ // escaped backslash rather than to an escape sequence: a stashed value holding a literal
+ // backslash followed by "uABCD" serializes with a doubled backslash, where the `u` is ordinary
+ // text and must be left alone (lowercasing it there would corrupt the value). Group 1 captures
+ // the (possibly empty) run of escaped backslashes so it can be re-emitted verbatim; group 2 is
+ // the hex to lowercase.
+ private static final Pattern UNICODE_ESCAPE_RE =
+ Pattern.compile("(?<!\\\\)((?:\\\\\\\\)*)\\\\u([0-9A-Fa-f]{4})");
+
+ private static ObjectMapper buildMapper() {
+ // Quote scalars (do NOT minimize quotes) so a string like "1.1" or a bool-like token
+ // is emitted quoted and never re-parsed as a number/boolean. Used for the WRITE path only;
+ // reads go through the YAML-1.2 SnakeYAML loader below.
+ YAMLFactory yf = YAMLFactory.builder()
+ .disable(YAMLGenerator.Feature.WRITE_DOC_START_MARKER)
+ .build();
+ return new ObjectMapper(yf);
+ }
+
+ // A SnakeYAML Resolver with YAML 1.2 boolean semantics: only true/false are booleans, NOT the
+ // YAML 1.1 tokens on/off/yes/no/y/n. A metric view join's bare `on:` key, and any "on"/"off"
+ // string value, must read back as a string rather than silently becoming a boolean (a YAML 1.1
+ // reader would coerce it).
+ private static final class Yaml12Resolver extends Resolver {
+ @Override
+ protected void addImplicitResolvers() {
+ addImplicitResolver(Tag.MERGE, MERGE, "<");
+ addImplicitResolver(Tag.NULL, NULL, "~nN ");
+ addImplicitResolver(Tag.NULL, EMPTY, null);
+ addImplicitResolver(Tag.BOOL, java.util.regex.Pattern.compile(
+ "^(?:true|True|TRUE|false|False|FALSE)$"), "tTfF");
+ addImplicitResolver(Tag.INT, INT, "-+0123456789");
+ addImplicitResolver(Tag.FLOAT, FLOAT, "-+0123456789.");
+ addImplicitResolver(Tag.TIMESTAMP, TIMESTAMP, "0123456789");
+ }
+ }
+
+ // SnakeYAML readers are stateful; create one per conversion instead of retaining thread-local
+ // state on caller threads. Yaml also copies settings from loaderOptions into its constructor, so
+ // the constructor and reader must receive the same configured instance.
+ private static Yaml newYamlReader() {
+ LoaderOptions loaderOptions = new LoaderOptions();
+ loaderOptions.setAllowDuplicateKeys(false);
+ DumperOptions dumperOptions = new DumperOptions();
+ return new Yaml(new SafeConstructor(loaderOptions), new Representer(dumperOptions),
+ dumperOptions, loaderOptions, new Yaml12Resolver());
+ }
+
+ static Object loadYaml(String s) {
+ return newYamlReader().load(s);
+ }
+
+ private OssieConverterCommon() {}
+
+ // -- typed accessors over parsed YAML (Object) ----------------------------
+ @SuppressWarnings("unchecked")
+ static Map<String, Object> asMap(Object x) {
+ if (x instanceof Map) {
+ return (Map<String, Object>) x;
+ }
+ return new LinkedHashMap<>();
+ }
+
+ @SuppressWarnings("unchecked")
+ static List<Object> asList(Object x) {
+ if (x instanceof List) {
+ return (List<Object>) x;
+ }
+ return new ArrayList<>();
+ }
+
+ static Object get(Map<String, Object> m, String k) {
+ return m.get(k);
+ }
+
+ static String str(Object x) {
+ if (x == null) {
+ return null;
+ }
+ return x.toString();
+ }
+
+ static List<String> strList(Object x) {
+ List<String> out = new ArrayList<>();
+ for (Object o : asList(x)) {
+ if (o != null) {
+ out.add(o.toString());
+ }
+ }
+ return out;
+ }
+
+ // -- helpers ---------------------------------------------------------------
+ static boolean isSimpleIdentifier(Object expr) {
+ return expr instanceof String && IDENTIFIER_RE.matcher(((String) expr).trim()).matches();
+ }
+
+ /**
+ * Applies {@code pattern -> replacement} to {@code sql}, but only to the parts of the expression
+ * that are actual SQL code -- spans inside string literals ({@code '...'}, {@code "..."}),
+ * backquoted identifiers, {@code -- line} comments, and {@code /* block *}{@code /} comments are
+ * copied through untouched.
+ *
+ * <p>Measure expressions are rewritten to add or strip a fact qualifier, and a blind
+ * {@code replaceAll} over the raw text also rewrites any occurrence inside a literal: a measure
+ * such as {@code SUM(IF(source.region = 'source.us', amt, 0))} would silently become
+ * {@code ... = 'us'}, changing the predicate and therefore the measure's value. Only code spans
+ * may be rewritten.
+ *
+ * <p>{@code replacement} is treated as a literal string, not as a regex replacement template.
+ */
+ static String replaceOutsideLiterals(String sql, Pattern pattern, String replacement) {
+ return replaceOutsideLiterals(sql, pattern, m -> replacement);
+ }
+
+ /**
+ * As {@link #replaceOutsideLiterals(String, Pattern, String)}, but each match is replaced with
+ * {@code replacer.apply(match)} -- for a rewrite whose result depends on what matched. The
+ * returned text is inserted literally, not as a regex replacement template.
+ */
+ static String replaceOutsideLiterals(
+ String sql, Pattern pattern, Function<MatchResult, String> replacer) {
+ StringBuilder out = new StringBuilder(sql.length());
+ int codeStart = 0;
+ for (int[] span : literalSpans(sql)) {
+ // Rewrite the code span that precedes this literal/comment, then copy the span verbatim.
+ out.append(rewriteLiterally(sql.substring(codeStart, span[0]), pattern, replacer));
+ out.append(sql, span[0], span[1]);
+ codeStart = span[1];
+ }
+ out.append(rewriteLiterally(sql.substring(codeStart), pattern, replacer));
+ return out.toString();
+ }
+
+ /**
+ * True when {@code pattern} matches the SQL code of {@code sql} -- the find-only counterpart of
+ * {@link #replaceOutsideLiterals}, with the same string-literal/comment blindness.
+ *
+ * <p>Needed wherever a decision is made from an expression's text: a bare
+ * {@code pattern.matcher(sql).find()} also sees inside literals, so a column name that only
+ * occurs in a string (as in {@code SUM(IF(region = 'us', amt, 0))}) reads as a reference to a
+ * column named {@code us}.
+ */
+ static boolean findOutsideLiterals(String sql, Pattern pattern) {
+ List<int[]> spans = literalSpans(sql);
+ if (spans.isEmpty()) {
+ return pattern.matcher(sql).find();
+ }
+ // Blank the literal/comment spans rather than removing them, so the surrounding code keeps its
+ // token boundaries (the pattern may rely on \b or a look-around).
+ StringBuilder code = new StringBuilder(sql);
+ for (int[] span : spans) {
+ for (int i = span[0]; i < span[1]; i++) {
+ code.setCharAt(i, ' ');
+ }
+ }
+ return pattern.matcher(code).find();
+ }
+
+ /**
+ * Start (inclusive) and end (exclusive) offsets of every span in {@code sql} that is not SQL
+ * code: string literals ({@code '...'}, {@code "..."}), backquoted identifiers, {@code -- line}
+ * comments, and {@code /* block *}{@code /} comments.
+ */
+ private static List<int[]> literalSpans(String sql) {
+ List<int[]> spans = new ArrayList<>();
+ int i = 0;
+ while (i < sql.length()) {
+ char c = sql.charAt(i);
+ int skipTo = -1;
+ if (c == '\'' || c == '"' || c == '`') {
+ skipTo = endOfQuoted(sql, i, c);
+ } else if (c == '-' && i + 1 < sql.length() && sql.charAt(i + 1) == '-') {
+ int nl = sql.indexOf('\n', i);
+ skipTo = nl < 0 ? sql.length() : nl;
+ } else if (c == '/' && i + 1 < sql.length() && sql.charAt(i + 1) == '*') {
+ int end = sql.indexOf("*/", i + 2);
+ skipTo = end < 0 ? sql.length() : end + 2;
+ }
+ if (skipTo < 0) {
+ i++;
+ continue;
+ }
+ spans.add(new int[] {i, skipTo});
+ i = skipTo;
+ }
+ return spans;
+ }
+
+ /** Index just past the quoted span starting at {@code start}; handles doubled-quote escapes. */
+ private static int endOfQuoted(String sql, int start, char quote) {
+ int i = start + 1;
+ while (i < sql.length()) {
+ char c = sql.charAt(i);
+ if (c == '\\' && quote != '`' && i + 1 < sql.length()) {
+ i += 2;
+ continue;
+ }
+ if (c == quote) {
+ // A doubled quote is an escaped quote, not the end of the span.
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i += 2;
+ continue;
+ }
+ return i + 1;
+ }
+ i++;
+ }
+ // Unterminated literal: treat the remainder as part of the span rather than rewriting it.
+ return sql.length();
+ }
+
+ private static String rewriteLiterally(
+ String code, Pattern pattern, Function<MatchResult, String> replacer) {
+ return pattern.matcher(code)
+ .replaceAll(m -> Matcher.quoteReplacement(replacer.apply(m)));
+ }
+
+ /**
+ * A pattern matching a maximal dotted run of {@code names} that qualifies a column reference --
+ * {@code parent.child.} -- with group 1 holding the run without its trailing dot. Null when
+ * {@code names} is empty.
+ *
+ * <p>The whole run is matched at once, rather than one name at a time, so a rewrite driven by
+ * this pattern can never re-match a qualifier it has just produced: rewriting
+ * {@code customer.region.} name-by-name would see the {@code region.} inside its own output and
+ * qualify it a second time. The required trailing dot keeps a column that merely shares a name
+ * with a dataset ({@code SUM(customer.region)}) from being read as a qualifier.
+ */
+ static Pattern qualifierChainPattern(Collection<String> names) {
+ if (names.isEmpty()) {
+ return null;
+ }
+ // Longest name first: regex alternation is ordered, so at a given position the longest name
+ // wins (`nation_x` must never match as `nation`).
+ List<String> sorted = new ArrayList<>(names);
+ sorted.sort((a, b) -> b.length() - a.length());
+ StringBuilder alternation = new StringBuilder();
+ for (String name : sorted) {
+ if (alternation.length() > 0) {
+ alternation.append('|');
+ }
+ alternation.append(Pattern.quote(name));
+ }
+ String one = "(?:" + alternation + ")";
+ return Pattern.compile("\\b(" + one + "(?:\\." + one + ")*)\\.");
+ }
+
+ /**
+ * Rewrites each qualifier run {@code chain} matches by mapping the run's LAST name through
+ * {@code resolve} -- the name closest to the column is the one being addressed, as
+ * {@code resolveColumn} treats a dimension's alias path. A run whose leaf {@code resolve} does
+ * not know is left alone. Literals and comments are skipped, and the pass is single, so the
+ * result is independent of the order the names happen to be in.
+ */
+ static String rewriteQualifiers(String expr, Pattern chain, Function<String, String> resolve) {
+ if (chain == null) {
+ return expr;
+ }
+ return replaceOutsideLiterals(expr, chain, m -> {
+ String leaf = lastIdentifier(m.group(1));
+ String resolved = leaf == null ? null : resolve.apply(leaf);
+ return resolved == null ? m.group() : resolved + ".";
+ });
+ }
+
+ // Presence is tested by key, so a legitimately falsy non-string value (0, false) is returned;
+ // a missing key, a null, or an empty/whitespace-only string is rejected.
+ static Object require(Map<String, Object> obj, String key, String what) {
+ if (!obj.containsKey(key) || obj.get(key) == null) {
+ throw new ConversionException(what + " is missing required '" + key + "'");
+ }
+ Object value = obj.get(key);
+ if (value instanceof String && ((String) value).trim().isEmpty()) {
+ throw new ConversionException(what + " has an empty '" + key + "'");
+ }
+ return value;
+ }
+
+ // Like require(), but the value must be a string.
+ static String requireStr(Map<String, Object> obj, String key, String what) {
+ Object v = require(obj, key, what);
+ if (v instanceof String) {
+ return (String) v;
+ }
+ throw new ConversionException(
+ what + ": '" + key + "' must be a string, got " + v.getClass().getSimpleName());
+ }
+
+ static String validateSource(Object source, String datasetName) {
+ String s = source == null ? "" : source.toString().trim();
+ if (s.isEmpty()) {
+ throw new ConversionException("Dataset '" + datasetName + "': missing/empty 'source'");
+ }
+ if (SELECT_WITH_RE.matcher(s).find()) {
+ return s;
+ }
+ String[] parts = s.split("\\.", -1);
+ boolean ok = parts.length == 3;
+ if (ok) {
+ for (String p : parts) {
+ if (p.isEmpty() || containsWhitespace(p)) {
+ ok = false;
+ break;
+ }
+ }
+ }
+ if (ok) {
+ return s;
+ }
+ throw new ConversionException("Dataset '" + datasetName + "': source '" + source
+ + "' must be a 3-part catalog.schema.table identifier or a SELECT/WITH subquery");
+ }
+
+ private static boolean containsWhitespace(String p) {
+ for (int i = 0; i < p.length(); i++) {
+ if (Character.isWhitespace(p.charAt(i))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ static String pickExpression(Object osiExpression, String scope) {
+ // Keep the raw (possibly non-string) values so
+ // the type check below can fire; select DATABRICKS-or-ANSI by truthiness (`or`), so a
+ // null/empty DATABRICKS expr falls through to ANSI; and raise on a non-string chosen
+ // value rather than silently coercing it.
+ if (osiExpression == null) {
+ return null;
+ }
+ if (!(osiExpression instanceof Map)) {
+ String reason = scope + ": 'expression' must be a mapping";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ Map<String, Object> expression = asMap(osiExpression);
+ Object dialectValues = get(expression, "dialects");
+ if (dialectValues == null) {
+ return null;
+ }
+ if (!(dialectValues instanceof List)) {
+ String reason = scope + ": 'expression.dialects' must be a list";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ Map<String, Object> dialects = new LinkedHashMap<>();
+ int index = 0;
+ for (Object d : (List<Object>) dialectValues) {
+ if (!(d instanceof Map)) {
+ String reason = scope + ": 'expression.dialects[" + index + "]' must be a mapping";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ Map<String, Object> dm = (Map<String, Object>) d;
+ dialects.put(str(get(dm, "dialect")), get(dm, "expression"));
+ index++;
+ }
+ Object chosen = truthy(dialects.get(DIALECT_DATABRICKS))
+ ? dialects.get(DIALECT_DATABRICKS) : dialects.get(DIALECT_ANSI);
+ if (chosen != null && !(chosen instanceof String)) {
+ throw new ConversionException(
+ "expression must be a string, got " + chosen.getClass().getSimpleName());
+ }
+ return (String) chosen; // null when neither dialect present -> caller warns and skips
+ }
+
+ /**
+ * Emptiness test used throughout the converter for optional YAML values: null, the empty string,
+ * an empty list/map, numeric zero, and boolean false are all treated as absent; everything else
+ * is present. Used for the DATABRICKS-or-ANSI expression fallthrough (so an empty or absent
+ * DATABRICKS expr falls through to ANSI) and for optional-field mapping (so an empty
+ * `comment`/`synonyms` is dropped rather than emitted as an empty value).
+ */
+ static boolean truthy(Object v) {
+ if (v == null) {
+ return false;
+ }
+ if (v instanceof String) {
+ return !((String) v).isEmpty();
+ }
+ if (v instanceof java.util.Collection) {
+ return !((java.util.Collection<?>) v).isEmpty();
+ }
+ if (v instanceof Map) {
+ return !((Map<?, ?>) v).isEmpty();
+ }
+ if (v instanceof Number) {
+ return ((Number) v).doubleValue() != 0.0;
+ }
+ if (v instanceof Boolean) {
+ return (Boolean) v;
+ }
+ return true;
+ }
+
+ static List<String> synonymsOf(Object aiContext) {
+ if (aiContext instanceof Map) {
+ return strList(get(asMap(aiContext), "synonyms"));
+ }
+ return new ArrayList<>();
+ }
+
+ static String mergeDescription(Object description, Object aiContext) {
+ String desc = str(description);
+ if (aiContext instanceof String && !((String) aiContext).trim().isEmpty()) {
+ String s = (String) aiContext;
+ // When both are present they are joined with a newline; otherwise the
+ // `if description` is a truthiness test, so an empty (or null) description returns the
+ // ai_context alone rather than prepending a stray newline.
+ return (desc != null && !desc.isEmpty()) ? desc + "\n" + s : s;
+ }
+ return desc;
+ }
+
+ private static List<Map<String, Object>> customExtensions(Map<String, Object> obj) {
+ Object value = get(obj, "custom_extensions");
+ if (value == null) {
+ return new ArrayList<>();
+ }
+ if (!(value instanceof List)) {
+ String reason = "'custom_extensions' must be a list";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ List<Map<String, Object>> extensions = new ArrayList<>();
+ int index = 0;
+ for (Object extension : (List<Object>) value) {
+ if (!(extension instanceof Map)) {
+ String reason = "'custom_extensions[" + index + "]' must be a mapping";
+ throw ConversionException.invalidInput(reason, reason);
+ }
+ extensions.add((Map<String, Object>) extension);
+ index++;
+ }
+ return extensions;
+ }
+
+ static Map<String, Object> readStash(Map<String, Object> obj) {
+ Map<String, Object> stash = null;
+ for (Map<String, Object> ext : customExtensions(obj)) {
+ if (VENDOR.equals(str(get(ext, "vendor_name")))) {
+ if (stash != null) {
Review Comment:
`readStash` throws when more than one DATABRICKS `custom_extensions` entry is present.
Python's `read_stash` just returns the first match and ignores later duplicates.
An Ossie YAML with two such entries that Python converts fine gets rejected here.
Worth aligning with the Python behavior (first-wins) rather than erroring?
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java:
##########
@@ -0,0 +1,581 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.DIALECT_DATABRICKS;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SELECT_WITH_RE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.lastIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * IMPORT direction: Metric View v1.1 YAML -> Apache Ossie semantic model. Shared helpers come
+ * from {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class MetricViewToOssie {
+
+ private static final String[] MODEL_STASH_KEYS = {"filter", "parameters", "materialization"};
+ private static final String[] JOIN_STASH_KEYS = {"rely", "cardinality"};
+ // Metric-View-only column fields with no Apache Ossie representation, preserved verbatim in the
+ // DATABRICKS stash so an import can restore them. `window` and `partition` are measure-only
+ // (they live on MeasureExpression); a dimension never carries them, so the shared key list is
+ // simply never hit for those on the dimension path.
+ private static final String[] COLUMN_STASH_KEYS = {"format", "window", "partition"};
+ // Measure-only fields with no Apache Ossie Metric representation. A dimension's display_name
+ // maps to the Field `label`, but the Ossie Metric schema has no `label`, so a measure's
+ // display_name is preserved in the DATABRICKS stash instead. Kept separate from
+ // COLUMN_STASH_KEYS so the dimension path (which already maps display_name to `label`) does not
+ // also stash it.
+ private static final String[] MEASURE_STASH_KEYS = {"display_name"};
+ private static final Pattern NON_EQUI_RE = Pattern.compile("[<>!]=|<>|[<>]");
+ private static final Pattern AND_SPLIT_RE = Pattern.compile("\\s+AND\\s+", Pattern.CASE_INSENSITIVE);
+ private static final Pattern EQ_CLAUSE_RE = Pattern.compile("^\\s*(.+?)\\s*=\\s*(.+?)\\s*$");
+
+ private MetricViewToOssie() {}
+
+ static Result convertMetricViewToOssie(String mvYamlStr, String modelName) {
+ Notices notices = new Notices();
+ if (mvYamlStr == null || mvYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(mvYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> view = asMap(parsed);
+ Object versionValue = get(view, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!MV_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Metric View version '" + version
+ + "'. This converter targets v" + MV_VERSION + " only.",
+ version);
+ }
+ Map<String, Object> model = convertView(view, modelName, notices);
+ Map<String, Object> out = new LinkedHashMap<>();
+ out.put("version", OSSIE_VERSION);
+ List<Object> models = new ArrayList<>();
+ models.add(model);
+ out.put("semantic_model", models);
+ return serializeOssie(out, notices);
+ }
+
+ static Result serializeOssie(Map<String, Object> out, Notices notices) {
+ try {
+ return new Result(MAPPER.writeValueAsString(out), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Apache Ossie YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertView(
+ Map<String, Object> view, String modelName, Notices notices) {
+ Object source = get(view, "source");
+ if (source == null || source.toString().isEmpty()) {
+ throw new ConversionException("Metric View is missing required 'source'");
+ }
+ // Same subquery test validateSource applies below, so the two cannot drift.
+ boolean isSql = SELECT_WITH_RE.matcher(source.toString().trim()).find();
+ String lastId = lastIdentifier(source);
+ String factName;
+ if (modelName != null) {
+ factName = modelName;
+ } else if (!isSql && lastId != null && isSimpleIdentifier(lastId)) {
+ factName = lastId;
+ } else {
+ factName = "metric_view";
+ }
+ validateSource(source, factName);
+
+ List<Map<String, Object>> datasets = new ArrayList<>();
+ Map<String, Object> factDs = new LinkedHashMap<>();
+ factDs.put("name", factName);
+ factDs.put("source", source);
+ datasets.add(factDs);
+ List<Map<String, Object>> relationships = new ArrayList<>();
+ // Joins with no schema-valid Ossie relationship (a non-decomposable `on`) are collected here
+ // and stashed under the model's DATABRICKS custom_extensions instead of `relationships`.
+ List<Map<String, Object>> complexJoins = new ArrayList<>();
+ Map<String, String> aliasToDataset = new HashMap<>();
+ aliasToDataset.put("source", factName);
+ aliasToDataset.put(factName, factName);
+ Set<String> seenNames = new HashSet<>();
+ seenNames.add(factName.trim().toLowerCase(Locale.ROOT));
+
+ walk(factName, "source", asList(get(view, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+
+ // `fields` is a v1.1 alias for `dimensions`: an empty `dimensions: []` falls through to
+ // `fields`, and the "both set" warning fires only when BOTH are non-empty (not merely
+ // present).
+ List<Object> dimList = asList(get(view, "dimensions"));
+ List<Object> fieldList = asList(get(view, "fields"));
+ if (!dimList.isEmpty() && !fieldList.isEmpty()) {
+ notices.warn("view", "both 'dimensions' and 'fields' are set; 'fields' is a v1.1 alias "
+ + "for 'dimensions', so the 'fields' list is ignored");
+ }
+ Map<String, List<Object>> fieldsByDataset = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasets) {
+ fieldsByDataset.put((String) d.get("name"), new ArrayList<>());
+ }
+ List<Object> dims = !dimList.isEmpty() ? dimList : fieldList;
+ for (Object dimObj : dims) {
+ Map<String, Object> dim = asMap(dimObj);
+ if (isWildcard(dim)) {
+ notices.warn("dimension", "wildcard column '" + str(get(dim, "expr"))
+ + "' has no Apache Ossie field representation; skipped");
+ continue;
+ }
+ Object[] converted = convertDimension(dim, aliasToDataset, factName);
+ fieldsByDataset.get((String) converted[0]).add(converted[1]);
+ }
+ for (Map<String, Object> d : datasets) {
+ List<Object> flds = fieldsByDataset.get((String) d.get("name"));
+ if (!flds.isEmpty()) {
+ d.put("fields", flds);
+ }
+ }
+
+ List<Object> metrics = new ArrayList<>();
+ // Fixed once the join tree is walked, so compile it once for all measures.
+ Pattern aliasHead = qualifierChainPattern(aliasToDataset.keySet());
+ for (Object mObj : asList(get(view, "measures"))) {
+ Map<String, Object> m = asMap(mObj);
+ if (isWildcard(m)) {
+ notices.warn("measure", "wildcard measure '" + str(get(m, "expr"))
+ + "' has no Apache Ossie metric representation; skipped");
+ continue;
+ }
+ metrics.add(convertMeasure(m, aliasHead, aliasToDataset));
+ }
+
+ Map<String, Object> model = new LinkedHashMap<>();
+ model.put("name", factName);
+ if (truthy(get(view, "comment"))) {
+ model.put("description", get(view, "comment"));
+ }
+ model.put("datasets", datasets);
+ if (!relationships.isEmpty()) {
+ model.put("relationships", relationships);
+ }
+ if (!metrics.isEmpty()) {
+ model.put("metrics", metrics);
+ }
+
+ Map<String, Object> modelStash = new LinkedHashMap<>();
+ for (String k : MODEL_STASH_KEYS) {
+ if (view.containsKey(k)) {
+ modelStash.put(k, view.get(k));
+ }
+ }
+ if (hasOtm(asList(get(view, "joins")))) {
+ modelStash.put(STASH_SOURCE_KEY, factName);
+ }
+ if (!complexJoins.isEmpty()) {
+ modelStash.put("complex_joins", complexJoins);
+ }
+ writeStash(model, modelStash);
+ return model;
+ }
+
+ private static void walk(String parentName, String parentAlias, List<Object> joins,
Review Comment:
The import direction (`walk`/`hasOtm`) never enforces `MAX_JOIN_NODES`, while export checks it before and during recursion. A Metric View with 250+ joins imports cleanly but then fails to re-export.
I believe it breaks the documented guarantee that anything that imports cleanly can be exported again.
Should we add the check on import too?
##########
converters/databricks/java/src/main/java/org/apache/ossie/converter/databricks/MetricViewToOssie.java:
##########
@@ -0,0 +1,581 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ossie.converter.databricks;
+
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_MANY_TO_ONE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.CARD_ONE_TO_MANY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.DIALECT_DATABRICKS;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MAPPER;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.MV_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.OSSIE_VERSION;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.SELECT_WITH_RE;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.STASH_SOURCE_KEY;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.asMap;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.get;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.isSimpleIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.loadYaml;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.lastIdentifier;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.qualifierChainPattern;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.requireStr;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.rewriteQualifiers;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.str;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.strList;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.truthy;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.validateSource;
+import static org.apache.ossie.converter.databricks.OssieConverterCommon.writeStash;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.ossie.converter.databricks.OssieConverter.ConversionException;
+import org.apache.ossie.converter.databricks.OssieConverter.Notices;
+import org.apache.ossie.converter.databricks.OssieConverter.Result;
+
+/**
+ * IMPORT direction: Metric View v1.1 YAML -> Apache Ossie semantic model. Shared helpers come
+ * from {@link OssieConverterCommon}; the public entry point is re-exported through
+ * {@link OssieConverter}.
+ */
+// Map-based YAML manipulation: casts of the parsed Object graph to Map/List are inherently
+// unchecked; the asMap/asList helpers guard them, so unchecked warnings here are expected.
+@SuppressWarnings("unchecked")
+final class MetricViewToOssie {
+
+ private static final String[] MODEL_STASH_KEYS = {"filter", "parameters", "materialization"};
+ private static final String[] JOIN_STASH_KEYS = {"rely", "cardinality"};
+ // Metric-View-only column fields with no Apache Ossie representation, preserved verbatim in the
+ // DATABRICKS stash so an import can restore them. `window` and `partition` are measure-only
+ // (they live on MeasureExpression); a dimension never carries them, so the shared key list is
+ // simply never hit for those on the dimension path.
+ private static final String[] COLUMN_STASH_KEYS = {"format", "window", "partition"};
+ // Measure-only fields with no Apache Ossie Metric representation. A dimension's display_name
+ // maps to the Field `label`, but the Ossie Metric schema has no `label`, so a measure's
+ // display_name is preserved in the DATABRICKS stash instead. Kept separate from
+ // COLUMN_STASH_KEYS so the dimension path (which already maps display_name to `label`) does not
+ // also stash it.
+ private static final String[] MEASURE_STASH_KEYS = {"display_name"};
+ private static final Pattern NON_EQUI_RE = Pattern.compile("[<>!]=|<>|[<>]");
+ private static final Pattern AND_SPLIT_RE = Pattern.compile("\\s+AND\\s+", Pattern.CASE_INSENSITIVE);
+ private static final Pattern EQ_CLAUSE_RE = Pattern.compile("^\\s*(.+?)\\s*=\\s*(.+?)\\s*$");
+
+ private MetricViewToOssie() {}
+
+ static Result convertMetricViewToOssie(String mvYamlStr, String modelName) {
+ Notices notices = new Notices();
+ if (mvYamlStr == null || mvYamlStr.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root", "the input is empty");
+ }
+ Object parsed;
+ try {
+ parsed = loadYaml(mvYamlStr);
+ } catch (Exception e) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: " + e.getMessage(),
+ "failed to parse YAML: " + e.getMessage(),
+ e);
+ }
+ if (!(parsed instanceof Map)) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: expected a mapping at the root",
+ "it is not a mapping at the root");
+ }
+ Map<String, Object> view = asMap(parsed);
+ Object versionValue = get(view, "version");
+ String version = str(versionValue);
+ if (versionValue == null || version.trim().isEmpty()) {
+ throw ConversionException.invalidInput(
+ "Invalid Metric View YAML: missing required 'version' field",
+ "it is missing the required 'version' field");
+ }
+ if (!MV_VERSION.equals(version)) {
+ throw ConversionException.unsupportedVersion(
+ "Unsupported Metric View version '" + version
+ + "'. This converter targets v" + MV_VERSION + " only.",
+ version);
+ }
+ Map<String, Object> model = convertView(view, modelName, notices);
+ Map<String, Object> out = new LinkedHashMap<>();
+ out.put("version", OSSIE_VERSION);
+ List<Object> models = new ArrayList<>();
+ models.add(model);
+ out.put("semantic_model", models);
+ return serializeOssie(out, notices);
+ }
+
+ static Result serializeOssie(Map<String, Object> out, Notices notices) {
+ try {
+ return new Result(MAPPER.writeValueAsString(out), notices.toList());
+ } catch (Exception e) {
+ throw ConversionException.internalError(
+ "failed to serialize Apache Ossie YAML: " + e.getMessage(), e);
+ }
+ }
+
+ private static Map<String, Object> convertView(
+ Map<String, Object> view, String modelName, Notices notices) {
+ Object source = get(view, "source");
+ if (source == null || source.toString().isEmpty()) {
+ throw new ConversionException("Metric View is missing required 'source'");
+ }
+ // Same subquery test validateSource applies below, so the two cannot drift.
+ boolean isSql = SELECT_WITH_RE.matcher(source.toString().trim()).find();
+ String lastId = lastIdentifier(source);
+ String factName;
+ if (modelName != null) {
+ factName = modelName;
+ } else if (!isSql && lastId != null && isSimpleIdentifier(lastId)) {
+ factName = lastId;
+ } else {
+ factName = "metric_view";
+ }
+ validateSource(source, factName);
+
+ List<Map<String, Object>> datasets = new ArrayList<>();
+ Map<String, Object> factDs = new LinkedHashMap<>();
+ factDs.put("name", factName);
+ factDs.put("source", source);
+ datasets.add(factDs);
+ List<Map<String, Object>> relationships = new ArrayList<>();
+ // Joins with no schema-valid Ossie relationship (a non-decomposable `on`) are collected here
+ // and stashed under the model's DATABRICKS custom_extensions instead of `relationships`.
+ List<Map<String, Object>> complexJoins = new ArrayList<>();
+ Map<String, String> aliasToDataset = new HashMap<>();
+ aliasToDataset.put("source", factName);
+ aliasToDataset.put(factName, factName);
+ Set<String> seenNames = new HashSet<>();
+ seenNames.add(factName.trim().toLowerCase(Locale.ROOT));
+
+ walk(factName, "source", asList(get(view, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+
+ // `fields` is a v1.1 alias for `dimensions`: an empty `dimensions: []` falls through to
+ // `fields`, and the "both set" warning fires only when BOTH are non-empty (not merely
+ // present).
+ List<Object> dimList = asList(get(view, "dimensions"));
+ List<Object> fieldList = asList(get(view, "fields"));
+ if (!dimList.isEmpty() && !fieldList.isEmpty()) {
+ notices.warn("view", "both 'dimensions' and 'fields' are set; 'fields' is a v1.1 alias "
+ + "for 'dimensions', so the 'fields' list is ignored");
+ }
+ Map<String, List<Object>> fieldsByDataset = new LinkedHashMap<>();
+ for (Map<String, Object> d : datasets) {
+ fieldsByDataset.put((String) d.get("name"), new ArrayList<>());
+ }
+ List<Object> dims = !dimList.isEmpty() ? dimList : fieldList;
+ for (Object dimObj : dims) {
+ Map<String, Object> dim = asMap(dimObj);
+ if (isWildcard(dim)) {
+ notices.warn("dimension", "wildcard column '" + str(get(dim, "expr"))
+ + "' has no Apache Ossie field representation; skipped");
+ continue;
+ }
+ Object[] converted = convertDimension(dim, aliasToDataset, factName);
+ fieldsByDataset.get((String) converted[0]).add(converted[1]);
+ }
+ for (Map<String, Object> d : datasets) {
+ List<Object> flds = fieldsByDataset.get((String) d.get("name"));
+ if (!flds.isEmpty()) {
+ d.put("fields", flds);
+ }
+ }
+
+ List<Object> metrics = new ArrayList<>();
+ // Fixed once the join tree is walked, so compile it once for all measures.
+ Pattern aliasHead = qualifierChainPattern(aliasToDataset.keySet());
+ for (Object mObj : asList(get(view, "measures"))) {
+ Map<String, Object> m = asMap(mObj);
+ if (isWildcard(m)) {
+ notices.warn("measure", "wildcard measure '" + str(get(m, "expr"))
+ + "' has no Apache Ossie metric representation; skipped");
+ continue;
+ }
+ metrics.add(convertMeasure(m, aliasHead, aliasToDataset));
+ }
+
+ Map<String, Object> model = new LinkedHashMap<>();
+ model.put("name", factName);
+ if (truthy(get(view, "comment"))) {
+ model.put("description", get(view, "comment"));
+ }
+ model.put("datasets", datasets);
+ if (!relationships.isEmpty()) {
+ model.put("relationships", relationships);
+ }
+ if (!metrics.isEmpty()) {
+ model.put("metrics", metrics);
+ }
+
+ Map<String, Object> modelStash = new LinkedHashMap<>();
+ for (String k : MODEL_STASH_KEYS) {
+ if (view.containsKey(k)) {
+ modelStash.put(k, view.get(k));
+ }
+ }
+ if (hasOtm(asList(get(view, "joins")))) {
+ modelStash.put(STASH_SOURCE_KEY, factName);
+ }
+ if (!complexJoins.isEmpty()) {
+ modelStash.put("complex_joins", complexJoins);
+ }
+ writeStash(model, modelStash);
+ return model;
+ }
+
+ private static void walk(String parentName, String parentAlias, List<Object> joins,
+ List<Map<String, Object>> datasets, List<Map<String, Object>> relationships,
+ List<Map<String, Object>> complexJoins, Map<String, String> aliasToDataset,
+ Set<String> seenNames, Notices notices) {
+ for (Object joinObj : joins) {
+ Map<String, Object> join = asMap(joinObj);
+ String child = requireStr(join, "name", "join");
+ if (child.trim().equalsIgnoreCase("source")) {
+ throw new ConversionException(
+ "Join name 'source' is reserved for the fact source; rename the join.");
+ }
+ if (!seenNames.add(child.trim().toLowerCase(Locale.ROOT))) {
+ throw new ConversionException("Duplicate dataset/join name '" + child
+ + "'; Metric View join names and the source must be distinct (case-insensitively).");
+ }
+ Map<String, Object> childDs = new LinkedHashMap<>();
+ childDs.put("name", child);
+ // Validated with the same rule the export applies to a dataset source, so a Metric View that
+ // imports cleanly is always exportable again (a 2-part join source used to import fine and
+ // then fail on the way back out).
+ String childSource = requireStr(join, "source", "join '" + child + "'");
+ validateSource(childSource, child);
+ childDs.put("source", childSource);
+ datasets.add(childDs);
+ aliasToDataset.put(child, child);
+ Map<String, Object> rel = convertJoin(join, parentName, parentAlias, child, notices);
+ // convertJoin omits from/to columns for a non-decomposable `on`; those entries have no valid
+ // Ossie relationship form, so they go to the model-level stash instead of `relationships`.
+ if (rel.containsKey("from_columns")) {
+ relationships.add(rel);
+ } else {
+ complexJoins.add(rel);
+ }
+ // rely.at_most_one_match on a many_to_one join -> recover a unique_key on the child.
+ Map<String, Object> rely = asMap(get(join, "rely"));
+ List<String> toCols = strList(get(rel, "to_columns"));
+ if (child.equals(str(get(rel, "to"))) && !toCols.isEmpty()
+ && Boolean.TRUE.equals(rely.get("at_most_one_match"))) {
+ List<Object> uk = new ArrayList<>();
+ uk.add(new ArrayList<>(toCols));
+ childDs.put("unique_keys", uk);
+ }
+ walk(child, child, asList(get(join, "joins")), datasets, relationships,
+ complexJoins, aliasToDataset, seenNames, notices);
+ }
+ }
+
+ private static boolean hasOtm(List<Object> joins) {
+ for (Object jObj : joins) {
+ Map<String, Object> j = asMap(jObj);
+ if (CARD_ONE_TO_MANY.equalsIgnoreCase(str(get(j, "cardinality")))) {
+ return true;
+ }
+ if (hasOtm(asList(get(j, "joins")))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static Map<String, Object> convertJoin(
+ Map<String, Object> join, String parentName, String parentAlias, String child,
+ Notices notices) {
+ boolean hasUsing = get(join, "using") != null && !asList(get(join, "using")).isEmpty();
+ boolean hasOn = str(get(join, "on")) != null && !str(get(join, "on")).isEmpty();
+ if (!hasUsing && !hasOn) {
+ throw new ConversionException("Join '" + child + "' has no join condition (empty or "
+ + "absent 'on'/'using'); condition-less (cross) joins have no Apache Ossie "
+ + "relationship representation.");
+ }
+ Object[] decomposed = decomposeOn(join, parentAlias, parentName, child);
+ List<String> parentCols = (List<String>) decomposed[0];
+ List<String> childCols = (List<String>) decomposed[1];
+ String rawOn = (String) decomposed[2];
+ if (rawOn != null) {
+ // The `on` is not an equi-join of simple `alias.column` pairs (it has a non-equi operator, a
+ // SQL-function-wrapped key, or an extra filter predicate), so an Apache Ossie relationship
+ // cannot represent it (from/to columns are required). Rather than abort the whole
+ // conversion, warn here; the columns-less entry built below is stashed under the model's
+ // custom_extensions (complex_joins) so it round-trips, instead of an invalid relationship.
+ notices.warn("join '" + child + "'", "non-equi or unsupported join condition ('on: " + rawOn
Review Comment:
This accepts a non-equi/filtered join `on` clause and preserves it via the new `complex_joins` stash, but the Python reference still raises `ConversionError` on the same input.
Is this divergence intentional/tracked, or should Java match Python's current (stricter) behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on PR #333:
URL: https://github.com/apache/ossie/pull/333#issuecomment-5456460354
Hi @jbonofre , fixes look good. Thanks! I also pushed one commit addressing some issues we found when we are building native sql support based on the converter. Hope you can review those as well. Also feel free to change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "GMorrisXBL (via GitHub)" <gi...@apache.org>.
GMorrisXBL commented on PR #333:
URL: https://github.com/apache/ossie/pull/333#issuecomment-5528065146
Thanks @Haoranli503!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [OSSIE][DATABRICKS] Add Java Converter and Reformat [ossie]
Posted by "Haoranli503 (via GitHub)" <gi...@apache.org>.
Haoranli503 commented on PR #333:
URL: https://github.com/apache/ossie/pull/333#issuecomment-5737216012
Pushed three commits.
a991e68 (fixes #422): cascade drop matched references to dropped fields case sensitively, so a metric like COUNT(DISTINCT REGION_NAME) survived referencing a dropped region_name as a dangling reference. It now matches case insensitively, in both the containment prefilter and the regex, across the propagation gate and the confirmation. Added a test with the reported repro.
53c59f6: merged main into the branch to bring it current with #396. Resolved the one converters/databricks/README.md conflict by keeping this branch's overview, since the Java and Python implementations live in their own subdirectories here.
416119f: migrated the Java converter to flat documents (the model at the document root) to match #396. Import reads the model at the root and rejects a legacy semantic_model wrapper, a missing string name, and root dialects/vendors; export emits the model at the root. Flattened all test inputs and fixtures and updated the export assertions to read the model at the root.
One note on verification for that last commit: I could not run mvn locally, so I compiled the converter and ran a driver covering flat import, the legacy rejection, flat export, and a round trip, and reconverted all three fixtures. The JUnit assertions still need a mvn test run as the final gate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ossie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org