You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu> on 2026/08/03 03:48:18 UTC
[XL] Change in asterixdb[master]: [NO ISSUE][FUN] Add okf_catalog() builtin
From Vivek Gangavarapu <ga...@gmail.com>:
Vivek Gangavarapu has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21484?usp=email )
Change subject: [NO ISSUE][FUN] Add okf_catalog() builtin
......................................................................
[NO ISSUE][FUN] Add okf_catalog() builtin
- user model changes: no
- storage format changes: no
- interface changes: yes
Adds a private datasource function that walks the Metadata.*
catalog on the CC and emits it as a linked graph of Open
Knowledge Format concept documents: one per dataverse, dataset,
datatype and secondary index. Each row carries the source_query
that reproduces it, so a consumer can re-ground a doc against the
live catalog rather than trusting a cached copy.
okf_catalog([dataverse[, exclude_marker]])
- OKFCatalog{Rewriter,Datasource,Function,Reader}: modeled on
ping() and dataset_resources(). The catalog walk runs on the
CC; rendered rows are shipped to a single NC location
- dataset docs carry the markdown schema (dotted paths for
nested fields, ? for optional), secondary indexes, ANALYZE
sample statistics, profiling queries and the recent completed
statements that reference the dataset
- the ANALYZE sample index is reported as statistics, not as an
access path
- a caller that runs its own statements against the walked
datasets can stamp them with a marker comment and pass it as
exclude_marker, so its own plumbing is not mined back into the
docs on the next walk. An empty first argument walks the whole
catalog, which is how such a caller asks for everything
- emitted SQL++ is escaped: values become quoted literals and
dataset names delimited identifiers, since a consumer may run
what it is handed
- registered as private in MetadataBuiltinFunctions
- unit tests for schema rendering, concept identities, workload
selection, escaping and the document shape, plus runtime tests
under misc/okf-catalog
Generated-by: Claude Code w/ Opus 4.8
Change-Id: Iff8891618f0cc715abb8b2fb8c0c35c54093c586
---
A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogDatasource.java
A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogFunction.java
A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogReader.java
A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogRewriter.java
M asterixdb/asterix-app/src/main/java/org/apache/asterix/util/MetadataBuiltinFunctions.java
A asterixdb/asterix-app/src/test/java/org/apache/asterix/app/function/OKFCatalogRewriterTest.java
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.1.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.2.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.3.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.4.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.5.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.6.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.7.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.8.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.9.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.2.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.3.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.4.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.5.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.8.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.9.adm
M asterixdb/asterix-app/src/test/resources/runtimets/sqlpp_queries.xml
22 files changed, 1,201 insertions(+), 0 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/84/21484/1
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogDatasource.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogDatasource.java
new file mode 100644
index 0000000..a7454bd
--- /dev/null
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogDatasource.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.app.function;
+
+import java.util.List;
+import java.util.Objects;
+
+import org.apache.asterix.metadata.api.IDatasourceFunction;
+import org.apache.asterix.metadata.declared.DataSourceId;
+import org.apache.asterix.metadata.declared.FunctionDataSource;
+import org.apache.asterix.metadata.declared.MetadataProvider;
+import org.apache.hyracks.algebricks.common.constraints.AlgebricksAbsolutePartitionConstraint;
+import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException;
+import org.apache.hyracks.algebricks.core.algebra.properties.INodeDomain;
+import org.apache.hyracks.util.annotations.AiProvenance;
+
+/**
+ * Datasource for {@code okf_catalog}. The concept documents are rendered once on
+ * the CC by {@link OKFCatalogRewriter} (the catalog lives there) and carried
+ * through to a single NC location for emission, so the walk runs exactly once.
+ */
+@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_4_8, tool = AiProvenance.Tool.CLAUDE_CODE_CLI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+public class OKFCatalogDatasource extends FunctionDataSource {
+
+ private static final DataSourceId OKF_CATALOG_DATASOURCE_ID = createDataSourceId(OKFCatalogRewriter.OKF_CATALOG);
+
+ private final List<String> concepts;
+
+ public OKFCatalogDatasource(INodeDomain domain, List<String> concepts) throws AlgebricksException {
+ super(OKF_CATALOG_DATASOURCE_ID, OKFCatalogRewriter.OKF_CATALOG, domain);
+ this.concepts = concepts;
+ }
+
+ @Override
+ protected IDatasourceFunction createFunction(MetadataProvider metadataProvider,
+ AlgebricksAbsolutePartitionConstraint locations) {
+ return new OKFCatalogFunction(AlgebricksAbsolutePartitionConstraint.randomLocation(locations.getLocations()),
+ concepts);
+ }
+
+ @Override
+ protected boolean sameFunctionDatasource(FunctionDataSource other) {
+ if (!Objects.equals(this.functionId, other.getFunctionId())) {
+ return false;
+ }
+ OKFCatalogDatasource that = (OKFCatalogDatasource) other;
+ return Objects.equals(this.concepts, that.concepts);
+ }
+}
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogFunction.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogFunction.java
new file mode 100644
index 0000000..6bacc79
--- /dev/null
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogFunction.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.app.function;
+
+import java.util.List;
+
+import org.apache.asterix.external.api.IRecordReader;
+import org.apache.asterix.metadata.declared.AbstractDatasourceFunction;
+import org.apache.hyracks.algebricks.common.constraints.AlgebricksAbsolutePartitionConstraint;
+import org.apache.hyracks.api.context.IHyracksTaskContext;
+import org.apache.hyracks.util.annotations.AiProvenance;
+
+/**
+ * NC-side handle for {@code okf_catalog}. Carries the pre-rendered OKF concept
+ * documents to the assigned partition and hands them to the reader for emission.
+ */
+@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_4_8, tool = AiProvenance.Tool.CLAUDE_CODE_CLI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+public class OKFCatalogFunction extends AbstractDatasourceFunction {
+
+ private static final long serialVersionUID = 1L;
+
+ private final List<String> concepts;
+
+ public OKFCatalogFunction(AlgebricksAbsolutePartitionConstraint locations, List<String> concepts) {
+ super(locations);
+ this.concepts = concepts;
+ }
+
+ @Override
+ public IRecordReader<char[]> createRecordReader(IHyracksTaskContext ctx, int partition) {
+ return new OKFCatalogReader(concepts);
+ }
+}
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogReader.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogReader.java
new file mode 100644
index 0000000..a759a3c
--- /dev/null
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogReader.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.app.function;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+
+import org.apache.asterix.external.api.IRawRecord;
+import org.apache.asterix.external.input.record.CharArrayRecord;
+import org.apache.hyracks.util.annotations.AiProvenance;
+
+/** Emits each pre-rendered OKF concept document as one JSON record. */
+@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_4_8, tool = AiProvenance.Tool.CLAUDE_CODE_CLI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+public class OKFCatalogReader extends FunctionReader {
+
+ private final Iterator<String> it;
+ private final CharArrayRecord record;
+
+ public OKFCatalogReader(List<String> concepts) {
+ this.it = concepts.iterator();
+ this.record = new CharArrayRecord();
+ }
+
+ @Override
+ public boolean hasNext() throws Exception {
+ return it.hasNext();
+ }
+
+ @Override
+ public IRawRecord<char[]> next() throws IOException, InterruptedException {
+ record.reset();
+ record.set(it.next());
+ record.endRecord();
+ return record;
+ }
+}
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogRewriter.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogRewriter.java
new file mode 100644
index 0000000..bc92f2e
--- /dev/null
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/OKFCatalogRewriter.java
@@ -0,0 +1,534 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.app.function;
+
+import static org.apache.hyracks.algebricks.core.algebra.functions.FunctionIdentifier.VARARGS;
+
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.asterix.common.api.IClientRequest;
+import org.apache.asterix.common.config.DatasetConfig.DatasetType;
+import org.apache.asterix.common.functions.FunctionConstants;
+import org.apache.asterix.common.metadata.DataverseName;
+import org.apache.asterix.metadata.declared.MetadataProvider;
+import org.apache.asterix.metadata.entities.Dataset;
+import org.apache.asterix.metadata.entities.Datatype;
+import org.apache.asterix.metadata.entities.Index;
+import org.apache.asterix.om.types.ARecordType;
+import org.apache.asterix.om.types.AUnionType;
+import org.apache.asterix.om.types.AbstractCollectionType;
+import org.apache.asterix.om.types.IAType;
+import org.apache.commons.lang3.mutable.Mutable;
+import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException;
+import org.apache.hyracks.algebricks.core.algebra.base.ILogicalExpression;
+import org.apache.hyracks.algebricks.core.algebra.base.IOptimizationContext;
+import org.apache.hyracks.algebricks.core.algebra.expressions.AbstractFunctionCallExpression;
+import org.apache.hyracks.algebricks.core.algebra.functions.FunctionIdentifier;
+import org.apache.hyracks.api.exceptions.SourceLocation;
+import org.apache.hyracks.util.annotations.AiProvenance;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+
+/**
+ * Rewriter for the {@code okf_catalog([dataverse[, exclude_marker]])} datasource
+ * function, which walks the {@code Metadata.*} catalog on the CC and emits it as
+ * a linked graph of Open Knowledge Format (OKF) concept documents — one row per
+ * concept:
+ * <ul>
+ * <li>one per <b>dataverse</b>, linking to its datasets,</li>
+ * <li>one per <b>dataset / view / external dataset</b> (schema, indexes, and
+ * ready-to-run SQL++),</li>
+ * <li>one per <b>datatype</b> (the schema contract, shared across
+ * datasets),</li>
+ * <li>one per <b>secondary index</b> (access-path knowledge).</li>
+ * </ul>
+ * Each row is a record with {@code subject}, {@code type}, {@code title},
+ * {@code description}, {@code resource}, {@code tags}, {@code timestamp},
+ * {@code source_query}, {@code text}, {@code links}, {@code profile_queries}
+ * and {@code observed_queries}.
+ * <p>
+ * The walk is pure metadata access, so every emitted doc is deterministic and
+ * self-grounding: each carries the {@code source_query} that reproduces it
+ * against the live catalog, and dataset docs additionally carry profiling
+ * queries (row count, sample) that a caller can run to re-ground the doc.
+ * <p>
+ * Dataset docs also fold in the recent completed statements that reference the
+ * dataset. A caller that runs its own statements against these datasets — a
+ * refresh pipeline grounding or advising a doc — would otherwise see its own
+ * plumbing mined back into the next walk, so it may stamp those statements with
+ * a marker comment and pass the same marker as {@code exclude_marker} to have
+ * them left out.
+ * <p>
+ * With no argument the whole catalog (minus the {@code Metadata} system
+ * dataverse) is walked; with a non-empty first argument only that dataverse is.
+ * An empty first argument walks everything, which is how a caller asks for the
+ * whole catalog while still supplying an {@code exclude_marker}.
+ */
+@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_4_8, tool = AiProvenance.Tool.CLAUDE_CODE_CLI, contributionKind = AiProvenance.ContributionKind.GENERATED, notes = "OKF catalog enrichment prototype: Metadata.* walk to a linked OKF concept-doc graph")
+public class OKFCatalogRewriter extends FunctionRewriter {
+
+ public static final FunctionIdentifier OKF_CATALOG = FunctionConstants.newAsterix("okf-catalog", VARARGS);
+ public static final OKFCatalogRewriter INSTANCE = new OKFCatalogRewriter(OKF_CATALOG);
+
+ private static final String TYPE_DATAVERSE = "AsterixDB Dataverse";
+ private static final String TYPE_DATASET = "AsterixDB Dataset";
+ private static final String TYPE_EXTERNAL_DATASET = "AsterixDB External Dataset";
+ private static final String TYPE_VIEW = "AsterixDB View";
+ private static final String TYPE_DATATYPE = "AsterixDB Datatype";
+ private static final String TYPE_INDEX = "AsterixDB Index";
+ private static final String SYSTEM_DATAVERSE = "Metadata";
+ private static final String RESOURCE_SCHEME = "asterix://";
+ private static final String BASE_TAG = "asterixdb";
+
+ private static final String FIELD_SUBJECT = "subject";
+ private static final String FIELD_TYPE = "type";
+ private static final String FIELD_TITLE = "title";
+ private static final String FIELD_DESCRIPTION = "description";
+ private static final String FIELD_RESOURCE = "resource";
+ private static final String FIELD_TAGS = "tags";
+ private static final String FIELD_TIMESTAMP = "timestamp";
+ private static final String FIELD_SOURCE_QUERY = "source_query";
+ private static final String FIELD_TEXT = "text";
+ private static final String FIELD_LINKS = "links";
+ private static final String FIELD_PROFILE_QUERIES = "profile_queries";
+ private static final String FIELD_OBSERVED_QUERIES = "observed_queries";
+
+ // fields of a completed request, as the request tracker renders them
+ private static final String REQUEST_STATEMENT = "statement";
+ private static final String REQUEST_TIME = "requestTime";
+ private static final String REQUEST_ELAPSED_TIME = "elapsedTime";
+ private static final int MAX_OBSERVED_QUERIES = 3;
+ private static final int MAX_OBSERVED_QUERY_LENGTH = 200;
+ // this function's own name: a catalog walk is not part of the observed workload
+ private static final String SELF_REFERENCE = "okf_catalog";
+
+ private static final ObjectMapper JSON = new ObjectMapper();
+
+ private OKFCatalogRewriter(FunctionIdentifier functionId) {
+ super(functionId);
+ }
+
+ @Override
+ public OKFCatalogDatasource toDatasource(IOptimizationContext context, AbstractFunctionCallExpression f)
+ throws AlgebricksException {
+ SourceLocation loc = f.getSourceLocation();
+ MetadataProvider metadataProvider = (MetadataProvider) context.getMetadataProvider();
+ List<Mutable<ILogicalExpression>> args = f.getArguments();
+ String dataverseFilter = args.isEmpty() ? null : getString(loc, args, 0);
+ String excludeMarker = args.size() < 2 ? null : getString(loc, args, 1);
+ return new OKFCatalogDatasource(context.getComputationNodeDomain(),
+ new BundleBuilder(metadataProvider, dataverseFilter, excludeMarker).build());
+ }
+
+ @Override
+ protected boolean invalidArgs(List<Mutable<ILogicalExpression>> args) {
+ return args.size() > 2;
+ }
+
+ /**
+ * Builds the OKF bundle for one invocation: walks the catalog once and
+ * renders the full linked concept graph with a single shared timestamp.
+ */
+ static final class BundleBuilder {
+
+ private final MetadataProvider metadataProvider;
+ private final String dataverseFilter;
+ // caller-supplied substring marking its own statements as plumbing
+ private final String excludeMarker;
+ private final String timestamp = Instant.now().toString();
+ // insertion-ordered so dataverse docs precede their members
+ private final Map<String, String> conceptsBySubject = new LinkedHashMap<>();
+ // dataverse subject -> dataset subjects, for the dataverse concept doc
+ private final Map<String, List<String>> dataverseMembers = new LinkedHashMap<>();
+ // completed cluster requests, lazily fetched once per bundle build
+ private List<JsonNode> workload;
+
+ BundleBuilder(MetadataProvider metadataProvider, String dataverseFilter, String excludeMarker) {
+ this.metadataProvider = metadataProvider;
+ this.dataverseFilter = dataverseFilter;
+ this.excludeMarker = excludeMarker;
+ }
+
+ List<String> build() throws AlgebricksException {
+ List<Dataset> datasets = new ArrayList<>(metadataProvider.getAllDatasets());
+ datasets.sort(Comparator.comparing((Dataset d) -> d.getDataverseName().getCanonicalForm())
+ .thenComparing(Dataset::getDatasetName));
+ for (Dataset dataset : datasets) {
+ String dataverse = dataset.getDataverseName().getCanonicalForm();
+ if (isFilteredOut(dataverse)) {
+ continue;
+ }
+ renderDatasetConcept(dataset);
+ }
+ renderDataverseConcepts();
+ return new ArrayList<>(conceptsBySubject.values());
+ }
+
+ private boolean isFilteredOut(String dataverse) {
+ return isFilteredOut(dataverse, dataverseFilter);
+ }
+
+ /** An absent or empty filter walks the whole catalog, minus the system dataverse. */
+ static boolean isFilteredOut(String dataverse, String dataverseFilter) {
+ if (dataverseFilter == null || dataverseFilter.isEmpty()) {
+ return SYSTEM_DATAVERSE.equals(dataverse);
+ }
+ return !dataverseFilter.equals(dataverse);
+ }
+
+ private void renderDatasetConcept(Dataset dataset) throws AlgebricksException {
+ DataverseName dv = dataset.getDataverseName();
+ String dataverse = dv.getCanonicalForm();
+ String name = dataset.getDatasetName();
+ String subject = dataverse + "." + name;
+ String conceptType = conceptTypeOf(dataset);
+
+ ARecordType recordType = resolveRecordType(dataset);
+ List<Index> allIndexes = dataset.getDatasetType() == DatasetType.INTERNAL
+ ? metadataProvider.getDatasetIndexes(dataset.getDatabaseName(), dv, name) : List.of();
+ // the ANALYZE sample index carries CBO stats; it is not an access path
+ List<Index> indexes = new ArrayList<>();
+ long sampleSourceCardinality = -1;
+ for (Index index : allIndexes) {
+ if (index.getIndexDetails() instanceof Index.SampleIndexDetails) {
+ sampleSourceCardinality =
+ ((Index.SampleIndexDetails) index.getIndexDetails()).getSourceCardinality();
+ } else {
+ indexes.add(index);
+ }
+ }
+
+ StringBuilder body = new StringBuilder();
+ body.append("# Schema\n\n");
+ renderSchema(body, recordType);
+ body.append("\n# Indexes\n\n");
+ if (indexes.isEmpty()) {
+ body.append("_none_\n");
+ } else {
+ for (Index index : indexes) {
+ body.append("- ").append(index.getIndexName()).append(index.isPrimaryIndex() ? " (PRIMARY)" : "")
+ .append("\n");
+ }
+ }
+ if (sampleSourceCardinality >= 0) {
+ // sample-based profiling: read the optimizer's own ANALYZE sample, no scan
+ body.append("\n# Statistics\n\n- Row count: ").append(sampleSourceCardinality).append(
+ " (from the ANALYZE sample; re-run ANALYZE or the profiling count query to re-ground)\n");
+ }
+ List<JsonNode> observedRequests = observedRequests(name);
+ if (!observedRequests.isEmpty()) {
+ body.append("\n# Observed queries\n\n");
+ for (JsonNode request : observedRequests) {
+ body.append("- `").append(oneLine(request.path(REQUEST_STATEMENT).asText())).append("` (")
+ .append(request.path(REQUEST_ELAPSED_TIME).asText("?")).append("s)\n");
+ }
+ }
+ body.append("\n# Queries\n\n");
+ String dataverseIdentifier = dv.toString();
+ body.append("Sample rows:\n\n```sql\n").append(sampleQuery(dataverseIdentifier, name)).append("\n```\n\n");
+ body.append("Row count (profiling; re-run to re-ground):\n\n```sql\n")
+ .append(countQuery(dataverseIdentifier, name)).append("\n```\n");
+
+ List<String> links = new ArrayList<>();
+ String typeSubject = datatypeSubject(dataset);
+ if (typeSubject != null) {
+ links.add(typeSubject);
+ }
+ for (Index index : indexes) {
+ if (!index.isPrimaryIndex()) {
+ links.add(indexSubject(subject, index));
+ }
+ }
+
+ ObjectNode doc = concept(subject, conceptType, name, describeDataset(conceptType, name, dataverse),
+ dataverse, sourceQuery("Dataset", "DataverseName", dataverse, "DatasetName", name), body.toString(),
+ links);
+ ArrayNode profileQueries = doc.putArray(FIELD_PROFILE_QUERIES);
+ profileQueries.add(countQuery(dataverseIdentifier, name));
+ profileQueries.add(sampleQuery(dataverseIdentifier, name));
+ // structured mirror of "# Observed queries" for downstream pipelines
+ // (grounding, ADVISE) that must not parse markdown
+ ArrayNode observed = doc.putArray(FIELD_OBSERVED_QUERIES);
+ for (JsonNode request : observedRequests) {
+ observed.add(request.path(REQUEST_STATEMENT).asText());
+ }
+ addConcept(subject, doc);
+
+ dataverseMembers.computeIfAbsent(dataverse, k -> new ArrayList<>()).add(subject);
+ renderDatatypeConcept(dataset, subject);
+ for (Index index : indexes) {
+ if (!index.isPrimaryIndex()) {
+ renderIndexConcept(dataverse, subject, index);
+ }
+ }
+ }
+
+ private void renderDatatypeConcept(Dataset dataset, String datasetSubject) throws AlgebricksException {
+ String subject = datatypeSubject(dataset);
+ if (subject == null) {
+ return;
+ }
+ String typeName = dataset.getItemTypeName();
+ String typeDataverse = dataset.getItemTypeDataverseName().getCanonicalForm();
+ if (conceptsBySubject.containsKey(subject)) {
+ // shared type: append this dataset as another back-link
+ return;
+ }
+ ARecordType recordType = resolveRecordType(dataset);
+ StringBuilder body = new StringBuilder("# Fields\n\n");
+ renderSchema(body, recordType);
+ ObjectNode doc = concept(subject, TYPE_DATATYPE, typeName,
+ "Datatype " + typeName + " in dataverse " + typeDataverse, typeDataverse,
+ sourceQuery("Datatype", "DataverseName", typeDataverse, "DatatypeName", typeName), body.toString(),
+ List.of(datasetSubject));
+ addConcept(subject, doc);
+ }
+
+ private void renderIndexConcept(String dataverse, String datasetSubject, Index index) {
+ String subject = indexSubject(datasetSubject, index);
+ StringBuilder body = new StringBuilder();
+ body.append("# Index\n\n- Type: ").append(index.getIndexType()).append("\n");
+ if (index.getIndexDetails() instanceof Index.ValueIndexDetails) {
+ List<List<String>> keys = ((Index.ValueIndexDetails) index.getIndexDetails()).getKeyFieldNames();
+ body.append("- Keys: ");
+ for (int i = 0; i < keys.size(); i++) {
+ body.append(i > 0 ? ", " : "").append(String.join(".", keys.get(i)));
+ }
+ body.append("\n");
+ }
+ ObjectNode doc = concept(subject, TYPE_INDEX, index.getIndexName(),
+ "Index " + index.getIndexName() + " on " + datasetSubject, dataverse,
+ sourceQuery("Index", "DataverseName", dataverse, "IndexName", index.getIndexName()),
+ body.toString(), List.of(datasetSubject));
+ addConcept(subject, doc);
+ }
+
+ private void renderDataverseConcepts() {
+ for (Map.Entry<String, List<String>> entry : dataverseMembers.entrySet()) {
+ String dataverse = entry.getKey();
+ StringBuilder body = new StringBuilder("# Datasets\n\n");
+ for (String member : entry.getValue()) {
+ body.append("- ").append(member).append("\n");
+ }
+ ObjectNode doc = concept(dataverse, TYPE_DATAVERSE, dataverse, "Dataverse " + dataverse, dataverse,
+ sourceQuery("Dataverse", "DataverseName", dataverse), body.toString(), entry.getValue());
+ addConcept(dataverse, doc);
+ }
+ }
+
+ ObjectNode concept(String subject, String conceptType, String title, String description, String dataverse,
+ String sourceQuery, String text, List<String> links) {
+ ObjectNode doc = JSON.createObjectNode();
+ doc.put(FIELD_SUBJECT, subject);
+ doc.put(FIELD_TYPE, conceptType);
+ doc.put(FIELD_TITLE, title);
+ doc.put(FIELD_DESCRIPTION, description);
+ doc.put(FIELD_RESOURCE, RESOURCE_SCHEME + subject.replace('.', '/'));
+ ArrayNode tags = doc.putArray(FIELD_TAGS);
+ tags.add(BASE_TAG);
+ tags.add(conceptType.substring(conceptType.lastIndexOf(' ') + 1).toLowerCase());
+ tags.add(dataverse);
+ doc.put(FIELD_TIMESTAMP, timestamp);
+ doc.put(FIELD_SOURCE_QUERY, sourceQuery);
+ doc.put(FIELD_TEXT, text);
+ ArrayNode linkArray = doc.putArray(FIELD_LINKS);
+ links.forEach(linkArray::add);
+ return doc;
+ }
+
+ private void addConcept(String subject, ObjectNode doc) {
+ conceptsBySubject.put(subject, doc.toString());
+ }
+
+ /**
+ * Folds the most recent completed cluster statements that reference this
+ * dataset into its doc. Best-effort enrichment; the walk never fails on it.
+ */
+ private List<JsonNode> observedRequests(String datasetName) {
+ return selectObserved(workload(), datasetName, excludeMarker);
+ }
+
+ /**
+ * The most recent completed statements referencing {@code datasetName},
+ * minus this function's own walks and, when the caller supplied a marker,
+ * minus the statements the caller stamped with it.
+ */
+ static List<JsonNode> selectObserved(List<JsonNode> workload, String datasetName, String excludeMarker) {
+ boolean excluding = excludeMarker != null && !excludeMarker.isEmpty();
+ List<JsonNode> observed = new ArrayList<>();
+ for (JsonNode request : workload) {
+ String statement = request.path(REQUEST_STATEMENT).asText("");
+ if (statement.isEmpty() || statement.contains(SELF_REFERENCE) || !statement.contains(datasetName)) {
+ continue;
+ }
+ if (excluding && statement.contains(excludeMarker)) {
+ continue;
+ }
+ observed.add(request);
+ }
+ observed.sort(Comparator.comparing((JsonNode r) -> r.path(REQUEST_TIME).asText()).reversed());
+ return observed.subList(0, Math.min(MAX_OBSERVED_QUERIES, observed.size()));
+ }
+
+ private List<JsonNode> workload() {
+ if (workload != null) {
+ return workload;
+ }
+ workload = new ArrayList<>();
+ try {
+ for (IClientRequest request : metadataProvider.getApplicationContext().getRequestTracker()
+ .getCompletedRequests()) {
+ workload.add(JSON.readTree(request.toJson()));
+ }
+ } catch (Exception e) {
+ // mining is optional enrichment; an unavailable tracker must not break the walk
+ }
+ return workload;
+ }
+
+ static String oneLine(String statement) {
+ String flat = statement.replaceAll("\\s+", " ").replace("`", "'").trim();
+ return flat.length() > MAX_OBSERVED_QUERY_LENGTH ? flat.substring(0, MAX_OBSERVED_QUERY_LENGTH) + "…"
+ : flat;
+ }
+
+ private ARecordType resolveRecordType(Dataset dataset) throws AlgebricksException {
+ Datatype datatype = metadataProvider.findTypeEntity(dataset.getItemTypeDatabaseName(),
+ dataset.getItemTypeDataverseName(), dataset.getItemTypeName());
+ if (datatype == null) {
+ return null;
+ }
+ IAType type = datatype.getDatatype();
+ return type instanceof ARecordType ? (ARecordType) type : null;
+ }
+
+ private String datatypeSubject(Dataset dataset) {
+ DataverseName typeDv = dataset.getItemTypeDataverseName();
+ if (typeDv == null || dataset.getItemTypeName() == null) {
+ return null;
+ }
+ return typeDv.getCanonicalForm() + "/type/" + dataset.getItemTypeName();
+ }
+
+ static String indexSubject(String datasetSubject, Index index) {
+ return datasetSubject + "/index/" + index.getIndexName();
+ }
+
+ static String conceptTypeOf(Dataset dataset) {
+ switch (dataset.getDatasetType()) {
+ case EXTERNAL:
+ return TYPE_EXTERNAL_DATASET;
+ case VIEW:
+ return TYPE_VIEW;
+ default:
+ return TYPE_DATASET;
+ }
+ }
+
+ static String describeDataset(String conceptType, String name, String dataverse) {
+ return conceptType.substring("AsterixDB ".length()) + " " + name + " in dataverse " + dataverse;
+ }
+
+ /**
+ * The catalog query that reproduces a concept, as
+ * {@code SELECT VALUE d FROM Metadata.`Entity` d WHERE d.Field = "value"}.
+ * Values arrive as alternating field/value pairs and are escaped, since a
+ * consumer may run what it is handed.
+ */
+ static String sourceQuery(String entity, String... fieldsAndValues) {
+ StringBuilder query =
+ new StringBuilder("SELECT VALUE d FROM Metadata.`").append(entity).append("` d WHERE ");
+ for (int i = 0; i < fieldsAndValues.length; i += 2) {
+ if (i > 0) {
+ query.append(" AND ");
+ }
+ query.append("d.").append(fieldsAndValues[i]).append(" = ")
+ .append(stringLiteral(fieldsAndValues[i + 1]));
+ }
+ return query.toString();
+ }
+
+ /** The dataverse arrives as its display form, which is already a valid multi-part identifier. */
+ static String sampleQuery(String dataverseIdentifier, String name) {
+ return "SELECT * FROM " + dataverseIdentifier + "." + delimitedIdentifier(name) + " LIMIT 10;";
+ }
+
+ static String countQuery(String dataverseIdentifier, String name) {
+ return "SELECT COUNT(*) AS cnt FROM " + dataverseIdentifier + "." + delimitedIdentifier(name) + ";";
+ }
+
+ /** A double-quoted SQL++ string literal; the value cannot end the literal early. */
+ static String stringLiteral(String value) {
+ return "\"" + value.replace("\\", "\\\\").replace("\"", "\\\"") + "\"";
+ }
+
+ /** A backtick-delimited SQL++ identifier; the name cannot end the identifier early. */
+ static String delimitedIdentifier(String name) {
+ return "`" + name.replace("\\", "\\\\").replace("`", "\\`") + "`";
+ }
+
+ /** Renders a (possibly nested) record type as a markdown field table with dotted paths. */
+ static void renderSchema(StringBuilder body, ARecordType recordType) {
+ if (recordType == null) {
+ body.append("_open type; schema not statically known_\n");
+ return;
+ }
+ body.append("| Field | Type |\n| --- | --- |\n");
+ renderFields(body, recordType, "");
+ }
+
+ private static void renderFields(StringBuilder body, ARecordType recordType, String prefix) {
+ String[] names = recordType.getFieldNames();
+ IAType[] types = recordType.getFieldTypes();
+ for (int i = 0; i < names.length; i++) {
+ IAType fieldType = types[i];
+ boolean optional = false;
+ if (fieldType instanceof AUnionType && ((AUnionType) fieldType).isUnknownableType()) {
+ optional = true;
+ fieldType = ((AUnionType) fieldType).getActualType();
+ }
+ String path = prefix + names[i];
+ body.append("| ").append(path).append(" | ").append(typeNameOf(fieldType)).append(optional ? "?" : "")
+ .append(" |\n");
+ if (fieldType instanceof ARecordType) {
+ renderFields(body, (ARecordType) fieldType, path + ".");
+ }
+ }
+ }
+
+ static String typeNameOf(IAType type) {
+ if (type instanceof ARecordType) {
+ return "object";
+ }
+ if (type instanceof AbstractCollectionType) {
+ IAType item = ((AbstractCollectionType) type).getItemType();
+ return "[" + typeNameOf(item) + "]";
+ }
+ return type.getTypeName();
+ }
+ }
+}
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/util/MetadataBuiltinFunctions.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/util/MetadataBuiltinFunctions.java
index 58bc863..31b91c4 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/util/MetadataBuiltinFunctions.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/util/MetadataBuiltinFunctions.java
@@ -25,6 +25,7 @@
import org.apache.asterix.app.function.DumpIndexRewriter;
import org.apache.asterix.app.function.FeedRewriter;
import org.apache.asterix.app.function.JobSummariesRewriter;
+import org.apache.asterix.app.function.OKFCatalogRewriter;
import org.apache.asterix.app.function.OpenRequestsRewriter;
import org.apache.asterix.app.function.PingRewriter;
import org.apache.asterix.app.function.QueryIndexRewriter;
@@ -61,6 +62,12 @@
BuiltinFunctions.addUnnestFun(StorageComponentsRewriter.STORAGE_COMPONENTS, false);
BuiltinFunctions.addDatasourceFunction(StorageComponentsRewriter.STORAGE_COMPONENTS,
StorageComponentsRewriter.INSTANCE, BuiltinFunctions.DataSourceFunctionProperty.MIN_MEMORY_BUDGET);
+ // OKF catalog function
+ BuiltinFunctions.addPrivateFunction(OKFCatalogRewriter.OKF_CATALOG,
+ (expression, env, mp) -> RecordUtil.FULLY_OPEN_RECORD_TYPE, true);
+ BuiltinFunctions.addUnnestFun(OKFCatalogRewriter.OKF_CATALOG, true);
+ BuiltinFunctions.addDatasourceFunction(OKFCatalogRewriter.OKF_CATALOG, OKFCatalogRewriter.INSTANCE,
+ BuiltinFunctions.DataSourceFunctionProperty.MIN_MEMORY_BUDGET);
// Ping function
BuiltinFunctions.addPrivateFunction(PingRewriter.PING,
(expression, env, mp) -> RecordUtil.FULLY_OPEN_RECORD_TYPE, true);
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/function/OKFCatalogRewriterTest.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/function/OKFCatalogRewriterTest.java
new file mode 100644
index 0000000..0281d25
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/function/OKFCatalogRewriterTest.java
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.app.function;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.asterix.app.function.OKFCatalogRewriter.BundleBuilder;
+import org.apache.asterix.common.config.DatasetConfig.DatasetType;
+import org.apache.asterix.metadata.entities.Dataset;
+import org.apache.asterix.metadata.entities.Index;
+import org.apache.asterix.om.types.AOrderedListType;
+import org.apache.asterix.om.types.ARecordType;
+import org.apache.asterix.om.types.AUnionType;
+import org.apache.asterix.om.types.BuiltinType;
+import org.apache.asterix.om.types.IAType;
+import org.apache.hyracks.util.annotations.AiProvenance;
+import org.junit.Test;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.JsonNodeFactory;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+
+@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_4_8, tool = AiProvenance.Tool.CLAUDE_CODE_CLI, contributionKind = AiProvenance.ContributionKind.TEST_GENERATED, notes = "Unit coverage for schema rendering, concept identities, doc shape and workload flattening")
+public class OKFCatalogRewriterTest {
+
+ private static ARecordType record(String name, String[] fields, IAType[] types, boolean open) {
+ return new ARecordType(name, fields, types, open);
+ }
+
+ @Test
+ public void typeNameOfRendersSqlppNames() {
+ assertEquals("string", BundleBuilder.typeNameOf(BuiltinType.ASTRING));
+ assertEquals("double", BundleBuilder.typeNameOf(BuiltinType.ADOUBLE));
+ assertEquals("int64", BundleBuilder.typeNameOf(BuiltinType.AINT64));
+ }
+
+ @Test
+ public void typeNameOfCollapsesRecordsToObjectAndNestsCollections() {
+ ARecordType inner = record("AddrT", new String[] { "city" }, new IAType[] { BuiltinType.ASTRING }, false);
+ assertEquals("object", BundleBuilder.typeNameOf(inner));
+ assertEquals("[string]", BundleBuilder.typeNameOf(new AOrderedListType(BuiltinType.ASTRING, "tags")));
+ assertEquals("[object]", BundleBuilder.typeNameOf(new AOrderedListType(inner, "addrs")));
+ }
+
+ @Test
+ public void renderSchemaFallsBackWhenTypeIsNotStaticallyKnown() {
+ StringBuilder body = new StringBuilder();
+ BundleBuilder.renderSchema(body, null);
+ assertEquals("_open type; schema not statically known_\n", body.toString());
+ }
+
+ @Test
+ public void renderSchemaEmitsDottedPathsAndOptionalMarkers() {
+ ARecordType addr = record("AddrT", new String[] { "city", "zip" },
+ new IAType[] { BuiltinType.ASTRING, AUnionType.createUnknownableType(BuiltinType.ASTRING) }, false);
+ ARecordType order = record("OrderT", new String[] { "id", "total", "addr" },
+ new IAType[] { BuiltinType.AINT64, BuiltinType.ADOUBLE, AUnionType.createUnknownableType(addr) }, true);
+
+ StringBuilder body = new StringBuilder();
+ BundleBuilder.renderSchema(body, order);
+ String schema = body.toString();
+
+ assertTrue(schema.startsWith("| Field | Type |\n| --- | --- |\n"));
+ // IAType exposes the internal spelling, so the SQL++ bigint renders as int64
+ assertTrue(schema.contains("| id | int64 |\n"));
+ assertTrue(schema.contains("| total | double |\n"));
+ assertTrue(schema.contains("| addr | object? |\n"));
+ assertTrue(schema.contains("| addr.city | string |\n"));
+ assertTrue(schema.contains("| addr.zip | string? |\n"));
+ }
+
+ @Test
+ public void oneLineFlattensWhitespaceAndNeutralizesBackticks() {
+ assertEquals("select * from 'ds';", BundleBuilder.oneLine("select *\n from `ds`;\n"));
+ }
+
+ @Test
+ public void oneLineTruncatesOverlongStatements() {
+ String statement = "select " + "x".repeat(300);
+ String flat = BundleBuilder.oneLine(statement);
+ assertEquals(201, flat.length());
+ assertTrue(flat.endsWith("…"));
+ }
+
+ @Test
+ public void profilingQueriesDelimitTheDatasetName() {
+ assertEquals("SELECT * FROM okftest.`my orders` LIMIT 10;", BundleBuilder.sampleQuery("okftest", "my orders"));
+ assertEquals("SELECT COUNT(*) AS cnt FROM okftest.`orders`;", BundleBuilder.countQuery("okftest", "orders"));
+ }
+
+ @Test
+ public void sourceQueryPairsFieldsWithEscapedValues() {
+ assertEquals("SELECT VALUE d FROM Metadata.`Dataverse` d WHERE d.DataverseName = \"okftest\"",
+ BundleBuilder.sourceQuery("Dataverse", "DataverseName", "okftest"));
+ assertEquals(
+ "SELECT VALUE d FROM Metadata.`Dataset` d "
+ + "WHERE d.DataverseName = \"okftest\" AND d.DatasetName = \"orders\"",
+ BundleBuilder.sourceQuery("Dataset", "DataverseName", "okftest", "DatasetName", "orders"));
+ }
+
+ @Test
+ public void aNameCannotBreakOutOfAStringLiteral() {
+ // a dataset named x" OR "1"="1 must not close the literal it lands in
+ assertEquals("\"x\\\" OR \\\"1\\\"=\\\"1\"", BundleBuilder.stringLiteral("x\" OR \"1\"=\"1"));
+ assertEquals("\"back\\\\slash\"", BundleBuilder.stringLiteral("back\\slash"));
+ assertEquals("SELECT VALUE d FROM Metadata.`Dataset` d WHERE d.DatasetName = \"a\\\"b\"",
+ BundleBuilder.sourceQuery("Dataset", "DatasetName", "a\"b"));
+ }
+
+ @Test
+ public void aNameCannotBreakOutOfADelimitedIdentifier() {
+ assertEquals("`plain`", BundleBuilder.delimitedIdentifier("plain"));
+ assertEquals("`tick\\`inside`", BundleBuilder.delimitedIdentifier("tick`inside"));
+ assertEquals("SELECT COUNT(*) AS cnt FROM dv.`a\\`b`;", BundleBuilder.countQuery("dv", "a`b"));
+ }
+
+ @Test
+ public void conceptSubjectsAreStableAndHierarchical() {
+ Index index = mock(Index.class);
+ when(index.getIndexName()).thenReturn("totalIdx");
+ assertEquals("okftest.orders/index/totalIdx", BundleBuilder.indexSubject("okftest.orders", index));
+ assertEquals("Dataset orders in dataverse okftest",
+ BundleBuilder.describeDataset("AsterixDB Dataset", "orders", "okftest"));
+ }
+
+ @Test
+ public void conceptTypeFollowsTheDatasetKind() {
+ assertEquals("AsterixDB Dataset", BundleBuilder.conceptTypeOf(datasetOfType(DatasetType.INTERNAL)));
+ assertEquals("AsterixDB External Dataset", BundleBuilder.conceptTypeOf(datasetOfType(DatasetType.EXTERNAL)));
+ assertEquals("AsterixDB View", BundleBuilder.conceptTypeOf(datasetOfType(DatasetType.VIEW)));
+ }
+
+ private static Dataset datasetOfType(DatasetType type) {
+ Dataset dataset = mock(Dataset.class);
+ when(dataset.getDatasetType()).thenReturn(type);
+ return dataset;
+ }
+
+ @Test
+ public void anAbsentOrEmptyDataverseWalksEverythingButTheSystemDataverse() {
+ assertFalse(BundleBuilder.isFilteredOut("okftest", null));
+ assertFalse(BundleBuilder.isFilteredOut("okftest", ""));
+ assertTrue(BundleBuilder.isFilteredOut("Metadata", null));
+ assertTrue(BundleBuilder.isFilteredOut("Metadata", ""));
+ }
+
+ @Test
+ public void aNonEmptyDataverseWalksOnlyThatDataverse() {
+ assertFalse(BundleBuilder.isFilteredOut("okftest", "okftest"));
+ assertTrue(BundleBuilder.isFilteredOut("other", "okftest"));
+ // an explicit request for the system dataverse is honoured
+ assertFalse(BundleBuilder.isFilteredOut("Metadata", "Metadata"));
+ }
+
+ @Test
+ public void observedStatementsExcludeWalksAndUnrelatedDatasets() {
+ List<JsonNode> workload = Arrays.asList(request("SELECT * FROM orders;", "3"),
+ request("SELECT * FROM okf_catalog('okftest') c;", "2"), request("SELECT * FROM other;", "1"));
+
+ List<JsonNode> observed = BundleBuilder.selectObserved(workload, "orders", null);
+
+ assertEquals(1, observed.size());
+ assertEquals("SELECT * FROM orders;", observed.get(0).get("statement").asText());
+ }
+
+ @Test
+ public void observedStatementsExcludeTheCallersOwnMarkedPlumbing() {
+ List<JsonNode> workload = Arrays.asList(request("SELECT COUNT(*) AS cnt FROM orders;", "2"),
+ request("/*okf*/ SELECT COUNT(*) AS cnt FROM orders;", "1"));
+
+ assertEquals(2, BundleBuilder.selectObserved(workload, "orders", null).size());
+ assertEquals(2, BundleBuilder.selectObserved(workload, "orders", "").size());
+
+ List<JsonNode> observed = BundleBuilder.selectObserved(workload, "orders", "/*okf*/");
+ assertEquals(1, observed.size());
+ assertEquals("SELECT COUNT(*) AS cnt FROM orders;", observed.get(0).get("statement").asText());
+ }
+
+ @Test
+ public void observedStatementsAreTheMostRecentFew() {
+ List<JsonNode> workload =
+ Arrays.asList(request("SELECT 1 FROM orders;", "1"), request("SELECT 2 FROM orders;", "2"),
+ request("SELECT 3 FROM orders;", "3"), request("SELECT 4 FROM orders;", "4"));
+
+ List<JsonNode> observed = BundleBuilder.selectObserved(workload, "orders", null);
+
+ assertEquals(3, observed.size());
+ assertEquals("SELECT 4 FROM orders;", observed.get(0).get("statement").asText());
+ assertEquals("SELECT 2 FROM orders;", observed.get(2).get("statement").asText());
+ }
+
+ private static JsonNode request(String statement, String requestTime) {
+ ObjectNode request = JsonNodeFactory.instance.objectNode();
+ request.put("statement", statement);
+ request.put("requestTime", requestTime);
+ return request;
+ }
+
+ @Test
+ public void conceptCarriesIdentityGroundingAndLinks() {
+ List<String> links = Arrays.asList("okftest/type/OrderT", "okftest.orders/index/totalIdx");
+ ObjectNode doc = new BundleBuilder(null, null, null).concept("okftest.orders", "AsterixDB Dataset", "orders",
+ "Dataset orders in dataverse okftest", "okftest", "SELECT VALUE d FROM Metadata.`Dataset` d",
+ "# Schema\n", links);
+
+ assertEquals("okftest.orders", doc.get("subject").asText());
+ assertEquals("AsterixDB Dataset", doc.get("type").asText());
+ assertEquals("orders", doc.get("title").asText());
+ assertEquals("asterix://okftest/orders", doc.get("resource").asText());
+ assertEquals("SELECT VALUE d FROM Metadata.`Dataset` d", doc.get("source_query").asText());
+ assertEquals("# Schema\n", doc.get("text").asText());
+ assertFalse(doc.get("timestamp").asText().isEmpty());
+
+ assertEquals(3, doc.get("tags").size());
+ assertEquals("asterixdb", doc.get("tags").get(0).asText());
+ assertEquals("dataset", doc.get("tags").get(1).asText());
+ assertEquals("okftest", doc.get("tags").get(2).asText());
+
+ assertEquals(2, doc.get("links").size());
+ assertEquals("okftest/type/OrderT", doc.get("links").get(0).asText());
+ }
+}
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.1.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.1.ddl.sqlpp
new file mode 100644
index 0000000..7df8eaf
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.1.ddl.sqlpp
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+drop dataverse okftest if exists;
+create dataverse okftest;
+
+use okftest;
+
+create type okftest.AddrT as closed {
+ city : string,
+ zip : string?
+};
+
+create type okftest.OrderT as open {
+ id : bigint,
+ total : double,
+ addr : AddrT?
+};
+
+create dataset orders(OrderT) primary key id;
+
+create index totalIdx on orders(total);
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.2.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.2.query.sqlpp
new file mode 100644
index 0000000..78d0264
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.2.query.sqlpp
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// one concept doc each for: dataverse, dataset, datatype, secondary index
+set `import-private-functions` `true`;
+
+select value count(*) from okf_catalog('okftest') c;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.3.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.3.query.sqlpp
new file mode 100644
index 0000000..b0b6996
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.3.query.sqlpp
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// link integrity: every link target resolves to a concept in the bundle
+set `import-private-functions` `true`;
+
+with bundle as (select value c from okf_catalog('okftest') c)
+select value count(*)
+from bundle b unnest b.links l
+where l not in (select value c.subject from bundle c);
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.4.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.4.query.sqlpp
new file mode 100644
index 0000000..64c4d18
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.4.query.sqlpp
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// stable concept identities
+set `import-private-functions` `true`;
+
+select value c.subject from okf_catalog('okftest') c order by c.subject;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.5.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.5.query.sqlpp
new file mode 100644
index 0000000..0f44cde
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.5.query.sqlpp
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// schema rendering: nested dotted paths, optional markers, SQL++ type names
+set `import-private-functions` `true`;
+
+select value contains(c.text, '| addr | object? |')
+ and contains(c.text, '| addr.city | string |')
+ and contains(c.text, '| addr.zip | string? |')
+ and contains(c.text, '| total | double |')
+from okf_catalog('okftest') c where c.subject = 'okftest.orders';
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.6.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.6.update.sqlpp
new file mode 100644
index 0000000..20cae51
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.6.update.sqlpp
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use okftest;
+
+insert into orders ([
+ {"id": 1, "total": 10.0, "addr": {"city": "Irvine", "zip": "92617"}},
+ {"id": 2, "total": 20.5, "addr": {"city": "Tustin"}},
+ {"id": 3, "total": 30.25}
+]);
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.7.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.7.ddl.sqlpp
new file mode 100644
index 0000000..e535858
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.7.ddl.sqlpp
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+analyze dataset okftest.orders;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.8.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.8.query.sqlpp
new file mode 100644
index 0000000..dbb4e1d
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.8.query.sqlpp
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// sample-based statistics from the ANALYZE sample; sample index is not an access path
+set `import-private-functions` `true`;
+
+// the exact cardinality is the sample's estimate, so only its presence is asserted
+select value contains(c.text, '# Statistics')
+ and contains(c.text, '- Row count: ')
+ and (not contains(c.text, 'sample_idx'))
+from okf_catalog('okftest') c where c.subject = 'okftest.orders';
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.9.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.9.query.sqlpp
new file mode 100644
index 0000000..0c9bc78
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/misc/okf-catalog/okf-catalog.9.query.sqlpp
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// the caller's own marked statements are left out of the workload mining;
+// the concept graph itself is unchanged by the marker
+set `import-private-functions` `true`;
+
+select value count(*) from okf_catalog('okftest', '/*okf*/') c;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.2.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.2.adm
new file mode 100644
index 0000000..b8626c4
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.2.adm
@@ -0,0 +1 @@
+4
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.3.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.3.adm
new file mode 100644
index 0000000..573541a
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.3.adm
@@ -0,0 +1 @@
+0
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.4.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.4.adm
new file mode 100644
index 0000000..a4f007d
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.4.adm
@@ -0,0 +1,4 @@
+"okftest"
+"okftest.orders"
+"okftest.orders/index/totalIdx"
+"okftest/type/OrderT"
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.5.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.5.adm
new file mode 100644
index 0000000..27ba77d
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.5.adm
@@ -0,0 +1 @@
+true
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.8.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.8.adm
new file mode 100644
index 0000000..27ba77d
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.8.adm
@@ -0,0 +1 @@
+true
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.9.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.9.adm
new file mode 100644
index 0000000..b8626c4
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/misc/okf-catalog/okf-catalog.9.adm
@@ -0,0 +1 @@
+4
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/sqlpp_queries.xml b/asterixdb/asterix-app/src/test/resources/runtimets/sqlpp_queries.xml
index 7c1dc97..2cf9500 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/sqlpp_queries.xml
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/sqlpp_queries.xml
@@ -7302,6 +7302,11 @@
</compilation-unit>
</test-case>
<test-case FilePath="misc">
+ <compilation-unit name="okf-catalog">
+ <output-dir compare="Text">okf-catalog</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="misc">
<compilation-unit name="ping">
<output-dir compare="Text">ping</output-dir>
</compilation-unit>
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21484?usp=email
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Iff8891618f0cc715abb8b2fb8c0c35c54093c586
Gerrit-Change-Number: 21484
Gerrit-PatchSet: 1
Gerrit-Owner: Vivek Gangavarapu <ga...@gmail.com>