You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sourcelume.apache.org by "CalvinKirs (via GitHub) via issues" <is...@sourcelume.apache.org> on 2026/10/31 09:59:43 UTC
[I] [Proposal] Introduce a metadata storage SPI [sourcelume-registry]
CalvinKirs opened a new issue, #31:
URL: https://github.com/apache/sourcelume-registry/issues/31
I'd like to introduce a Java SPI between Registry's application services and its metadata system. Atlas would become the first plugin implementation, with DataHub used to check that the interfaces work beyond Atlas. Switching implementations would change the plugin and its configuration, without adding product-specific branches to the ingest, validation, or query services.
A deployment would continue to use one metadata system to store and query Registry records. The plugin would provide both `RecordStore` and `RecordSearch`; those interfaces describe different access patterns over the same records.
## 1. Motivation and scope
Registry currently stores the original JSON-LD, record type, validation status, validation issues, and content digest in Atlas entities. Atlas also serves record reads, reference lookups, and search.
The existing [AtlasAdapter](https://github.com/apache/sourcelume-registry/blob/4d32c8c2239c4e2bc5a0498dfb453ab2b3c7ca9d/sourcelume-registry-core/src/main/java/org/apache/sourcelume/registry/core/AtlasAdapter.java) abstracts some client calls, but callers still deal with physical entity types, GUIDs, and type definitions. Startup and packaging also depend directly on Atlas. Adding another metadata system therefore reaches into the core and runtime modules.
The proposed boundary has four goals:
- Keep record identity, logical type, validation, and application queries in Core.
- Keep product models, identity mapping, clients, authentication, and error translation in each plugin.
- Let the runtime select and initialize a plugin through a common SPI.
- Retain a deployment with one metadata system and one persistence target per submission.
The first version covers record submission, correction of incomplete records, point reads, batched reference lookups, and search by logical type and validation status. Cross-system replication, a separate record database, projection pipelines, graph queries, full-text search, deletion, and edits to validated records are outside this proposal.
The [current ingest contract](https://github.com/apache/sourcelume-registry/blob/4d32c8c2239c4e2bc5a0498dfb453ab2b3c7ca9d/docs/ingest.md#design-notes) already documents a race between checking a record and writing it. The SPI makes that guarantee explicit and lets a deployment require a stronger guarantee where the selected implementation supports it.
## 2. Architecture
```mermaid
flowchart TB
HTTP[HTTP resources] --> INGEST[IngestService]
HTTP --> QUERY[RecordQueryService]
HTTP --> VALIDATION[ValidationService / RecordTypeRegistry]
INGEST --> VALIDATION
INGEST --> STORE[RecordStore SPI]
QUERY --> STORE
QUERY --> SEARCH[RecordSearch SPI]
STORE --> PLUGIN[Selected RegistryPlugin]
SEARCH --> PLUGIN
PLUGIN --> SYSTEM[Atlas or DataHub]
VALIDATION --> VALIDATORS[RecordValidator implementations]
BOOT[PluginBootstrap] --> LOADER[PluginLoader]
LOADER --> PROVIDER[RegistryPluginProvider]
PROVIDER -->|open| PLUGIN
LOADER -.->|check write guarantee| STORE
BOOT -.->|own and close| PLUGIN
```
`RecordStore` serves authoritative point reads and writes. `RecordSearch` uses the metadata system's search index, which may lag behind writes. Both come from the same plugin instance and use the same namespace, model version, and identity mapping.
| Layer | Responsibility | Product knowledge |
|---|---|---|
| HTTP API | Request parsing, response DTOs, HTTP status codes | None |
| Core | Classification, validation, ingest rules, reference resolution, query orchestration | None |
| SPI | Plugin lifecycle, record access, requests, results, and failure semantics | No vendor types or native queries |
| Product plugin | Identity mapping, model initialization, storage, search, client lifecycle | Private to that implementation |
| Runtime | Configuration, provider discovery, startup checks, service construction, health | Provider ID and opaque plugin options |
| Distribution | Assemble the runtime, one product plugin, and validators | Maven coordinates and defaults |
The separation has a direct purpose:
- A single persistence target keeps submission and failure handling straightforward.
- Separate store and search interfaces prevent reference resolution from depending on an eventually consistent index.
- A complete `RecordEnvelope` lets each plugin persist the original document and its validation result together.
- A provider creates the configured instance; the runtime owns its lifecycle.
- Startup checks prevent a deployment from silently accepting weaker write guarantees than it requires.
If the selected metadata system is unavailable, record operations are unavailable too. This is the same deployment dependency Registry has today.
## 3. Maven modules and dependencies
The parent remains `org.apache.sourcelume:sourcelume-registry`, with Java 21 as the baseline.
```text
sourcelume-registry/
├── pom.xml
├── sourcelume-registry-spi/
├── sourcelume-registry-common/
├── sourcelume-registry-core/
├── sourcelume-registry-validation-jsonschema/
├── sourcelume-registry-validation-shacl/
├── sourcelume-registry-runtime-quarkus/
├── sourcelume-registry-plugin-atlas/
├── sourcelume-registry-plugin-datahub/
├── sourcelume-registry-spi-tck/
├── sourcelume-registry-distribution-atlas/
└── sourcelume-registry-distribution-datahub/
```
| Module | Main package under `org.apache.sourcelume.registry` | Dependencies and contents |
|---|---|---|
| `spi` | `spi` | JDK only. No Quarkus, Jackson, `common`, or vendor SDK dependency. |
| `common` | `common.model`, `common.spec` | SPI, Sourcelume Spec models and resources, JSON-LD parsing. Product DTOs move out. |
| `core` | `core.ingest`, `core.query`, `core.validation` | SPI and common. Application services and validation contracts. |
| `validation-*` | `validation.*` | Core's validation contract, common, and the relevant validation engine. |
| `runtime-quarkus` | `runtime.quarkus.resource`, `.bootstrap`, `.config`, `.producer` | Core, SPI, and Quarkus. Built as a regular JAR. |
| `plugin-atlas` | `plugin.atlas` | SPI, private HTTP/JSON dependencies, private typedef resources. |
| `plugin-datahub` | `plugin.datahub` | SPI, private HTTP/JSON dependencies, model verification. |
| `spi-tck` | `spi.tck` | SPI and test libraries. Test dependency of plugins; excluded from runtime distributions. |
| `distribution-*` | Configuration and packaging | Runtime, the selected product plugin, and validation implementations. Produces the Quarkus application. |
```mermaid
flowchart BT
COMMON[common] --> SPI[spi]
CORE[core] --> SPI
CORE --> COMMON
VALIDATORS[validation-jsonschema / validation-shacl] --> CORE
RUNTIME[runtime-quarkus] --> CORE
RUNTIME --> SPI
ATLAS[plugin-atlas] --> SPI
DATAHUB[plugin-datahub] --> SPI
TCK[spi-tck] --> SPI
DIST_A[distribution-atlas] --> RUNTIME
DIST_A --> ATLAS
DIST_A --> VALIDATORS
DIST_D[distribution-datahub] --> RUNTIME
DIST_D --> DATAHUB
DIST_D --> VALIDATORS
```
Arrows show Maven dependencies. Product plugins depend on SPI, not on Core or Quarkus. Core does not generate Atlas attributes or DataHub aspects. A plugin can extract an optional display title through its own mapper.
The current `atlas-adapter` implementation moves into `plugin-atlas`, which also owns the Atlas typedef resources. Atlas-specific startup and configuration handling move out of the generic runtime.
The parent POM manages versions and build plugins. A product plugin declares its own client dependencies and a test dependency on the TCK:
```xml
<dependencies>
<dependency>
<groupId>org.apache.sourcelume</groupId>
<artifactId>sourcelume-registry-spi</artifactId>
</dependency>
<dependency>
<groupId>org.apache.sourcelume</groupId>
<artifactId>sourcelume-registry-spi-tck</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
```
The Atlas distribution assembles these artifacts:
```xml
<dependencies>
<dependency>
<groupId>org.apache.sourcelume</groupId>
<artifactId>sourcelume-registry-runtime-quarkus</artifactId>
</dependency>
<dependency>
<groupId>org.apache.sourcelume</groupId>
<artifactId>sourcelume-registry-plugin-atlas</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.sourcelume</groupId>
<artifactId>sourcelume-registry-validation-jsonschema</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.sourcelume</groupId>
<artifactId>sourcelume-registry-validation-shacl</artifactId>
<scope>runtime</scope>
</dependency>
</dependencies>
```
The DataHub distribution selects `plugin-datahub` instead. The Quarkus build execution belongs to the distribution module. The runtime JAR supplies a Jandex index, or the distribution includes it explicitly:
```properties
quarkus.index-dependency.registry-runtime.group-id=org.apache.sourcelume
quarkus.index-dependency.registry-runtime.artifact-id=sourcelume-registry-runtime-quarkus
```
Providers are discovered through `ServiceLoader`. The distribution packages their service descriptors and model resources, and the runtime initializes the selected plugin at startup.
DataHub also requires a server-side model artifact. It lives with the DataHub plugin under `model/` and is built with the matching DataHub model toolchain. The deliverables are the model archive, `entity-registry.yaml`, and installation instructions. Deployment of that artifact belongs to the DataHub installation; the Registry plugin verifies it during startup.
## 4. SPI and record contract
All public SPI types live in `org.apache.sourcelume.registry.spi`, with one type per source file.
### 4.1 Entry points
```java
public interface RegistryPluginProvider {
PluginDescriptor descriptor();
RegistryPlugin open(PluginContext context, CallContext startupContext);
}
```
```java
public interface RegistryPlugin extends AutoCloseable {
RecordStore recordStore();
RecordSearch recordSearch();
Health health(CallContext context);
@Override
void close();
}
```
```java
public interface RecordStore {
StoreLimits limits();
/**
* Returns the non-null write guarantee this configured instance actually provides.
* The value is fixed for the lifetime of the instance. The runtime checks it
* against the deployment requirement before exposing the plugin to core services.
*/
WriteGuarantee writeGuarantee();
Optional<StoredRecord> get(RecordId id, CallContext context);
Map<RecordId, RecordHeadResult> heads(Set<RecordId> ids, CallContext context);
WriteResult write(WriteRequest request, CallContext context);
}
```
```java
public interface RecordSearch {
SearchPage search(SearchQuery query, CallContext context);
}
```
```java
public interface SecretResolver {
char[] resolve(String reference, CallContext context);
}
```
`RegistryPluginProvider` is the discovery entry point. Its public no-argument constructor and `descriptor()` perform no I/O. `open()` parses the plugin's options, resolves credentials, creates the client, and checks the required model before returning a usable instance.
| API | Contract |
|---|---|
| `descriptor()` | Stable provider ID, SPI major version, and implementation version. Duplicate provider IDs fail startup. |
| `open(context, startupContext)` | Opens one configured instance. Cleans up resources if initialization fails. |
| `recordStore()` | Required; returns the same store instance on each call. |
| `recordSearch()` | Required; uses the same record namespace and identity mapping as the store. |
| `health(context)` | Checks the dependencies and model required for record operations, without writing probe records. |
| `close()` | Idempotently releases the client, connections, and internal threads. Store and search are not closed separately. |
| `resolve(reference, context)` | Resolves a credential reference. The caller owns the returned character array and clears it after use where practical. |
`PluginContext.instanceId` identifies the configured instance. The persistent record namespace is a separate plugin option; changing the instance name does not move data. Options are immutable and interpreted only by the selected provider.
Initialization defaults to `VERIFY_ONLY`. `CREATE_IF_MISSING` permits creation of missing models through supported product APIs. It does not permit overwriting incompatible models or modifying the server installation. Atlas can create typedefs through REST. DataHub requires its model extension to be installed beforehand in either mode.
### 4.2 Record access
| Method | Semantics |
|---|---|
| `limits()` | Maximum logical ID size in UTF-8 bytes, document size in bytes, and lookup batch size. Limits must account for the product's encoded representation. |
| `writeGuarantee()` | The guarantee this configured instance actually provides. Non-null and fixed for its lifetime. |
| `get(id, context)` | Returns a complete stored snapshot, or `Optional.empty()` when absent. Reads the record store rather than the asynchronous search index. |
| `heads(ids, context)` | Returns one outcome for every input ID: `Found`, `Missing`, or `Failed`. A failure affecting the whole request may throw `PluginException`. |
| `write(request, context)` | Persists a complete envelope. Returns `Written` or a business-level `Rejected`; technical failures throw `PluginException`. |
Store and search instances must support concurrent requests. `CallContext` carries an absolute deadline and a request ID. The remaining budget applies to connection establishment, remote calls, and local lock waits. A plugin must not start another remote operation after the deadline. The request ID is for tracing.
Record IDs are exact logical identifiers. Plugins must not case-fold, normalize Unicode, decode IRIs, or truncate them. All logical record types share one ID space within a namespace. A plugin resolves the physical object from the logical ID; callers never supply an Atlas entity type or DataHub URN.
When a plugin derives a physical key from a hash, it also stores the original ID and checks it on reads and before writes. An identity mismatch is `DATA_CORRUPTION`.
`Revision` is opaque. Core passes it back unchanged and does not interpret it as an Atlas version, GUID, or DataHub aspect counter. `RecordHead` is a summary of one stored snapshot for reference resolution and conflict responses.
### 4.3 Write conditions and guarantees
`Absent` means create only. `AtRevision` means correct an `INCOMPLETE` record using the revision the caller previously read. Every request must supply a condition. The document, bound type, validation result, and digest are saved together.
| Existing record | Condition | Candidate type | Result |
|---|---|---|---|
| Missing | `Absent` | Any recognized type | `CREATED` |
| Missing | `AtRevision` | Any | `NOT_FOUND` |
| Present | Any | Different from the bound type | `TYPE_MISMATCH` |
| `VALIDATED` | Any | Same type | `IMMUTABLE` |
| `INCOMPLETE` | `Absent` | Same type | `ALREADY_EXISTS` |
| `INCOMPLETE` | `AtRevision`, mismatch | Same type | `STALE_REVISION` |
| `INCOMPLETE` | `AtRevision`, match | Same type | `REPLACED`, replacing the complete envelope |
Conditions are evaluated in this order. Core may reject an obvious conflict early, but the plugin must enforce the condition again in the actual write path. Core must not replace the caller's revision with a newer one obtained through an internal read.
`CHECK_THEN_WRITE` performs an authoritative read and checks the condition before issuing a write. There is a race window between those operations. The Atlas implementation uses this guarantee. A process-local lock does not provide protection across Registry instances or external writers.
`ATOMIC_CONDITIONAL` requires the storage service to enforce the condition atomically with the write. At most one concurrent correction using the same revision succeeds. DataHub's conditional aspect writes are a candidate implementation, subject to verification against the deployed version and configuration.
The runtime setting `sourcelume.metadata.requiredWriteGuarantee` specifies the deployment requirement. It defaults to `check-then-write`.
| Required guarantee | Actual guarantee | Startup result |
|---|---|---|
| `CHECK_THEN_WRITE` | `CHECK_THEN_WRITE` | Accept, with the documented race window. |
| `CHECK_THEN_WRITE` | `ATOMIC_CONDITIONAL` | Accept, retaining the stronger guarantee. |
| `ATOMIC_CONDITIONAL` | `CHECK_THEN_WRITE` | Close the instance and fail with `INCOMPATIBLE`. |
| `ATOMIC_CONDITIONAL` | `ATOMIC_CONDITIONAL` | Accept. |
`PluginLoader` checks this after `open()` and before handing the instance to `PluginBootstrap`. An incompatible instance never reaches Core. The requirement is runtime configuration, not a plugin option, and cannot change what a plugin claims to support. Invalid configuration is rejected before opening the plugin; a null capability is a plugin contract error. Matching uses explicit enum cases rather than ordinal ordering.
`Written` contains the snapshot produced by the write. A subsequent point read must observe that completed write or a later version. Search visibility may lag. The response does not guarantee that another writer has not already advanced the record.
### 4.4 Failures and retries
| Failure code | Meaning and handling |
|---|---|
| `UNAVAILABLE` | A dependency cannot be used. Internal write retries are allowed only when the request is known not to have been sent. |
| `TIMEOUT` | The deadline expired and the write outcome is not ambiguous. |
| `UNKNOWN_WRITE` | The write may have committed, but the result cannot be confirmed. Return the request ID; the caller should read the record before deciding what to do next. |
| `AUTHENTICATION`, `PERMISSION_DENIED` | The configured service credentials are invalid or lack permission. These are dependency failures, not authentication failures of the HTTP caller. |
| `INVALID_REQUEST` | Invalid SPI arguments. Distinguish user input errors from internal mapping defects. |
| `LIMIT_EXCEEDED` | A document, batch, capacity, or rate limit was exceeded. |
| `DATA_CORRUPTION` | Stored identity, content, or digest is inconsistent. Do not return a partial record as a successful read. |
| `INCOMPATIBLE` | The required model, model version, or service capability is missing. |
| `CURSOR_EXPIRED` | The cursor is no longer usable; restart the search. |
`PluginException` carries a failure code and a safe message; the cause is available for internal logging. Vendor response bodies, credentials, queries, and SDK stack traces stay out of HTTP responses. Product status codes must be translated using the actual error semantics, rather than treating every `409`, `412`, or `422` as a revision conflict.
After an ambiguous write, the client must not silently retry with a fresh revision or a newly computed validation result. This proposal does not add a replay store or promise exactly-once submission.
### 4.5 Search
`SearchQuery` contains optional logical type and record status filters, a page limit, and an optional cursor. The default limit is 50.
- Type and status are exact filters combined with AND. A plugin must not silently ignore either filter.
- Searches are restricted to the configured namespace. HTTP clients cannot supply a native query, physical type, or different namespace.
- `SearchPage` returns candidate logical IDs, optional display titles, and an optional next cursor. Core reads current records and filters candidates that no longer exist or no longer match.
- The limit applies to candidates. The HTTP result may be shorter after filtering, and must retain the candidate page's next cursor.
- A cursor is opaque to Core and bound to the namespace, query parameters, page position, and format version. Reusing it with a different query is `INVALID_REQUEST`; an expired cursor is `CURSOR_EXPIRED`.
- Pagination is best effort during concurrent changes. It does not promise a snapshot, a strongly consistent total count, or freedom from duplicate or omitted hits.
Store and search can use separate implementation classes, but share the model and identity codec. Search results expose logical IDs, not GUIDs, URNs, or product entities.
### 4.6 Documents and validation results
`RawDocument` preserves the exact valid UTF-8 input bytes. Construction and access use defensive copies. Core parses JSON and identifies the logical record type; the plugin stores the bytes or a reversible encoding without reserializing the document.
`RecordEnvelope` contains the ID, bound type, original document, and complete validation result. `ValidationStamp` records the Spec version, configured validator chain, executed stages, validation resource digests, and validation time.
A correction replaces the entire document and validation result, including issues and the stamp. An empty issue list clears previous issues. Optional display fields that disappear from the new record are also cleared. Original documents and complete validation results are kept out of full-text indexes so index limits cannot alter the stored content.
`INCOMPLETE` requires an explanation of the validation failure. Unavailable validation resources or validator execution failures are system errors and must not be persisted as an `INCOMPLETE` verdict.
### 4.7 Public value types and usage
The following declarations list every SPI value type and its fields. Each declaration belongs in its own file. Collections are immutable, and `Optional` fields are never null. Construction performs local structural checks; Core and the write implementation enforce the record rules above.
<details>
<summary>Complete SPI value types</summary>
```java
public record Absent() implements WriteCondition {}
public record AtRevision(Revision revision) implements WriteCondition {}
public record CallContext(Instant deadline, String requestId) {}
public enum ConflictCode { ALREADY_EXISTS, IMMUTABLE, TYPE_MISMATCH, STALE_REVISION, NOT_FOUND }
public record Failed(FailureCode code, String message) implements RecordHeadResult {}
public enum FailureCode { UNAVAILABLE, TIMEOUT, AUTHENTICATION, PERMISSION_DENIED, INVALID_REQUEST, LIMIT_EXCEEDED, DATA_CORRUPTION, UNKNOWN_WRITE, INCOMPATIBLE, CURSOR_EXPIRED }
public record Found(RecordHead head) implements RecordHeadResult {}
public record Health(boolean ready, String code, String message) {}
public enum InitializationMode { VERIFY_ONLY, CREATE_IF_MISSING }
public record Issue(String validator, String severity, String path, String message) {}
public record Missing() implements RecordHeadResult {}
public record PluginContext(
String instanceId,
Map<String, String> options, SecretResolver secrets,
InitializationMode initializationMode) {}
public record PluginDescriptor(String id, int spiMajor, String implementationVersion) {}
public final class PluginException extends RuntimeException {
public PluginException(FailureCode code, String safeMessage, Throwable cause);
public FailureCode code();
}
public final class RawDocument {
public RawDocument(byte[] utf8);
public int size();
public byte[] copyBytes();
public String sha256();
}
public record RecordEnvelope(
RecordId id, RecordTypeId boundType, RawDocument document, ValidationResult validationResult) {}
public record RecordHead(
RecordId id, RecordTypeId boundType, RecordStatus status,
String sha256, Revision revision) {}
public sealed interface RecordHeadResult permits Found, Missing, Failed {}
public record RecordId(String value) {}
public enum RecordStatus { VALIDATED, INCOMPLETE }
public record RecordTypeId(String value) {}
public record Rejected(
ConflictCode code, Optional<RecordHead> current,
String message) implements WriteResult {}
public record Revision(String value) {}
public record SearchCursor(String value) {}
public record SearchHit(RecordId id, Optional<String> title) {}
public record SearchPage(List<SearchHit> hits, Optional<SearchCursor> next) {}
public record SearchQuery(
Optional<RecordTypeId> type, Optional<RecordStatus> status,
int limit, Optional<SearchCursor> after) {}
public record StoreLimits(int maxRecordIdBytes, int maxDocumentBytes, int maxLookupBatchSize) {}
public record StoredRecord(RecordEnvelope record, Revision revision) {}
public record ValidationResult(RecordStatus status, List<Issue> issues, ValidationStamp stamp) {}
public record ValidationStamp(
String specVersion, List<ValidatorIdentity> chain,
List<String> executedStages, Map<String, String> artifactSha256,
Instant validatedAt) {}
public record ValidatorIdentity(String id, String implementationVersion) {}
public sealed interface WriteCondition permits Absent, AtRevision {}
public enum WriteDisposition { CREATED, REPLACED }
public enum WriteGuarantee { CHECK_THEN_WRITE, ATOMIC_CONDITIONAL }
public record WriteRequest(RecordEnvelope candidate, WriteCondition condition) {}
public sealed interface WriteResult permits Written, Rejected {}
public record Written(StoredRecord stored, WriteDisposition disposition) implements WriteResult {}
```
The context and request types expose these construction methods. Each type has a static `builder()`; setters return that type's nested `Builder`, and `build()` returns the enclosing type.
| Type | Setters | Result |
|---|---|---|
| `PluginContext` | `instanceId(String value)`; `options(Map<String, String> value)`; `option(String key, String value)`; `secrets(SecretResolver value)`; `initializationMode(InitializationMode value)` | `PluginContext build()` |
| `CallContext` | `deadline(Instant value)`; `requestId(String value)` | `CallContext build()` |
| `ValidationStamp` | `specVersion(String value)`; `chain(List<ValidatorIdentity> value)`; `executedStages(List<String> value)`; `artifactSha256(Map<String, String> value)`; `validatedAt(Instant value)` | `ValidationStamp build()` |
| `ValidationResult` | `status(RecordStatus value)`; `issues(List<Issue> value)`; `stamp(ValidationStamp value)` | `ValidationResult build()` |
| `RecordEnvelope` | `id(RecordId value)`; `boundType(RecordTypeId value)`; `document(RawDocument value)`; `validationResult(ValidationResult value)` | `RecordEnvelope build()` |
| `WriteRequest` | `candidate(RecordEnvelope value)`; `condition(WriteCondition value)` | `WriteRequest build()` |
| `SearchQuery` | `type(RecordTypeId value)`; `status(RecordStatus value)`; `limit(int value)`; `after(SearchCursor value)` | `SearchQuery build()` |
</details>
For example:
```java
PluginContext pluginContext = PluginContext.builder()
.instanceId("primary")
.option("endpoint", endpoint)
.option("namespace", "sourcelume")
.secrets(secretResolver)
.initializationMode(InitializationMode.VERIFY_ONLY)
.build();
CallContext call = CallContext.builder()
.deadline(Instant.now().plusSeconds(10))
.requestId(requestId)
.build();
RecordEnvelope envelope = RecordEnvelope.builder()
.id(recordId)
.boundType(recordType)
.document(new RawDocument(rawBytes))
.validationResult(validationResult)
.build();
WriteRequest request = WriteRequest.builder()
.candidate(envelope)
.condition(new Absent())
.build();
WriteResult result = plugin.recordStore().write(request, call);
SearchQuery query = SearchQuery.builder()
.type(recordType)
.status(RecordStatus.VALIDATED)
.limit(50)
.build();
SearchPage page = plugin.recordSearch().search(query, call);
```
## 5. Application services and HTTP API
### 5.1 Core services
These classes belong to Core. Product plugins do not depend on them.
| Class | Public API | Responsibility |
|---|---|---|
| `IngestService` | `WriteResult submit(byte[] raw, WriteCondition condition, CallContext context)` | Classify, validate, construct the envelope, and call the store. |
| `RecordClassifier` | `ClassifiedRecord classify(RawDocument document)` | Parse the Spec ID and logical type; reject malformed or unrecognized input. |
| `ValidationService` | `ValidationResult validate(RawDocument document, RecordTypeId type, CallContext context)` | Run the configured validation stages and collect the verdict and stamp. |
| `RecordTypeRegistry` | `List<RecordTypeDefinition> list()`; `Optional<RecordTypeDefinition> find(RecordTypeId id)` | Expose Spec types and resources. |
| `RecordQueryService` | `Optional<RecordView> get(RecordId id, CallContext context)` | Return the stored record and current reference resolutions. |
| `RecordQueryService` | `Optional<RawRecord> raw(RecordId id, CallContext context)` | Return the exact stored document and revision. |
| `RecordQueryService` | `RecordSearchResult search(SearchQuery query, CallContext context)` | Search for candidates, read current records, and produce application views. |
| `ReferenceResolver` | `List<ReferenceResolution> resolve(StoredRecord record, CallContext context)` | Extract references and use `heads()` in bounded batches. |
| `RecordValidator` | `String id()`; `int order()`; `String implementationVersion()`; `ValidationStageResult validate(RawDocument document, RecordTypeId type, CallContext context)` | Existing validation extension point; validators do not write records. |
Core result types are separate from the storage SPI:
| Type | Fields |
|---|---|
| `ClassifiedRecord` | `RecordId id`, `RecordTypeId boundType` |
| `RecordTypeDefinition` | `RecordTypeId id`, `String specVersion`, `Map<String,String> resources` |
| `RecordView` | `StoredRecord stored`, `List<ReferenceResolution> references` |
| `RawRecord` | `RecordId id`, `RawDocument document`, `Revision revision` |
| `RecordSearchResult` | `List<RecordView> records`, `Optional<SearchCursor> next` |
| `ReferenceResolution` | `String path`, `RecordId id`, `RecordTypeId expectedType`, `ReferenceStatus status`, `Optional<RecordHead> actual`, `Optional<FailureCode> failure` |
| `ReferenceStatus` | `RESOLVED`, `MISSING`, `TYPE_MISMATCH`, `FAILED` |
| `ValidationStageResult` | `boolean conforms`, `List<Issue> issues`, `Map<String,String> artifactSha256` |
| `RecordInputException` | `InputErrorCode code`, safe message. Codes: `INVALID_UTF8`, `INVALID_JSON`, `MISSING_ID`, `MISSING_TYPE`, `UNKNOWN_TYPE`. |
| `ValidationExecutionException` | `ValidationFailureCode code`, safe message, internal cause. Codes: `RESOURCE_UNAVAILABLE`, `TIMEOUT`, `INTERNAL_ERROR`. |
Resolved and type-mismatched references include the current head. Missing references include neither a head nor a failure code. Failed lookups include a failure code and must not be reported as missing. An incomplete record returns its stored validation result without trying to interpret invalid business fields as a complete reference graph.
Services receive their dependencies explicitly:
```java
IngestService ingest = IngestService.builder()
.recordStore(plugin.recordStore())
.recordClassifier(classifier)
.validationService(validation)
.build();
ReferenceResolver references = ReferenceResolver.builder()
.recordStore(plugin.recordStore())
.recordTypeRegistry(recordTypes)
.build();
RecordQueryService query = RecordQueryService.builder()
.recordStore(plugin.recordStore())
.recordSearch(plugin.recordSearch())
.referenceResolver(references)
.build();
```
Service construction performs local checks and does not open product connections.
### 5.2 Runtime classes
| Class | Methods | Responsibility |
|---|---|---|
| `PluginBootstrap` | `void initialize()`; `RegistryPlugin plugin()`; `void close()` | Own the selected instance and its application lifetime. |
| `PluginLoader` | `RegistryPlugin load(RegistryPluginConfiguration config, CallContext startup)` | Discover and select a provider, open it, check required interfaces and write guarantees, and return the instance. |
| `RegistryPluginConfiguration` | `String provider()`; `String instanceId()`; `InitializationMode initialize()`; `WriteGuarantee requiredWriteGuarantee()`; `Map<String,String> options()` | Typed runtime configuration. |
| `CoreServicesProducer` | `IngestService ingestService()`; `RecordQueryService recordQueryService()`; `ValidationService validationService()`; `RecordTypeRegistry recordTypeRegistry()` | Construct application services using the initialized plugin. |
| `PluginReadinessCheck` | `HealthCheckResponse call()` | Report plugin health, provider ID, required and actual guarantees, and safe failure codes. |
| `CallContextFactory` | `CallContext create(String requestId)` | Validate or generate the request ID and set the deadline. |
| `RegistryExceptionMapper` | `Response toResponse(Throwable failure)` | Map application and plugin failures to HTTP responses. |
`PluginBootstrap` is an application-scoped startup bean. Initialization uses `@PostConstruct`, and cleanup uses `@PreDestroy`. Application service producers use `@Produces`, `@Singleton`, and `@Startup`; calling `PluginBootstrap.plugin()` establishes the initialization dependency. The application completes plugin checks and service construction before becoming ready. This follows the [Quarkus startup lifecycle](https://quarkus.io/guides/lifecycle/#using-startup-to-initialize-a-cdi-bean-at-application-startup).
Ownership transfers once: the provider cleans up a failed `open()`; the loader closes an opened instance if a subsequent check fails; the bootstrap owns the instance after `load()` returns. Cleanup failures are suppressed on the original startup exception. Core services and health checks borrow the instance and do not close it.
### 5.3 HTTP endpoints
These routes define the proposed application boundary. IDs use query parameters so `/`, `?`, and `#` inside an IRI do not become path syntax.
| Endpoint | Request | Successful response | Entry point |
|---|---|---|---|
| `POST /records` | JSON-LD body; optional `expectedRevision` query parameter | `201` for create, `200` for correction; `RecordWriteResponse` | `RecordResource.submit` → `IngestService.submit` |
| `POST /records/validate` | JSON-LD body | `200 ValidationResponse`, including a negative validation verdict | `RecordValidationResource.validate` → classifier and validation service |
| `GET /records/by-id?id=...` | Exact logical ID | `200 RecordResponse`; `404` if absent | `RecordResource.get` → `RecordQueryService.get` |
| `GET /records/raw?id=...` | Exact logical ID | `200 application/ld+json`, exact stored bytes | `RecordResource.raw` → `RecordQueryService.raw` |
| `GET /record-types` | None | `200 RecordTypeResponse[]` | `RecordTypeResource.list` → `RecordTypeRegistry.list` |
| `GET /records/search` | Optional `type`, `status`, `after`; `limit` defaults to 50 | `200 RecordSearchResponse` | `RecordSearchResource.search` → `RecordQueryService.search` |
A submission without `expectedRevision` uses `Absent`; one with it uses `AtRevision`. Repeated submissions do not implicitly overwrite an incomplete record. The version condition remains subject to the selected plugin's write guarantee.
HTTP search defaults to `VALIDATED`; callers can explicitly request `INCOMPLETE`. Invalid limits, statuses, cursors, and unknown Spec types are rejected. Native identifiers, model names, and namespace selection are not part of the HTTP contract.
The raw-document operation addresses the byte-preservation use case in #29. Batched heads support the read-time reference resolution discussed in #24, while keeping that path independent of search-index freshness.
### 5.4 Response DTOs
HTTP DTOs belong to the runtime and are not SPI transport objects.
| DTO | Fields |
|---|---|
| `RecordWriteResponse` | `RecordResponse record`, `WriteDisposition disposition` |
| `RecordResponse` | `String id`, `String type`, `String revision`, `String rawJsonLd`, `String sha256`, `ValidationResultDto validation`, `ReferenceResolutionDto[] references` |
| `ValidationResponse` | `String id`, `String type`, `String sha256`, `ValidationResultDto validation` |
| `ValidationResultDto` | `String status`, `ValidationIssueDto[] issues`, `ValidationStampDto stamp` |
| `ValidationIssueDto` | `String validator`, `String severity`, `String path`, `String message` |
| `ValidationStampDto` | `String specVersion`, `ValidatorIdentityDto[] chain`, `String[] executedStages`, `Map<String,String> artifactSha256`, `String validatedAt` in ISO 8601 |
| `ValidatorIdentityDto` | `String id`, `String implementationVersion` |
| `RecordHeadDto` | `String id`, `String type`, `String status`, `String sha256`, `String revision` |
| `ReferenceResolutionDto` | `String path`, `String id`, `String expectedType`, `String status`, optional `RecordHeadDto actual`, optional `String failure` |
| `RecordSearchResponse` | `RecordResponse[] records`, optional `String next` |
| `RecordTypeResponse` | `String id`, `String specVersion`, `Map<String,String> resources` |
| `ErrorResponse` | `String code`, `String message`, `String requestId`, optional `RecordHeadDto current` |
Submission responses contain an empty `references` list so a successful write does not depend on follow-up lookups. GET and search responses fill it through `RecordQueryService`. `rawJsonLd` is the decoded original UTF-8 content, not JSON reconstructed from a tree.
Raw responses include `X-Record-Revision`, encoded as unpadded Base64URL of the UTF-8 revision, and `X-Record-Sha256`, the lowercase hexadecimal digest of the original bytes. Clients decode the revision header before passing `expectedRevision`; JSON responses carry the original revision string.
| Failure | HTTP response |
|---|---|
| Invalid input, missing ID/type, unknown type, invalid query parameter | `400` |
| Document exceeds the allowed size | `413` |
| Record absent on GET | `404` |
| Business write conflict, including a missing target for conditional correction | `409`, with a safe current head when available |
| Expired cursor | `410` |
| Unavailable plugin, service authentication/permission/model failure, or ambiguous write | `503` |
| Timeout without an ambiguous write outcome | `504` |
| Corrupt product response | `502` |
| Validator execution failure or internal mapping defect | `500` |
Validation failures produce a verdict and issues. Infrastructure failures produce an error response.
## 6. Plugin implementations
### 6.1 Classes and responsibilities
| Atlas class | DataHub class | Responsibility |
|---|---|---|
| `AtlasPluginProvider` | `DataHubPluginProvider` | Parse options, construct the client, initialize or verify the model, and return the plugin. |
| `AtlasPlugin` | `DataHubPlugin` | Own store, search, and client; implement health and close. |
| `AtlasRecordStore` | `DataHubRecordStore` | Implement point reads, batched heads, and complete-envelope writes. |
| `AtlasRecordSearch` | `DataHubRecordSearch` | Translate filters and pagination; return logical IDs. |
| `AtlasSchemaManager` | `DataHubModelVerifier` | Create or check typedefs for Atlas; verify the installed model extension for DataHub. Called during `open()`. |
| `AtlasRecordMapper` | `DataHubRecordMapper` | Encode/decode the envelope and search fields; validate identity and digest. |
| `AtlasIdentityCodec` | `DataHubIdentityCodec` | Map namespace plus logical ID to the private physical key. |
| `AtlasClient` | `DataHubClient` | Handle protocol, authentication, deadlines, and product error translation. |
These are implementation classes, not additional public SPI interfaces. Store and search share the client and identity codec. Model registration is part of initialization, not a step repeated for each record request.
### 6.2 Atlas
All logical record types map to one private Atlas entity type, `sourcelume_record`. The entity represents the provenance record itself. The logical record type is an attribute, so lookup by ID does not require Core to choose a physical entity type.
| Atlas attribute | Mapping |
|---|---|
| `qualifiedName` | Deterministic short key derived from namespace and exact logical ID; unique and indexed. |
| `namespace` | Fixed plugin namespace; exact filter. |
| `recordId` | Original logical ID, preserved for identity checks; not indexed for full-text search. |
| `recordType` | Bound logical type; exact filter. |
| `recordStatus` | `VALIDATED` or `INCOMPLETE`; separate from Atlas `ACTIVE`/`DELETED`. |
| `title` | Optional display title, explicitly cleared when absent in a replacement. |
| `envelope` | Versioned private JSON containing the reversibly encoded document, identity, and full validation result; not indexed. |
| `sha256` | Original document digest, verified after decoding. |
| `registryRevision` | New opaque plugin-generated revision for condition checks. |
Atlas's public entity API does not turn an entity version into an atomic compare-and-set condition. This implementation declares `CHECK_THEN_WRITE`. See the [Entity REST API](https://atlas.apache.org/api/v2/resource_EntityREST.html) and [Atlas 2.5 entity store](https://github.com/apache/atlas/blob/release-2.5.0/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java).
The original ID and envelope must retain their exact content. Indexable long strings can be subject to length handling in Atlas's storage mapping, so these preservation fields are kept out of the index and verified on reads. See [EntityGraphMapper](https://github.com/apache/atlas/blob/release-2.5.0/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java).
| SPI operation | Atlas operation | Mapping requirement |
|---|---|---|
| `open` | Read typedefs; create missing definitions when allowed | Check field types, indexes, and uniqueness. Reject incompatible existing models. |
| `get` | GET `/api/atlas/v2/entity/uniqueAttribute/type/sourcelume_record`, using `attr:qualifiedName` | Validate namespace, ID, type, and digest. Do not expose a deleted entity as a normal record. |
| `heads` | Batch unique-attribute reads, or bounded parallel point reads | Preserve per-ID failure outcomes. |
| `write` | Authoritative read, rule check, then POST `/api/atlas/v2/entity/bulk` | Replace all owned record fields in one write. |
| `search` | POST `/api/atlas/v2/search/basic` | Fix entity type and namespace, apply logical type/status filters, explicitly request `recordId` and `title`. |
| `health` | Service and required-model checks | A missing user record is not a health failure. |
### 6.3 DataHub
The DataHub plugin uses a dedicated `sourcelumeRecord` entity. The complete record is held in one `sourcelumeRecordInfo` aspect so the document, validation result, and searchable attributes are updated together.
| Model element | Mapping |
|---|---|
| `sourcelumeRecordKey` | Deterministic ID derived from namespace and logical record ID. |
| URN | `urn:li:sourcelumeRecord:<digest>`, private to the plugin. |
| `sourcelumeRecordInfo.namespace` | Exact searchable namespace. |
| `sourcelumeRecordInfo.recordId` | Original logical ID for validation and search-result translation. |
| `recordType`, `recordStatus` | Exact searchable fields; validation status is separate from removal state. |
| `title` | Optional display field; cleared when omitted by a replacement. |
| `envelope` | Versioned string containing the original bytes and complete validation result; not marked searchable. |
| `sha256` | Original document digest. |
| `systemMetadata.version` | Candidate conditional-write revision for this aspect, kept as an opaque string. |
The entity and aspect definitions require PDL, `entity-registry.yaml`, and a model artifact installed in GMS and the relevant indexing components. The provider verifies the extension through supported configuration/model APIs and fails with `INCOMPATIBLE` if it is missing. See [DataHub custom model deployment](https://github.com/datahub-project/datahub/blob/v1.7.0/metadata-models-custom/README.md).
| SPI operation | DataHub operation | Mapping requirement |
|---|---|---|
| `open` | GET `/config` and verify the model registration | Connection configuration alone is insufficient if the model extension is absent. |
| `get` | GET `/entitiesV2/{urn}?aspects=List(sourcelumeRecordInfo)` | Read the aspect and its system metadata; distinguish a missing record aspect from unrelated aspects on an entity. |
| `heads` | Supported batch reads or bounded parallel aspect reads | Return an outcome for every requested logical ID. |
| `write` | Synchronous MCP ingest of the complete aspect | Replace the document, verdict, and search attributes together. |
| `Absent` | Conditional aspect creation; the prototype uses `If-Version-Match=-1` | Reject an existing record aspect. |
| `AtRevision` | Authoritative aspect read and MCP `headers.If-Version-Match` | Use `systemMetadata.version`; translate a confirmed condition failure to `Rejected`. |
| `search` | GMS entity search with exact filters and pagination | Support the custom entity without depending on a built-in GraphQL entity enum. |
DataHub's [conditional-write contract](https://archive.docs.datahub.com/docs/1.7.0/advanced/mcp-mcl#conditional-writes) provides an implementation path. An `ATOMIC_CONDITIONAL` declaration still requires verification of the deployed version, conditional validation configuration, conflict handling, concurrent creates and corrections, and authoritative reads. The historical aspect version parameter is not the conditional-write revision.
Protocol details remain inside `DataHubClient`:
- In the tested 1.7 responses, `systemMetadata.version` is a string. It is passed back unchanged in the MCP's internal headers.
- Rest.li represents the MCP's `aspect.value` as bytes in a string. The document must be encoded as UTF-8 bytes and then represented using Rest.li's byte encoding; a plain Java string containing non-ASCII JSON is not equivalent.
- A version-condition failure can arrive as HTTP `422`. The client checks the specific version failure and reads the current head; it does not treat every `422` as a conflict.
- Search criteria use `values`, for example `{"field":"recordStatus","condition":"EQUAL","values":["VALIDATED"]}`. The older singular `value` field can be ignored despite a successful response.
Search indexing is asynchronous, which fits the search contract above. DataHub retains its normal database, messaging, and search dependencies. Registry does not add another persistence system or synchronization service. A custom DataHub UI view may require additional UI work; Registry record access uses the service APIs.
### 6.4 Registration and configuration
Each plugin includes this service descriptor:
```text
META-INF/services/org.apache.sourcelume.registry.spi.RegistryPluginProvider
```
For Atlas, its contents are:
```text
org.apache.sourcelume.registry.plugin.atlas.AtlasPluginProvider
```
Example Atlas configuration:
```yaml
sourcelume:
metadata:
provider: atlas
instanceId: primary
initialize: verify-only
requiredWriteGuarantee: check-then-write
options:
endpoint: http://atlas:21000
namespace: sourcelume
username: registry
passwordRef: secret:atlas-password
```
Example DataHub configuration:
```yaml
sourcelume:
metadata:
provider: datahub
instanceId: primary
initialize: verify-only
requiredWriteGuarantee: check-then-write
options:
endpoint: http://datahub-gms:8080
namespace: sourcelume
tokenRef: secret:datahub-token
modelRegistryId: sourcelume-record-model
modelVersion: "1"
```
The runtime interprets `provider`, `instanceId`, `initialize`, and `requiredWriteGuarantee`. The selected provider interprets `options` and resolves credential references through `SecretResolver`. Credentials are excluded from configuration snapshots.
`requiredWriteGuarantee` accepts `check-then-write` and `atomic-conditional`, defaulting to the former. Both examples use the guarantee currently declared by the prototypes. A verified implementation can advertise atomic conditional writes; setting the requirement alone cannot upgrade an implementation's behavior.
The namespace is part of persistent identity. Changing it selects a different record space and does not migrate existing data.
### 6.5 Adapter feasibility checks
The SPI draft and independent adapter prototypes compile on Java 21. The prototypes have been exercised against Atlas 2.5.0 and DataHub 1.7.0 with the custom model installed.
Both passed checks for creation, exact document preservation with payloads over 140 KB, incomplete-record correction, clearing old validation issues, sequential conflict rejection, batched lookup outcomes, and type/status-filtered pagination. Provider discovery, repeated initialization, health checks, and reuse of store/search instances were also exercised.
The DataHub prototype additionally passed ten concurrent-create races and ten concurrent corrections from the same revision, with one successful write in each race. It still conservatively reports `CHECK_THEN_WRITE`: these cases do not establish the complete atomic contract for every deployment. Atlas also reports `CHECK_THEN_WRITE`.
These checks use synthetic validation results to exercise persistence. They do not cover the production Quarkus wiring, the real Spec validation chain, credential rotation, or the full failure and recovery behavior. The startup capability check described here is part of the proposed runtime design, not a claim that production integration has shipped.
## 7. Execution flows
### 7.1 Startup and shutdown
```mermaid
sequenceDiagram
participant Q as Quarkus
participant B as PluginBootstrap
participant L as PluginLoader
participant P as RegistryPluginProvider
participant X as RegistryPlugin
participant C as CoreServicesProducer
Q->>B: initialize()
B->>L: load(configuration, startupContext)
L->>L: Validate configuration and discover provider
L->>P: open(pluginContext, startupContext)
P->>P: Open client and verify model
P-->>L: RegistryPlugin
L->>X: Get store, search, and write guarantee
L->>L: Match requiredWriteGuarantee
alt Contract or guarantee does not match
L->>X: close()
L-->>B: Startup failure
B-->>Q: Abort startup
else Checks pass
L-->>B: RegistryPlugin
B->>B: Retain instance
end
Note over Q,C: Continue only after successful initialization
Q->>C: Construct application services
C->>B: plugin()
B-->>C: RegistryPlugin
C->>X: recordStore() / recordSearch()
C->>C: Construct ingest and query services
Q->>Q: Stop accepting requests and drain in-flight work
Q->>B: close()
B->>X: close()
```
Shutdown stops new requests and waits for in-flight work within the shutdown budget before closing the plugin. It does not wait indefinitely for retries. An interrupted write with an uncertain outcome remains `UNKNOWN_WRITE`.
### 7.2 Submission
```mermaid
sequenceDiagram
participant U as Client
participant H as RecordResource
participant I as IngestService
participant V as ValidationService
participant S as RecordStore
participant M as Metadata system
U->>H: POST /records, optional expectedRevision
H->>I: submit(raw, condition, context)
I->>I: Preserve bytes and classify logical type
I->>V: validate(document, type, context)
V-->>I: ValidationResult
I->>I: Construct complete RecordEnvelope
I->>S: write(request, context)
S->>M: Check condition and persist using declared guarantee
M-->>S: Stored result or failure
S-->>I: Written / Rejected, or PluginException
I-->>H: Result
H-->>U: 201 / 200, conflict, or error
```
The plugin does not call validators. The application service does not construct product-specific entities or aspects. Submission completes after the record write; it does not wait for search indexing or reference lookups.
### 7.3 Reads and reference resolution
`RecordResource.get` calls `RecordQueryService.get`, which reads the stored record through `RecordStore.get`. `ReferenceResolver` extracts references from a valid record and calls `heads()` in batches bounded by `StoreLimits`. Each reference retains its own resolved, missing, mismatched, or failed outcome.
The raw-document endpoint calls `RecordQueryService.raw` and returns the stored bytes without resolving references or reserializing the document.
### 7.4 Search
`RecordSearchResource.search` constructs a logical `SearchQuery`. `RecordQueryService` asks `RecordSearch` for candidates, reads their current records through `RecordStore`, removes stale candidates, and resolves references for the retained records. It returns the plugin's next cursor even when filtering leaves a short page.
Feedback on the SPI boundary, write guarantees, and the Atlas/DataHub mappings would be welcome before starting the production refactor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sourcelume.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@sourcelume.apache.org
For additional commands, e-mail: issues-help@sourcelume.apache.org
Re: [I] [Proposal] Introduce a metadata storage SPI [sourcelume-registry]
Posted by "CalvinKirs (via GitHub) via issues" <is...@sourcelume.apache.org>.
CalvinKirs commented on issue #31:
URL: https://github.com/apache/sourcelume-registry/issues/31#issuecomment-6605201416
Thanks, Achim, for taking the time to review this and explain the concrete implications. The tooling examples and the reverse-query use cases are particularly helpful. I agree that the proposal needs a clearer rollout plan and capability boundaries before the production refactor.
1. **Sequence and compatibility.** I’ll separate the SPI extraction, storage-model change, public API changes, and production DataHub integration into independently reviewable steps. Before fixing that sequence, I’ll document how the SPI maps to the existing model, including revisions and validation metadata that older records do not contain. The storage-model step will explicitly address migration versus re-ingestion. The DataHub prototype can continue checking the abstraction while its production integration remains a separate step.
2. **HTTP behavior and tooling.** Looking at the current tooling, Maven and Python also depend on the top-level `recordStatus`, so moving it could hide incomplete publications in all three tools. During the extraction, I intend to preserve the current validation status codes, response fields, correction behavior, and routes, including the recently added document endpoint. That needs an explicit compatibility path: an existing correction without a revision cannot simply become an `Absent` submission. Incompatible changes should have a versioned contract and coordinated tooling updates, with tests covering negative verdicts and missing required response fields.
3. **Lineage and search.** I agree that the reverse-query path needs to be addressed in this design. I’ll add defined, backend-neutral facets for relevant provenance fields and references, and make a named optional graph capability explicit in the proposal. Its relationship semantics and final Java interface still need discussion, but its role should be clear: a rebuildable projection of stored records, with referenced records’ current validation status resolved at read time. Native Atlas and DataHub mappings need validation against those semantics rather than an assumption that existing lineage APIs fit every Sourcelume reference.
4. **Detached signatures.** Agreed that the storage contract needs to be settled with the attestation work before finalizing the envelope and new model. It should cover which document revision a signature belongs to, how it is supplied and retrieved, and how correction replaces or clears it. A bounded sidecar representation is a useful option to work through together; the signature format and verification policy can be agreed with that work.
5. **Atlas concurrency.** You’re right that documenting the race alone is insufficient deployment guidance. My proposed initial baseline is one active Registry writer per namespace, with per-ID serialization of the condition check and write, and no bypass writers. Atlas would still declare `CHECK_THEN_WRITE`. Read-back verification may detect mismatches but cannot prevent a later overwrite, and ambiguous writes need explicit recovery behavior because a remote write may still be in flight after a timeout. I’ll make these assumptions and limitations explicit while retaining the startup check for deployments requiring atomic conditional writes.
I’ll also account for the existing dataset/type routes in the compatibility plan and record the summary-search, unchanged-resubmission, and withdrawal points as separate follow-ups, so their scope is clear. Thanks again for helping identify where the proposal needs to be more concrete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sourcelume.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@sourcelume.apache.org
For additional commands, e-mail: issues-help@sourcelume.apache.org
Re: [I] [Proposal] Introduce a metadata storage SPI [sourcelume-registry]
Posted by "ANierbeck (via GitHub) via issues" <is...@sourcelume.apache.org>.
ANierbeck commented on issue #31:
URL: https://github.com/apache/sourcelume-registry/issues/31#issuecomment-6631584248
Thanks Calvin, this covers what I was hoping for. Three things from my side,
mostly answers to the open points you raised:
**Correction without a revision.** The legacy `POST /records` could map onto
the new contract in Core: read the head; absent → `Absent`; `INCOMPLETE` →
`AtRevision(head)`; `VALIDATED` → 409 as today. Existing clients keep working
and the lost-update protection comes for free.
**Ambiguous writes.** Since the digest covers the exact bytes, a retry of
identical bytes is safe. After a timeout, read the head: same digest means the
write landed (success), a different digest means someone else won. Only the
remaining case needs `UNKNOWN_WRITE`. This also fits the idempotent
resubmit you listed as a follow-up.
**Signatures and lineage.** For the storage contract, I'd treat a signature as
an opaque attachment (media type + bytes) bound to the digest of the stored
document and replaced together with it. The signature format stays with the
attestation thread, so the SPI doesn't need to know it. For the graph
capability, it may help to first list the edges we actually have
(`trainingData`, `custodyChain`, ...) and say per edge what it means, before
looking at the Atlas/DataHub mappings. These are claims made in a signed
record, not observed data flows, which is why the native lineage models may not
fit one to one.
Happy to review the mapping document or a design-doc PR when it's ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sourcelume.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@sourcelume.apache.org
For additional commands, e-mail: issues-help@sourcelume.apache.org
Re: [I] [Proposal] Introduce a metadata storage SPI [sourcelume-registry]
Posted by "ANierbeck (via GitHub) via issues" <is...@sourcelume.apache.org>.
ANierbeck commented on issue #31:
URL: https://github.com/apache/sourcelume-registry/issues/31#issuecomment-6545167702
Read it end to end, so here are my thoughts on the proposal. To be transparent: I worked through it together with an AI assistant to discuss the implications. What follows is what came out of that discussion, and the English is polished by the AI.
A few things I'd ask to sort out before the production refactor:
1. Phase it. As written, this lands four changes at once: the internal SPI, a new Atlas data model, a new public HTTP API, and a second backend. I'd sequence them: (1) SPI + Atlas plugin behind the unchanged HTTP API, plus the TCK; (2) the new Atlas model, with an explicit decision what happens to existing entities (migrate or re-ingest — early stage, either is fine, but say so); (3) HTTP changes, versioned and coordinated with the tooling plugins; (4) the DataHub plugin. Each phase is independently reviewable; the first one is the low-risk core.
2. Lineage needs a place in the SPI, even if it stays empty in v1. This is my main one. Lineage was one of the founding ideas of the project, and it is what the site promises (lineage tracking, graph queries, GraphQL). The proposal puts graph queries out of scope but doesn't say what that means for lineage, and nothing working is lost, since the registry creates no relationships today. But the spec is graph-shaped (trainingData, custodyChain, references across record types), and what we decided so far covers one direction only: a record's references are resolved when it is read. The other direction ("which models were trained on dataset X?", "which records under license Y?", "which records name this agent?") has no query path in the proposal: search is by type and status, and the attributes indexed today (licenseId, sourceUri, modelUri, ...) are gone once everything lives in the envelope.
What makes this concrete rather than hypothetical: the second backend in this very proposal. DataHub's core features are exactly the ones at stake — table- and column-level lineage, faceted Elasticsearch search, and a GraphQL API over its metadata graph. As drafted, the plugin maps records into one opaque aspect, and none of that can surface. The SPI is sized by what both backends share as storage, but both backends are lineage and search systems first — the common denominator is higher than the proposal uses it for.
Two additions would keep that door open without touching the proposal's goals:
Facets: Core extracts defined values (license IRI, referenced dataset ids, creator) into the envelope, plugins index them, and SearchQuery gets an optional facet filter. That alone answers the reverse questions, and it avoids a second data model migration later.
A named, optional graph capability. If it exists, it should be a derived projection that can be rebuilt from the stored envelopes, never the source of truth, and it should not store status: that stays a read-time lookup, as we settled in the references thread (https://lists.apache.org/thread/lnfvvw3v4hkfhwlcpoz6sbp772cy2y69). Atlas could map it to relationships/lineage, DataHub to upstream lineage. Empty in v1, but named, so lineage is documented as deliberately later rather than silently out.
3. The HTTP changes break clients silently. Solvable, but it needs to be sequenced with the tooling: the plugins (Maven, Python, CMake) mostly branch on status codes rather than on response fields. /records/validate would answer 200 for a negative verdict where it answers 422 today — all three plugins would wave non-conforming records through with green builds — and recordStatus moves under record.validation.status, which the CMake module reads at the top level (Sourcelume.cmake), so INCOMPLETE detection would silently stop working. Corrections requiring expectedRevision also means the plugins get a 409 where they currently fix-and-resubmit. Either an API version boundary or a coordinated tooling release, and plugin tests that catch "field no longer there", not just "request failed".
4. No place for signatures. The attestation thread is converging on a detached signature stored with the record and replaced together with it on resubmission — exactly the envelope semantics, but there is no field for it. Since RecordEnvelope is a record, adding one later is an SPI break. A generic slot now (e.g. named attachments with media type and bytes, replaced with the document) would keep Attest from having to reopen the SPI next month.
5. CHECK_THEN_WRITE leaves the core promise unguarded on Atlas. Two concurrent Absent submissions of the same id both see "missing" and both answer 201 CREATED; one record is silently lost, and the client holds a digest for bytes that were never stored. The race is documented, but as the default of the reference implementation it undermines "VALIDATED never changes" and the sign-the-stored-bytes model. At minimum: state a single-writer deployment assumption and add a per-id lock in the plugin (the proposal dismisses locks only because they don't protect across instances — a documented single-writer assumption makes them correct), and/or verify the write after it lands and report UNKNOWN_WRITE on mismatch.
Smaller notes: /api/v1/datasets and /api/v1/types disappear without a word. Dropping the GUID route is consistent with the SPI's "no GUIDs" rule, but these are the only versioned routes we have, and the new surface (/records/by-id, /records/raw, ...) carries no version at all. Since we are working towards a 1.0, I'd version the API from the start (e.g. under /v1), so later incompatible changes become a new version rather than a silent break; deletion/withdrawal is out of scope even though data-protection removals will eventually be a legal requirement — worth one sentence on the intended path; search responses embed the full document per hit — a head/summary view with /raw for the bytes would serve the endpoint better; a byte-identical resubmit of a stored record could be answered as success (UNCHANGED) so CI publishes and ambiguous-write retries don't need conflict handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sourcelume.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@sourcelume.apache.org
For additional commands, e-mail: issues-help@sourcelume.apache.org