You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polaris.apache.org by Yufei Gu <fl...@gmail.com> on 2026/07/08 18:20:26 UTC
[DISCUSS] Semantic Model REST API payload representation
Hi folks,
Following JB's suggestion, I'd like to start a dedicated discussion on the
REST API payload representation for semantic models.
I think there are three possible approaches:
1.
Represent the semantic model as a raw string.
2.
Represent the semantic model as an opaque JSON document.
3.
Model the semantic model structure directly in the REST specification.
I think it's helpful to separate the REST API from Polaris' internal
representation. The REST API is the long-term contract with clients, while
the internal representation can evolve independently.
I'm comfortable with either option 1 or option 2. Both avoid coupling the
REST API to the Ossie schema and allow Polaris to validate the payload
based on the semantic model type and version while preserving the document
through write and read operations.
My concern is with option 3. Since the Ossie schema is versioned and
expected to evolve, modeling the full semantic model structure directly in
the REST specification would tightly couple the Polaris REST API to Ossie
versions. Every Ossie schema evolution could require changes to the REST
specification, generated clients, and potentially client applications.
Between options 1 and 2, I think there is an additional tradeoff.
An opaque JSON document assumes that semantic models are always represented
as JSON. While that works well for Ossie today, Polaris may support other
semantic model formats in the future. For example, OKF[1] is defined as
Markdown rather than JSON. Using a raw string keeps the REST API
independent of any particular document format, allowing Polaris to support
JSON, Markdown, or other representations without changing the API contract.
So my current view is:
-
Option 1 provides the greatest flexibility and is format agnostic.
-
Option 2 is a natural choice if we want to optimize specifically for
JSON based semantic models.
-
Option 3 provides strong typing, but at the cost of coupling the REST
API to Ossie schema evolution.
I'm happy with either option 1 or option 2, but I'd avoid option 3 for the
reasons above.
Thoughts?
1.
https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing
Thanks,
Yufei
Re: [DISCUSS] Semantic Model REST API payload representation
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Yufei,
I believe even with Option 2 the REST API response structure can be defined
with an "envelope" around the semantic payload to allow both plain JSON and
non-JSON formats to be represented. The latter will obviously need to be
encoded inside the JSON response envelope.
Cheers,
Dmitri.
On Mon, Jul 13, 2026 at 2:13 PM Yufei Gu <fl...@gmail.com> wrote:
> Hi Dmitri, Thanks! I think option 2 is a reasonable choice. The advantage I
> see with option 1 is future extensibility. If we later decide to support
> another semantic model format that isn't JSON, such as OKF, option 2 would
> require a REST spec change, while option 1 would not. That's why I slightly
> prefer option 1, although I think either option is much better than
> modeling the semantic model structure directly in the REST API. What do you
> think?
>
> Thanks,
> Yufei
>
>
> On Mon, Jul 13, 2026 at 11:10 AM Yufei Gu <fl...@gmail.com> wrote:
>
> > Hi Robert,
> >
> > Thanks for the thoughtful feedback.
> >
> > I agree that the current API is narrower than a complete semantic layer.
> > Today it's primarily about managing the lifecycle of semantic model
> > documents. Discovery, search, and client consumption are all important
> > topics, but I think they're broader than the Ossie semantic model itself.
> > We touched on some of these in the last community sync[1], and I'm happy
> to
> > continue that discussion in a separate thread.
> >
> > I also agree that we should be careful not to overstate what this initial
> > API provides. Calling it a beta semantic model registry or document
> hosting
> > API seems reasonable.
> >
> > That said, I think those questions are orthogonal to the payload
> > representation discussion. My original topic was simply whether the REST
> > API should treat the semantic model as a raw document, an opaque JSON
> > payload, or model the schema directly. Since the consumption story is
> still
> > evolving, I think keeping the REST contract loosely coupled to the
> > underlying semantic model specification gives us the most flexibility.
> >
> > [1]
> >
> https://drive.google.com/file/d/1hxYkk2t-BcnFOk8eJG9NYCHfjkOXg3Iz/view?usp=sharing
> >
> > Thanks,
> >
> > Yufei
> >
> > On Mon, Jul 13, 2026 at 6:16 AM Robert Stupp <sn...@snazy.de> wrote:
> >
> >> Hi,
> >>
> >> I think the payload representation question depends on the client model
> we
> >> expect this API to support.
> >>
> >> I support Polaris hosting Apache Ossie semantic-model documents as a
> beta
> >> foundation. That seems like a useful first step while Ossie itself is
> >> still
> >> evolving.
> >>
> >> But I do not think we should describe the current API as enabling AI
> >> tools,
> >> BI tools, or human semantic-model discovery yet.
> >>
> >> The merged API is primarily namespace/name CRUD for a semantic-model
> >> document.
> >> That is opaque document storage and exact point-retrieval.
> >>
> >> It does not enable clients to discover the right semantic model from a
> >> query,
> >> nor to find semantic models by table, metric, domain, user, or
> capability.
> >>
> >> It also does not define a standard consuming API or tool contract,
> >> search/indexing contract, freshness model, or current/trusted/certified
> >> model
> >> semantics.
> >>
> >> That distinction matters because the REST API is the user-facing
> contract.
> >>
> >> If the beta API is intended only as opaque document hosting, I think the
> >> spec
> >> and docs should say that clearly, and users/clients should not infer
> >> broader
> >> discovery or interoperability semantics from the CRUD API.
> >>
> >> I am not asking to solve the full semantic-layer story immediately.
> >>
> >> I am asking that durable implementation work does not get ahead of the
> >> client-consumption story.
> >>
> >> The client model should drive the persistent data model, not the other
> way
> >> around.
> >>
> >> Once semantic models are stored as durable Polaris entities, choices
> >> around
> >> identity, versioning, validation, indexing, size limits, source-table
> >> references, and freshness become much harder to change.
> >>
> >> For now, I would be comfortable describing this as beta Apache Ossie
> >> document
> >> hosting / semantic-model registry work.
> >>
> >> I would not yet be comfortable describing it as enabling AI, BI, or
> human
> >> semantic workflows until the discovery and client-consumption story
> >> exists.
> >>
> >> Robert
> >>
> >>
> >> On Fri, Jul 10, 2026 at 3:10 PM Dmitri Bourlatchkov <di...@apache.org>
> >> wrote:
> >>
> >> > Hi Yufei,
> >> >
> >> > Thanks for starting this thread!
> >> >
> >> > I tend to think it is best to represent Ossie data as direct JSON
> >> without
> >> > defining its structure in the Polaris OpenAPI spec (Ossie schemas are
> >> > controlled by Ossie, not Polaris). I believe this corresponds to
> Option
> >> 2
> >> > from your email.
> >> >
> >> > Polaris code that implements the new API will then interpret the Ossie
> >> > parts according to the declared version of the Ossie spec.
> >> >
> >> > With that in mind, the API should clearly state the format (Ossie or
> >> OKF)
> >> > and the specification version of the semantic data sub-object
> >> (apologies if
> >> > it has that already, I'm behind on the related PR updates). Obviously
> >> OKF
> >> > will have a different representation in the payload, but this should
> not
> >> > prevent Ossie from leveraging JSON synergies.
> >> >
> >> > Anand's work on solving a similar problem in the Metrics API [4115]
> may
> >> be
> >> > reusable here.
> >> >
> >> > [4115] https://github.com/apache/polaris/pull/4115
> >> >
> >> > Cheers,
> >> > Dmitri.
> >> >
> >> >
> >> > On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <fl...@gmail.com> wrote:
> >> >
> >> > > Hi folks,
> >> > >
> >> > > Following JB's suggestion, I'd like to start a dedicated discussion
> on
> >> > the
> >> > > REST API payload representation for semantic models.
> >> > >
> >> > > I think there are three possible approaches:
> >> > >
> >> > > 1.
> >> > >
> >> > > Represent the semantic model as a raw string.
> >> > > 2.
> >> > >
> >> > > Represent the semantic model as an opaque JSON document.
> >> > > 3.
> >> > >
> >> > > Model the semantic model structure directly in the REST
> >> specification.
> >> > >
> >> > > I think it's helpful to separate the REST API from Polaris' internal
> >> > > representation. The REST API is the long-term contract with clients,
> >> > while
> >> > > the internal representation can evolve independently.
> >> > >
> >> > > I'm comfortable with either option 1 or option 2. Both avoid
> coupling
> >> the
> >> > > REST API to the Ossie schema and allow Polaris to validate the
> payload
> >> > > based on the semantic model type and version while preserving the
> >> > document
> >> > > through write and read operations.
> >> > >
> >> > > My concern is with option 3. Since the Ossie schema is versioned and
> >> > > expected to evolve, modeling the full semantic model structure
> >> directly
> >> > in
> >> > > the REST specification would tightly couple the Polaris REST API to
> >> Ossie
> >> > > versions. Every Ossie schema evolution could require changes to the
> >> REST
> >> > > specification, generated clients, and potentially client
> applications.
> >> > >
> >> > > Between options 1 and 2, I think there is an additional tradeoff.
> >> > >
> >> > > An opaque JSON document assumes that semantic models are always
> >> > represented
> >> > > as JSON. While that works well for Ossie today, Polaris may support
> >> other
> >> > > semantic model formats in the future. For example, OKF[1] is defined
> >> as
> >> > > Markdown rather than JSON. Using a raw string keeps the REST API
> >> > > independent of any particular document format, allowing Polaris to
> >> > support
> >> > > JSON, Markdown, or other representations without changing the API
> >> > contract.
> >> > >
> >> > > So my current view is:
> >> > >
> >> > > -
> >> > >
> >> > > Option 1 provides the greatest flexibility and is format
> agnostic.
> >> > > -
> >> > >
> >> > > Option 2 is a natural choice if we want to optimize specifically
> >> for
> >> > > JSON based semantic models.
> >> > > -
> >> > >
> >> > > Option 3 provides strong typing, but at the cost of coupling the
> >> REST
> >> > > API to Ossie schema evolution.
> >> > >
> >> > > I'm happy with either option 1 or option 2, but I'd avoid option 3
> for
> >> > the
> >> > > reasons above.
> >> > >
> >> > > Thoughts?
> >> > >
> >> > > 1.
> >> > >
> >> > >
> >> >
> >>
> https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Yufei
> >> > >
> >> >
> >>
> >
>
Re: [DISCUSS] Semantic Model REST API payload representation
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Yufei,
Thanks for the pointer.
The existing spec version field in the Ossie JSON schema seems to be
sufficient for identifying the format. Polaris does not have to duplicate
it.
At this point, I do not see a strong reason for Polaris to use envelopes in
Ossie JSON payloads.
Cheers,
Dmitri.
On Fri, Aug 14, 2026 at 5:42 PM Yufei Gu <fl...@gmail.com> wrote:
> > I'm not up-to-date with Ossie though... Is it explicit about the spec
> version
> in its JSON data representation?
>
> Yes. Check the Ossie spec in the repo https://github.com/apache/ossie for
> more details. And Polaris side, both my design doc and and REST spec have
> covered that.
>
> > If not, then Polaris will have to indicate the version of the Ossie spec
> somehow.
> This will probably require an envelope in the response anyway.
>
> Could you clarify the envelope approach here? I explicitly asked about this
> in my last emails and provided examples to facilitate discussion. Would you
> mind taking a look and provide more details of what's in your mind?
>
> Yufei
>
>
> On Fri, Aug 14, 2026 at 2:12 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Yufei,
> >
> > I'm fine with JSON response payloads following the Apache Ossie JSON
> > structure in v1 API and deferring other payload types to future API
> > revisions.
> >
> > I'm not up-to-date with Ossie though... Is it explicit about the spec
> > version in its JSON data representation?
> >
> > If not, then Polaris will have to indicate the version of the Ossie spec
> > somehow. This will probably require an envelope in the response anyway.
> >
> > The alternative is to revision the Polaris API (v1, v2, v3, etc.) for
> every
> > Ossie spec change, which seems impractical to me.
> >
> > WDYT?
> >
> > Thanks,
> > Dmitri.
> >
> > On Tue, Jul 21, 2026 at 9:37 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Thanks for the discussion.
> > >
> > > Dmitri, could you clarify what the envelope would look like?
> > >
> > > My current understanding is that it might look roughly like this:
> > >
> > > {
> > > "format": "ossie",
> > > "version": "1.0",
> > > "payload": {
> > > "...": "..."
> > > }
> > > }
> > >
> > > For a non JSON format, the payload would instead need to be
> > > represented as an encoded string:
> > >
> > > {
> > > "format": "okf",
> > > "version": "1.0",
> > > "payload": "encoded document content"
> > > }
> > >
> > > If that is the intended design, I am not sure the envelope gives us
> > > much beyond option 1, as clients still need format specific handling.
> > >
> > > So the clearest choices seem to be either:
> > > 1. Direct JSON for Ossie, with no envelope, that seems to be what JB
> > > proposed. In that case, we'd have to change the spec or introduce a v2
> > > endpoint when we introduce a new format like OKF.
> > > 2. a format-agnostic envelope with an opaque string payload.
> > >
> > > Am I missing another benefit of the envelope approach?
> > >
> > > Thanks,
> > > Yufei
> > >
> > > Yufei
> > >
> > >
> > > On Thu, Jul 16, 2026 at 4:22 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I prefer to store and expose the semantic API using a direct JSON
> > > > payload format ("raw").
> > > >
> > > > I also suggest that we focus exclusively on the Apache Ossie format
> > > > for now, without any wrapping.
> > > >
> > > > Regarding the evolution of the Ossie specification, I believe it is
> > > > better to store raw JSON and handle the plumbing on our end. This
> > > > approach allows any tool, including AI tools, to directly consume the
> > > > Ossie definition.
> > > >
> > > > Regards,
> > > > JB
> > > >
> > > > On Mon, Jul 13, 2026 at 8:13 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > > >
> > > > > Hi Dmitri, Thanks! I think option 2 is a reasonable choice. The
> > > advantage I
> > > > > see with option 1 is future extensibility. If we later decide to
> > > support
> > > > > another semantic model format that isn't JSON, such as OKF, option
> 2
> > > would
> > > > > require a REST spec change, while option 1 would not. That's why I
> > > slightly
> > > > > prefer option 1, although I think either option is much better than
> > > > > modeling the semantic model structure directly in the REST API.
> What
> > > do you
> > > > > think?
> > > > >
> > > > > Thanks,
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Mon, Jul 13, 2026 at 11:10 AM Yufei Gu <fl...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Robert,
> > > > > >
> > > > > > Thanks for the thoughtful feedback.
> > > > > >
> > > > > > I agree that the current API is narrower than a complete semantic
> > > layer.
> > > > > > Today it's primarily about managing the lifecycle of semantic
> model
> > > > > > documents. Discovery, search, and client consumption are all
> > > important
> > > > > > topics, but I think they're broader than the Ossie semantic model
> > > itself.
> > > > > > We touched on some of these in the last community sync[1], and
> I'm
> > > happy to
> > > > > > continue that discussion in a separate thread.
> > > > > >
> > > > > > I also agree that we should be careful not to overstate what this
> > > initial
> > > > > > API provides. Calling it a beta semantic model registry or
> document
> > > hosting
> > > > > > API seems reasonable.
> > > > > >
> > > > > > That said, I think those questions are orthogonal to the payload
> > > > > > representation discussion. My original topic was simply whether
> the
> > > REST
> > > > > > API should treat the semantic model as a raw document, an opaque
> > JSON
> > > > > > payload, or model the schema directly. Since the consumption
> story
> > > is still
> > > > > > evolving, I think keeping the REST contract loosely coupled to
> the
> > > > > > underlying semantic model specification gives us the most
> > > flexibility.
> > > > > >
> > > > > > [1]
> > > > > >
> > >
> >
> https://drive.google.com/file/d/1hxYkk2t-BcnFOk8eJG9NYCHfjkOXg3Iz/view?usp=sharing
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > > On Mon, Jul 13, 2026 at 6:16 AM Robert Stupp <sn...@snazy.de>
> > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> I think the payload representation question depends on the
> client
> > > model we
> > > > > >> expect this API to support.
> > > > > >>
> > > > > >> I support Polaris hosting Apache Ossie semantic-model documents
> as
> > > a beta
> > > > > >> foundation. That seems like a useful first step while Ossie
> itself
> > > is
> > > > > >> still
> > > > > >> evolving.
> > > > > >>
> > > > > >> But I do not think we should describe the current API as
> enabling
> > AI
> > > > > >> tools,
> > > > > >> BI tools, or human semantic-model discovery yet.
> > > > > >>
> > > > > >> The merged API is primarily namespace/name CRUD for a
> > semantic-model
> > > > > >> document.
> > > > > >> That is opaque document storage and exact point-retrieval.
> > > > > >>
> > > > > >> It does not enable clients to discover the right semantic model
> > > from a
> > > > > >> query,
> > > > > >> nor to find semantic models by table, metric, domain, user, or
> > > capability.
> > > > > >>
> > > > > >> It also does not define a standard consuming API or tool
> contract,
> > > > > >> search/indexing contract, freshness model, or
> > > current/trusted/certified
> > > > > >> model
> > > > > >> semantics.
> > > > > >>
> > > > > >> That distinction matters because the REST API is the user-facing
> > > contract.
> > > > > >>
> > > > > >> If the beta API is intended only as opaque document hosting, I
> > > think the
> > > > > >> spec
> > > > > >> and docs should say that clearly, and users/clients should not
> > infer
> > > > > >> broader
> > > > > >> discovery or interoperability semantics from the CRUD API.
> > > > > >>
> > > > > >> I am not asking to solve the full semantic-layer story
> > immediately.
> > > > > >>
> > > > > >> I am asking that durable implementation work does not get ahead
> of
> > > the
> > > > > >> client-consumption story.
> > > > > >>
> > > > > >> The client model should drive the persistent data model, not the
> > > other way
> > > > > >> around.
> > > > > >>
> > > > > >> Once semantic models are stored as durable Polaris entities,
> > choices
> > > > > >> around
> > > > > >> identity, versioning, validation, indexing, size limits,
> > > source-table
> > > > > >> references, and freshness become much harder to change.
> > > > > >>
> > > > > >> For now, I would be comfortable describing this as beta Apache
> > Ossie
> > > > > >> document
> > > > > >> hosting / semantic-model registry work.
> > > > > >>
> > > > > >> I would not yet be comfortable describing it as enabling AI, BI,
> > or
> > > human
> > > > > >> semantic workflows until the discovery and client-consumption
> > story
> > > > > >> exists.
> > > > > >>
> > > > > >> Robert
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Jul 10, 2026 at 3:10 PM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Yufei,
> > > > > >> >
> > > > > >> > Thanks for starting this thread!
> > > > > >> >
> > > > > >> > I tend to think it is best to represent Ossie data as direct
> > JSON
> > > > > >> without
> > > > > >> > defining its structure in the Polaris OpenAPI spec (Ossie
> > schemas
> > > are
> > > > > >> > controlled by Ossie, not Polaris). I believe this corresponds
> to
> > > Option
> > > > > >> 2
> > > > > >> > from your email.
> > > > > >> >
> > > > > >> > Polaris code that implements the new API will then interpret
> the
> > > Ossie
> > > > > >> > parts according to the declared version of the Ossie spec.
> > > > > >> >
> > > > > >> > With that in mind, the API should clearly state the format
> > (Ossie
> > > or
> > > > > >> OKF)
> > > > > >> > and the specification version of the semantic data sub-object
> > > > > >> (apologies if
> > > > > >> > it has that already, I'm behind on the related PR updates).
> > > Obviously
> > > > > >> OKF
> > > > > >> > will have a different representation in the payload, but this
> > > should not
> > > > > >> > prevent Ossie from leveraging JSON synergies.
> > > > > >> >
> > > > > >> > Anand's work on solving a similar problem in the Metrics API
> > > [4115] may
> > > > > >> be
> > > > > >> > reusable here.
> > > > > >> >
> > > > > >> > [4115] https://github.com/apache/polaris/pull/4115
> > > > > >> >
> > > > > >> > Cheers,
> > > > > >> > Dmitri.
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <flyrain000@gmail.com
> >
> > > wrote:
> > > > > >> >
> > > > > >> > > Hi folks,
> > > > > >> > >
> > > > > >> > > Following JB's suggestion, I'd like to start a dedicated
> > > discussion on
> > > > > >> > the
> > > > > >> > > REST API payload representation for semantic models.
> > > > > >> > >
> > > > > >> > > I think there are three possible approaches:
> > > > > >> > >
> > > > > >> > > 1.
> > > > > >> > >
> > > > > >> > > Represent the semantic model as a raw string.
> > > > > >> > > 2.
> > > > > >> > >
> > > > > >> > > Represent the semantic model as an opaque JSON document.
> > > > > >> > > 3.
> > > > > >> > >
> > > > > >> > > Model the semantic model structure directly in the REST
> > > > > >> specification.
> > > > > >> > >
> > > > > >> > > I think it's helpful to separate the REST API from Polaris'
> > > internal
> > > > > >> > > representation. The REST API is the long-term contract with
> > > clients,
> > > > > >> > while
> > > > > >> > > the internal representation can evolve independently.
> > > > > >> > >
> > > > > >> > > I'm comfortable with either option 1 or option 2. Both avoid
> > > coupling
> > > > > >> the
> > > > > >> > > REST API to the Ossie schema and allow Polaris to validate
> the
> > > payload
> > > > > >> > > based on the semantic model type and version while
> preserving
> > > the
> > > > > >> > document
> > > > > >> > > through write and read operations.
> > > > > >> > >
> > > > > >> > > My concern is with option 3. Since the Ossie schema is
> > > versioned and
> > > > > >> > > expected to evolve, modeling the full semantic model
> structure
> > > > > >> directly
> > > > > >> > in
> > > > > >> > > the REST specification would tightly couple the Polaris REST
> > > API to
> > > > > >> Ossie
> > > > > >> > > versions. Every Ossie schema evolution could require changes
> > to
> > > the
> > > > > >> REST
> > > > > >> > > specification, generated clients, and potentially client
> > > applications.
> > > > > >> > >
> > > > > >> > > Between options 1 and 2, I think there is an additional
> > > tradeoff.
> > > > > >> > >
> > > > > >> > > An opaque JSON document assumes that semantic models are
> > always
> > > > > >> > represented
> > > > > >> > > as JSON. While that works well for Ossie today, Polaris may
> > > support
> > > > > >> other
> > > > > >> > > semantic model formats in the future. For example, OKF[1] is
> > > defined
> > > > > >> as
> > > > > >> > > Markdown rather than JSON. Using a raw string keeps the REST
> > API
> > > > > >> > > independent of any particular document format, allowing
> > Polaris
> > > to
> > > > > >> > support
> > > > > >> > > JSON, Markdown, or other representations without changing
> the
> > > API
> > > > > >> > contract.
> > > > > >> > >
> > > > > >> > > So my current view is:
> > > > > >> > >
> > > > > >> > > -
> > > > > >> > >
> > > > > >> > > Option 1 provides the greatest flexibility and is format
> > > agnostic.
> > > > > >> > > -
> > > > > >> > >
> > > > > >> > > Option 2 is a natural choice if we want to optimize
> > > specifically
> > > > > >> for
> > > > > >> > > JSON based semantic models.
> > > > > >> > > -
> > > > > >> > >
> > > > > >> > > Option 3 provides strong typing, but at the cost of
> > coupling
> > > the
> > > > > >> REST
> > > > > >> > > API to Ossie schema evolution.
> > > > > >> > >
> > > > > >> > > I'm happy with either option 1 or option 2, but I'd avoid
> > > option 3 for
> > > > > >> > the
> > > > > >> > > reasons above.
> > > > > >> > >
> > > > > >> > > Thoughts?
> > > > > >> > >
> > > > > >> > > 1.
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > >
> >
> https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > >
> > > > > >> > > Yufei
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > >
> >
>
Re: [DISCUSS] Semantic Model REST API payload representation
Posted by Robert Stupp <sn...@snazy.de>.
Hi,
I'd like to return to the client-consumption concern I raised earlier [1].
When I raised it, the initial REST scaffolding had already merged in PR
#4816 [2], while the durable `SEMANTIC_MODEL` entity and JDBC-backed CRUD
were still under review in PR #4961 [3].
I repeated the concern directly on #4961 before it merged [4].
Those endpoints are technically callable, but Polaris still does not
provide a usable path for any of the advertised AI-agent, BI-catalog, or
multi-engine workflows.
A bespoke client that already knows the catalog, namespace, model name,
payload format, and what to do with the returned document can store and
retrieve it.
That is storage plumbing for a low-level namespace/name document registry.
Polaris does not currently provide the discovery or consuming integration
needed to turn it into any of the end-to-end workflows named in the design
[5].
The response to my earlier concern was that discovery and consumption are
orthogonal and can evolve independently [6].
I do not think that resolves the concern.
The expected consumer tells us whether raw Ossie JSON, an envelope, or
another representation is the useful contract.
It also affects durable choices around identity, versioning, validation,
indexing, and discovery.
The API and persistence implementation have now merged while payload
representation is still under discussion and the consumption flow remains
undefined.
Before adding more persistent model or API surface, could we write down and
validate one concrete end-to-end flow?
It should identify who publishes a model, how an intended consumer
discovers the right one, which client or API it uses, how it interprets the
payload and version, and which responsibilities belong to Polaris, Ossie,
or the consuming tool.
I used "beta" for document hosting in my earlier reply, and the feature is
currently labeled that way [7].
On reflection, I do not think that "beta" label accurately describes the
current end-to-end maturity.
To me, "beta" means that an intended workflow exists and is usable for
evaluation, even if it is rough and may change.
Here, the registry-level CRUD exists, but the intended consumer workflow is
still undefined.
Until one exists, I think "experimental semantic-model document registry"
is the more accurate description.
There is also a concrete persistence consequence here.
An Ossie document has no reason to remain a reasonably small metadata value
for the generic properties bag.
The current implementation stores the entire serialized document as a
string inside the entity properties map, without a semantic-model-specific
size bound [8].
That makes whole-document reads and rewrites, cache pressure, and any
future projections or indexes part of the current persistence decision, not
an orthogonal follow-up.
Cheers,
Robert
[1] https://lists.apache.org/thread/wppf8yyl25zf8wcrtq8pxd11gmgq2bzb
[2] https://github.com/apache/polaris/pull/4816
[3] https://github.com/apache/polaris/pull/4961
[4] https://github.com/apache/polaris/pull/4961#issuecomment-5166815635
[5]
https://docs.google.com/document/d/1ZdI-1w_5LbyCMhvUhLCtOt-N1Z89L2P-oiGLaYayCZg/edit
[6] https://lists.apache.org/thread/lbxzhcz88j5zgs1pnt6qnko2qth2s8oo
[7] https://github.com/apache/polaris/pull/4983
[8]
https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/semantic/SemanticModelEntity.java#L32-L39
On Fri, Aug 14, 2026 at 11:41 PM Yufei Gu <fl...@gmail.com> wrote:
> > I'm not up-to-date with Ossie though... Is it explicit about the spec
> version
> in its JSON data representation?
>
> Yes. Check the Ossie spec in the repo https://github.com/apache/ossie for
> more details. And Polaris side, both my design doc and and REST spec have
> covered that.
>
> > If not, then Polaris will have to indicate the version of the Ossie spec
> somehow.
> This will probably require an envelope in the response anyway.
>
> Could you clarify the envelope approach here? I explicitly asked about this
> in my last emails and provided examples to facilitate discussion. Would you
> mind taking a look and provide more details of what's in your mind?
>
> Yufei
>
>
> On Fri, Aug 14, 2026 at 2:12 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Yufei,
> >
> > I'm fine with JSON response payloads following the Apache Ossie JSON
> > structure in v1 API and deferring other payload types to future API
> > revisions.
> >
> > I'm not up-to-date with Ossie though... Is it explicit about the spec
> > version in its JSON data representation?
> >
> > If not, then Polaris will have to indicate the version of the Ossie spec
> > somehow. This will probably require an envelope in the response anyway.
> >
> > The alternative is to revision the Polaris API (v1, v2, v3, etc.) for
> every
> > Ossie spec change, which seems impractical to me.
> >
> > WDYT?
> >
> > Thanks,
> > Dmitri.
> >
> > On Tue, Jul 21, 2026 at 9:37 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Thanks for the discussion.
> > >
> > > Dmitri, could you clarify what the envelope would look like?
> > >
> > > My current understanding is that it might look roughly like this:
> > >
> > > {
> > > "format": "ossie",
> > > "version": "1.0",
> > > "payload": {
> > > "...": "..."
> > > }
> > > }
> > >
> > > For a non JSON format, the payload would instead need to be
> > > represented as an encoded string:
> > >
> > > {
> > > "format": "okf",
> > > "version": "1.0",
> > > "payload": "encoded document content"
> > > }
> > >
> > > If that is the intended design, I am not sure the envelope gives us
> > > much beyond option 1, as clients still need format specific handling.
> > >
> > > So the clearest choices seem to be either:
> > > 1. Direct JSON for Ossie, with no envelope, that seems to be what JB
> > > proposed. In that case, we'd have to change the spec or introduce a v2
> > > endpoint when we introduce a new format like OKF.
> > > 2. a format-agnostic envelope with an opaque string payload.
> > >
> > > Am I missing another benefit of the envelope approach?
> > >
> > > Thanks,
> > > Yufei
> > >
> > > Yufei
> > >
> > >
> > > On Thu, Jul 16, 2026 at 4:22 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I prefer to store and expose the semantic API using a direct JSON
> > > > payload format ("raw").
> > > >
> > > > I also suggest that we focus exclusively on the Apache Ossie format
> > > > for now, without any wrapping.
> > > >
> > > > Regarding the evolution of the Ossie specification, I believe it is
> > > > better to store raw JSON and handle the plumbing on our end. This
> > > > approach allows any tool, including AI tools, to directly consume the
> > > > Ossie definition.
> > > >
> > > > Regards,
> > > > JB
> > > >
> > > > On Mon, Jul 13, 2026 at 8:13 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > > >
> > > > > Hi Dmitri, Thanks! I think option 2 is a reasonable choice. The
> > > advantage I
> > > > > see with option 1 is future extensibility. If we later decide to
> > > support
> > > > > another semantic model format that isn't JSON, such as OKF, option
> 2
> > > would
> > > > > require a REST spec change, while option 1 would not. That's why I
> > > slightly
> > > > > prefer option 1, although I think either option is much better than
> > > > > modeling the semantic model structure directly in the REST API.
> What
> > > do you
> > > > > think?
> > > > >
> > > > > Thanks,
> > > > > Yufei
> > > > >
> > > > >
> > > > > On Mon, Jul 13, 2026 at 11:10 AM Yufei Gu <fl...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Robert,
> > > > > >
> > > > > > Thanks for the thoughtful feedback.
> > > > > >
> > > > > > I agree that the current API is narrower than a complete semantic
> > > layer.
> > > > > > Today it's primarily about managing the lifecycle of semantic
> model
> > > > > > documents. Discovery, search, and client consumption are all
> > > important
> > > > > > topics, but I think they're broader than the Ossie semantic model
> > > itself.
> > > > > > We touched on some of these in the last community sync[1], and
> I'm
> > > happy to
> > > > > > continue that discussion in a separate thread.
> > > > > >
> > > > > > I also agree that we should be careful not to overstate what this
> > > initial
> > > > > > API provides. Calling it a beta semantic model registry or
> document
> > > hosting
> > > > > > API seems reasonable.
> > > > > >
> > > > > > That said, I think those questions are orthogonal to the payload
> > > > > > representation discussion. My original topic was simply whether
> the
> > > REST
> > > > > > API should treat the semantic model as a raw document, an opaque
> > JSON
> > > > > > payload, or model the schema directly. Since the consumption
> story
> > > is still
> > > > > > evolving, I think keeping the REST contract loosely coupled to
> the
> > > > > > underlying semantic model specification gives us the most
> > > flexibility.
> > > > > >
> > > > > > [1]
> > > > > >
> > >
> >
> https://drive.google.com/file/d/1hxYkk2t-BcnFOk8eJG9NYCHfjkOXg3Iz/view?usp=sharing
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > > On Mon, Jul 13, 2026 at 6:16 AM Robert Stupp <sn...@snazy.de>
> > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> I think the payload representation question depends on the
> client
> > > model we
> > > > > >> expect this API to support.
> > > > > >>
> > > > > >> I support Polaris hosting Apache Ossie semantic-model documents
> as
> > > a beta
> > > > > >> foundation. That seems like a useful first step while Ossie
> itself
> > > is
> > > > > >> still
> > > > > >> evolving.
> > > > > >>
> > > > > >> But I do not think we should describe the current API as
> enabling
> > AI
> > > > > >> tools,
> > > > > >> BI tools, or human semantic-model discovery yet.
> > > > > >>
> > > > > >> The merged API is primarily namespace/name CRUD for a
> > semantic-model
> > > > > >> document.
> > > > > >> That is opaque document storage and exact point-retrieval.
> > > > > >>
> > > > > >> It does not enable clients to discover the right semantic model
> > > from a
> > > > > >> query,
> > > > > >> nor to find semantic models by table, metric, domain, user, or
> > > capability.
> > > > > >>
> > > > > >> It also does not define a standard consuming API or tool
> contract,
> > > > > >> search/indexing contract, freshness model, or
> > > current/trusted/certified
> > > > > >> model
> > > > > >> semantics.
> > > > > >>
> > > > > >> That distinction matters because the REST API is the user-facing
> > > contract.
> > > > > >>
> > > > > >> If the beta API is intended only as opaque document hosting, I
> > > think the
> > > > > >> spec
> > > > > >> and docs should say that clearly, and users/clients should not
> > infer
> > > > > >> broader
> > > > > >> discovery or interoperability semantics from the CRUD API.
> > > > > >>
> > > > > >> I am not asking to solve the full semantic-layer story
> > immediately.
> > > > > >>
> > > > > >> I am asking that durable implementation work does not get ahead
> of
> > > the
> > > > > >> client-consumption story.
> > > > > >>
> > > > > >> The client model should drive the persistent data model, not the
> > > other way
> > > > > >> around.
> > > > > >>
> > > > > >> Once semantic models are stored as durable Polaris entities,
> > choices
> > > > > >> around
> > > > > >> identity, versioning, validation, indexing, size limits,
> > > source-table
> > > > > >> references, and freshness become much harder to change.
> > > > > >>
> > > > > >> For now, I would be comfortable describing this as beta Apache
> > Ossie
> > > > > >> document
> > > > > >> hosting / semantic-model registry work.
> > > > > >>
> > > > > >> I would not yet be comfortable describing it as enabling AI, BI,
> > or
> > > human
> > > > > >> semantic workflows until the discovery and client-consumption
> > story
> > > > > >> exists.
> > > > > >>
> > > > > >> Robert
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Jul 10, 2026 at 3:10 PM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Yufei,
> > > > > >> >
> > > > > >> > Thanks for starting this thread!
> > > > > >> >
> > > > > >> > I tend to think it is best to represent Ossie data as direct
> > JSON
> > > > > >> without
> > > > > >> > defining its structure in the Polaris OpenAPI spec (Ossie
> > schemas
> > > are
> > > > > >> > controlled by Ossie, not Polaris). I believe this corresponds
> to
> > > Option
> > > > > >> 2
> > > > > >> > from your email.
> > > > > >> >
> > > > > >> > Polaris code that implements the new API will then interpret
> the
> > > Ossie
> > > > > >> > parts according to the declared version of the Ossie spec.
> > > > > >> >
> > > > > >> > With that in mind, the API should clearly state the format
> > (Ossie
> > > or
> > > > > >> OKF)
> > > > > >> > and the specification version of the semantic data sub-object
> > > > > >> (apologies if
> > > > > >> > it has that already, I'm behind on the related PR updates).
> > > Obviously
> > > > > >> OKF
> > > > > >> > will have a different representation in the payload, but this
> > > should not
> > > > > >> > prevent Ossie from leveraging JSON synergies.
> > > > > >> >
> > > > > >> > Anand's work on solving a similar problem in the Metrics API
> > > [4115] may
> > > > > >> be
> > > > > >> > reusable here.
> > > > > >> >
> > > > > >> > [4115] https://github.com/apache/polaris/pull/4115
> > > > > >> >
> > > > > >> > Cheers,
> > > > > >> > Dmitri.
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <flyrain000@gmail.com
> >
> > > wrote:
> > > > > >> >
> > > > > >> > > Hi folks,
> > > > > >> > >
> > > > > >> > > Following JB's suggestion, I'd like to start a dedicated
> > > discussion on
> > > > > >> > the
> > > > > >> > > REST API payload representation for semantic models.
> > > > > >> > >
> > > > > >> > > I think there are three possible approaches:
> > > > > >> > >
> > > > > >> > > 1.
> > > > > >> > >
> > > > > >> > > Represent the semantic model as a raw string.
> > > > > >> > > 2.
> > > > > >> > >
> > > > > >> > > Represent the semantic model as an opaque JSON document.
> > > > > >> > > 3.
> > > > > >> > >
> > > > > >> > > Model the semantic model structure directly in the REST
> > > > > >> specification.
> > > > > >> > >
> > > > > >> > > I think it's helpful to separate the REST API from Polaris'
> > > internal
> > > > > >> > > representation. The REST API is the long-term contract with
> > > clients,
> > > > > >> > while
> > > > > >> > > the internal representation can evolve independently.
> > > > > >> > >
> > > > > >> > > I'm comfortable with either option 1 or option 2. Both avoid
> > > coupling
> > > > > >> the
> > > > > >> > > REST API to the Ossie schema and allow Polaris to validate
> the
> > > payload
> > > > > >> > > based on the semantic model type and version while
> preserving
> > > the
> > > > > >> > document
> > > > > >> > > through write and read operations.
> > > > > >> > >
> > > > > >> > > My concern is with option 3. Since the Ossie schema is
> > > versioned and
> > > > > >> > > expected to evolve, modeling the full semantic model
> structure
> > > > > >> directly
> > > > > >> > in
> > > > > >> > > the REST specification would tightly couple the Polaris REST
> > > API to
> > > > > >> Ossie
> > > > > >> > > versions. Every Ossie schema evolution could require changes
> > to
> > > the
> > > > > >> REST
> > > > > >> > > specification, generated clients, and potentially client
> > > applications.
> > > > > >> > >
> > > > > >> > > Between options 1 and 2, I think there is an additional
> > > tradeoff.
> > > > > >> > >
> > > > > >> > > An opaque JSON document assumes that semantic models are
> > always
> > > > > >> > represented
> > > > > >> > > as JSON. While that works well for Ossie today, Polaris may
> > > support
> > > > > >> other
> > > > > >> > > semantic model formats in the future. For example, OKF[1] is
> > > defined
> > > > > >> as
> > > > > >> > > Markdown rather than JSON. Using a raw string keeps the REST
> > API
> > > > > >> > > independent of any particular document format, allowing
> > Polaris
> > > to
> > > > > >> > support
> > > > > >> > > JSON, Markdown, or other representations without changing
> the
> > > API
> > > > > >> > contract.
> > > > > >> > >
> > > > > >> > > So my current view is:
> > > > > >> > >
> > > > > >> > > -
> > > > > >> > >
> > > > > >> > > Option 1 provides the greatest flexibility and is format
> > > agnostic.
> > > > > >> > > -
> > > > > >> > >
> > > > > >> > > Option 2 is a natural choice if we want to optimize
> > > specifically
> > > > > >> for
> > > > > >> > > JSON based semantic models.
> > > > > >> > > -
> > > > > >> > >
> > > > > >> > > Option 3 provides strong typing, but at the cost of
> > coupling
> > > the
> > > > > >> REST
> > > > > >> > > API to Ossie schema evolution.
> > > > > >> > >
> > > > > >> > > I'm happy with either option 1 or option 2, but I'd avoid
> > > option 3 for
> > > > > >> > the
> > > > > >> > > reasons above.
> > > > > >> > >
> > > > > >> > > Thoughts?
> > > > > >> > >
> > > > > >> > > 1.
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > >
> >
> https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > >
> > > > > >> > > Yufei
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > >
> >
>
Re: [DISCUSS] Semantic Model REST API payload representation
Posted by Yufei Gu <fl...@gmail.com>.
> I'm not up-to-date with Ossie though... Is it explicit about the spec version
in its JSON data representation?
Yes. Check the Ossie spec in the repo https://github.com/apache/ossie for
more details. And Polaris side, both my design doc and and REST spec have
covered that.
> If not, then Polaris will have to indicate the version of the Ossie spec somehow.
This will probably require an envelope in the response anyway.
Could you clarify the envelope approach here? I explicitly asked about this
in my last emails and provided examples to facilitate discussion. Would you
mind taking a look and provide more details of what's in your mind?
Yufei
On Fri, Aug 14, 2026 at 2:12 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi Yufei,
>
> I'm fine with JSON response payloads following the Apache Ossie JSON
> structure in v1 API and deferring other payload types to future API
> revisions.
>
> I'm not up-to-date with Ossie though... Is it explicit about the spec
> version in its JSON data representation?
>
> If not, then Polaris will have to indicate the version of the Ossie spec
> somehow. This will probably require an envelope in the response anyway.
>
> The alternative is to revision the Polaris API (v1, v2, v3, etc.) for every
> Ossie spec change, which seems impractical to me.
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> On Tue, Jul 21, 2026 at 9:37 PM Yufei Gu <fl...@gmail.com> wrote:
>
> > Thanks for the discussion.
> >
> > Dmitri, could you clarify what the envelope would look like?
> >
> > My current understanding is that it might look roughly like this:
> >
> > {
> > "format": "ossie",
> > "version": "1.0",
> > "payload": {
> > "...": "..."
> > }
> > }
> >
> > For a non JSON format, the payload would instead need to be
> > represented as an encoded string:
> >
> > {
> > "format": "okf",
> > "version": "1.0",
> > "payload": "encoded document content"
> > }
> >
> > If that is the intended design, I am not sure the envelope gives us
> > much beyond option 1, as clients still need format specific handling.
> >
> > So the clearest choices seem to be either:
> > 1. Direct JSON for Ossie, with no envelope, that seems to be what JB
> > proposed. In that case, we'd have to change the spec or introduce a v2
> > endpoint when we introduce a new format like OKF.
> > 2. a format-agnostic envelope with an opaque string payload.
> >
> > Am I missing another benefit of the envelope approach?
> >
> > Thanks,
> > Yufei
> >
> > Yufei
> >
> >
> > On Thu, Jul 16, 2026 at 4:22 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> > >
> > > Hi everyone,
> > >
> > > I prefer to store and expose the semantic API using a direct JSON
> > > payload format ("raw").
> > >
> > > I also suggest that we focus exclusively on the Apache Ossie format
> > > for now, without any wrapping.
> > >
> > > Regarding the evolution of the Ossie specification, I believe it is
> > > better to store raw JSON and handle the plumbing on our end. This
> > > approach allows any tool, including AI tools, to directly consume the
> > > Ossie definition.
> > >
> > > Regards,
> > > JB
> > >
> > > On Mon, Jul 13, 2026 at 8:13 PM Yufei Gu <fl...@gmail.com> wrote:
> > > >
> > > > Hi Dmitri, Thanks! I think option 2 is a reasonable choice. The
> > advantage I
> > > > see with option 1 is future extensibility. If we later decide to
> > support
> > > > another semantic model format that isn't JSON, such as OKF, option 2
> > would
> > > > require a REST spec change, while option 1 would not. That's why I
> > slightly
> > > > prefer option 1, although I think either option is much better than
> > > > modeling the semantic model structure directly in the REST API. What
> > do you
> > > > think?
> > > >
> > > > Thanks,
> > > > Yufei
> > > >
> > > >
> > > > On Mon, Jul 13, 2026 at 11:10 AM Yufei Gu <fl...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Robert,
> > > > >
> > > > > Thanks for the thoughtful feedback.
> > > > >
> > > > > I agree that the current API is narrower than a complete semantic
> > layer.
> > > > > Today it's primarily about managing the lifecycle of semantic model
> > > > > documents. Discovery, search, and client consumption are all
> > important
> > > > > topics, but I think they're broader than the Ossie semantic model
> > itself.
> > > > > We touched on some of these in the last community sync[1], and I'm
> > happy to
> > > > > continue that discussion in a separate thread.
> > > > >
> > > > > I also agree that we should be careful not to overstate what this
> > initial
> > > > > API provides. Calling it a beta semantic model registry or document
> > hosting
> > > > > API seems reasonable.
> > > > >
> > > > > That said, I think those questions are orthogonal to the payload
> > > > > representation discussion. My original topic was simply whether the
> > REST
> > > > > API should treat the semantic model as a raw document, an opaque
> JSON
> > > > > payload, or model the schema directly. Since the consumption story
> > is still
> > > > > evolving, I think keeping the REST contract loosely coupled to the
> > > > > underlying semantic model specification gives us the most
> > flexibility.
> > > > >
> > > > > [1]
> > > > >
> >
> https://drive.google.com/file/d/1hxYkk2t-BcnFOk8eJG9NYCHfjkOXg3Iz/view?usp=sharing
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Yufei
> > > > >
> > > > > On Mon, Jul 13, 2026 at 6:16 AM Robert Stupp <sn...@snazy.de>
> wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> I think the payload representation question depends on the client
> > model we
> > > > >> expect this API to support.
> > > > >>
> > > > >> I support Polaris hosting Apache Ossie semantic-model documents as
> > a beta
> > > > >> foundation. That seems like a useful first step while Ossie itself
> > is
> > > > >> still
> > > > >> evolving.
> > > > >>
> > > > >> But I do not think we should describe the current API as enabling
> AI
> > > > >> tools,
> > > > >> BI tools, or human semantic-model discovery yet.
> > > > >>
> > > > >> The merged API is primarily namespace/name CRUD for a
> semantic-model
> > > > >> document.
> > > > >> That is opaque document storage and exact point-retrieval.
> > > > >>
> > > > >> It does not enable clients to discover the right semantic model
> > from a
> > > > >> query,
> > > > >> nor to find semantic models by table, metric, domain, user, or
> > capability.
> > > > >>
> > > > >> It also does not define a standard consuming API or tool contract,
> > > > >> search/indexing contract, freshness model, or
> > current/trusted/certified
> > > > >> model
> > > > >> semantics.
> > > > >>
> > > > >> That distinction matters because the REST API is the user-facing
> > contract.
> > > > >>
> > > > >> If the beta API is intended only as opaque document hosting, I
> > think the
> > > > >> spec
> > > > >> and docs should say that clearly, and users/clients should not
> infer
> > > > >> broader
> > > > >> discovery or interoperability semantics from the CRUD API.
> > > > >>
> > > > >> I am not asking to solve the full semantic-layer story
> immediately.
> > > > >>
> > > > >> I am asking that durable implementation work does not get ahead of
> > the
> > > > >> client-consumption story.
> > > > >>
> > > > >> The client model should drive the persistent data model, not the
> > other way
> > > > >> around.
> > > > >>
> > > > >> Once semantic models are stored as durable Polaris entities,
> choices
> > > > >> around
> > > > >> identity, versioning, validation, indexing, size limits,
> > source-table
> > > > >> references, and freshness become much harder to change.
> > > > >>
> > > > >> For now, I would be comfortable describing this as beta Apache
> Ossie
> > > > >> document
> > > > >> hosting / semantic-model registry work.
> > > > >>
> > > > >> I would not yet be comfortable describing it as enabling AI, BI,
> or
> > human
> > > > >> semantic workflows until the discovery and client-consumption
> story
> > > > >> exists.
> > > > >>
> > > > >> Robert
> > > > >>
> > > > >>
> > > > >> On Fri, Jul 10, 2026 at 3:10 PM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Yufei,
> > > > >> >
> > > > >> > Thanks for starting this thread!
> > > > >> >
> > > > >> > I tend to think it is best to represent Ossie data as direct
> JSON
> > > > >> without
> > > > >> > defining its structure in the Polaris OpenAPI spec (Ossie
> schemas
> > are
> > > > >> > controlled by Ossie, not Polaris). I believe this corresponds to
> > Option
> > > > >> 2
> > > > >> > from your email.
> > > > >> >
> > > > >> > Polaris code that implements the new API will then interpret the
> > Ossie
> > > > >> > parts according to the declared version of the Ossie spec.
> > > > >> >
> > > > >> > With that in mind, the API should clearly state the format
> (Ossie
> > or
> > > > >> OKF)
> > > > >> > and the specification version of the semantic data sub-object
> > > > >> (apologies if
> > > > >> > it has that already, I'm behind on the related PR updates).
> > Obviously
> > > > >> OKF
> > > > >> > will have a different representation in the payload, but this
> > should not
> > > > >> > prevent Ossie from leveraging JSON synergies.
> > > > >> >
> > > > >> > Anand's work on solving a similar problem in the Metrics API
> > [4115] may
> > > > >> be
> > > > >> > reusable here.
> > > > >> >
> > > > >> > [4115] https://github.com/apache/polaris/pull/4115
> > > > >> >
> > > > >> > Cheers,
> > > > >> > Dmitri.
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <fl...@gmail.com>
> > wrote:
> > > > >> >
> > > > >> > > Hi folks,
> > > > >> > >
> > > > >> > > Following JB's suggestion, I'd like to start a dedicated
> > discussion on
> > > > >> > the
> > > > >> > > REST API payload representation for semantic models.
> > > > >> > >
> > > > >> > > I think there are three possible approaches:
> > > > >> > >
> > > > >> > > 1.
> > > > >> > >
> > > > >> > > Represent the semantic model as a raw string.
> > > > >> > > 2.
> > > > >> > >
> > > > >> > > Represent the semantic model as an opaque JSON document.
> > > > >> > > 3.
> > > > >> > >
> > > > >> > > Model the semantic model structure directly in the REST
> > > > >> specification.
> > > > >> > >
> > > > >> > > I think it's helpful to separate the REST API from Polaris'
> > internal
> > > > >> > > representation. The REST API is the long-term contract with
> > clients,
> > > > >> > while
> > > > >> > > the internal representation can evolve independently.
> > > > >> > >
> > > > >> > > I'm comfortable with either option 1 or option 2. Both avoid
> > coupling
> > > > >> the
> > > > >> > > REST API to the Ossie schema and allow Polaris to validate the
> > payload
> > > > >> > > based on the semantic model type and version while preserving
> > the
> > > > >> > document
> > > > >> > > through write and read operations.
> > > > >> > >
> > > > >> > > My concern is with option 3. Since the Ossie schema is
> > versioned and
> > > > >> > > expected to evolve, modeling the full semantic model structure
> > > > >> directly
> > > > >> > in
> > > > >> > > the REST specification would tightly couple the Polaris REST
> > API to
> > > > >> Ossie
> > > > >> > > versions. Every Ossie schema evolution could require changes
> to
> > the
> > > > >> REST
> > > > >> > > specification, generated clients, and potentially client
> > applications.
> > > > >> > >
> > > > >> > > Between options 1 and 2, I think there is an additional
> > tradeoff.
> > > > >> > >
> > > > >> > > An opaque JSON document assumes that semantic models are
> always
> > > > >> > represented
> > > > >> > > as JSON. While that works well for Ossie today, Polaris may
> > support
> > > > >> other
> > > > >> > > semantic model formats in the future. For example, OKF[1] is
> > defined
> > > > >> as
> > > > >> > > Markdown rather than JSON. Using a raw string keeps the REST
> API
> > > > >> > > independent of any particular document format, allowing
> Polaris
> > to
> > > > >> > support
> > > > >> > > JSON, Markdown, or other representations without changing the
> > API
> > > > >> > contract.
> > > > >> > >
> > > > >> > > So my current view is:
> > > > >> > >
> > > > >> > > -
> > > > >> > >
> > > > >> > > Option 1 provides the greatest flexibility and is format
> > agnostic.
> > > > >> > > -
> > > > >> > >
> > > > >> > > Option 2 is a natural choice if we want to optimize
> > specifically
> > > > >> for
> > > > >> > > JSON based semantic models.
> > > > >> > > -
> > > > >> > >
> > > > >> > > Option 3 provides strong typing, but at the cost of
> coupling
> > the
> > > > >> REST
> > > > >> > > API to Ossie schema evolution.
> > > > >> > >
> > > > >> > > I'm happy with either option 1 or option 2, but I'd avoid
> > option 3 for
> > > > >> > the
> > > > >> > > reasons above.
> > > > >> > >
> > > > >> > > Thoughts?
> > > > >> > >
> > > > >> > > 1.
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> >
> https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > >
> > > > >> > > Yufei
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> >
>
Re: [DISCUSS] Semantic Model REST API payload representation
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Yufei,
I'm fine with JSON response payloads following the Apache Ossie JSON
structure in v1 API and deferring other payload types to future API
revisions.
I'm not up-to-date with Ossie though... Is it explicit about the spec
version in its JSON data representation?
If not, then Polaris will have to indicate the version of the Ossie spec
somehow. This will probably require an envelope in the response anyway.
The alternative is to revision the Polaris API (v1, v2, v3, etc.) for every
Ossie spec change, which seems impractical to me.
WDYT?
Thanks,
Dmitri.
On Tue, Jul 21, 2026 at 9:37 PM Yufei Gu <fl...@gmail.com> wrote:
> Thanks for the discussion.
>
> Dmitri, could you clarify what the envelope would look like?
>
> My current understanding is that it might look roughly like this:
>
> {
> "format": "ossie",
> "version": "1.0",
> "payload": {
> "...": "..."
> }
> }
>
> For a non JSON format, the payload would instead need to be
> represented as an encoded string:
>
> {
> "format": "okf",
> "version": "1.0",
> "payload": "encoded document content"
> }
>
> If that is the intended design, I am not sure the envelope gives us
> much beyond option 1, as clients still need format specific handling.
>
> So the clearest choices seem to be either:
> 1. Direct JSON for Ossie, with no envelope, that seems to be what JB
> proposed. In that case, we'd have to change the spec or introduce a v2
> endpoint when we introduce a new format like OKF.
> 2. a format-agnostic envelope with an opaque string payload.
>
> Am I missing another benefit of the envelope approach?
>
> Thanks,
> Yufei
>
> Yufei
>
>
> On Thu, Jul 16, 2026 at 4:22 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >
> > Hi everyone,
> >
> > I prefer to store and expose the semantic API using a direct JSON
> > payload format ("raw").
> >
> > I also suggest that we focus exclusively on the Apache Ossie format
> > for now, without any wrapping.
> >
> > Regarding the evolution of the Ossie specification, I believe it is
> > better to store raw JSON and handle the plumbing on our end. This
> > approach allows any tool, including AI tools, to directly consume the
> > Ossie definition.
> >
> > Regards,
> > JB
> >
> > On Mon, Jul 13, 2026 at 8:13 PM Yufei Gu <fl...@gmail.com> wrote:
> > >
> > > Hi Dmitri, Thanks! I think option 2 is a reasonable choice. The
> advantage I
> > > see with option 1 is future extensibility. If we later decide to
> support
> > > another semantic model format that isn't JSON, such as OKF, option 2
> would
> > > require a REST spec change, while option 1 would not. That's why I
> slightly
> > > prefer option 1, although I think either option is much better than
> > > modeling the semantic model structure directly in the REST API. What
> do you
> > > think?
> > >
> > > Thanks,
> > > Yufei
> > >
> > >
> > > On Mon, Jul 13, 2026 at 11:10 AM Yufei Gu <fl...@gmail.com>
> wrote:
> > >
> > > > Hi Robert,
> > > >
> > > > Thanks for the thoughtful feedback.
> > > >
> > > > I agree that the current API is narrower than a complete semantic
> layer.
> > > > Today it's primarily about managing the lifecycle of semantic model
> > > > documents. Discovery, search, and client consumption are all
> important
> > > > topics, but I think they're broader than the Ossie semantic model
> itself.
> > > > We touched on some of these in the last community sync[1], and I'm
> happy to
> > > > continue that discussion in a separate thread.
> > > >
> > > > I also agree that we should be careful not to overstate what this
> initial
> > > > API provides. Calling it a beta semantic model registry or document
> hosting
> > > > API seems reasonable.
> > > >
> > > > That said, I think those questions are orthogonal to the payload
> > > > representation discussion. My original topic was simply whether the
> REST
> > > > API should treat the semantic model as a raw document, an opaque JSON
> > > > payload, or model the schema directly. Since the consumption story
> is still
> > > > evolving, I think keeping the REST contract loosely coupled to the
> > > > underlying semantic model specification gives us the most
> flexibility.
> > > >
> > > > [1]
> > > >
> https://drive.google.com/file/d/1hxYkk2t-BcnFOk8eJG9NYCHfjkOXg3Iz/view?usp=sharing
> > > >
> > > > Thanks,
> > > >
> > > > Yufei
> > > >
> > > > On Mon, Jul 13, 2026 at 6:16 AM Robert Stupp <sn...@snazy.de> wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I think the payload representation question depends on the client
> model we
> > > >> expect this API to support.
> > > >>
> > > >> I support Polaris hosting Apache Ossie semantic-model documents as
> a beta
> > > >> foundation. That seems like a useful first step while Ossie itself
> is
> > > >> still
> > > >> evolving.
> > > >>
> > > >> But I do not think we should describe the current API as enabling AI
> > > >> tools,
> > > >> BI tools, or human semantic-model discovery yet.
> > > >>
> > > >> The merged API is primarily namespace/name CRUD for a semantic-model
> > > >> document.
> > > >> That is opaque document storage and exact point-retrieval.
> > > >>
> > > >> It does not enable clients to discover the right semantic model
> from a
> > > >> query,
> > > >> nor to find semantic models by table, metric, domain, user, or
> capability.
> > > >>
> > > >> It also does not define a standard consuming API or tool contract,
> > > >> search/indexing contract, freshness model, or
> current/trusted/certified
> > > >> model
> > > >> semantics.
> > > >>
> > > >> That distinction matters because the REST API is the user-facing
> contract.
> > > >>
> > > >> If the beta API is intended only as opaque document hosting, I
> think the
> > > >> spec
> > > >> and docs should say that clearly, and users/clients should not infer
> > > >> broader
> > > >> discovery or interoperability semantics from the CRUD API.
> > > >>
> > > >> I am not asking to solve the full semantic-layer story immediately.
> > > >>
> > > >> I am asking that durable implementation work does not get ahead of
> the
> > > >> client-consumption story.
> > > >>
> > > >> The client model should drive the persistent data model, not the
> other way
> > > >> around.
> > > >>
> > > >> Once semantic models are stored as durable Polaris entities, choices
> > > >> around
> > > >> identity, versioning, validation, indexing, size limits,
> source-table
> > > >> references, and freshness become much harder to change.
> > > >>
> > > >> For now, I would be comfortable describing this as beta Apache Ossie
> > > >> document
> > > >> hosting / semantic-model registry work.
> > > >>
> > > >> I would not yet be comfortable describing it as enabling AI, BI, or
> human
> > > >> semantic workflows until the discovery and client-consumption story
> > > >> exists.
> > > >>
> > > >> Robert
> > > >>
> > > >>
> > > >> On Fri, Jul 10, 2026 at 3:10 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > >> wrote:
> > > >>
> > > >> > Hi Yufei,
> > > >> >
> > > >> > Thanks for starting this thread!
> > > >> >
> > > >> > I tend to think it is best to represent Ossie data as direct JSON
> > > >> without
> > > >> > defining its structure in the Polaris OpenAPI spec (Ossie schemas
> are
> > > >> > controlled by Ossie, not Polaris). I believe this corresponds to
> Option
> > > >> 2
> > > >> > from your email.
> > > >> >
> > > >> > Polaris code that implements the new API will then interpret the
> Ossie
> > > >> > parts according to the declared version of the Ossie spec.
> > > >> >
> > > >> > With that in mind, the API should clearly state the format (Ossie
> or
> > > >> OKF)
> > > >> > and the specification version of the semantic data sub-object
> > > >> (apologies if
> > > >> > it has that already, I'm behind on the related PR updates).
> Obviously
> > > >> OKF
> > > >> > will have a different representation in the payload, but this
> should not
> > > >> > prevent Ossie from leveraging JSON synergies.
> > > >> >
> > > >> > Anand's work on solving a similar problem in the Metrics API
> [4115] may
> > > >> be
> > > >> > reusable here.
> > > >> >
> > > >> > [4115] https://github.com/apache/polaris/pull/4115
> > > >> >
> > > >> > Cheers,
> > > >> > Dmitri.
> > > >> >
> > > >> >
> > > >> > On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > > >> >
> > > >> > > Hi folks,
> > > >> > >
> > > >> > > Following JB's suggestion, I'd like to start a dedicated
> discussion on
> > > >> > the
> > > >> > > REST API payload representation for semantic models.
> > > >> > >
> > > >> > > I think there are three possible approaches:
> > > >> > >
> > > >> > > 1.
> > > >> > >
> > > >> > > Represent the semantic model as a raw string.
> > > >> > > 2.
> > > >> > >
> > > >> > > Represent the semantic model as an opaque JSON document.
> > > >> > > 3.
> > > >> > >
> > > >> > > Model the semantic model structure directly in the REST
> > > >> specification.
> > > >> > >
> > > >> > > I think it's helpful to separate the REST API from Polaris'
> internal
> > > >> > > representation. The REST API is the long-term contract with
> clients,
> > > >> > while
> > > >> > > the internal representation can evolve independently.
> > > >> > >
> > > >> > > I'm comfortable with either option 1 or option 2. Both avoid
> coupling
> > > >> the
> > > >> > > REST API to the Ossie schema and allow Polaris to validate the
> payload
> > > >> > > based on the semantic model type and version while preserving
> the
> > > >> > document
> > > >> > > through write and read operations.
> > > >> > >
> > > >> > > My concern is with option 3. Since the Ossie schema is
> versioned and
> > > >> > > expected to evolve, modeling the full semantic model structure
> > > >> directly
> > > >> > in
> > > >> > > the REST specification would tightly couple the Polaris REST
> API to
> > > >> Ossie
> > > >> > > versions. Every Ossie schema evolution could require changes to
> the
> > > >> REST
> > > >> > > specification, generated clients, and potentially client
> applications.
> > > >> > >
> > > >> > > Between options 1 and 2, I think there is an additional
> tradeoff.
> > > >> > >
> > > >> > > An opaque JSON document assumes that semantic models are always
> > > >> > represented
> > > >> > > as JSON. While that works well for Ossie today, Polaris may
> support
> > > >> other
> > > >> > > semantic model formats in the future. For example, OKF[1] is
> defined
> > > >> as
> > > >> > > Markdown rather than JSON. Using a raw string keeps the REST API
> > > >> > > independent of any particular document format, allowing Polaris
> to
> > > >> > support
> > > >> > > JSON, Markdown, or other representations without changing the
> API
> > > >> > contract.
> > > >> > >
> > > >> > > So my current view is:
> > > >> > >
> > > >> > > -
> > > >> > >
> > > >> > > Option 1 provides the greatest flexibility and is format
> agnostic.
> > > >> > > -
> > > >> > >
> > > >> > > Option 2 is a natural choice if we want to optimize
> specifically
> > > >> for
> > > >> > > JSON based semantic models.
> > > >> > > -
> > > >> > >
> > > >> > > Option 3 provides strong typing, but at the cost of coupling
> the
> > > >> REST
> > > >> > > API to Ossie schema evolution.
> > > >> > >
> > > >> > > I'm happy with either option 1 or option 2, but I'd avoid
> option 3 for
> > > >> > the
> > > >> > > reasons above.
> > > >> > >
> > > >> > > Thoughts?
> > > >> > >
> > > >> > > 1.
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing
> > > >> > >
> > > >> > > Thanks,
> > > >> > >
> > > >> > > Yufei
> > > >> > >
> > > >> >
> > > >>
> > > >
>
Re: [DISCUSS] Semantic Model REST API payload representation
Posted by Yufei Gu <fl...@gmail.com>.
Thanks for the discussion.
Dmitri, could you clarify what the envelope would look like?
My current understanding is that it might look roughly like this:
{
"format": "ossie",
"version": "1.0",
"payload": {
"...": "..."
}
}
For a non JSON format, the payload would instead need to be
represented as an encoded string:
{
"format": "okf",
"version": "1.0",
"payload": "encoded document content"
}
If that is the intended design, I am not sure the envelope gives us
much beyond option 1, as clients still need format specific handling.
So the clearest choices seem to be either:
1. Direct JSON for Ossie, with no envelope, that seems to be what JB
proposed. In that case, we'd have to change the spec or introduce a v2
endpoint when we introduce a new format like OKF.
2. a format-agnostic envelope with an opaque string payload.
Am I missing another benefit of the envelope approach?
Thanks,
Yufei
Yufei
On Thu, Jul 16, 2026 at 4:22 AM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi everyone,
>
> I prefer to store and expose the semantic API using a direct JSON
> payload format ("raw").
>
> I also suggest that we focus exclusively on the Apache Ossie format
> for now, without any wrapping.
>
> Regarding the evolution of the Ossie specification, I believe it is
> better to store raw JSON and handle the plumbing on our end. This
> approach allows any tool, including AI tools, to directly consume the
> Ossie definition.
>
> Regards,
> JB
>
> On Mon, Jul 13, 2026 at 8:13 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > Hi Dmitri, Thanks! I think option 2 is a reasonable choice. The advantage I
> > see with option 1 is future extensibility. If we later decide to support
> > another semantic model format that isn't JSON, such as OKF, option 2 would
> > require a REST spec change, while option 1 would not. That's why I slightly
> > prefer option 1, although I think either option is much better than
> > modeling the semantic model structure directly in the REST API. What do you
> > think?
> >
> > Thanks,
> > Yufei
> >
> >
> > On Mon, Jul 13, 2026 at 11:10 AM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Hi Robert,
> > >
> > > Thanks for the thoughtful feedback.
> > >
> > > I agree that the current API is narrower than a complete semantic layer.
> > > Today it's primarily about managing the lifecycle of semantic model
> > > documents. Discovery, search, and client consumption are all important
> > > topics, but I think they're broader than the Ossie semantic model itself.
> > > We touched on some of these in the last community sync[1], and I'm happy to
> > > continue that discussion in a separate thread.
> > >
> > > I also agree that we should be careful not to overstate what this initial
> > > API provides. Calling it a beta semantic model registry or document hosting
> > > API seems reasonable.
> > >
> > > That said, I think those questions are orthogonal to the payload
> > > representation discussion. My original topic was simply whether the REST
> > > API should treat the semantic model as a raw document, an opaque JSON
> > > payload, or model the schema directly. Since the consumption story is still
> > > evolving, I think keeping the REST contract loosely coupled to the
> > > underlying semantic model specification gives us the most flexibility.
> > >
> > > [1]
> > > https://drive.google.com/file/d/1hxYkk2t-BcnFOk8eJG9NYCHfjkOXg3Iz/view?usp=sharing
> > >
> > > Thanks,
> > >
> > > Yufei
> > >
> > > On Mon, Jul 13, 2026 at 6:16 AM Robert Stupp <sn...@snazy.de> wrote:
> > >
> > >> Hi,
> > >>
> > >> I think the payload representation question depends on the client model we
> > >> expect this API to support.
> > >>
> > >> I support Polaris hosting Apache Ossie semantic-model documents as a beta
> > >> foundation. That seems like a useful first step while Ossie itself is
> > >> still
> > >> evolving.
> > >>
> > >> But I do not think we should describe the current API as enabling AI
> > >> tools,
> > >> BI tools, or human semantic-model discovery yet.
> > >>
> > >> The merged API is primarily namespace/name CRUD for a semantic-model
> > >> document.
> > >> That is opaque document storage and exact point-retrieval.
> > >>
> > >> It does not enable clients to discover the right semantic model from a
> > >> query,
> > >> nor to find semantic models by table, metric, domain, user, or capability.
> > >>
> > >> It also does not define a standard consuming API or tool contract,
> > >> search/indexing contract, freshness model, or current/trusted/certified
> > >> model
> > >> semantics.
> > >>
> > >> That distinction matters because the REST API is the user-facing contract.
> > >>
> > >> If the beta API is intended only as opaque document hosting, I think the
> > >> spec
> > >> and docs should say that clearly, and users/clients should not infer
> > >> broader
> > >> discovery or interoperability semantics from the CRUD API.
> > >>
> > >> I am not asking to solve the full semantic-layer story immediately.
> > >>
> > >> I am asking that durable implementation work does not get ahead of the
> > >> client-consumption story.
> > >>
> > >> The client model should drive the persistent data model, not the other way
> > >> around.
> > >>
> > >> Once semantic models are stored as durable Polaris entities, choices
> > >> around
> > >> identity, versioning, validation, indexing, size limits, source-table
> > >> references, and freshness become much harder to change.
> > >>
> > >> For now, I would be comfortable describing this as beta Apache Ossie
> > >> document
> > >> hosting / semantic-model registry work.
> > >>
> > >> I would not yet be comfortable describing it as enabling AI, BI, or human
> > >> semantic workflows until the discovery and client-consumption story
> > >> exists.
> > >>
> > >> Robert
> > >>
> > >>
> > >> On Fri, Jul 10, 2026 at 3:10 PM Dmitri Bourlatchkov <di...@apache.org>
> > >> wrote:
> > >>
> > >> > Hi Yufei,
> > >> >
> > >> > Thanks for starting this thread!
> > >> >
> > >> > I tend to think it is best to represent Ossie data as direct JSON
> > >> without
> > >> > defining its structure in the Polaris OpenAPI spec (Ossie schemas are
> > >> > controlled by Ossie, not Polaris). I believe this corresponds to Option
> > >> 2
> > >> > from your email.
> > >> >
> > >> > Polaris code that implements the new API will then interpret the Ossie
> > >> > parts according to the declared version of the Ossie spec.
> > >> >
> > >> > With that in mind, the API should clearly state the format (Ossie or
> > >> OKF)
> > >> > and the specification version of the semantic data sub-object
> > >> (apologies if
> > >> > it has that already, I'm behind on the related PR updates). Obviously
> > >> OKF
> > >> > will have a different representation in the payload, but this should not
> > >> > prevent Ossie from leveraging JSON synergies.
> > >> >
> > >> > Anand's work on solving a similar problem in the Metrics API [4115] may
> > >> be
> > >> > reusable here.
> > >> >
> > >> > [4115] https://github.com/apache/polaris/pull/4115
> > >> >
> > >> > Cheers,
> > >> > Dmitri.
> > >> >
> > >> >
> > >> > On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <fl...@gmail.com> wrote:
> > >> >
> > >> > > Hi folks,
> > >> > >
> > >> > > Following JB's suggestion, I'd like to start a dedicated discussion on
> > >> > the
> > >> > > REST API payload representation for semantic models.
> > >> > >
> > >> > > I think there are three possible approaches:
> > >> > >
> > >> > > 1.
> > >> > >
> > >> > > Represent the semantic model as a raw string.
> > >> > > 2.
> > >> > >
> > >> > > Represent the semantic model as an opaque JSON document.
> > >> > > 3.
> > >> > >
> > >> > > Model the semantic model structure directly in the REST
> > >> specification.
> > >> > >
> > >> > > I think it's helpful to separate the REST API from Polaris' internal
> > >> > > representation. The REST API is the long-term contract with clients,
> > >> > while
> > >> > > the internal representation can evolve independently.
> > >> > >
> > >> > > I'm comfortable with either option 1 or option 2. Both avoid coupling
> > >> the
> > >> > > REST API to the Ossie schema and allow Polaris to validate the payload
> > >> > > based on the semantic model type and version while preserving the
> > >> > document
> > >> > > through write and read operations.
> > >> > >
> > >> > > My concern is with option 3. Since the Ossie schema is versioned and
> > >> > > expected to evolve, modeling the full semantic model structure
> > >> directly
> > >> > in
> > >> > > the REST specification would tightly couple the Polaris REST API to
> > >> Ossie
> > >> > > versions. Every Ossie schema evolution could require changes to the
> > >> REST
> > >> > > specification, generated clients, and potentially client applications.
> > >> > >
> > >> > > Between options 1 and 2, I think there is an additional tradeoff.
> > >> > >
> > >> > > An opaque JSON document assumes that semantic models are always
> > >> > represented
> > >> > > as JSON. While that works well for Ossie today, Polaris may support
> > >> other
> > >> > > semantic model formats in the future. For example, OKF[1] is defined
> > >> as
> > >> > > Markdown rather than JSON. Using a raw string keeps the REST API
> > >> > > independent of any particular document format, allowing Polaris to
> > >> > support
> > >> > > JSON, Markdown, or other representations without changing the API
> > >> > contract.
> > >> > >
> > >> > > So my current view is:
> > >> > >
> > >> > > -
> > >> > >
> > >> > > Option 1 provides the greatest flexibility and is format agnostic.
> > >> > > -
> > >> > >
> > >> > > Option 2 is a natural choice if we want to optimize specifically
> > >> for
> > >> > > JSON based semantic models.
> > >> > > -
> > >> > >
> > >> > > Option 3 provides strong typing, but at the cost of coupling the
> > >> REST
> > >> > > API to Ossie schema evolution.
> > >> > >
> > >> > > I'm happy with either option 1 or option 2, but I'd avoid option 3 for
> > >> > the
> > >> > > reasons above.
> > >> > >
> > >> > > Thoughts?
> > >> > >
> > >> > > 1.
> > >> > >
> > >> > >
> > >> >
> > >> https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing
> > >> > >
> > >> > > Thanks,
> > >> > >
> > >> > > Yufei
> > >> > >
> > >> >
> > >>
> > >
Re: [DISCUSS] Semantic Model REST API payload representation
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi everyone,
I prefer to store and expose the semantic API using a direct JSON
payload format ("raw").
I also suggest that we focus exclusively on the Apache Ossie format
for now, without any wrapping.
Regarding the evolution of the Ossie specification, I believe it is
better to store raw JSON and handle the plumbing on our end. This
approach allows any tool, including AI tools, to directly consume the
Ossie definition.
Regards,
JB
On Mon, Jul 13, 2026 at 8:13 PM Yufei Gu <fl...@gmail.com> wrote:
>
> Hi Dmitri, Thanks! I think option 2 is a reasonable choice. The advantage I
> see with option 1 is future extensibility. If we later decide to support
> another semantic model format that isn't JSON, such as OKF, option 2 would
> require a REST spec change, while option 1 would not. That's why I slightly
> prefer option 1, although I think either option is much better than
> modeling the semantic model structure directly in the REST API. What do you
> think?
>
> Thanks,
> Yufei
>
>
> On Mon, Jul 13, 2026 at 11:10 AM Yufei Gu <fl...@gmail.com> wrote:
>
> > Hi Robert,
> >
> > Thanks for the thoughtful feedback.
> >
> > I agree that the current API is narrower than a complete semantic layer.
> > Today it's primarily about managing the lifecycle of semantic model
> > documents. Discovery, search, and client consumption are all important
> > topics, but I think they're broader than the Ossie semantic model itself.
> > We touched on some of these in the last community sync[1], and I'm happy to
> > continue that discussion in a separate thread.
> >
> > I also agree that we should be careful not to overstate what this initial
> > API provides. Calling it a beta semantic model registry or document hosting
> > API seems reasonable.
> >
> > That said, I think those questions are orthogonal to the payload
> > representation discussion. My original topic was simply whether the REST
> > API should treat the semantic model as a raw document, an opaque JSON
> > payload, or model the schema directly. Since the consumption story is still
> > evolving, I think keeping the REST contract loosely coupled to the
> > underlying semantic model specification gives us the most flexibility.
> >
> > [1]
> > https://drive.google.com/file/d/1hxYkk2t-BcnFOk8eJG9NYCHfjkOXg3Iz/view?usp=sharing
> >
> > Thanks,
> >
> > Yufei
> >
> > On Mon, Jul 13, 2026 at 6:16 AM Robert Stupp <sn...@snazy.de> wrote:
> >
> >> Hi,
> >>
> >> I think the payload representation question depends on the client model we
> >> expect this API to support.
> >>
> >> I support Polaris hosting Apache Ossie semantic-model documents as a beta
> >> foundation. That seems like a useful first step while Ossie itself is
> >> still
> >> evolving.
> >>
> >> But I do not think we should describe the current API as enabling AI
> >> tools,
> >> BI tools, or human semantic-model discovery yet.
> >>
> >> The merged API is primarily namespace/name CRUD for a semantic-model
> >> document.
> >> That is opaque document storage and exact point-retrieval.
> >>
> >> It does not enable clients to discover the right semantic model from a
> >> query,
> >> nor to find semantic models by table, metric, domain, user, or capability.
> >>
> >> It also does not define a standard consuming API or tool contract,
> >> search/indexing contract, freshness model, or current/trusted/certified
> >> model
> >> semantics.
> >>
> >> That distinction matters because the REST API is the user-facing contract.
> >>
> >> If the beta API is intended only as opaque document hosting, I think the
> >> spec
> >> and docs should say that clearly, and users/clients should not infer
> >> broader
> >> discovery or interoperability semantics from the CRUD API.
> >>
> >> I am not asking to solve the full semantic-layer story immediately.
> >>
> >> I am asking that durable implementation work does not get ahead of the
> >> client-consumption story.
> >>
> >> The client model should drive the persistent data model, not the other way
> >> around.
> >>
> >> Once semantic models are stored as durable Polaris entities, choices
> >> around
> >> identity, versioning, validation, indexing, size limits, source-table
> >> references, and freshness become much harder to change.
> >>
> >> For now, I would be comfortable describing this as beta Apache Ossie
> >> document
> >> hosting / semantic-model registry work.
> >>
> >> I would not yet be comfortable describing it as enabling AI, BI, or human
> >> semantic workflows until the discovery and client-consumption story
> >> exists.
> >>
> >> Robert
> >>
> >>
> >> On Fri, Jul 10, 2026 at 3:10 PM Dmitri Bourlatchkov <di...@apache.org>
> >> wrote:
> >>
> >> > Hi Yufei,
> >> >
> >> > Thanks for starting this thread!
> >> >
> >> > I tend to think it is best to represent Ossie data as direct JSON
> >> without
> >> > defining its structure in the Polaris OpenAPI spec (Ossie schemas are
> >> > controlled by Ossie, not Polaris). I believe this corresponds to Option
> >> 2
> >> > from your email.
> >> >
> >> > Polaris code that implements the new API will then interpret the Ossie
> >> > parts according to the declared version of the Ossie spec.
> >> >
> >> > With that in mind, the API should clearly state the format (Ossie or
> >> OKF)
> >> > and the specification version of the semantic data sub-object
> >> (apologies if
> >> > it has that already, I'm behind on the related PR updates). Obviously
> >> OKF
> >> > will have a different representation in the payload, but this should not
> >> > prevent Ossie from leveraging JSON synergies.
> >> >
> >> > Anand's work on solving a similar problem in the Metrics API [4115] may
> >> be
> >> > reusable here.
> >> >
> >> > [4115] https://github.com/apache/polaris/pull/4115
> >> >
> >> > Cheers,
> >> > Dmitri.
> >> >
> >> >
> >> > On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <fl...@gmail.com> wrote:
> >> >
> >> > > Hi folks,
> >> > >
> >> > > Following JB's suggestion, I'd like to start a dedicated discussion on
> >> > the
> >> > > REST API payload representation for semantic models.
> >> > >
> >> > > I think there are three possible approaches:
> >> > >
> >> > > 1.
> >> > >
> >> > > Represent the semantic model as a raw string.
> >> > > 2.
> >> > >
> >> > > Represent the semantic model as an opaque JSON document.
> >> > > 3.
> >> > >
> >> > > Model the semantic model structure directly in the REST
> >> specification.
> >> > >
> >> > > I think it's helpful to separate the REST API from Polaris' internal
> >> > > representation. The REST API is the long-term contract with clients,
> >> > while
> >> > > the internal representation can evolve independently.
> >> > >
> >> > > I'm comfortable with either option 1 or option 2. Both avoid coupling
> >> the
> >> > > REST API to the Ossie schema and allow Polaris to validate the payload
> >> > > based on the semantic model type and version while preserving the
> >> > document
> >> > > through write and read operations.
> >> > >
> >> > > My concern is with option 3. Since the Ossie schema is versioned and
> >> > > expected to evolve, modeling the full semantic model structure
> >> directly
> >> > in
> >> > > the REST specification would tightly couple the Polaris REST API to
> >> Ossie
> >> > > versions. Every Ossie schema evolution could require changes to the
> >> REST
> >> > > specification, generated clients, and potentially client applications.
> >> > >
> >> > > Between options 1 and 2, I think there is an additional tradeoff.
> >> > >
> >> > > An opaque JSON document assumes that semantic models are always
> >> > represented
> >> > > as JSON. While that works well for Ossie today, Polaris may support
> >> other
> >> > > semantic model formats in the future. For example, OKF[1] is defined
> >> as
> >> > > Markdown rather than JSON. Using a raw string keeps the REST API
> >> > > independent of any particular document format, allowing Polaris to
> >> > support
> >> > > JSON, Markdown, or other representations without changing the API
> >> > contract.
> >> > >
> >> > > So my current view is:
> >> > >
> >> > > -
> >> > >
> >> > > Option 1 provides the greatest flexibility and is format agnostic.
> >> > > -
> >> > >
> >> > > Option 2 is a natural choice if we want to optimize specifically
> >> for
> >> > > JSON based semantic models.
> >> > > -
> >> > >
> >> > > Option 3 provides strong typing, but at the cost of coupling the
> >> REST
> >> > > API to Ossie schema evolution.
> >> > >
> >> > > I'm happy with either option 1 or option 2, but I'd avoid option 3 for
> >> > the
> >> > > reasons above.
> >> > >
> >> > > Thoughts?
> >> > >
> >> > > 1.
> >> > >
> >> > >
> >> >
> >> https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Yufei
> >> > >
> >> >
> >>
> >
Re: [DISCUSS] Semantic Model REST API payload representation
Posted by Yufei Gu <fl...@gmail.com>.
Hi Dmitri, Thanks! I think option 2 is a reasonable choice. The advantage I
see with option 1 is future extensibility. If we later decide to support
another semantic model format that isn't JSON, such as OKF, option 2 would
require a REST spec change, while option 1 would not. That's why I slightly
prefer option 1, although I think either option is much better than
modeling the semantic model structure directly in the REST API. What do you
think?
Thanks,
Yufei
On Mon, Jul 13, 2026 at 11:10 AM Yufei Gu <fl...@gmail.com> wrote:
> Hi Robert,
>
> Thanks for the thoughtful feedback.
>
> I agree that the current API is narrower than a complete semantic layer.
> Today it's primarily about managing the lifecycle of semantic model
> documents. Discovery, search, and client consumption are all important
> topics, but I think they're broader than the Ossie semantic model itself.
> We touched on some of these in the last community sync[1], and I'm happy to
> continue that discussion in a separate thread.
>
> I also agree that we should be careful not to overstate what this initial
> API provides. Calling it a beta semantic model registry or document hosting
> API seems reasonable.
>
> That said, I think those questions are orthogonal to the payload
> representation discussion. My original topic was simply whether the REST
> API should treat the semantic model as a raw document, an opaque JSON
> payload, or model the schema directly. Since the consumption story is still
> evolving, I think keeping the REST contract loosely coupled to the
> underlying semantic model specification gives us the most flexibility.
>
> [1]
> https://drive.google.com/file/d/1hxYkk2t-BcnFOk8eJG9NYCHfjkOXg3Iz/view?usp=sharing
>
> Thanks,
>
> Yufei
>
> On Mon, Jul 13, 2026 at 6:16 AM Robert Stupp <sn...@snazy.de> wrote:
>
>> Hi,
>>
>> I think the payload representation question depends on the client model we
>> expect this API to support.
>>
>> I support Polaris hosting Apache Ossie semantic-model documents as a beta
>> foundation. That seems like a useful first step while Ossie itself is
>> still
>> evolving.
>>
>> But I do not think we should describe the current API as enabling AI
>> tools,
>> BI tools, or human semantic-model discovery yet.
>>
>> The merged API is primarily namespace/name CRUD for a semantic-model
>> document.
>> That is opaque document storage and exact point-retrieval.
>>
>> It does not enable clients to discover the right semantic model from a
>> query,
>> nor to find semantic models by table, metric, domain, user, or capability.
>>
>> It also does not define a standard consuming API or tool contract,
>> search/indexing contract, freshness model, or current/trusted/certified
>> model
>> semantics.
>>
>> That distinction matters because the REST API is the user-facing contract.
>>
>> If the beta API is intended only as opaque document hosting, I think the
>> spec
>> and docs should say that clearly, and users/clients should not infer
>> broader
>> discovery or interoperability semantics from the CRUD API.
>>
>> I am not asking to solve the full semantic-layer story immediately.
>>
>> I am asking that durable implementation work does not get ahead of the
>> client-consumption story.
>>
>> The client model should drive the persistent data model, not the other way
>> around.
>>
>> Once semantic models are stored as durable Polaris entities, choices
>> around
>> identity, versioning, validation, indexing, size limits, source-table
>> references, and freshness become much harder to change.
>>
>> For now, I would be comfortable describing this as beta Apache Ossie
>> document
>> hosting / semantic-model registry work.
>>
>> I would not yet be comfortable describing it as enabling AI, BI, or human
>> semantic workflows until the discovery and client-consumption story
>> exists.
>>
>> Robert
>>
>>
>> On Fri, Jul 10, 2026 at 3:10 PM Dmitri Bourlatchkov <di...@apache.org>
>> wrote:
>>
>> > Hi Yufei,
>> >
>> > Thanks for starting this thread!
>> >
>> > I tend to think it is best to represent Ossie data as direct JSON
>> without
>> > defining its structure in the Polaris OpenAPI spec (Ossie schemas are
>> > controlled by Ossie, not Polaris). I believe this corresponds to Option
>> 2
>> > from your email.
>> >
>> > Polaris code that implements the new API will then interpret the Ossie
>> > parts according to the declared version of the Ossie spec.
>> >
>> > With that in mind, the API should clearly state the format (Ossie or
>> OKF)
>> > and the specification version of the semantic data sub-object
>> (apologies if
>> > it has that already, I'm behind on the related PR updates). Obviously
>> OKF
>> > will have a different representation in the payload, but this should not
>> > prevent Ossie from leveraging JSON synergies.
>> >
>> > Anand's work on solving a similar problem in the Metrics API [4115] may
>> be
>> > reusable here.
>> >
>> > [4115] https://github.com/apache/polaris/pull/4115
>> >
>> > Cheers,
>> > Dmitri.
>> >
>> >
>> > On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <fl...@gmail.com> wrote:
>> >
>> > > Hi folks,
>> > >
>> > > Following JB's suggestion, I'd like to start a dedicated discussion on
>> > the
>> > > REST API payload representation for semantic models.
>> > >
>> > > I think there are three possible approaches:
>> > >
>> > > 1.
>> > >
>> > > Represent the semantic model as a raw string.
>> > > 2.
>> > >
>> > > Represent the semantic model as an opaque JSON document.
>> > > 3.
>> > >
>> > > Model the semantic model structure directly in the REST
>> specification.
>> > >
>> > > I think it's helpful to separate the REST API from Polaris' internal
>> > > representation. The REST API is the long-term contract with clients,
>> > while
>> > > the internal representation can evolve independently.
>> > >
>> > > I'm comfortable with either option 1 or option 2. Both avoid coupling
>> the
>> > > REST API to the Ossie schema and allow Polaris to validate the payload
>> > > based on the semantic model type and version while preserving the
>> > document
>> > > through write and read operations.
>> > >
>> > > My concern is with option 3. Since the Ossie schema is versioned and
>> > > expected to evolve, modeling the full semantic model structure
>> directly
>> > in
>> > > the REST specification would tightly couple the Polaris REST API to
>> Ossie
>> > > versions. Every Ossie schema evolution could require changes to the
>> REST
>> > > specification, generated clients, and potentially client applications.
>> > >
>> > > Between options 1 and 2, I think there is an additional tradeoff.
>> > >
>> > > An opaque JSON document assumes that semantic models are always
>> > represented
>> > > as JSON. While that works well for Ossie today, Polaris may support
>> other
>> > > semantic model formats in the future. For example, OKF[1] is defined
>> as
>> > > Markdown rather than JSON. Using a raw string keeps the REST API
>> > > independent of any particular document format, allowing Polaris to
>> > support
>> > > JSON, Markdown, or other representations without changing the API
>> > contract.
>> > >
>> > > So my current view is:
>> > >
>> > > -
>> > >
>> > > Option 1 provides the greatest flexibility and is format agnostic.
>> > > -
>> > >
>> > > Option 2 is a natural choice if we want to optimize specifically
>> for
>> > > JSON based semantic models.
>> > > -
>> > >
>> > > Option 3 provides strong typing, but at the cost of coupling the
>> REST
>> > > API to Ossie schema evolution.
>> > >
>> > > I'm happy with either option 1 or option 2, but I'd avoid option 3 for
>> > the
>> > > reasons above.
>> > >
>> > > Thoughts?
>> > >
>> > > 1.
>> > >
>> > >
>> >
>> https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing
>> > >
>> > > Thanks,
>> > >
>> > > Yufei
>> > >
>> >
>>
>
Re: [DISCUSS] Semantic Model REST API payload representation
Posted by Yufei Gu <fl...@gmail.com>.
Hi Robert,
Thanks for the thoughtful feedback.
I agree that the current API is narrower than a complete semantic layer.
Today it's primarily about managing the lifecycle of semantic model
documents. Discovery, search, and client consumption are all important
topics, but I think they're broader than the Ossie semantic model itself.
We touched on some of these in the last community sync[1], and I'm happy to
continue that discussion in a separate thread.
I also agree that we should be careful not to overstate what this initial
API provides. Calling it a beta semantic model registry or document hosting
API seems reasonable.
That said, I think those questions are orthogonal to the payload
representation discussion. My original topic was simply whether the REST
API should treat the semantic model as a raw document, an opaque JSON
payload, or model the schema directly. Since the consumption story is still
evolving, I think keeping the REST contract loosely coupled to the
underlying semantic model specification gives us the most flexibility.
[1]
https://drive.google.com/file/d/1hxYkk2t-BcnFOk8eJG9NYCHfjkOXg3Iz/view?usp=sharing
Thanks,
Yufei
On Mon, Jul 13, 2026 at 6:16 AM Robert Stupp <sn...@snazy.de> wrote:
> Hi,
>
> I think the payload representation question depends on the client model we
> expect this API to support.
>
> I support Polaris hosting Apache Ossie semantic-model documents as a beta
> foundation. That seems like a useful first step while Ossie itself is still
> evolving.
>
> But I do not think we should describe the current API as enabling AI tools,
> BI tools, or human semantic-model discovery yet.
>
> The merged API is primarily namespace/name CRUD for a semantic-model
> document.
> That is opaque document storage and exact point-retrieval.
>
> It does not enable clients to discover the right semantic model from a
> query,
> nor to find semantic models by table, metric, domain, user, or capability.
>
> It also does not define a standard consuming API or tool contract,
> search/indexing contract, freshness model, or current/trusted/certified
> model
> semantics.
>
> That distinction matters because the REST API is the user-facing contract.
>
> If the beta API is intended only as opaque document hosting, I think the
> spec
> and docs should say that clearly, and users/clients should not infer
> broader
> discovery or interoperability semantics from the CRUD API.
>
> I am not asking to solve the full semantic-layer story immediately.
>
> I am asking that durable implementation work does not get ahead of the
> client-consumption story.
>
> The client model should drive the persistent data model, not the other way
> around.
>
> Once semantic models are stored as durable Polaris entities, choices around
> identity, versioning, validation, indexing, size limits, source-table
> references, and freshness become much harder to change.
>
> For now, I would be comfortable describing this as beta Apache Ossie
> document
> hosting / semantic-model registry work.
>
> I would not yet be comfortable describing it as enabling AI, BI, or human
> semantic workflows until the discovery and client-consumption story exists.
>
> Robert
>
>
> On Fri, Jul 10, 2026 at 3:10 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Yufei,
> >
> > Thanks for starting this thread!
> >
> > I tend to think it is best to represent Ossie data as direct JSON without
> > defining its structure in the Polaris OpenAPI spec (Ossie schemas are
> > controlled by Ossie, not Polaris). I believe this corresponds to Option 2
> > from your email.
> >
> > Polaris code that implements the new API will then interpret the Ossie
> > parts according to the declared version of the Ossie spec.
> >
> > With that in mind, the API should clearly state the format (Ossie or OKF)
> > and the specification version of the semantic data sub-object (apologies
> if
> > it has that already, I'm behind on the related PR updates). Obviously OKF
> > will have a different representation in the payload, but this should not
> > prevent Ossie from leveraging JSON synergies.
> >
> > Anand's work on solving a similar problem in the Metrics API [4115] may
> be
> > reusable here.
> >
> > [4115] https://github.com/apache/polaris/pull/4115
> >
> > Cheers,
> > Dmitri.
> >
> >
> > On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <fl...@gmail.com> wrote:
> >
> > > Hi folks,
> > >
> > > Following JB's suggestion, I'd like to start a dedicated discussion on
> > the
> > > REST API payload representation for semantic models.
> > >
> > > I think there are three possible approaches:
> > >
> > > 1.
> > >
> > > Represent the semantic model as a raw string.
> > > 2.
> > >
> > > Represent the semantic model as an opaque JSON document.
> > > 3.
> > >
> > > Model the semantic model structure directly in the REST
> specification.
> > >
> > > I think it's helpful to separate the REST API from Polaris' internal
> > > representation. The REST API is the long-term contract with clients,
> > while
> > > the internal representation can evolve independently.
> > >
> > > I'm comfortable with either option 1 or option 2. Both avoid coupling
> the
> > > REST API to the Ossie schema and allow Polaris to validate the payload
> > > based on the semantic model type and version while preserving the
> > document
> > > through write and read operations.
> > >
> > > My concern is with option 3. Since the Ossie schema is versioned and
> > > expected to evolve, modeling the full semantic model structure directly
> > in
> > > the REST specification would tightly couple the Polaris REST API to
> Ossie
> > > versions. Every Ossie schema evolution could require changes to the
> REST
> > > specification, generated clients, and potentially client applications.
> > >
> > > Between options 1 and 2, I think there is an additional tradeoff.
> > >
> > > An opaque JSON document assumes that semantic models are always
> > represented
> > > as JSON. While that works well for Ossie today, Polaris may support
> other
> > > semantic model formats in the future. For example, OKF[1] is defined as
> > > Markdown rather than JSON. Using a raw string keeps the REST API
> > > independent of any particular document format, allowing Polaris to
> > support
> > > JSON, Markdown, or other representations without changing the API
> > contract.
> > >
> > > So my current view is:
> > >
> > > -
> > >
> > > Option 1 provides the greatest flexibility and is format agnostic.
> > > -
> > >
> > > Option 2 is a natural choice if we want to optimize specifically for
> > > JSON based semantic models.
> > > -
> > >
> > > Option 3 provides strong typing, but at the cost of coupling the
> REST
> > > API to Ossie schema evolution.
> > >
> > > I'm happy with either option 1 or option 2, but I'd avoid option 3 for
> > the
> > > reasons above.
> > >
> > > Thoughts?
> > >
> > > 1.
> > >
> > >
> >
> https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing
> > >
> > > Thanks,
> > >
> > > Yufei
> > >
> >
>
Re: [DISCUSS] Semantic Model REST API payload representation
Posted by Robert Stupp <sn...@snazy.de>.
Hi,
I think the payload representation question depends on the client model we
expect this API to support.
I support Polaris hosting Apache Ossie semantic-model documents as a beta
foundation. That seems like a useful first step while Ossie itself is still
evolving.
But I do not think we should describe the current API as enabling AI tools,
BI tools, or human semantic-model discovery yet.
The merged API is primarily namespace/name CRUD for a semantic-model
document.
That is opaque document storage and exact point-retrieval.
It does not enable clients to discover the right semantic model from a
query,
nor to find semantic models by table, metric, domain, user, or capability.
It also does not define a standard consuming API or tool contract,
search/indexing contract, freshness model, or current/trusted/certified
model
semantics.
That distinction matters because the REST API is the user-facing contract.
If the beta API is intended only as opaque document hosting, I think the
spec
and docs should say that clearly, and users/clients should not infer broader
discovery or interoperability semantics from the CRUD API.
I am not asking to solve the full semantic-layer story immediately.
I am asking that durable implementation work does not get ahead of the
client-consumption story.
The client model should drive the persistent data model, not the other way
around.
Once semantic models are stored as durable Polaris entities, choices around
identity, versioning, validation, indexing, size limits, source-table
references, and freshness become much harder to change.
For now, I would be comfortable describing this as beta Apache Ossie
document
hosting / semantic-model registry work.
I would not yet be comfortable describing it as enabling AI, BI, or human
semantic workflows until the discovery and client-consumption story exists.
Robert
On Fri, Jul 10, 2026 at 3:10 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi Yufei,
>
> Thanks for starting this thread!
>
> I tend to think it is best to represent Ossie data as direct JSON without
> defining its structure in the Polaris OpenAPI spec (Ossie schemas are
> controlled by Ossie, not Polaris). I believe this corresponds to Option 2
> from your email.
>
> Polaris code that implements the new API will then interpret the Ossie
> parts according to the declared version of the Ossie spec.
>
> With that in mind, the API should clearly state the format (Ossie or OKF)
> and the specification version of the semantic data sub-object (apologies if
> it has that already, I'm behind on the related PR updates). Obviously OKF
> will have a different representation in the payload, but this should not
> prevent Ossie from leveraging JSON synergies.
>
> Anand's work on solving a similar problem in the Metrics API [4115] may be
> reusable here.
>
> [4115] https://github.com/apache/polaris/pull/4115
>
> Cheers,
> Dmitri.
>
>
> On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <fl...@gmail.com> wrote:
>
> > Hi folks,
> >
> > Following JB's suggestion, I'd like to start a dedicated discussion on
> the
> > REST API payload representation for semantic models.
> >
> > I think there are three possible approaches:
> >
> > 1.
> >
> > Represent the semantic model as a raw string.
> > 2.
> >
> > Represent the semantic model as an opaque JSON document.
> > 3.
> >
> > Model the semantic model structure directly in the REST specification.
> >
> > I think it's helpful to separate the REST API from Polaris' internal
> > representation. The REST API is the long-term contract with clients,
> while
> > the internal representation can evolve independently.
> >
> > I'm comfortable with either option 1 or option 2. Both avoid coupling the
> > REST API to the Ossie schema and allow Polaris to validate the payload
> > based on the semantic model type and version while preserving the
> document
> > through write and read operations.
> >
> > My concern is with option 3. Since the Ossie schema is versioned and
> > expected to evolve, modeling the full semantic model structure directly
> in
> > the REST specification would tightly couple the Polaris REST API to Ossie
> > versions. Every Ossie schema evolution could require changes to the REST
> > specification, generated clients, and potentially client applications.
> >
> > Between options 1 and 2, I think there is an additional tradeoff.
> >
> > An opaque JSON document assumes that semantic models are always
> represented
> > as JSON. While that works well for Ossie today, Polaris may support other
> > semantic model formats in the future. For example, OKF[1] is defined as
> > Markdown rather than JSON. Using a raw string keeps the REST API
> > independent of any particular document format, allowing Polaris to
> support
> > JSON, Markdown, or other representations without changing the API
> contract.
> >
> > So my current view is:
> >
> > -
> >
> > Option 1 provides the greatest flexibility and is format agnostic.
> > -
> >
> > Option 2 is a natural choice if we want to optimize specifically for
> > JSON based semantic models.
> > -
> >
> > Option 3 provides strong typing, but at the cost of coupling the REST
> > API to Ossie schema evolution.
> >
> > I'm happy with either option 1 or option 2, but I'd avoid option 3 for
> the
> > reasons above.
> >
> > Thoughts?
> >
> > 1.
> >
> >
> https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing
> >
> > Thanks,
> >
> > Yufei
> >
>
Re: [DISCUSS] Semantic Model REST API payload representation
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Yufei,
Thanks for starting this thread!
I tend to think it is best to represent Ossie data as direct JSON without
defining its structure in the Polaris OpenAPI spec (Ossie schemas are
controlled by Ossie, not Polaris). I believe this corresponds to Option 2
from your email.
Polaris code that implements the new API will then interpret the Ossie
parts according to the declared version of the Ossie spec.
With that in mind, the API should clearly state the format (Ossie or OKF)
and the specification version of the semantic data sub-object (apologies if
it has that already, I'm behind on the related PR updates). Obviously OKF
will have a different representation in the payload, but this should not
prevent Ossie from leveraging JSON synergies.
Anand's work on solving a similar problem in the Metrics API [4115] may be
reusable here.
[4115] https://github.com/apache/polaris/pull/4115
Cheers,
Dmitri.
On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <fl...@gmail.com> wrote:
> Hi folks,
>
> Following JB's suggestion, I'd like to start a dedicated discussion on the
> REST API payload representation for semantic models.
>
> I think there are three possible approaches:
>
> 1.
>
> Represent the semantic model as a raw string.
> 2.
>
> Represent the semantic model as an opaque JSON document.
> 3.
>
> Model the semantic model structure directly in the REST specification.
>
> I think it's helpful to separate the REST API from Polaris' internal
> representation. The REST API is the long-term contract with clients, while
> the internal representation can evolve independently.
>
> I'm comfortable with either option 1 or option 2. Both avoid coupling the
> REST API to the Ossie schema and allow Polaris to validate the payload
> based on the semantic model type and version while preserving the document
> through write and read operations.
>
> My concern is with option 3. Since the Ossie schema is versioned and
> expected to evolve, modeling the full semantic model structure directly in
> the REST specification would tightly couple the Polaris REST API to Ossie
> versions. Every Ossie schema evolution could require changes to the REST
> specification, generated clients, and potentially client applications.
>
> Between options 1 and 2, I think there is an additional tradeoff.
>
> An opaque JSON document assumes that semantic models are always represented
> as JSON. While that works well for Ossie today, Polaris may support other
> semantic model formats in the future. For example, OKF[1] is defined as
> Markdown rather than JSON. Using a raw string keeps the REST API
> independent of any particular document format, allowing Polaris to support
> JSON, Markdown, or other representations without changing the API contract.
>
> So my current view is:
>
> -
>
> Option 1 provides the greatest flexibility and is format agnostic.
> -
>
> Option 2 is a natural choice if we want to optimize specifically for
> JSON based semantic models.
> -
>
> Option 3 provides strong typing, but at the cost of coupling the REST
> API to Ossie schema evolution.
>
> I'm happy with either option 1 or option 2, but I'd avoid option 3 for the
> reasons above.
>
> Thoughts?
>
> 1.
>
> https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing
>
> Thanks,
>
> Yufei
>