topics metadata, OWL ontologies and AVRO

[Nicolae Marasoiu <ni...@ovoenergy.com>]

Hello,

We're trying an "indexing" of the data sources, such as Kafka topics
mostly, so that various teams and services could identify potential
existing sources of data for specific needs.

We're trying to label topics with producing teams, generic entities names
(currently topic names don't always reflect entities in a clear
convention), and other such meta information, in order to be able to search
for a specific data type and get a result of some topics that may contain
it.

On the other hand, of course many teams express almost the same concepts
and domain model objects in slightly different ways.

I am thinking how could we define or formalise the similarities and
differences to relate those local domains (bounded contexts) and to allow
search along similarities like "this is a Customer" and "used in business
flow x", generated by team y, in context z.

An idea that comes is semantic web and ontologies. An OWL ontology could
express those similarities and yet allow for decentralised domains with
separate ontologies or just one with separate components in the graph for
each team I do not know.

At the lower level of singular concepts and basic relationships, this would
map to AVRO schemas. I realise this may entail some duplication unless we
find a way to generate them from OWL.

Do you have any ideas, experience, hints how to proceed with a cross team
project of common approach to modelling in a formal way the common
properties the otherwise different objects in different domains have?

How would it be related to AVRO, if generating AVRO schemas from OWL would
work?

If you think it is realistic, did you hear about it or does it make sense
to you that that would enable cross organisational share and reuse of
domain and enhanced discoverability of resources, data and services along
some similar properties like entity types, flows, etc?

Please ask me, I think it is still potentially fuzzy,
Thank you very much,
Nicolae Marasoiu
-- 
Thank you,
Nicolae Marasoiu
Scala Engineer
Orion, OVO Group