linking different topics AVRO schemas with OWL/RDFS ontologies

[Nicolae Marasoiu <ni...@ovoenergy.com>]

Hello kafka community,

We use Apache Kafka and microservices publishing and subscribing to the
data exposed by other microservices. We use AVRO. Different microservices
publish one or more topics like events topics or facts/entities/current
state topics. And consume from others.

Currently the data is a bit heterogenous and a lot of guesswork happens in
terms of "is there any microservice publishing the data i need?", "where to
find a data source for customer like entities?", "what data does team x
publish?".

We are exploring semantic web & ontologies for:

1. linking different topics AVRO schemas with OWL/RDFS ontologies

2. marking metadata on topics to be searched for

3. giving more semantics to data and allow more ways to navigate it for
users across the enterprise

It seems to me that the problem starts perhaps with proper modelling. Since
AVRO is already a schema (for JSON documents), I would like to explore ways
to go into the semantic layer of relating various concepts exposed by these
microservices so that later on people and microservices can query this set
of relations and find data sources that they would not normally be aware
of, thus increasing productivity.

Do you know of any project that integrated Kafka or AVRO with OWL
Ontologies?

Or more generally in the land of microservices do you know of pragmatic and
nice application of ontologies to help offer visibility on data and various
parties can search for "customer like entities" for instance, across the
organisation?

Do you see other technologies for achieving our goals of increasing
transparency of domains and their interconnections?

Thank you very much, for more context:

https://www.quora.com/unanswered/How-can-AVRO-schemas-used-in-Kafka-topics-be-reused-to-generate-OWL-ontologies-and-how-can-OWL-ontologies-be-used-to-generate-AVRO-schemas-What-are-other-ways-to-inter-op

https://www.quora.com/unanswered/How-would-you-mix-OWL-ontologies-with-AVRO-schemas-needed-in-Apache-Kafka-to-describe-the-same-domain-without-much-duplication

https://www.quora.com/unanswered/What-are-the-ways-to-define-a-domain-model-that-can-be-freely-shared-across-organisations-and-be-laterally-developed-in-a-decentralised-fashion-allow-for-different-ways-of-modelling-the-Customer-Is-OWL-ontologies-a

https://www.quora.com/What-kind-of-metadata-would-you-use-to-associate-data-sources-like-Kafka-topics-or-db-tables-that-they-represent-Entity-X-and-they-are-owned-by-team-Y-Would-OWL-Ontologies-be-ok-to-model-entities-across


Thank you very much,
Nicu
-- 
Thank you,
Nicolae Marasoiu
Scala Engineer
Orion, OVO Group