Info abou microservices architecture

[Piergiorgio Lucidi <pi...@apache.org>]

Hi Airflowers,
we are designing a new platform totally based on a microservices
architecture.

How could we manage an ingestion process with AirFlow in a microservices
architecture?

Could you please share best practices to achieve this?

Thank you for your support.
Cheers,
PJ

Re: Info abou microservices architecture

[Piergiorgio Lucidi <pi...@apache.org>]

One of the first goals is to manage data ingestion from different data
sources to write the outcome in Apache Hive.
Consider having different reader and writer components as microservices.

The main question is about how to integrate Airflow with microservices such
as readers and writers?

We are wondering how to interact with microservices from Airflow tasks, can
we use a queue management service such as Kafka or RabbitMQ? The
communication should be bidirectional, imagine to start a job
(Airflow->microservices) and the last involved microservice at the end
should send a notification to Airflow (microservice->Airflow)

Considering Mx a specific microservice, we should have something like the
following chaining invocation calls exchanging events: Airflow -> M1 -> M2
-> M3 -> final Airflow notification event. Does this make sense for  an
architecture with Airflow?

This specific section of the chaining M1 -> M2 -> M3 is implemented by
events without any involvement of Airflow.
Thank you for any advice :)
Cheers,
PJ

Il giorno mar 23 nov 2021 alle ore 11:50 Piergiorgio Lucidi <
piergiorgio@apache.org> ha scritto:

> Hi Airflowers,
> we are designing a new platform totally based on a microservices
> architecture.
>
> How could we manage an ingestion process with AirFlow in a microservices
> architecture?
>
> Could you please share best practices to achieve this?
>
> Thank you for your support.
> Cheers,
> PJ
>


-- 
Piergiorgio