Airflow installations on on-premise servers and Postgres in AWS

[Sunil Khaire <su...@gmail.com>]

Hi Team ,

Is it possible to use Amazon postgres RDS database as a backend for airflow
installation running on-premise ? If yes how ?

Thanks in advance.

Regards ,
Sunil Khaire

Re: Airflow installations on on-premise servers and Postgres in AWS

[Ash Berlin-Taylor <as...@apache.org>]

Sure it's possible if that's what you want. The performance of it will largely depend on your latency between your data centre and AWS.

To make it work you need to ensure that the RDS instance is accessible from your on-prem machines, either by putting the RDS in a public subnet, or ensuring the right kind of routing (Direct Connect?) is set up to allow access to RDS without opening it to IP traffic from the whole world. The exact details of this is not really something we can help with, it's very specific to your infrastructure and company IT policies.
Next you'd need to create a non-superuser user inside the RDS - Airflow needs to create tables (but shouldn't be creating new users). Look at the AWS docs for this.
Once you've done that, then you configure with something like sql_alchemy_conn = postgresql://USER:PASS@instance.abc123.us-west-1.rds.amazonaws.com/airflow
-ash
On Jun 18 2020, at 10:24 am, Sunil Khaire <su...@gmail.com> wrote:
> Hi Team ,
>
> Is it possible to use Amazon postgres RDS database as a backend for airflow installation running on-premise ? If yes how ?
>
> Thanks in advance.
>
> Regards ,
> Sunil Khaire
>