You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by ra...@gmail.com, ra...@gmail.com on 2018/12/10 15:10:45 UTC

Recommended backend metastore for Airflow

Hi All,

It seems that Airflow supports mysql, postgresql and mssql as backend store. Any recommendation on using one over other. We are expecting to run 1000(s) of concurrent Dags which would generate heavy load on backend store.
Any pointer on this would be useful.

Thanks,
Raman Gupta

Re: Recommended backend metastore for Airflow

Posted by airflowuser <ai...@protonmail.com.INVALID>.

Definitely PostgreSQL.

https://www.2ndquadrant.com/en/postgresql/postgresql-vs-mysql/



‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, December 10, 2018 5:10 PM, ramandumcs@gmail.com <ra...@gmail.com> wrote:

> Hi All,
>
> It seems that Airflow supports mysql, postgresql and mssql as backend store. Any recommendation on using one over other. We are expecting to run 1000(s) of concurrent Dags which would generate heavy load on backend store.
> Any pointer on this would be useful.
>
> Thanks,
> Raman Gupta

Re: Recommended backend metastore for Airflow

Posted by ra...@gmail.com, ra...@gmail.com.

Thanks Ash,
We are trying to run 1000 concurrent Dags and are facing scalability issues with mysql. So we are exploring other backend stores pgsql and mssql.
Any recommendation on airflow config like heartbeat interval, pool size etc..  to support this much workload

Thanks,
Raman Gupta

On 2018/12/10 15:12:41, Ash Berlin-Taylor <as...@apache.org> wrote: 
> Postgres.
> 
> Friends don't let friends use MySQL is my personal rule.
> 
> (I can get in to the reasons if you'd like, but the short version is I find Postgres has more compliant behaviour with SQL standard, and a much better query planner.)
> 
> -ash
> 
> > On 10 Dec 2018, at 15:10, ramandumcs@gmail.com wrote:
> > 
> > Hi All,
> > 
> > It seems that Airflow supports mysql, postgresql and mssql as backend store. Any recommendation on using one over other. We are expecting to run 1000(s) of concurrent Dags which would generate heavy load on backend store.
> > Any pointer on this would be useful.
> > 
> > Thanks,
> > Raman Gupta
> 
>

Re: Recommended backend metastore for Airflow

Posted by Ash Berlin-Taylor <as...@apache.org>.

Postgres.

Friends don't let friends use MySQL is my personal rule.

(I can get in to the reasons if you'd like, but the short version is I find Postgres has more compliant behaviour with SQL standard, and a much better query planner.)

-ash

> On 10 Dec 2018, at 15:10, ramandumcs@gmail.com wrote:
> 
> Hi All,
> 
> It seems that Airflow supports mysql, postgresql and mssql as backend store. Any recommendation on using one over other. We are expecting to run 1000(s) of concurrent Dags which would generate heavy load on backend store.
> Any pointer on this would be useful.
> 
> Thanks,
> Raman Gupta