You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Brandon White <br...@freenome.com> on 2016/10/27 18:40:32 UTC
Encrypting Airflow Communications
From what I see, Airflow communicates with a couple sources:
1) SQL Store
2) Celery Broker
Does Airflow have any configurations which make it easy to encrypt all of
its communications or do we need to build custom solutions into Airflow?
--
This e-mail is private and confidential and is for the addressee only. If
misdirected, please notify us by telephone, confirming that it has been
deleted from your system and any hard copies destroyed. You are strictly
prohibited from using, printing, distributing or disseminating it or any
information contained in it save to the intended recipient.
Re: Encrypting Airflow Communications
Posted by Bolke de Bruin <bd...@gmail.com>.
You can setup ssl and encryption for all these connections. For celery you need a broker that supports encryption (i.e. Rabbitmq) and for the db both Postgres and MySQL support ssl connections.
I don't think a change in airflow is needed for this except configuration.
Bolke
Sent from my iPhone
> On 27 Oct 2016, at 20:40, Brandon White <br...@freenome.com> wrote:
>
> From what I see, Airflow communicates with a couple sources:
>
> 1) SQL Store
> 2) Celery Broker
>
> Does Airflow have any configurations which make it easy to encrypt all of
> its communications or do we need to build custom solutions into Airflow?
>
> --
> This e-mail is private and confidential and is for the addressee only. If
> misdirected, please notify us by telephone, confirming that it has been
> deleted from your system and any hard copies destroyed. You are strictly
> prohibited from using, printing, distributing or disseminating it or any
> information contained in it save to the intended recipient.
Re: Encrypting Airflow Communications
Posted by siddharth anand <sa...@apache.org>.
This might be relevant as well if you want to learn more about VPC :
https://www.youtube.com/watch?v=HexrVfuIY1k
On Thu, Oct 27, 2016 at 6:39 PM, siddharth anand <sa...@apache.org> wrote:
> Hmn.. it looks like traffic within AWS's VPC is not encrypted.. so using
> TLS between all services is a need.
>
> <http://goog_420699131>
> http://serverfault.com/questions/573115/traffic-in-a-
> aws-virtual-private-cloud
>
> This video discusses the design of an AWS VPC.
> https://www.youtube.com/watch?v=Zd5hsL-JNY4
>
> -s
>
> On Thu, Oct 27, 2016 at 6:20 PM, siddharth anand <sa...@apache.org>
> wrote:
>
>> I haven't looked into it but would welcome a PR if you were to propose
>> one. We use SQL Alchemy for our ORM, so you may want to look at that for 1)
>> above.
>>
>> We (Agari) run in AWS and run all our EC2-based services (e.g. Airflow
>> servers and the DB) within a VPC. I suspect the folks running in GCP have a
>> similar solution but I don't know enough about GCP.
>>
>> However, this won't help folks running Airflow in their own data centers
>> or in other shared environments.
>>
>>
>> -s
>>
>> On Thu, Oct 27, 2016 at 11:40 AM, Brandon White <
>> brandon.white@freenome.com> wrote:
>>
>>> From what I see, Airflow communicates with a couple sources:
>>>
>>> 1) SQL Store
>>> 2) Celery Broker
>>>
>>> Does Airflow have any configurations which make it easy to encrypt all of
>>> its communications or do we need to build custom solutions into Airflow?
>>>
>>> --
>>> This e-mail is private and confidential and is for the addressee only. If
>>> misdirected, please notify us by telephone, confirming that it has been
>>> deleted from your system and any hard copies destroyed. You are strictly
>>> prohibited from using, printing, distributing or disseminating it or any
>>> information contained in it save to the intended recipient.
>>>
>>
>>
>
Re: Encrypting Airflow Communications
Posted by siddharth anand <sa...@apache.org>.
Hmn.. it looks like traffic within AWS's VPC is not encrypted.. so using
TLS between all services is a need.
<goog_420699131>
http://serverfault.com/questions/573115/traffic-in-a-aws-virtual-private-cloud
This video discusses the design of an AWS VPC.
https://www.youtube.com/watch?v=Zd5hsL-JNY4
-s
On Thu, Oct 27, 2016 at 6:20 PM, siddharth anand <sa...@apache.org> wrote:
> I haven't looked into it but would welcome a PR if you were to propose
> one. We use SQL Alchemy for our ORM, so you may want to look at that for 1)
> above.
>
> We (Agari) run in AWS and run all our EC2-based services (e.g. Airflow
> servers and the DB) within a VPC. I suspect the folks running in GCP have a
> similar solution but I don't know enough about GCP.
>
> However, this won't help folks running Airflow in their own data centers
> or in other shared environments.
>
>
> -s
>
> On Thu, Oct 27, 2016 at 11:40 AM, Brandon White <
> brandon.white@freenome.com> wrote:
>
>> From what I see, Airflow communicates with a couple sources:
>>
>> 1) SQL Store
>> 2) Celery Broker
>>
>> Does Airflow have any configurations which make it easy to encrypt all of
>> its communications or do we need to build custom solutions into Airflow?
>>
>> --
>> This e-mail is private and confidential and is for the addressee only. If
>> misdirected, please notify us by telephone, confirming that it has been
>> deleted from your system and any hard copies destroyed. You are strictly
>> prohibited from using, printing, distributing or disseminating it or any
>> information contained in it save to the intended recipient.
>>
>
>
Re: Encrypting Airflow Communications
Posted by siddharth anand <sa...@apache.org>.
I haven't looked into it but would welcome a PR if you were to propose one.
We use SQL Alchemy for our ORM, so you may want to look at that for 1)
above.
We (Agari) run in AWS and run all our EC2-based services (e.g. Airflow
servers and the DB) within a VPC. I suspect the folks running in GCP have a
similar solution but I don't know enough about GCP.
However, this won't help folks running Airflow in their own data centers or
in other shared environments.
-s
On Thu, Oct 27, 2016 at 11:40 AM, Brandon White <br...@freenome.com>
wrote:
> From what I see, Airflow communicates with a couple sources:
>
> 1) SQL Store
> 2) Celery Broker
>
> Does Airflow have any configurations which make it easy to encrypt all of
> its communications or do we need to build custom solutions into Airflow?
>
> --
> This e-mail is private and confidential and is for the addressee only. If
> misdirected, please notify us by telephone, confirming that it has been
> deleted from your system and any hard copies destroyed. You are strictly
> prohibited from using, printing, distributing or disseminating it or any
> information contained in it save to the intended recipient.
>