You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by jeeyoung kim <je...@gmail.com> on 2018/02/26 19:22:47 UTC

Recommended way of piping credentials / sensitive information on templated operators

Hi everyone,

I’m wondering how people work around accidentally writing credentials on
bash operator template page / logs.

For example, I may have PostgreSQL operator to copy data into Redshift.

COPY TABLE_NAME from 's3://.../something.manifest.json'
access_key_id '{{ params.AWS_ACCESS_KEY }}'
secret_access_key '{{ params.AWS_SECRET_KEY }}'

Or a command that exports from mongo

mongoexport \
  --assertExists \
  -h {{ connection.host }} \
  {% if connection.login %} -u {{ connection.login }} {% endif %}\
  {% if connection.get_password() %} -p {{ connection.get_password()
}} {% endif %}\
  -d {{ connection.schema }}
  ...

However, when this operator is executed (or when the template is rendered
on the UI), the credentials are written to the log files / clearly visible
on the UI, which is problematic.

There are many other cases where this can happen, and I’m wondering what is
a solution for it.

What would be ideal is:

   - Prevent credentials from accidentally being shown in “show rendered
   template” screen.
   - Prevent credentials from being written to the logs.

Thanks.

-Jeeyoung Kim

Re: Recommended way of piping credentials / sensitive information on templated operators

Posted by Shoumitra Srivastava <sh...@gmail.com>.

Hey jeeyoung,

Specifically for Redshift, we worked around that problem by using iam_roles
instead of credentials.

-Shoumitra

On Mon, Feb 26, 2018 at 11:22 AM, jeeyoung kim <je...@gmail.com> wrote:

> Hi everyone,
>
> I’m wondering how people work around accidentally writing credentials on
> bash operator template page / logs.
>
> For example, I may have PostgreSQL operator to copy data into Redshift.
>
> COPY TABLE_NAME from 's3://.../something.manifest.json'
> access_key_id '{{ params.AWS_ACCESS_KEY }}'
> secret_access_key '{{ params.AWS_SECRET_KEY }}'
>
> Or a command that exports from mongo
>
> mongoexport \
>   --assertExists \
>   -h {{ connection.host }} \
>   {% if connection.login %} -u {{ connection.login }} {% endif %}\
>   {% if connection.get_password() %} -p {{ connection.get_password()
> }} {% endif %}\
>   -d {{ connection.schema }}
>   ...
>
> However, when this operator is executed (or when the template is rendered
> on the UI), the credentials are written to the log files / clearly visible
> on the UI, which is problematic.
>
> There are many other cases where this can happen, and I’m wondering what is
> a solution for it.
>
> What would be ideal is:
>
>    - Prevent credentials from accidentally being shown in “show rendered
>    template” screen.
>    - Prevent credentials from being written to the logs.
>
> Thanks.
>
> -Jeeyoung Kim
> 
>

Re: Recommended way of piping credentials / sensitive information on templated operators

Posted by Shah Altaf <me...@gmail.com>.

Would like to hear a little more from others, or any recommendations around
this.  We've got other systems like SQL connections, SSH with user:pass,
API tokens.

Is there any recommended way to hide these from the output logs?



On Mon, Feb 26, 2018 at 10:41 PM Hbw <br...@heisenbergwoodworking.com>
wrote:

> Aws profiles on the workers - the creds are on the machines, but not
> exposed.  Boto/cli takes these profile names instead of access key/secret
> for just this kind of use case.
>
> Sent from a device with less than stellar autocorrect
>
> > On Feb 26, 2018, at 1:22 PM, jeeyoung kim <je...@gmail.com> wrote:
> >
> > Hi everyone,
> >
> > I’m wondering how people work around accidentally writing credentials on
> > bash operator template page / logs.
> >
> > For example, I may have PostgreSQL operator to copy data into Redshift.
> >
> > COPY TABLE_NAME from 's3://.../something.manifest.json'
> > access_key_id '{{ params.AWS_ACCESS_KEY }}'
> > secret_access_key '{{ params.AWS_SECRET_KEY }}'
> >
> > Or a command that exports from mongo
> >
> > mongoexport \
> >  --assertExists \
> >  -h {{ connection.host }} \
> >  {% if connection.login %} -u {{ connection.login }} {% endif %}\
> >  {% if connection.get_password() %} -p {{ connection.get_password()
> > }} {% endif %}\
> >  -d {{ connection.schema }}
> >  ...
> >
> > However, when this operator is executed (or when the template is rendered
> > on the UI), the credentials are written to the log files / clearly
> visible
> > on the UI, which is problematic.
> >
> > There are many other cases where this can happen, and I’m wondering what
> is
> > a solution for it.
> >
> > What would be ideal is:
> >
> >   - Prevent credentials from accidentally being shown in “show rendered
> >   template” screen.
> >   - Prevent credentials from being written to the logs.
> >
> > Thanks.
> >
> > -Jeeyoung Kim
> > 
>

Re: Recommended way of piping credentials / sensitive information on templated operators

Posted by Hbw <br...@heisenbergwoodworking.com>.

Aws profiles on the workers - the creds are on the machines, but not exposed.  Boto/cli takes these profile names instead of access key/secret for just this kind of use case.

Sent from a device with less than stellar autocorrect

> On Feb 26, 2018, at 1:22 PM, jeeyoung kim <je...@gmail.com> wrote:
> 
> Hi everyone,
> 
> I’m wondering how people work around accidentally writing credentials on
> bash operator template page / logs.
> 
> For example, I may have PostgreSQL operator to copy data into Redshift.
> 
> COPY TABLE_NAME from 's3://.../something.manifest.json'
> access_key_id '{{ params.AWS_ACCESS_KEY }}'
> secret_access_key '{{ params.AWS_SECRET_KEY }}'
> 
> Or a command that exports from mongo
> 
> mongoexport \
>  --assertExists \
>  -h {{ connection.host }} \
>  {% if connection.login %} -u {{ connection.login }} {% endif %}\
>  {% if connection.get_password() %} -p {{ connection.get_password()
> }} {% endif %}\
>  -d {{ connection.schema }}
>  ...
> 
> However, when this operator is executed (or when the template is rendered
> on the UI), the credentials are written to the log files / clearly visible
> on the UI, which is problematic.
> 
> There are many other cases where this can happen, and I’m wondering what is
> a solution for it.
> 
> What would be ideal is:
> 
>   - Prevent credentials from accidentally being shown in “show rendered
>   template” screen.
>   - Prevent credentials from being written to the logs.
> 
> Thanks.
> 
> -Jeeyoung Kim
>