You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by "Erb, Stephan" <St...@blue-yonder.com> on 2016/01/11 15:04:42 UTC
Managing Secrets
Hi everyone,
I'd like to explore mechanisms to provide secrets to applications deployed on Mesos (e.g. via Marathon or Apache Aurora).
The classical workflow of having the passwords encrypted in the local git repository and only decrypted during deployment (for example via ansible vault) tends to fall too short. From the user perspective the deployment is done once its job has been submitted to the service scheduler. Then however, the passwords would be unsecured when traveling through the various Mesos and scheduler layers. It might potentially been visible to anyone having access to either the scheduler or the Mesos UI.
One possible solution is to provide an out-of-band mechanism to distribute passwords to all slaves and to protect those passwords using the regular Linux permissions, making them only accessible to authorized tasks. Other solutions might involve the integration of tools like HashiCorp Vault or Square Keywhiz.
Is anyone willing to share his/her experience with handling such secrets?
Thanks and Best Regards,
Stephan
Re: Managing Secrets
Posted by Alexandre Mclean <al...@gmail.com>.
Hashicorp's Vault seems to be the most promising solution for this.
Mantl has integrated it with Mesos and Marathon already, so I would look at
their stack to see how it all fits together:
https://mantl.io
On Mon, Jan 11, 2016 at 10:21 AM, Luke Amdor <lu...@banno.com> wrote:
> We've been using Hashicorp's Vault (https://vaultproject.io) to secure
> secrets for our applications deployed via Marathon. Currently we've just
> locked down Marathon and use vault's token auth per application to access
> secrets. However, we've done a lot of work in the last few months to
> authenticate apps via checking Mesos and Marathon to see if the task is
> actually authentic and then will trust it:
> https://github.com/Banno/vault/commit/62deeb6866abd8ea95a9f2f2fe60c0f605075494
>
> Overall Vault has been a great addition to our stack.
>
> On Mon, Jan 11, 2016 at 8:28 AM, Robert Vežnaver <
> robert.veznaver@gmail.com> wrote:
>
>> Hi,
>>
>> We give each app a private/public key pair, and let the developers
>> encrypt any environment variable with their app's public key (we also
>> base64 encode the ciphertext).
>> Then instead of calling the usual "/bin/sh", we call a small bash
>> wrapper script which decrypts the desired environment variables prior
>> to running the app.
>>
>> Note that our cluster is managed by Chef, so we ensure that:
>> - both the user and the private key are set up
>> - each app is running as its own user
>> - each app may access only its own private key
>>
>> As an added bonus, the wrapper script:
>> - wraps stdout and stderr and forwards them to the local syslog (which
>> then forwards them to Elasticsearch)
>> - automatically decrypts any environment variable that starts with
>> "SECRET_" (so they don't have to specify anything when calling the
>> command)
>>
>> Hope this gives you an idea or two,
>>
>> Cheers,
>> Robert
>>
>>
>> On 11 January 2016 at 15:04, Erb, Stephan <St...@blue-yonder.com>
>> wrote:
>> > Hi everyone,
>> >
>> > I'd like to explore mechanisms to provide secrets to applications
>> deployed on Mesos (e.g. via Marathon or Apache Aurora).
>> >
>> > The classical workflow of having the passwords encrypted in the local
>> git repository and only decrypted during deployment (for example via
>> ansible vault) tends to fall too short. From the user perspective the
>> deployment is done once its job has been submitted to the service
>> scheduler. Then however, the passwords would be unsecured when traveling
>> through the various Mesos and scheduler layers. It might potentially been
>> visible to anyone having access to either the scheduler or the Mesos UI.
>> >
>> > One possible solution is to provide an out-of-band mechanism to
>> distribute passwords to all slaves and to protect those passwords using the
>> regular Linux permissions, making them only accessible to authorized tasks.
>> Other solutions might involve the integration of tools like HashiCorp Vault
>> or Square Keywhiz.
>> >
>> > Is anyone willing to share his/her experience with handling such
>> secrets?
>> >
>> > Thanks and Best Regards,
>> > Stephan
>>
>
>
>
> --
> *Luke Amdor* | Platform Lead Architect | *Banno*
> Des Moines IA 50309 | Cell 515.231.4033
>
--
Alexandre
Re: Managing Secrets
Posted by Luke Amdor <lu...@banno.com>.
We've been using Hashicorp's Vault (https://vaultproject.io) to secure
secrets for our applications deployed via Marathon. Currently we've just
locked down Marathon and use vault's token auth per application to access
secrets. However, we've done a lot of work in the last few months to
authenticate apps via checking Mesos and Marathon to see if the task is
actually authentic and then will trust it:
https://github.com/Banno/vault/commit/62deeb6866abd8ea95a9f2f2fe60c0f605075494
Overall Vault has been a great addition to our stack.
On Mon, Jan 11, 2016 at 8:28 AM, Robert Vežnaver <ro...@gmail.com>
wrote:
> Hi,
>
> We give each app a private/public key pair, and let the developers
> encrypt any environment variable with their app's public key (we also
> base64 encode the ciphertext).
> Then instead of calling the usual "/bin/sh", we call a small bash
> wrapper script which decrypts the desired environment variables prior
> to running the app.
>
> Note that our cluster is managed by Chef, so we ensure that:
> - both the user and the private key are set up
> - each app is running as its own user
> - each app may access only its own private key
>
> As an added bonus, the wrapper script:
> - wraps stdout and stderr and forwards them to the local syslog (which
> then forwards them to Elasticsearch)
> - automatically decrypts any environment variable that starts with
> "SECRET_" (so they don't have to specify anything when calling the
> command)
>
> Hope this gives you an idea or two,
>
> Cheers,
> Robert
>
>
> On 11 January 2016 at 15:04, Erb, Stephan <St...@blue-yonder.com>
> wrote:
> > Hi everyone,
> >
> > I'd like to explore mechanisms to provide secrets to applications
> deployed on Mesos (e.g. via Marathon or Apache Aurora).
> >
> > The classical workflow of having the passwords encrypted in the local
> git repository and only decrypted during deployment (for example via
> ansible vault) tends to fall too short. From the user perspective the
> deployment is done once its job has been submitted to the service
> scheduler. Then however, the passwords would be unsecured when traveling
> through the various Mesos and scheduler layers. It might potentially been
> visible to anyone having access to either the scheduler or the Mesos UI.
> >
> > One possible solution is to provide an out-of-band mechanism to
> distribute passwords to all slaves and to protect those passwords using the
> regular Linux permissions, making them only accessible to authorized tasks.
> Other solutions might involve the integration of tools like HashiCorp Vault
> or Square Keywhiz.
> >
> > Is anyone willing to share his/her experience with handling such secrets?
> >
> > Thanks and Best Regards,
> > Stephan
>
--
*Luke Amdor* | Platform Lead Architect | *Banno*
Des Moines IA 50309 | Cell 515.231.4033
Re: Managing Secrets
Posted by Robert Vežnaver <ro...@gmail.com>.
Hi,
We give each app a private/public key pair, and let the developers
encrypt any environment variable with their app's public key (we also
base64 encode the ciphertext).
Then instead of calling the usual "/bin/sh", we call a small bash
wrapper script which decrypts the desired environment variables prior
to running the app.
Note that our cluster is managed by Chef, so we ensure that:
- both the user and the private key are set up
- each app is running as its own user
- each app may access only its own private key
As an added bonus, the wrapper script:
- wraps stdout and stderr and forwards them to the local syslog (which
then forwards them to Elasticsearch)
- automatically decrypts any environment variable that starts with
"SECRET_" (so they don't have to specify anything when calling the
command)
Hope this gives you an idea or two,
Cheers,
Robert
On 11 January 2016 at 15:04, Erb, Stephan <St...@blue-yonder.com> wrote:
> Hi everyone,
>
> I'd like to explore mechanisms to provide secrets to applications deployed on Mesos (e.g. via Marathon or Apache Aurora).
>
> The classical workflow of having the passwords encrypted in the local git repository and only decrypted during deployment (for example via ansible vault) tends to fall too short. From the user perspective the deployment is done once its job has been submitted to the service scheduler. Then however, the passwords would be unsecured when traveling through the various Mesos and scheduler layers. It might potentially been visible to anyone having access to either the scheduler or the Mesos UI.
>
> One possible solution is to provide an out-of-band mechanism to distribute passwords to all slaves and to protect those passwords using the regular Linux permissions, making them only accessible to authorized tasks. Other solutions might involve the integration of tools like HashiCorp Vault or Square Keywhiz.
>
> Is anyone willing to share his/her experience with handling such secrets?
>
> Thanks and Best Regards,
> Stephan