Hashicorp Vault - update

[Richard Downer <ri...@apache.org>]

Hello all,

Apache Brooklyn has supported Hashicorp Vault as an "externalized
configuration" provider[1]. Vault. This was implemented in 2015 - at that
time, Vault was at version 0.3.1 or thereabouts - still a fairly early
product.

Fast forward to 2021 and Vault is at 1.6.0 and several of its early
features have been deprecated and replaced with improved features, but
Brooklyn has stood still. In recent PRs[2][3] I've updated the Vault
provider to fix the breaking changes between versions 0.3 and 1.6, and to
ensure that Brooklyn still works with a near-out-of-the-box configuration
of Vault.

So Brooklyn is now back at the "minimum" support for Vault - but it's not
at the "recommended" level.

This is a complicated subject, and I don't claim to fully understand it
all. There's also quite a few different options, which vary depending on
the environment that Brooklyn is running in. Below I summarise the
resources available. I'm not proposing any immediate development effort
right now, but if there is demand for using a better Vault implementation,
the resources here will be a starting point for discussing implementations.

The "Secure Introduction" problem - how you get a secret to an app for it
to use to access Vault when you are trying to avoid storing secrets on the
app - is talked about at [4]. A number of options are proposed there,
depending on the deployment environment. For example, AWS-specific services
can be used, if deployment is onto AWS.

"Approle"[5][6] is Hashicorp's recommendation for apps to access Vault, but
that requires a "trusted entity" to act as an agent to get a secret
delivered to Brooklyn. Vault's tutorials describe Chef as a possible entity
- it obtains a secret for each new client it sets up, and deploys it into
the app it is deploying.

Response wrapping[7] provides a way to deliver a secret to the app. Rather
than deliver the secret directly, it's placed in a Vault "cubbyhole" and
what is delivered is a single-use, short-TTL token. The short TTL means
that an attacker has a very short time window to retrieve the secret. The
single-use nature means that if an attacker is able to successfully get the
secret, the app will know that if it can't retrieve the secret that it has
been compromised.


I suspect that the large variation of options and deployment environment
makes a "one size fits all" approach quite difficult to implement. But
interested in other people's thoughts.

Richard.


[1]
https://brooklyn.apache.org/v/latest/ops/externalized-configuration.html#vault
[2] https://github.com/apache/brooklyn-server/pull/1136
[3] https://github.com/apache/brooklyn-docs/pull/313
[4]
https://learn.hashicorp.com/tutorials/vault/secure-introduction?in=vault/app-integration
[5] https://www.vaultproject.io/docs/auth/approle
[6] https://learn.hashicorp.com/tutorials/vault/approle
[7] https://www.vaultproject.io/docs/concepts/response-wrapping.html

Re: Hashicorp Vault - update

[Geoff Macartney <ge...@gmail.com>]

Hi Richard,

Just to get back to you on this topic. Thanks for the email first of
all, it is indeed a great starting point for discussion, as you say.
I've been doing some work with Vault recently, though I'm far from an
expert, and the "secure introduction" problem is indeed one of the key
problems to solve.

From an Apache Brooklyn perspective I would say that Brooklyn is
pretty well placed to work with Vault, though it might require us to
do some development on top of the improvements you have made recently.
To me Brooklyn itself is ideally placed to be the "trusted
orchestrator" in a deployment using Vault for secrets management.

Yes, Brooklyn itself would need to have a way to authenticate with
Vault in the first place, but that is no different from the case of
using, e.g. Terraform as a trusted orchestrator, as in their example
([4] in your email).  There always needs to be a way to authenticate
any orchestrator. We could imagine a process that requires the user to
enter Vault credentials in order to launch Brooklyn, and stores them
in memory thereafter.

Once we have done that, Brooklyn can do the interactions with Vault to
obtain tokens to give to any entities it provisions, as they are being
created. These tokens can be short lived, ideally using Vault response
wrapping so that they are single use. They would then be exchanged by
the provisioned entity with Vault in order to get the secrets that
will be used to authenticate regular communications, e.g. Vault
AppRole tokens. (It could be other things like X509 certs, but I think
the AppRole tokens are likely to be typical. You would then use the
AppRole token to log in to Vault from the entity, and retrieve your
X509 certs or any other secrets.)

All of that sort of thing is fiddly to implement if you have to do it
on a case by case basis in the services you write, especially if you
have lots of microservices. If we come up with a useful set of
primitives, we can provide mechanisms to automate a lot of this for
our users.

An obvious Brooklyn entity type to create for this would be one to
manage the Vault Agent [1]. For containerised deployments on
Kubernetes, we could have an entity that would add the Vault Agent
Injector [2]. Imagine being able to configure an entity for deployment
to a KubernetesLocation with a "VaultAgentInjectorInitializer",
perhaps with one or two config keys, in order for Brooklyn to
automatically take care of adding the Vault annotations [3] to the
deployment descriptor for your entity, all without you having to write
a line of them yourself. (They can be quite verbose.)

I don't believe there is one size that fits all, and if we are
thinking about this we should think more of creating a Vault toolkit
than a "solution". But it would also be fairly "Brooklyn Style" if we
then also put those tools together in a simple way that would enable
easy use of Brooklyn as a trusted orchestrator, with a conventional
pattern of use such as the one I outlined in the fourth paragraph
above.

Just my 2¢.

Regards
Geoff


[1] https://www.vaultproject.io/docs/agent
[2] https://www.vaultproject.io/docs/platform/k8s/injector
[3] https://www.vaultproject.io/docs/platform/k8s/injector/annotations





On Fri, 15 Jan 2021 at 13:55, Richard Downer <ri...@apache.org> wrote:
>
> Hello all,
>
> Apache Brooklyn has supported Hashicorp Vault as an "externalized
> configuration" provider[1]. Vault. This was implemented in 2015 - at that
> time, Vault was at version 0.3.1 or thereabouts - still a fairly early
> product.
>
> Fast forward to 2021 and Vault is at 1.6.0 and several of its early
> features have been deprecated and replaced with improved features, but
> Brooklyn has stood still. In recent PRs[2][3] I've updated the Vault
> provider to fix the breaking changes between versions 0.3 and 1.6, and to
> ensure that Brooklyn still works with a near-out-of-the-box configuration
> of Vault.
>
> So Brooklyn is now back at the "minimum" support for Vault - but it's not
> at the "recommended" level.
>
> This is a complicated subject, and I don't claim to fully understand it
> all. There's also quite a few different options, which vary depending on
> the environment that Brooklyn is running in. Below I summarise the
> resources available. I'm not proposing any immediate development effort
> right now, but if there is demand for using a better Vault implementation,
> the resources here will be a starting point for discussing implementations.
>
> The "Secure Introduction" problem - how you get a secret to an app for it
> to use to access Vault when you are trying to avoid storing secrets on the
> app - is talked about at [4]. A number of options are proposed there,
> depending on the deployment environment. For example, AWS-specific services
> can be used, if deployment is onto AWS.
>
> "Approle"[5][6] is Hashicorp's recommendation for apps to access Vault, but
> that requires a "trusted entity" to act as an agent to get a secret
> delivered to Brooklyn. Vault's tutorials describe Chef as a possible entity
> - it obtains a secret for each new client it sets up, and deploys it into
> the app it is deploying.
>
> Response wrapping[7] provides a way to deliver a secret to the app. Rather
> than deliver the secret directly, it's placed in a Vault "cubbyhole" and
> what is delivered is a single-use, short-TTL token. The short TTL means
> that an attacker has a very short time window to retrieve the secret. The
> single-use nature means that if an attacker is able to successfully get the
> secret, the app will know that if it can't retrieve the secret that it has
> been compromised.
>
>
> I suspect that the large variation of options and deployment environment
> makes a "one size fits all" approach quite difficult to implement. But
> interested in other people's thoughts.
>
> Richard.
>
>
> [1]
> https://brooklyn.apache.org/v/latest/ops/externalized-configuration.html#vault
> [2] https://github.com/apache/brooklyn-server/pull/1136
> [3] https://github.com/apache/brooklyn-docs/pull/313
> [4]
> https://learn.hashicorp.com/tutorials/vault/secure-introduction?in=vault/app-integration
> [5] https://www.vaultproject.io/docs/auth/approle
> [6] https://learn.hashicorp.com/tutorials/vault/approle
> [7] https://www.vaultproject.io/docs/concepts/response-wrapping.html