Handling secrets in Jenkins

[Marco de Abreu <ma...@googlemail.com>]

Hello,

I'm planning to release the entire setup (based on Terraform) and
configuration of the current CI system to a public repository in order give
the community the possibility to influence the development and give
suggestions. In order to protect credentials, I've considered storing these
secrets in Amazon KMS or another similar system while the actual
configuration is stored in the publicly accessible repository.

Unfortunately, there are quite a lot of credentials embedded into the
Jenkins configuration files. I have created an analysis at
https://cwiki.apache.org/confluence/display/MXNET/Proposal%3A+Architecture,
see 'Spoiler: File analysis and categorization'.

While directories like 'secrets' are clear and *only* contain secrets, most
of the other files mix actual configuration with credentials. In general,
Jenkins offers a method to define credentials in a central repository so
that they can be reference at other locations. Unfortunately, many plugins
are not making use of this method and rather store the encrypted value in
the configuration files. A bit of background: All plugins use the same
mechanism based on hudson.util.Secret (see
https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/util/Secret.java).
This mechanism is using AES encryption, while the used key is stored at
'secrets/master.key'.

This example is part of the configuration for the GitHub SSO plugin, stored
in the central Jenkins configuration file config.xml:

  <securityRealm class="org.jenkinsci.plugins.GithubSecurityRealm">
    <githubWebUri>https://github.com</githubWebUri>
    <githubApiUri>https://api.github.com</githubApiUri>
    <clientID>1234567890</clientID>
    <clientSecret>{SomeEncryptedText==}</clientSecret>
    <oauthScopes>read:org,user:email</oauthScopes>
  </securityRealm>

While it would be desirable to have the central config.xml stored in the
public repository, entries like this theoretically classify the file as
secret. If you take another look at the analysis, you'll see that this way
would classify quite a lot of files as secret and thus render it useless to
have the configuration released to the public. But there is one catch:
There are only two ways to decrypt these values:
1. Gain access to the file at 'secrets/master.key'
2. Gain the ability to execute the script 'def decrypted =
hudson.util.Secret.fromString("{SomeEncryptedText==}").getPlainText()'

Luckily, the Jenkins developers are aware of these cases and implemented a
few security measurements. First of all, all groovy scripts (aka the
Jenkinsfile) are executed in a sandboxed environment and thus unable to
access the underlying filesystem. Another method would be to run a job on
the master and read the file at 'secrets/master.key', but since we have 0
executor slots available on the master, no user-defined jobs can be
executed. Method 1 is thus not possible. So what about the script I've
mentioned above? Jenkins restricts the access to certain APIs when executed
inside user-defined scripts like the Jenkinsfile. I have tried to access
this API by modifying the Jenkinsfile inside a PR at
https://github.com/apache/incubator-mxnet/pull/9331 and the result was the
following message:
"caught
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException:
Scripts not permitted to use staticMethod hudson.util.Secret fromString
java.lang.String"
This means that a user is not able to submit a PR to decrypt a secret. The
last way to execute this script is the Script-Console, but considering that
it is only accessible as an Administrator, I will put this one aside. This
means that method 2 is also not possible.


Since these are the only two ways to decrypt a secret, we're theoretically
safe to publish these encrypted values as long as the master.key stays
secret. Other than that, there's no way to decrypt these values.

Basically we have to following options:
1. Rely on the encryption and do not hide encrypted values. This means only
the actual secrets will be stored in KMS and the entire configuration can
be posted to the public repository. This method relies on the ability to
keep the master.key a secret - only people with access to the underlying
AWS account would be able to access it.
2. Keep the entire Jenkins configuration in a private repository and only
grant certain people access. At the moment, this would only be the
administrators of the CI (Meghna, Gautam and myself) as well as Chris
Olivier due to the constraints mentioned in
https://lists.apache.org/thread.html/b28653380fc179a8cc1410d12661a7b1636c09e39a155df157b51bc3@%3Cdev.mxnet.apache.org%3E
.

What does the community think?

Best regards,
Marco

Re: Handling secrets in Jenkins

[Mu Li <mu...@gmail.com>]

I suggest to use 2) for simplicity, especially given Jenkins is not very
secure in some aspects. I remember about a half year ago I forgot to
upgrade the jenkins server in time, someone then installed a coin miner on
all slaves..

On Sat, Jan 6, 2018 at 2:23 PM, Marco de Abreu <marco.g.abreu@googlemail.com
> wrote:

> Hello,
>
> I'm planning to release the entire setup (based on Terraform) and
> configuration of the current CI system to a public repository in order give
> the community the possibility to influence the development and give
> suggestions. In order to protect credentials, I've considered storing these
> secrets in Amazon KMS or another similar system while the actual
> configuration is stored in the publicly accessible repository.
>
> Unfortunately, there are quite a lot of credentials embedded into the
> Jenkins configuration files. I have created an analysis at
> https://cwiki.apache.org/confluence/display/MXNET/Proposal%3A+Architecture
> ,
> see 'Spoiler: File analysis and categorization'.
>
> While directories like 'secrets' are clear and *only* contain secrets, most
> of the other files mix actual configuration with credentials. In general,
> Jenkins offers a method to define credentials in a central repository so
> that they can be reference at other locations. Unfortunately, many plugins
> are not making use of this method and rather store the encrypted value in
> the configuration files. A bit of background: All plugins use the same
> mechanism based on hudson.util.Secret (see
> https://github.com/jenkinsci/jenkins/blob/master/core/src/
> main/java/hudson/util/Secret.java).
> This mechanism is using AES encryption, while the used key is stored at
> 'secrets/master.key'.
>
> This example is part of the configuration for the GitHub SSO plugin, stored
> in the central Jenkins configuration file config.xml:
>
>   <securityRealm class="org.jenkinsci.plugins.GithubSecurityRealm">
>     <githubWebUri>https://github.com</githubWebUri>
>     <githubApiUri>https://api.github.com</githubApiUri>
>     <clientID>1234567890</clientID>
>     <clientSecret>{SomeEncryptedText==}</clientSecret>
>     <oauthScopes>read:org,user:email</oauthScopes>
>   </securityRealm>
>
> While it would be desirable to have the central config.xml stored in the
> public repository, entries like this theoretically classify the file as
> secret. If you take another look at the analysis, you'll see that this way
> would classify quite a lot of files as secret and thus render it useless to
> have the configuration released to the public. But there is one catch:
> There are only two ways to decrypt these values:
> 1. Gain access to the file at 'secrets/master.key'
> 2. Gain the ability to execute the script 'def decrypted =
> hudson.util.Secret.fromString("{SomeEncryptedText==}").getPlainText()'
>
> Luckily, the Jenkins developers are aware of these cases and implemented a
> few security measurements. First of all, all groovy scripts (aka the
> Jenkinsfile) are executed in a sandboxed environment and thus unable to
> access the underlying filesystem. Another method would be to run a job on
> the master and read the file at 'secrets/master.key', but since we have 0
> executor slots available on the master, no user-defined jobs can be
> executed. Method 1 is thus not possible. So what about the script I've
> mentioned above? Jenkins restricts the access to certain APIs when executed
> inside user-defined scripts like the Jenkinsfile. I have tried to access
> this API by modifying the Jenkinsfile inside a PR at
> https://github.com/apache/incubator-mxnet/pull/9331 and the result was the
> following message:
> "caught
> org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException:
> Scripts not permitted to use staticMethod hudson.util.Secret fromString
> java.lang.String"
> This means that a user is not able to submit a PR to decrypt a secret. The
> last way to execute this script is the Script-Console, but considering that
> it is only accessible as an Administrator, I will put this one aside. This
> means that method 2 is also not possible.
>
>
> Since these are the only two ways to decrypt a secret, we're theoretically
> safe to publish these encrypted values as long as the master.key stays
> secret. Other than that, there's no way to decrypt these values.
>
> Basically we have to following options:
> 1. Rely on the encryption and do not hide encrypted values. This means only
> the actual secrets will be stored in KMS and the entire configuration can
> be posted to the public repository. This method relies on the ability to
> keep the master.key a secret - only people with access to the underlying
> AWS account would be able to access it.
> 2. Keep the entire Jenkins configuration in a private repository and only
> grant certain people access. At the moment, this would only be the
> administrators of the CI (Meghna, Gautam and myself) as well as Chris
> Olivier due to the constraints mentioned in
> https://lists.apache.org/thread.html/b28653380fc179a8cc1410d12661a7
> b1636c09e39a155df157b51bc3@%3Cdev.mxnet.apache.org%3E
> .
>
> What does the community think?
>
> Best regards,
> Marco
>