Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

Our Release Managers keep having troubles running Maven's release plugin.  It tends to fail for them when downloading dependencies from Maven Central or uploading artifacts to repository.a.o.  It runs just fine for me on the US west coast.  This got me wondering if we could run our release steps on builds.a.o in hopes that the builds servers have better connections to Maven Central and repository.a.o.  I think we need to use a Windows node.  

Maven's release plugins commit and push to Git and upload to repository.a.o.  I saw that some folks have a node that can commit to the a.o website SVN.  Is anyone already doing releases from builds?  What issues are there, if any?

Thanks,
-Alex

Re: Can we package release artifacts on builds.a.o?

On 2018/12/08 17:43:37, Alex Harui <ah...@adobe.com.INVALID> wrote: 
> Gavin, Alan, Karl,
> 
> Thanks for the information.
> 
> This email implies that there is a Jenkins node that can commit something.  What creds are used for that?  Is there a buildbot user?
> https://lists.apache.org/thread.html/efed1ff44fbfe5770ea1574b2f53a5295ae8326c5a3a5feb9f88cd48@%3Cbuilds.apache.org%3E
> 
> If so, I was imagining the following workflow:
> 
> 1) Jenkins runs Maven release.  I forgot about the PGP signing part.  If there is no way to skip it, then can a buildbot "user" PGP sign it?

So I have long had an idea in the back of my head (I discussed it a little with Hervé Boutemy) to leverage the work I did on the Jenkins SSH Agent plugin to allow it to act as a GPG agent (since GPG Agent protocol is just an extension of the SSH Agent protocol)

Or rather to act as a GPG Agent proxy.

The user would run a command locally to connect a tunnel of their local GPG Agent process to Jenkins and then on to the specific build agent (probably by having a build step that outputs a one time key and pauses the build until connected)... thus the one time key allows you to ensure that only the build agent you selected gets access to your GPG Agent.

This would enable builds on an ASF owned Jenkins that have been signed by the user triggering the build without their GPG key ever leaving their machine.

> 2) RM downloads the artifacts and verifies them.  The source package has to match the tag so I think that would detect any injections from other stuff running in Jenkins or elsewhere on the build server.  There's been a recent discussion on reproducible binaries and if this workflow is approved I would make our binaries are reproducible, and that should again detect any injections from the build server.
> 3) RM adds his/her PGP signature to the artifacts.  Not sure if there is a Maven way to do that.

Actually I had another idea (I briefly discussed with Brian Fox) namely if Nexus allowed appending GPG signatures to staged artifacts... perhaps could be written as a nexus plugin.

This would mean that the build artifacts could start off unsigned... or signed with a one-time key of the build agent. As each vote is cast by committers, they would upload the signatures to the relevant artifacts in the staging repo. When the vote is closed and the artifacts are released they would be signed by everyone who voted rather than the one signature of the RM.

> 4) Voting and other steps follow from there.
> 
> These would not be continuously running jobs.  They would have to be kicked off manually so it shouldn't add significant load, and we would know which commits came from buildbot so we could detect if anything went funky.
> 
> Thoughts?
> -Alex
> 
> On 12/8/18, 7:54 AM, "Gavin McDonald" <ip...@gmail.com> wrote:
> 
>     additionally, nobody should have their creds stored anyway other than their
>     own machine.
>     
>     On Sat, Dec 8, 2018 at 3:49 PM Allen Wittenauer
>     <aw...@effectivemachines.com.invalid> wrote:
>     
>     >
>     >
>     > > On Dec 7, 2018, at 11:56 PM, Alex Harui <ah...@adobe.com.INVALID>
>     > wrote:
>     > >
>     > >
>     > >
>     > > On 12/7/18, 10:49 PM, "Allen Wittenauer" <aw...@effectivemachines.com.INVALID>
>     > wrote:
>     > >
>     > >
>     > >
>     > >> On Dec 7, 2018, at 10:22 PM, Alex Harui <ah...@adobe.com.INVALID>
>     > wrote:
>     > >>
>     > >> Maven's release plugins commit and push to Git and upload to
>     > repository.a.o.  I saw that some folks have a node that can commit to the
>     > a.o website SVN.  Is anyone already doing releases from builds?  What
>     > issues are there, if any?
>     > >
>     > >       It's just flat out not secure enough to do a release on.
>     > >
>     > > Can you give me an example of how it isn't secure enough?
>     >
>     >
>     >         The primary purpose of these servers is to run untested,
>     > unverified code.
>     >
>     >         Jenkins has some very sharp security corners that makes it
>     > trivially un-trustable.  Something easy to understand: when Jenkins is
>     > configured to run multiple builds on a node, all builds on that node run in
>     > the same user space. Because there is no separation between executors, it's
>     > very possible for anyone to execute something that modifies another running
>     > build.  For example, probably the biggest bang for the least amount of work
>     > would be to replace jars in the shared maven cache.
>     >
>     >         [... and no, Docker doesn't help.]
>     >
>     >         There are other, bigger problems, but I'd rather not put that out
>     > in the public.
>     >
>     >
>     >
>     
>     -- 
>     Gav...
>     
> 
> 

Re: Can we package release artifacts on builds.a.o?

[Christofer Dutz <ch...@c-ware.de>]

Just have a look at the Jenkinsfile of the PLC4X projects develop branch.

The build has an additional Parameter:
-DaltDeploymentRepository=snapshot-repo::default::file:./local-snapshots-dir clean deploy

The deploy step then use the wagon plugin to upload.

You'll figure it out the rest ...

Chris

Outlook for Android<https://aka.ms/ghei36> herunterladen

________________________________
From: Hervé BOUTEMY <he...@free.fr>
Sent: Tuesday, January 8, 2019 2:49:19 AM
To: builds@apache.org
Subject: Re: Can we package release artifacts on builds.a.o?

ok, the issue for log4net is different than Royale

a few thoughts:
- perhaps you should limit your targets, since some targets look really old:
this would decrease the number of configurations needed for the RM

- I suppose the official source release artifact is the same for every
platform: it's only the convenience binaries that require the platforms, and
the requirement is not only for building but also for testing. Perhaps the
efforts on convenience binaries should be decoupled from source, eventually
with waves of binaries = make binaries available for a few recent 1st class
platforms during the release, then other platforms available later based on
feedback and help got from the community

Choosing to support so many platform version, with very old ones, comes at a
price: IMHO, making efforts to reduce supported scope will help you
And if old platforms are really badly needed by many users, these users could
be turned into contributors for the builds

my 2 cents...

Regards,

Hervé

Le lundi 7 janvier 2019, 13:23:59 CET Dominik Psenner a écrit :
> On 2019-01-07 08:51, Alex Harui wrote:
> > The workflow I envision is this:
> >
> > 1.  RM runs Jenkins job on builds@ to create release branch, generate
> > artifacts , tag the repo, push artifacts to Nexus staging and
> > dist.a.o/dev/Royale 2.  RM downloads artifacts to verify them, adds PGP
> > signature and calls for vote 3.  PMC downloads artifacts, verifies that
> > the source packages matches the tag and performs other checks required by
> > ASF release policy.
> This roughly matches the vision I have for log4net, a subproject of the
> apache logging pmc. log4net targets several .net frameworks (.net 2.0,
> .net 3.5, .net 4.0, .net 4.0 cp, .net 4.5, netstandard 1.3, mono 2.0,
> mono 3.5) and shall run on both linux and windows operating systems.
> Requiring a developer to arrange at least four machines to maintain the
> source code is off the table. Further the onboarding procedure for new
> committers shall be as painless as possible. Otherwise it makes it just
> harder to grow the community. So far we managed to build and test the
> source code using jenkins on a sensible array of environments. I was
> able to implement that pipeline because I had to stay in bed for two
> months to heal some wounds. During this period I was able to work on
> this for an hour or two each day. Currently we do have a release branch
> and the binaries were built by jenkins. We currently lack the human
> resources to test these binaries, even though I have great confidence
> that the binaries work. If we had ready-to-use environments it would be
> a great addition to the project. It would allow us to continue with step
> 2 of the above workflow. I do however have no idea how this could be
> achieved. Docker could be a piece of the puzzle, but not for the ancient
> .net frameworks mentioned earlier. Last but not least I do hope that the
> efforts put into the building and testing automation attracts new
> developers and allows the project to continue existing. The alternative
> is stagnation and becoming obsolete over time.

Re: Can we package release artifacts on builds.a.o?

[Hervé BOUTEMY <he...@free.fr>]

ok, the issue for log4net is different than Royale

a few thoughts:
- perhaps you should limit your targets, since some targets look really old: 
this would decrease the number of configurations needed for the RM

- I suppose the official source release artifact is the same for every 
platform: it's only the convenience binaries that require the platforms, and 
the requirement is not only for building but also for testing. Perhaps the 
efforts on convenience binaries should be decoupled from source, eventually 
with waves of binaries = make binaries available for a few recent 1st class 
platforms during the release, then other platforms available later based on 
feedback and help got from the community

Choosing to support so many platform version, with very old ones, comes at a 
price: IMHO, making efforts to reduce supported scope will help you
And if old platforms are really badly needed by many users, these users could 
be turned into contributors for the builds

my 2 cents...

Regards,

Hervé

Le lundi 7 janvier 2019, 13:23:59 CET Dominik Psenner a écrit :
> On 2019-01-07 08:51, Alex Harui wrote:
> > The workflow I envision is this:
> > 
> > 1.  RM runs Jenkins job on builds@ to create release branch, generate
> > artifacts , tag the repo, push artifacts to Nexus staging and
> > dist.a.o/dev/Royale 2.  RM downloads artifacts to verify them, adds PGP
> > signature and calls for vote 3.  PMC downloads artifacts, verifies that
> > the source packages matches the tag and performs other checks required by
> > ASF release policy.
> This roughly matches the vision I have for log4net, a subproject of the
> apache logging pmc. log4net targets several .net frameworks (.net 2.0,
> .net 3.5, .net 4.0, .net 4.0 cp, .net 4.5, netstandard 1.3, mono 2.0,
> mono 3.5) and shall run on both linux and windows operating systems.
> Requiring a developer to arrange at least four machines to maintain the
> source code is off the table. Further the onboarding procedure for new
> committers shall be as painless as possible. Otherwise it makes it just
> harder to grow the community. So far we managed to build and test the
> source code using jenkins on a sensible array of environments. I was
> able to implement that pipeline because I had to stay in bed for two
> months to heal some wounds. During this period I was able to work on
> this for an hour or two each day. Currently we do have a release branch
> and the binaries were built by jenkins. We currently lack the human
> resources to test these binaries, even though I have great confidence
> that the binaries work. If we had ready-to-use environments it would be
> a great addition to the project. It would allow us to continue with step
> 2 of the above workflow. I do however have no idea how this could be
> achieved. Docker could be a piece of the puzzle, but not for the ancient
> .net frameworks mentioned earlier. Last but not least I do hope that the
> efforts put into the building and testing automation attracts new
> developers and allows the project to continue existing. The alternative
> is stagnation and becoming obsolete over time.

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

On 1/7/19, 1:20 AM, "Christofer Dutz" <ch...@c-ware.de> wrote:

    Just adding my thoughts to the problem:
    
    Whenever I did a Maven training in the past, the core mantra I tried my students to understand was: 
    "If it's hard to do with Maven, you're probably doing it wrong."
    Ok ... if you're working on builds for non-java, things do get quite a lot trickier, but still solvable.
    
    Even if this sounds pretty blunt, in the past about 10 years of doing a lot of projects with Maven, in the end it has proven to be very true. 
    
    I would suggest to get someone involved, who's a Maven expert, to help you diagnose and fix the issues you are having instead of adding a patch that works around the core problem.
    I doubt Maven is actively preventing only the Royale project from doing releases from outside the US, so I guess it's a configuration problem.
    
    So I would suggest to get some of the Maven folks on one of the Maven lists to help you.
    
This sounds like a tangential topic to me.  There is more to the story than I have so far described in order to try to reduce the amount of stuff folks have to read.  Maybe it is a configuration issue, but I did not mention that I was able to reproduce upload issues to Nexus by also trying to stream video over Roku on my home network when uploading.  And also I did not mention the amount of time spent helping new RMs get configured to even run both Ant and Maven and the tests.  Clearly, the default configurations didn't make others successful.   A busy network seems to increase the chance of failure.  Having it all set up on a single computer that all RM's could share that is known to work would help solve this problem.

Having a single shared computer for RM's doesn't appear to violate any ASF policy.  I don't see any security risks, but maybe someone will come up with one.  But to share this computer, we need the computer to be able to commit stuff, either with a special account, or by swapping in credentials.

Thanks,
-Alex        
    
    

Re: Can we package release artifacts on builds.a.o?

[Christofer Dutz <ch...@c-ware.de>]

Just adding my thoughts to the problem:

Whenever I did a Maven training in the past, the core mantra I tried my students to understand was: 
"If it's hard to do with Maven, you're probably doing it wrong."
Ok ... if you're working on builds for non-java, things do get quite a lot trickier, but still solvable.

Even if this sounds pretty blunt, in the past about 10 years of doing a lot of projects with Maven, in the end it has proven to be very true. 

I would suggest to get someone involved, who's a Maven expert, to help you diagnose and fix the issues you are having instead of adding a patch that works around the core problem.
I doubt Maven is actively preventing only the Royale project from doing releases from outside the US, so I guess it's a configuration problem.

So I would suggest to get some of the Maven folks on one of the Maven lists to help you.

Chris







Am 07.01.19, 08:59 schrieb "Alex Harui" <ah...@adobe.com.INVALID>:

    Let me re-summarize, since I think people are not reading the whole thread or the JIRA issue linked upthread.
    
    I'm only concerned about Royale, which is the project I work on.
    
    On 1/6/19, 11:12 PM, "Hervé BOUTEMY" <he...@free.fr> wrote:
    
        > I don't have a strong opinion on the above, but I'm very concerned
        > about a requirement of a bot pushing to SCM repos.
        +1
        adding that there are 2 levels of concerns for the scm repos:
        1. the source repo (at least for tagging), which is either svn or git
        2. the dist repo, which is svn, for release publication
        
        
        in addition to this scm repos write access issues, there is also the gpg 
        private key access, when signing the release
        
        
        last topic to me: releasing at Apache is a 2 phases process:
        1. staging, that includes the real build, to open the 72h voting period
        2. publishing once vote approved, where no build happens but management of 
        release area and many other parts like issue tracker
        
    
    Royale is mainly interested in #1.  Let me re-summarize, since I think people are not reading the whole thread or the JIRA issue linked upthread.
    
    Royale has been unsuccessful getting anyone other than me to be the RM.  All 3 others who have tried ran into upload/download issues running Maven's release plugin on their machines.  So, Royale is interested in having a Windows computer configured to generate their release artifacts that all Royale RMs could use.    This Windows computer will run Maven's release plugin which must be allowed to make commits.  I can understand why many of you don't see why Royale needs this, but I think we do, and some other projects have expressed interest as well.
    
    To be specific, Royale not only produces Maven artifacts, but Ant tasks as well and the RM pushes those Ant artifacts to dist.a.o as a convenience binary.  So committing to dist.a.o is included in this request.  We haven't gotten far enough to know if we have an issue with #2, publishing after the vote.  I doubt it since #2 does not require transferring lots of data from the RM's computer.  We also haven't gotten far enough to know if the bot would need to PGP sign the artifacts.  I would hope not, but maybe Maven's release plugin or Nexus has some restriction on that.
    
    The workflow I envision is this:
    
    1.  RM runs Jenkins job on builds@ to create release branch, generate artifacts , tag the repo, push artifacts to Nexus staging and dist.a.o/dev/Royale
    2.  RM downloads artifacts to verify them, adds PGP signature and calls for vote
    3.  PMC downloads artifacts, verifies that the source packages matches the tag and performs other checks required by ASF release policy.
    
    
    I'm a client framework developer, and finding security holes is not something I have to do often, so I am truly interested in getting feedback from others on potential exploits in my proposal.  To me, there is enough security in builds.a.o such that the bot could make commits and the PMC could review those changes and catch rogue commits.  
    
    I'm also interested in other ways to effectively create a "single computer" that is configured correctly that other committers could use to generate these artifacts.  If there was a way to temporarily place your Git/SVN creds on this remote "single computer" that would probably be sufficient for me.  Maven seems to want an SSH key on that computer in order to push to git.
    
    Thanks,
    -Alex
        
    
    

Re: Can we package release artifacts on builds.a.o?

[Dominik Psenner <dp...@gmail.com>]

On 2019-01-07 08:51, Alex Harui wrote:
> The workflow I envision is this:
>
> 1.  RM runs Jenkins job on builds@ to create release branch, generate artifacts , tag the repo, push artifacts to Nexus staging and dist.a.o/dev/Royale
> 2.  RM downloads artifacts to verify them, adds PGP signature and calls for vote
> 3.  PMC downloads artifacts, verifies that the source packages matches the tag and performs other checks required by ASF release policy.

This roughly matches the vision I have for log4net, a subproject of the 
apache logging pmc. log4net targets several .net frameworks (.net 2.0, 
.net 3.5, .net 4.0, .net 4.0 cp, .net 4.5, netstandard 1.3, mono 2.0, 
mono 3.5) and shall run on both linux and windows operating systems. 
Requiring a developer to arrange at least four machines to maintain the 
source code is off the table. Further the onboarding procedure for new 
committers shall be as painless as possible. Otherwise it makes it just 
harder to grow the community. So far we managed to build and test the 
source code using jenkins on a sensible array of environments. I was 
able to implement that pipeline because I had to stay in bed for two 
months to heal some wounds. During this period I was able to work on 
this for an hour or two each day. Currently we do have a release branch 
and the binaries were built by jenkins. We currently lack the human 
resources to test these binaries, even though I have great confidence 
that the binaries work. If we had ready-to-use environments it would be 
a great addition to the project. It would allow us to continue with step 
2 of the above workflow. I do however have no idea how this could be 
achieved. Docker could be a piece of the puzzle, but not for the ancient 
.net frameworks mentioned earlier. Last but not least I do hope that the 
efforts put into the building and testing automation attracts new 
developers and allows the project to continue existing. The alternative 
is stagnation and becoming obsolete over time.

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

Let me re-summarize, since I think people are not reading the whole thread or the JIRA issue linked upthread.

I'm only concerned about Royale, which is the project I work on.

On 1/6/19, 11:12 PM, "Hervé BOUTEMY" <he...@free.fr> wrote:

    > I don't have a strong opinion on the above, but I'm very concerned
    > about a requirement of a bot pushing to SCM repos.
    +1
    adding that there are 2 levels of concerns for the scm repos:
    1. the source repo (at least for tagging), which is either svn or git
    2. the dist repo, which is svn, for release publication
    
    
    in addition to this scm repos write access issues, there is also the gpg 
    private key access, when signing the release
    
    
    last topic to me: releasing at Apache is a 2 phases process:
    1. staging, that includes the real build, to open the 72h voting period
    2. publishing once vote approved, where no build happens but management of 
    release area and many other parts like issue tracker
    

Royale is mainly interested in #1.  Let me re-summarize, since I think people are not reading the whole thread or the JIRA issue linked upthread.

Royale has been unsuccessful getting anyone other than me to be the RM.  All 3 others who have tried ran into upload/download issues running Maven's release plugin on their machines.  So, Royale is interested in having a Windows computer configured to generate their release artifacts that all Royale RMs could use.    This Windows computer will run Maven's release plugin which must be allowed to make commits.  I can understand why many of you don't see why Royale needs this, but I think we do, and some other projects have expressed interest as well.

To be specific, Royale not only produces Maven artifacts, but Ant tasks as well and the RM pushes those Ant artifacts to dist.a.o as a convenience binary.  So committing to dist.a.o is included in this request.  We haven't gotten far enough to know if we have an issue with #2, publishing after the vote.  I doubt it since #2 does not require transferring lots of data from the RM's computer.  We also haven't gotten far enough to know if the bot would need to PGP sign the artifacts.  I would hope not, but maybe Maven's release plugin or Nexus has some restriction on that.

The workflow I envision is this:

1.  RM runs Jenkins job on builds@ to create release branch, generate artifacts , tag the repo, push artifacts to Nexus staging and dist.a.o/dev/Royale
2.  RM downloads artifacts to verify them, adds PGP signature and calls for vote
3.  PMC downloads artifacts, verifies that the source packages matches the tag and performs other checks required by ASF release policy.


I'm a client framework developer, and finding security holes is not something I have to do often, so I am truly interested in getting feedback from others on potential exploits in my proposal.  To me, there is enough security in builds.a.o such that the bot could make commits and the PMC could review those changes and catch rogue commits.  

I'm also interested in other ways to effectively create a "single computer" that is configured correctly that other committers could use to generate these artifacts.  If there was a way to temporarily place your Git/SVN creds on this remote "single computer" that would probably be sufficient for me.  Maven seems to want an SSH key on that computer in order to push to git.

Thanks,
-Alex
    

Re: Can we package release artifacts on builds.a.o?

[Hervé BOUTEMY <he...@free.fr>]

Le lundi 7 janvier 2019, 03:57:53 CET Roman Shaposhnik a écrit :
> On Sun, Jan 6, 2019 at 6:50 PM Alex Harui <ah...@adobe.com.invalid> wrote:
> > OK, apparently Infra doesn't want to discuss this in a JIRA issue so I
> > will try to continue it here and bug people with emails if the thread
> > stagnates like it did last time.
> > 
> > I'm unclear what questions and problems are of concern here specific to
> > this ask.  IMO: 1) ASF Release Policy currently allows artifacts to be
> > packaged on other hardware.  It just has to be verified on
> > RM/PMC-controlled hardware 2) There is no packaging specific security
> > risk.  Rogue executions via Jenkins are either possible or not possible
> > and there are plenty of other juicy targets for rogue executions besides
> > release artifacts that are verifiable.
> I don't have a strong opinion on the above, but I'm very concerned
> about a requirement of a bot pushing to SCM repos.
+1
adding that there are 2 levels of concerns for the scm repos:
1. the source repo (at least for tagging), which is either svn or git
2. the dist repo, which is svn, for release publication


in addition to this scm repos write access issues, there is also the gpg 
private key access, when signing the release


last topic to me: releasing at Apache is a 2 phases process:
1. staging, that includes the real build, to open the 72h voting period
2. publishing once vote approved, where no build happens but management of 
release area and many other parts like issue tracker

I don't get precisely which project requires to ease phase 1 or phase 2

In Maven project release procedure [1], phase 1 = staging is really simple and 
I would not see the value of putting a button on Jenkins vs the curent process 
on committer's machine
This is phase 2 that is complex: but this phase is hard to automate, because 
the diversity of systems to update, little variations per sub-projects (Maven 
has near 100 different components to release [2])

I'm not clear which Apache project has which improvement in mind.
What I know is that staging requirements are quite different from publishing 
ones, and probably we're not all aligned on what each project is trying to 
improve

Regards,

Hervé

[1] https://maven.apache.org/developers/release/maven-project-release-procedure.html

[2] https://maven.apache.org/scm.html#Maven_Sources_Overview
> 
> Thanks,
> Roman.

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

On 1/6/19, 8:30 PM, "Dave Fisher" <da...@comcast.net> wrote:

    
    
    Sent from my iPhone
    
    > On Jan 6, 2019, at 7:53 PM, Roman Shaposhnik <ro...@shaposhnik.org> wrote:
    > 
    >> On Sun, Jan 6, 2019 at 7:38 PM Alex Harui <ah...@adobe.com.invalid> wrote:
    >> 
    >> 
    >> 
    >> On 1/6/19, 6:58 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
    >> 
    >>>    On Sun, Jan 6, 2019 at 6:50 PM Alex Harui <ah...@adobe.com.invalid> wrote:
    >>> 
    >>> OK, apparently Infra doesn't want to discuss this in a JIRA issue so I will try to continue it here and bug people with emails if the thread stagnates like it did last time.
    >>> 
    >>> I'm unclear what questions and problems are of concern here specific to this ask.  IMO:
    >>> 1) ASF Release Policy currently allows artifacts to be packaged on other hardware.  It just has to be verified on RM/PMC-controlled hardware
    >>> 2) There is no packaging specific security risk.  Rogue executions via Jenkins are either possible or not possible and there are plenty of other juicy targets for rogue executions besides release artifacts that are verifiable.
    >> 
    >>    I don't have a strong opinion on the above, but I'm very concerned
    >>    about a requirement of a bot pushing to SCM repos.
    >> 
    >> Please explain your concern.
    > 
    > ASF lives and dies by how well it can track IP provenance in what we release.
    > That's why any non-committer interactions around SCM will give me pause.
    
    Releases are explicitly approved by a PMC. How can the build system results be approved by the PMC? Safely and confidently?
    
The source code in the source package would match the tag in SCM.

    > 
    >> A bot is already allowed to commit to the website repos, AIUI.
    > 
    > Two things:
    >   1. can you give me real-world examples of that?
    
    Website publishing is not an act of the whole PMC. It can be triggered on commit / done as a committer’s act.
    
    >   2. website repos are much lower on my list of priorities than code
    > repos (see above for reasoning)
    
    Agreed. I see that this question of “Release” to be  worth discussion. IMO what we are really discussing is automatically releasing build system produced convenience binaries.

I am not proposing a way to automatically release artifacts produced by the build system.  The job would be manually started.  The artifacts must be validated/verified by the PMC.  ASF release policy requires that.  builds@ policy could require that jobs that produce artifacts are not automatically triggered.
    
    Can we allow build system produced convenience binaries? If so must we hold votes? If not then what level of scrutiny must the PMC provide?
    
I am not proposing changing ASF policy on convenience binaries.  It should be up to the RM/PMC just like convenience binaries produced on an RM's computer.  Nobody knows how good/bad they are.  I will put effort into making Royale's binaries reproducible if build system generated artifacts are allowed.

Please try to avoid adding other discussion topics like whether convenience binaries require a vote to this thread.  I would like to only focus on whether a buildbot could make commits as part of packaging artifacts.

Thanks,
-Alex
 

Re: Can we package release artifacts on builds.a.o?

[Dave Fisher <da...@comcast.net>]

Sent from my iPhone

> On Jan 6, 2019, at 7:53 PM, Roman Shaposhnik <ro...@shaposhnik.org> wrote:
> 
>> On Sun, Jan 6, 2019 at 7:38 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>> 
>> 
>> 
>> On 1/6/19, 6:58 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
>> 
>>>    On Sun, Jan 6, 2019 at 6:50 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>>> 
>>> OK, apparently Infra doesn't want to discuss this in a JIRA issue so I will try to continue it here and bug people with emails if the thread stagnates like it did last time.
>>> 
>>> I'm unclear what questions and problems are of concern here specific to this ask.  IMO:
>>> 1) ASF Release Policy currently allows artifacts to be packaged on other hardware.  It just has to be verified on RM/PMC-controlled hardware
>>> 2) There is no packaging specific security risk.  Rogue executions via Jenkins are either possible or not possible and there are plenty of other juicy targets for rogue executions besides release artifacts that are verifiable.
>> 
>>    I don't have a strong opinion on the above, but I'm very concerned
>>    about a requirement of a bot pushing to SCM repos.
>> 
>> Please explain your concern.
> 
> ASF lives and dies by how well it can track IP provenance in what we release.
> That's why any non-committer interactions around SCM will give me pause.

Releases are explicitly approved by a PMC. How can the build system results be approved by the PMC? Safely and confidently?

> 
>> A bot is already allowed to commit to the website repos, AIUI.
> 
> Two things:
>   1. can you give me real-world examples of that?

Website publishing is not an act of the whole PMC. It can be triggered on commit / done as a committer’s act.

>   2. website repos are much lower on my list of priorities than code
> repos (see above for reasoning)

Agreed. I see that this question of “Release” to be  worth discussion. IMO what we are really discussing is automatically releasing build system produced convenience binaries.

Can we allow build system produced convenience binaries? If so must we hold votes? If not then what level of scrutiny must the PMC provide?

Regards,
Dave

> 
> Thanks,
> Roman.

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

On 1/6/19, 8:28 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
    > All commits, even PR's from non-commiters accepted by a committer are supposed to be reviewed, AIUI.  So if the bot makes a commit to the repo, the PMC is responsible for reviewing it.  In Royale's case, the bot should only be changing pom.xml files and making tags and branches, so a bad bot commit should be easy to spot and detection may even be tool-able.
    
    In theory -- yes. In CTR project -- may be not. Either way this is
    something I'd be only comfortable
    allowing if there's a great benefit of allowing it. Which I'm still
    failing to see, frankly.
    
It would help Royale.  As I said in the JIRA issue, our potential RM's have trouble getting configured, so having one box configured to crank releases would a great benefit to our community.  It sounds like some other projects want this too.


    >     > A bot is already allowed to commit to the website repos, AIUI.
    >
    >     Two things:
    >        1. can you give me real-world examples of that?
    >
    > See the beginning of this thread.  I posted this link to an old email:
    >
    > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2Fefed1ff44fbfe5770ea1574b2f53a5295ae8326c5a3a5feb9f88cd48%40%253Cbuilds.apache.org%253E&amp;data=02%7C01%7Caharui%40adobe.com%7C3dbf3ea90f45481777d508d67458928a%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636824321126950349&amp;sdata=uBOFbAk79PTSE%2B3jzWVOE4081bUUHy7PYrnzEDEwqYE%3D&amp;reserved=0
    >
    > And Karl Heinz Marbaise seemed to say that Maven is doing it.
    >
    > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbuilds.apache.org%2Fview%2FM-R%2Fview%2FMaven%2Fjob%2Fmaven-box%2Fjob%2Fmaven-site%2F&amp;data=02%7C01%7Caharui%40adobe.com%7C3dbf3ea90f45481777d508d67458928a%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636824321126950349&amp;sdata=tq1%2FNB0mEYbLKPShdNXXkVXrsZyxssIUpRfph7mvbxk%3D&amp;reserved=0
    >
    > Also note that in Royale's case, the Jenkins job would not be triggered.  It would be manually started.  So one requirement of allowing packaging jobs could be that artifact packaging jobs cannot be automatically triggered by repo changes or date/time.  That would better ensure that the PMC has reviewed any bot changes.
    
    I'm still not following. Can you link to the commits that originate from a bot?
    
The projects I work on do not use this feature.  Hopefully Karl or others who use this feature can supply commits.  I'm not sure why you need them though.

Thanks,
-Alex
 

Re: Can we package release artifacts on builds.a.o?

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Sun, Jan 6, 2019 at 8:20 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>
>
>
> On 1/6/19, 7:54 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
>
>     On Sun, Jan 6, 2019 at 7:38 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>     >
>     >
>     >
>     > On 1/6/19, 6:58 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
>     >
>     >     On Sun, Jan 6, 2019 at 6:50 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>     >     >
>     >     > OK, apparently Infra doesn't want to discuss this in a JIRA issue so I will try to continue it here and bug people with emails if the thread stagnates like it did last time.
>     >     >
>     >     > I'm unclear what questions and problems are of concern here specific to this ask.  IMO:
>     >     > 1) ASF Release Policy currently allows artifacts to be packaged on other hardware.  It just has to be verified on RM/PMC-controlled hardware
>     >     > 2) There is no packaging specific security risk.  Rogue executions via Jenkins are either possible or not possible and there are plenty of other juicy targets for rogue executions besides release artifacts that are verifiable.
>     >
>     >     I don't have a strong opinion on the above, but I'm very concerned
>     >     about a requirement of a bot pushing to SCM repos.
>     >
>     > Please explain your concern.
>
>     ASF lives and dies by how well it can track IP provenance in what we release.
>     That's why any non-committer interactions around SCM will give me pause.
>
> All commits, even PR's from non-commiters accepted by a committer are supposed to be reviewed, AIUI.  So if the bot makes a commit to the repo, the PMC is responsible for reviewing it.  In Royale's case, the bot should only be changing pom.xml files and making tags and branches, so a bad bot commit should be easy to spot and detection may even be tool-able.

In theory -- yes. In CTR project -- may be not. Either way this is
something I'd be only comfortable
allowing if there's a great benefit of allowing it. Which I'm still
failing to see, frankly.

>     > A bot is already allowed to commit to the website repos, AIUI.
>
>     Two things:
>        1. can you give me real-world examples of that?
>
> See the beginning of this thread.  I posted this link to an old email:
>
> https://lists.apache.org/thread.html/efed1ff44fbfe5770ea1574b2f53a5295ae8326c5a3a5feb9f88cd48@%3Cbuilds.apache.org%3E
>
> And Karl Heinz Marbaise seemed to say that Maven is doing it.
>
> https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven-site/
>
> Also note that in Royale's case, the Jenkins job would not be triggered.  It would be manually started.  So one requirement of allowing packaging jobs could be that artifact packaging jobs cannot be automatically triggered by repo changes or date/time.  That would better ensure that the PMC has reviewed any bot changes.

I'm still not following. Can you link to the commits that originate from a bot?

Thanks,
Roman.

Re: Can we package release artifacts on builds.a.o?

[sebb <se...@gmail.com>]

Regarding automatic commits to websites:

These are very different from code, for at least two reasons:
1) the website is not used as a dependency by other projects (apart
from web indexes, of course)
2) the website source code is not updated by Jenkins, so the next
build will revert any changes.

Re: Can we package release artifacts on builds.a.o?

[Hervé BOUTEMY <he...@free.fr>]

FYI, I managed to build Royale using Maven using Royale instructions [1]
The only missing instruction was about mvn install-file for airglobal-20.0.swc.

I measured what you deploy to the repository: with 206 MB of content for asjs containing a distribution of 2 files of 50 MB each, I can understand that any network issue can cause a failure here and that's unpleasant.

I just added a documentation in maven-deploy-plugin on how to deal with network issues [2]: I'm pretty sure the local staging directory strategy could perfectly match your case.

Hope this helps,

Hervé

[1] https://github.com/apache/royale-asjs/wiki/Build-Apache-Royale-with-Maven

[2] https://maven.apache.org/plugins-archives/maven-deploy-plugin-LATEST/examples/deploy-network-issues.html

Le mercredi 9 janvier 2019, 08:54:10 CET Hervé BOUTEMY a écrit :
> did anybody try the intersting proposal from Christofer (doing like PCL4X)?
> 
> looks promising for Royale
> 
> Regards,
> 
> Hervé
> 
> Le mardi 8 janvier 2019, 02:35:07 CET Hervé BOUTEMY a écrit :
> > yes, given Royale issue looks like network connectivity reliability (and I
> > suppose numerous and large artifacts), deploying to a local file:// based
> > repository then having a pure upload step from file:// rlocal repository
> > to
> > network based staging repository could be a solution that would be less
> > intrusive than changing the full process
> > 
> > I don't have immediately a plugin in mind for such a process but I'll
> > investigate
> > 
> > Regards,
> > 
> > Hervé
> > 
> > Le lundi 7 janvier 2019, 22:18:50 CET Christofer Dutz a écrit :
> > > Hi Alex,
> > > 
> > > Ways to do bad stuff with just a pom.xml:
> > > - simply adding a dependency to a vulnerable library, even an
> > > intentionally
> > > staged malicious one.
> >  
> >  - Adding an evec-maven-plugin to execute anything on
> >  
> > > the host machine - Generate code
> > > - Like I introduced into the FlexJS maven build: Patch/Modify source
> > > files
> > > Guess the is what I could think of in 5 minutes...
> > > 
> > > Ways to do bad stuff by just changing one-line versions:
> > > And changing the version of a dependency to a known vulnerable version
> > > would be such a one-liner.
> >  
> >  I'm currently introducing vulnerability checks into
> >  
> > > all of my builds, so I'm bumping dependencies to unvulnerable versions
> > > ...
> > > doing this the other way around would introduce vulnerabilities.
> > > 
> > > Connectivity problems:
> > > Regarding network problems ... on my way to Montreal I staged the first
> > > RC
> > > for Apache PLC4X in a plane ...
> >  
> >  it took about 3 hours to upload cause of
> >  
> > > network problems and latencies. Maven usually works around connectivity
> > > problems quite nicely and reliably.
> > > So all in all I would suggest you sort out the problems in the build
> > > with
> > > someone with experience.
> >  
> >  I already told Carlos how he could deploy to a
> >  
> > > local directory during the release itself and then use another plugin to
> > > stage that release independently.
> > > If it aborts, you just re-start the deployment and close the repo as
> > > soon
> > > as all passes.
> > > 
> > > Chris
> > > 
> > > Am 07.01.19, 19:39 schrieb "Alex Harui" <ah...@adobe.com.INVALID>:
> > >     Hi Greg,
> > >     
> > >     Thanks for the history.  I agree with the general problem, however,
> > >     for
> > > 
> > > Royale, I think the problem is constrained, but I could be wrong.  I
> > > don't
> > > think there are exploits from things like missing semicolons and other
> > > code
> > > exploits that can be executed against pom.xml files, so the Royale
> > > reviewers are first looking to see if bot changed any other files. 
> > > Maybe
> > > Maven experts can tell us what kinds of exploit could be hacked into a
> > > pom.xml.
> > > 
> > >     Could you answer another question?  What is the current state of
> > >     SVN/Git
> > > 
> > > integration?  Could we spin up an SVN clone of our Git repos, restrict
> > > the
> > > bot via SVN, then sync back from SVN to Git (all from Jenkins)?
> > > 
> > >     Thanks,
> > >     -Alex
> > >     
> > >     On 1/7/19, 10:30 AM, "Greg Stein" <gs...@gmail.com> wrote:
> > >         On Mon, Jan 7, 2019 at 12:23 PM Alex Harui
> > > 
> > > <ah...@adobe.com.invalid> wrote:
> > >         >...
> > >         >
> > >         > I still don't get why allowing a bot to commit to a Git repo
> > >         > isn't
> > >         > auditable.  The changes should all be text and sent to
> > >         > commits@
> > >         > and the
> > >         > RMs job is to verify that those commits are ok before putting
> > >         > the
> > >         > artifacts
> >  
> >  up for vote.  I'd even try to  make an email rule that
> >  
> > >         > checks for commits from buildbot and flags changes to files
> > >         > that
> > >         > are outside of what we expected.
> > >         
> > >         The historic position of the Foundation is "no ability to commit
> > > 
> > > without a
> >  
> >  matched ICLA". That is different from "we'll audit any commits
> >  
> > > made by $bot". The trust meter is rather different between those
> > > positions,
> > > specifically with the "what if nobody reviews? what if a commit is
> > > missed?
> > > what if that added semicolon is missed, yet opens a vuln?" ... With the
> > > "matched ICLA" position, the Foundation has the assurance of *that*
> > > committer, that everything is Good. ... Yet a bot cannot make any such
> > > assurances, despite any "best effort" of the PMC to review the bot's
> > > work.
> > > 
> > >         It is likely a solvable problem! My comments here are to outline
> > >         history/policy, rather than to say "NO". These are just the
> > > 
> > > parameters of
> >  
> >  the problem space.
> >  
> > >         Cheers,
> > >         -g
> > >         InfraAdmin

Re: Can we package release artifacts on builds.a.o?

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Wed, Jan 9, 2019 at 2:00 AM Greg Stein <gs...@gmail.com> wrote:
>
> On Wed, Jan 9, 2019 at 2:12 AM Alex Harui <ah...@adobe.com.invalid> wrote:
>
> > Here's my current summary:
> >
> > I think at least 3 projects are interested in sharing a computer to build
> > all or some of their release artifacts.
> >
> > I don't know who actually wears an Infra hat other than Greg, but his
> > response was maybe.
> >
>
> My answers/responses are official Infra replies. Gavin responded earlier in
> the thread, and is also part of Infra.
>
> But the most important person on this thread, and who will say whether this
> can be allowed, is Roman with his VP Legal Affairs role. In fact, I regard
> Infra as Legal's "hands". Much of what we do is rooted in legal concerns of
> the Foundation. (the other half is keeping shared tools operating)
>
> >...
>
> > these ideas as well.  Maybe they will, maybe they won't.  But to me, the
> > point is that it will be save the community much time and energy if we have
> > one shared machine that can crank artifacts.  Most of us change computers
> > every few years and have to get set up all over again.
>
>
> My non-official reply to the above is "sounds brittle". If you plan to set
> up a single machine to release artifacts, and the machine blows up ... then
> how do you set it up again? Whatever set up is written down (or
> containerized, or scripted, or...) would be the same for each RM, no?
>
> Under this  "do more work" logic, builds.a.o doesn't have to exist.
> > Communities can do more work and set up their own Jenkins somewhere.
> >
>
> Oh, that would be awesome. Jenkins is a headache. We'd love that.
>
> But nah... Infra will maintain our Jenkins installation for the long and
> foreseeable future. Very many projects depend upon it, and that's what
> we're here for: stand up shared tools, and keep them operating.
>
>
> > If someone with an Infra hat on tells me to go away,
>
>
> Nope, I won't say that. But I've tried to share what the official concerns
> are with bot accounts. The short answer is that we can't trace their
> commits to an ICLA, nor restrict their ability to sneak in bad code.
> Reliance upon human review is a non-starter, as there is no audit trail
> that a human *performed* that review.
>
> Thus, Infra/Legal leaves it to you/community to find a solution that meets
> the Foundation's concerns about provenance. For example, maybe the bot just
> creates a PR, for a human to review and merge (thus, we have an ICLA
> matched to the merge of the work).

I see this as one of the workable solutions here.

> Note that git-at-Apache was initially designed and set up by volunteers.
> When the boundaries of what Infra provides needs to be pushed, we like to
> turn to volunteers to drive that. We've got a boundless set of work
> already, so when a project says "I'd like to do $X", then we say "figure
> out how to do it, and then work with us".

Huge +1 to the above.

Thanks,
Roman.

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

On 1/10/19, 4:41 PM, "Dave Fisher" <da...@comcast.net> wrote:

    Hi -
    
    Sent from my iPhone
    
    > On Jan 10, 2019, at 4:18 PM, Roman Shaposhnik <ro...@shaposhnik.org> wrote:
    > 
    >> On Thu, Jan 10, 2019 at 12:45 AM Alex Harui <ah...@adobe.com.invalid> wrote:
    >> 
    >> 
    >> 
    >> On 1/9/19, 7:35 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
    >> 
    >>>    On Wed, Jan 9, 2019 at 11:38 AM Alex Harui <ah...@adobe.com.invalid> wrote:
    >>> 
    >>> Hi Greg,
    >>> 
    >>> You may have missed some other infra-technical questions upthread that might help us fashion a solution.  I'll repeat them here:
    >>> 
    >>> 1) What is the state of Git->SVN and SVN->Git integration?  Could our job clone git to SVN, have the bot make changes in SVN with the additional restrictions as you said SVN could do, then sync back up to Git (including tags as well)?
    >>> 2) What would be the impact of infra creating a "RoyalePMC" committer account?
    >> 
    >>    That is definitely not allowed. PMC members are expected to be human
    >>    beings with ICLAs on file with ASF.
    >> 
    >> The only allowed users of the RoyalePMC account would be human PMC
    >> members (technically, anyone with access to private@royale).
    >> Commits from RoyalePMC would therefore have somebody's ICLA behind it.
    
    As a Royale PMC member I’m not comfortable with being on this hook.
    
    > 
    > Then just do that from under individual accounts.
    
    +1
    
    I would be comfortable with Approved PMC member credentials like are used for the handful of VM sysadmins from the OpenOffice PMC.

Where can I find out more about "Approved PMC member credentials"?
    
    > 
    >> 
    >>    In fact, I would go as far as to say that any PMC member willingly
    >>    disseminating his or her credentials for *others* to use is likely to
    >>    be considered for a an action from the board.
    >> 
    >> I would agree that PMC members should not share their credentials with others,
    >> hence the idea of having a RoyalePMC account, so no human has to share or
    >> transfer credentials to the build machine.   Is it important to know exactly which
    >> individual committed something or just that somebody with an ICLA committed something, and why?
    > 
    > It is very important to establish IP provenance down to an individual.
    
    And accountability.
    
Could someone answer why?

Thanks,
-Alex 

Re: Can we package release artifacts on builds.a.o?

[Dave Fisher <da...@comcast.net>]

Hi -

Sent from my iPhone

> On Jan 10, 2019, at 4:18 PM, Roman Shaposhnik <ro...@shaposhnik.org> wrote:
> 
>> On Thu, Jan 10, 2019 at 12:45 AM Alex Harui <ah...@adobe.com.invalid> wrote:
>> 
>> 
>> 
>> On 1/9/19, 7:35 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
>> 
>>>    On Wed, Jan 9, 2019 at 11:38 AM Alex Harui <ah...@adobe.com.invalid> wrote:
>>> 
>>> Hi Greg,
>>> 
>>> You may have missed some other infra-technical questions upthread that might help us fashion a solution.  I'll repeat them here:
>>> 
>>> 1) What is the state of Git->SVN and SVN->Git integration?  Could our job clone git to SVN, have the bot make changes in SVN with the additional restrictions as you said SVN could do, then sync back up to Git (including tags as well)?
>>> 2) What would be the impact of infra creating a "RoyalePMC" committer account?
>> 
>>    That is definitely not allowed. PMC members are expected to be human
>>    beings with ICLAs on file with ASF.
>> 
>> The only allowed users of the RoyalePMC account would be human PMC
>> members (technically, anyone with access to private@royale).
>> Commits from RoyalePMC would therefore have somebody's ICLA behind it.

As a Royale PMC member I’m not comfortable with being on this hook.

> 
> Then just do that from under individual accounts.

+1

I would be comfortable with Approved PMC member credentials like are used for the handful of VM sysadmins from the OpenOffice PMC.

> 
>> 
>>    In fact, I would go as far as to say that any PMC member willingly
>>    disseminating his or her credentials for *others* to use is likely to
>>    be considered for a an action from the board.
>> 
>> I would agree that PMC members should not share their credentials with others,
>> hence the idea of having a RoyalePMC account, so no human has to share or
>> transfer credentials to the build machine.   Is it important to know exactly which
>> individual committed something or just that somebody with an ICLA committed something, and why?
> 
> It is very important to establish IP provenance down to an individual.

And accountability.

Regards,
Dave


> Thanks,
> Roman.

Re: Can we package release artifacts on builds.a.o?

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Thu, Jan 10, 2019 at 12:45 AM Alex Harui <ah...@adobe.com.invalid> wrote:
>
>
>
> On 1/9/19, 7:35 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
>
>     On Wed, Jan 9, 2019 at 11:38 AM Alex Harui <ah...@adobe.com.invalid> wrote:
>     >
>     > Hi Greg,
>     >
>     > You may have missed some other infra-technical questions upthread that might help us fashion a solution.  I'll repeat them here:
>     >
>     > 1) What is the state of Git->SVN and SVN->Git integration?  Could our job clone git to SVN, have the bot make changes in SVN with the additional restrictions as you said SVN could do, then sync back up to Git (including tags as well)?
>     > 2) What would be the impact of infra creating a "RoyalePMC" committer account?
>
>     That is definitely not allowed. PMC members are expected to be human
>     beings with ICLAs on file with ASF.
>
> The only allowed users of the RoyalePMC account would be human PMC
> members (technically, anyone with access to private@royale).
> Commits from RoyalePMC would therefore have somebody's ICLA behind it.

Then just do that from under individual accounts.

>
>     In fact, I would go as far as to say that any PMC member willingly
>     disseminating his or her credentials for *others* to use is likely to
>     be considered for a an action from the board.
>
> I would agree that PMC members should not share their credentials with others,
> hence the idea of having a RoyalePMC account, so no human has to share or
> transfer credentials to the build machine.   Is it important to know exactly which
> individual committed something or just that somebody with an ICLA committed something, and why?

It is very important to establish IP provenance down to an individual.

Thanks,
Roman.

Re: Can we package release artifacts on builds.a.o?

[Dominik Psenner <dp...@gmail.com>]

On 2019-01-10 11:11, stephen.alan.connolly@gmail.com wrote:
> Yes, but they may not be the people the board has tasked with ensuring that all released source code has been +1'd by at least three PMC members.

A working solution could be that during a release, a release-vote branch 
is maintained and everyone that +1s a release adds and pushes his 
signature to the release binaries to that branch. As soon as there are 
enough +1s to pass the vote, the release-vote branch can then be merged 
to master. When jenkins runs the pipeline on a master branch, the 
Jenkinsfile can then be written such that it builds and uploads the 
binaries to dist, publishes the website etc. In this scenario, write 
operations to the repository are always done by a human. The bot just 
does whatever the human tells him to do.

Re: Can we package release artifacts on builds.a.o?

[Greg Stein <gs...@gmail.com>]

On Thu, Jan 10, 2019 at 4:58 AM Dominik Psenner <dp...@gmail.com> wrote:

> On 2019-01-10 11:49, Stephen Connolly wrote:
> > That would meen, though, that the PMC would need to re-encrypt the file
> > every time the PMC changes or any time a PMC member loses their GPG key
> >
> > Note to self: e.g. see
> >
> http://laurent.bachelier.name/2013/03/gpg-encryption-to-multiple-recipients/
> > for example of how to encrypt a file for multiple recipients.
>
> A even trickier situation is when there's no recipient left (because
> went emeritus, death, ..) to decrypt the credentials. :-) It's wise to
> add new pmc members to recipients as soon as the new pmc member arrives.
>

Should anything along these lines be chosen, this wouldn't be a problem.
Infra would be one of the multiple recipients and/or we could easily
generate new credentials when the old creds are lost to stagnation.

Cheers,
-g

Re: Can we package release artifacts on builds.a.o?

[Dominik Psenner <dp...@gmail.com>]

On 2019-01-10 11:49, Stephen Connolly wrote:
> That would meen, though, that the PMC would need to re-encrypt the file
> every time the PMC changes or any time a PMC member loses their GPG key
>
> Note to self: e.g. see
> http://laurent.bachelier.name/2013/03/gpg-encryption-to-multiple-recipients/
> for example of how to encrypt a file for multiple recipients.

A even trickier situation is when there's no recipient left (because 
went emeritus, death, ..) to decrypt the credentials. :-) It's wise to 
add new pmc members to recipients as soon as the new pmc member arrives.

Re: Can we package release artifacts on builds.a.o?

[Stephen Connolly <st...@gmail.com>]

On Thu, 10 Jan 2019 at 10:44, Stephen Connolly <
stephen.alan.connolly@gmail.com> wrote:

> On Thu, 10 Jan 2019 at 10:31, Dominik Psenner <dp...@gmail.com> wrote:
>
>> On 2019-01-10 11:24, Alex Harui wrote:
>> > Stephen are you saying that we can't trust ASF Members?  That we have
>> to fear that at least one ASF member will not be able to resist the urge to
>> leverage the RoyalePMC account for evil?
>> > I'm sure we can find some other way to distribute credentials if that's
>> true, but I would think there are juicier targets for a rogue ASF member,
>> like leveraging Jenkins.
>>
>> -1, credentials are confidential. Credentials may be committed to a
>> repository to prevent accidental deletion, but shall be gpg encrypted to
>> the recipients who are allowed to read them. This implies that a bot is
>> never going to be able to decrypt those credentials.
>>
>>
> Thanks for that. I had forgotten that one could GPG encrypt the
> credentials that would be committed to /private/... so at least that would
> mean that only the intended recipients would be able to decrypt them which
> would limit the secrets to the Royale PMC.
>

That would meen, though, that the PMC would need to re-encrypt the file
every time the PMC changes or any time a PMC member loses their GPG key

Note to self: e.g. see
http://laurent.bachelier.name/2013/03/gpg-encryption-to-multiple-recipients/
for example of how to encrypt a file for multiple recipients.

Re: Can we package release artifacts on builds.a.o?

[Stephen Connolly <st...@gmail.com>]

On Thu, 10 Jan 2019 at 10:31, Dominik Psenner <dp...@gmail.com> wrote:

> On 2019-01-10 11:24, Alex Harui wrote:
> > Stephen are you saying that we can't trust ASF Members?  That we have to
> fear that at least one ASF member will not be able to resist the urge to
> leverage the RoyalePMC account for evil?
> > I'm sure we can find some other way to distribute credentials if that's
> true, but I would think there are juicier targets for a rogue ASF member,
> like leveraging Jenkins.
>
> -1, credentials are confidential. Credentials may be committed to a
> repository to prevent accidental deletion, but shall be gpg encrypted to
> the recipients who are allowed to read them. This implies that a bot is
> never going to be able to decrypt those credentials.
>
>
Thanks for that. I had forgotten that one could GPG encrypt the credentials
that would be committed to /private/... so at least that would mean that
only the intended recipients would be able to decrypt them which would
limit the secrets to the Royale PMC.

Re: Can we package release artifacts on builds.a.o?

[Dominik Psenner <dp...@gmail.com>]

On 2019-01-10 11:24, Alex Harui wrote:
> Stephen are you saying that we can't trust ASF Members?  That we have to fear that at least one ASF member will not be able to resist the urge to leverage the RoyalePMC account for evil?
> I'm sure we can find some other way to distribute credentials if that's true, but I would think there are juicier targets for a rogue ASF member, like leveraging Jenkins.

-1, credentials are confidential. Credentials may be committed to a 
repository to prevent accidental deletion, but shall be gpg encrypted to 
the recipients who are allowed to read them. This implies that a bot is 
never going to be able to decrypt those credentials.

Re: Can we package release artifacts on builds.a.o?

[Stephen Connolly <st...@gmail.com>]

On Thu, 10 Jan 2019 at 10:24, Alex Harui <ah...@adobe.com.invalid> wrote:

> Stephen are you saying that we can't trust ASF Members?  That we have to
> fear that at least one ASF member will not be able to resist the urge to
> leverage the RoyalePMC account for evil?
> I'm sure we can find some other way to distribute credentials if that's
> true, but I would think there are juicier targets for a rogue ASF member,
> like leveraging Jenkins.
>
> My 2 cents,
> -Alex
>

I think when you have grown above ~500 people you have to acknowledge the
potential for rogue actors. That does not mean there currently are any...
but if we are building systems that have legal requirements to meet, we
cannot eschew the possibility of rogue actors just because we hope everyone
is good.

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

Stephen are you saying that we can't trust ASF Members?  That we have to fear that at least one ASF member will not be able to resist the urge to leverage the RoyalePMC account for evil?
I'm sure we can find some other way to distribute credentials if that's true, but I would think there are juicier targets for a rogue ASF member, like leveraging Jenkins.

My 2 cents,
-Alex

On 1/10/19, 2:11 AM, "stephen.alan.connolly@" <gmail.com stephen.alan.connolly@gmail.com> wrote:

    
    
    On 2019/01/10 08:45:19, Alex Harui <ah...@adobe.com.INVALID> wrote: 
    > 
    > 
    > On 1/9/19, 7:35 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
    > 
    >     On Wed, Jan 9, 2019 at 11:38 AM Alex Harui <ah...@adobe.com.invalid> wrote:
    >     >
    >     > Hi Greg,
    >     >
    >     > You may have missed some other infra-technical questions upthread that might help us fashion a solution.  I'll repeat them here:
    >     >
    >     > 1) What is the state of Git->SVN and SVN->Git integration?  Could our job clone git to SVN, have the bot make changes in SVN with the additional restrictions as you said SVN could do, then sync back up to Git (including tags as well)?
    >     > 2) What would be the impact of infra creating a "RoyalePMC" committer account?
    >     
    >     That is definitely not allowed. PMC members are expected to be human
    >     beings with ICLAs on file with ASF.
    >     
    > The only allowed users of the RoyalePMC account would be human PMC members (technically, anyone with access to private@royale
    
    That's quite a lot of people: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Ffoundation%2Fmembers.html&amp;data=02%7C01%7Caharui%40adobe.com%7C33f7cbec128f4660474e08d676e3fe7b%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636827118935854043&amp;sdata=gqZuG0b6LJIw2%2BoH6AB0v9ejpr4OSeIyJubN48%2F6ux0%3D&amp;reserved=0
    
    Or had you forgotten that all members of the ASF implicitly have access to all TLP's private@ mailing lists?
    
    Now you might be able to put stuff in the project's private section of Subversion... but IIRC all commits there are notified back to private@ so that wouldn't deter somebody else.
    
    > ).  Commits from RoyalePMC would therefore have somebody's ICLA behind it.
    
    Yes, but they may not be the people the board has tasked with ensuring that all released source code has been +1'd by at least three PMC members.
    
    > 
    >     In fact, I would go as far as to say that any PMC member willingly
    >     disseminating his or her credentials for *others* to use is likely to
    >     be considered for a an action from the board.
    >     
    > I would agree that PMC members should not share their credentials with others, hence the idea of having a RoyalePMC account, so no human has to share or transfer credentials to the build machine.   Is it important to know exactly which individual committed something or just that somebody with an ICLA committed something, and why?
    > 
    > Thanks,
    > -Alex
    > 
    > 
    

Re: Can we package release artifacts on builds.a.o?

On 2019/01/10 08:45:19, Alex Harui <ah...@adobe.com.INVALID> wrote: 
> 
> 
> On 1/9/19, 7:35 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
> 
>     On Wed, Jan 9, 2019 at 11:38 AM Alex Harui <ah...@adobe.com.invalid> wrote:
>     >
>     > Hi Greg,
>     >
>     > You may have missed some other infra-technical questions upthread that might help us fashion a solution.  I'll repeat them here:
>     >
>     > 1) What is the state of Git->SVN and SVN->Git integration?  Could our job clone git to SVN, have the bot make changes in SVN with the additional restrictions as you said SVN could do, then sync back up to Git (including tags as well)?
>     > 2) What would be the impact of infra creating a "RoyalePMC" committer account?
>     
>     That is definitely not allowed. PMC members are expected to be human
>     beings with ICLAs on file with ASF.
>     
> The only allowed users of the RoyalePMC account would be human PMC members (technically, anyone with access to private@royale

That's quite a lot of people: https://www.apache.org/foundation/members.html

Or had you forgotten that all members of the ASF implicitly have access to all TLP's private@ mailing lists?

Now you might be able to put stuff in the project's private section of Subversion... but IIRC all commits there are notified back to private@ so that wouldn't deter somebody else.

> ).  Commits from RoyalePMC would therefore have somebody's ICLA behind it.

Yes, but they may not be the people the board has tasked with ensuring that all released source code has been +1'd by at least three PMC members.

> 
>     In fact, I would go as far as to say that any PMC member willingly
>     disseminating his or her credentials for *others* to use is likely to
>     be considered for a an action from the board.
>     
> I would agree that PMC members should not share their credentials with others, hence the idea of having a RoyalePMC account, so no human has to share or transfer credentials to the build machine.   Is it important to know exactly which individual committed something or just that somebody with an ICLA committed something, and why?
> 
> Thanks,
> -Alex
> 
> 

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

On 1/9/19, 7:35 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:

    On Wed, Jan 9, 2019 at 11:38 AM Alex Harui <ah...@adobe.com.invalid> wrote:
    >
    > Hi Greg,
    >
    > You may have missed some other infra-technical questions upthread that might help us fashion a solution.  I'll repeat them here:
    >
    > 1) What is the state of Git->SVN and SVN->Git integration?  Could our job clone git to SVN, have the bot make changes in SVN with the additional restrictions as you said SVN could do, then sync back up to Git (including tags as well)?
    > 2) What would be the impact of infra creating a "RoyalePMC" committer account?
    
    That is definitely not allowed. PMC members are expected to be human
    beings with ICLAs on file with ASF.
    
The only allowed users of the RoyalePMC account would be human PMC members (technically, anyone with access to private@royale).  Commits from RoyalePMC would therefore have somebody's ICLA behind it.

    In fact, I would go as far as to say that any PMC member willingly
    disseminating his or her credentials for *others* to use is likely to
    be considered for a an action from the board.
    
I would agree that PMC members should not share their credentials with others, hence the idea of having a RoyalePMC account, so no human has to share or transfer credentials to the build machine.   Is it important to know exactly which individual committed something or just that somebody with an ICLA committed something, and why?

Thanks,
-Alex

Re: Can we package release artifacts on builds.a.o?

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Wed, Jan 9, 2019 at 11:38 AM Alex Harui <ah...@adobe.com.invalid> wrote:
>
> Hi Greg,
>
> You may have missed some other infra-technical questions upthread that might help us fashion a solution.  I'll repeat them here:
>
> 1) What is the state of Git->SVN and SVN->Git integration?  Could our job clone git to SVN, have the bot make changes in SVN with the additional restrictions as you said SVN could do, then sync back up to Git (including tags as well)?
> 2) What would be the impact of infra creating a "RoyalePMC" committer account?

That is definitely not allowed. PMC members are expected to be human
beings with ICLAs on file with ASF.

In fact, I would go as far as to say that any PMC member willingly
disseminating his or her credentials for *others* to use is likely to
be considered for a an action from the board.

Thanks,
Roman.

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

Hi Greg,

You may have missed some other infra-technical questions upthread that might help us fashion a solution.  I'll repeat them here:

1) What is the state of Git->SVN and SVN->Git integration?  Could our job clone git to SVN, have the bot make changes in SVN with the additional restrictions as you said SVN could do, then sync back up to Git (including tags as well)?
2) What would be the impact of infra creating a "RoyalePMC" committer account?  The passwords and keys would be shared on private@royale.  I would then set up my Windows Azure VM with that account and crank artifacts from there.  Then you will know there is an ICLA behind each commit, but you may not know which ICLA, although it might be deduced from who promised to be the RM around that time.

Thanks,
-Alex

On 1/9/19, 2:00 AM, "Greg Stein" <gs...@gmail.com> wrote:

    On Wed, Jan 9, 2019 at 2:12 AM Alex Harui <ah...@adobe.com.invalid> wrote:
    
    > Here's my current summary:
    >
    > I think at least 3 projects are interested in sharing a computer to build
    > all or some of their release artifacts.
    >
    > I don't know who actually wears an Infra hat other than Greg, but his
    > response was maybe.
    >
    
    My answers/responses are official Infra replies. Gavin responded earlier in
    the thread, and is also part of Infra.
    
    But the most important person on this thread, and who will say whether this
    can be allowed, is Roman with his VP Legal Affairs role. In fact, I regard
    Infra as Legal's "hands". Much of what we do is rooted in legal concerns of
    the Foundation. (the other half is keeping shared tools operating)
    
    >...
    
    > these ideas as well.  Maybe they will, maybe they won't.  But to me, the
    > point is that it will be save the community much time and energy if we have
    > one shared machine that can crank artifacts.  Most of us change computers
    > every few years and have to get set up all over again.
    
    
    My non-official reply to the above is "sounds brittle". If you plan to set
    up a single machine to release artifacts, and the machine blows up ... then
    how do you set it up again? Whatever set up is written down (or
    containerized, or scripted, or...) would be the same for each RM, no?
    
    Under this  "do more work" logic, builds.a.o doesn't have to exist.
    > Communities can do more work and set up their own Jenkins somewhere.
    >
    
    Oh, that would be awesome. Jenkins is a headache. We'd love that.
    
    But nah... Infra will maintain our Jenkins installation for the long and
    foreseeable future. Very many projects depend upon it, and that's what
    we're here for: stand up shared tools, and keep them operating.
    
    
    > If someone with an Infra hat on tells me to go away,
    
    
    Nope, I won't say that. But I've tried to share what the official concerns
    are with bot accounts. The short answer is that we can't trace their
    commits to an ICLA, nor restrict their ability to sneak in bad code.
    Reliance upon human review is a non-starter, as there is no audit trail
    that a human *performed* that review.
    
    Thus, Infra/Legal leaves it to you/community to find a solution that meets
    the Foundation's concerns about provenance. For example, maybe the bot just
    creates a PR, for a human to review and merge (thus, we have an ICLA
    matched to the merge of the work).
    
    Note that git-at-Apache was initially designed and set up by volunteers.
    When the boundaries of what Infra provides needs to be pushed, we like to
    turn to volunteers to drive that. We've got a boundless set of work
    already, so when a project says "I'd like to do $X", then we say "figure
    out how to do it, and then work with us".
    
    Cheers,
    Greg
    InfraAdmin, ASF
    

Re: Can we package release artifacts on builds.a.o?

[Greg Stein <gs...@gmail.com>]

On Wed, Jan 9, 2019 at 2:12 AM Alex Harui <ah...@adobe.com.invalid> wrote:

> Here's my current summary:
>
> I think at least 3 projects are interested in sharing a computer to build
> all or some of their release artifacts.
>
> I don't know who actually wears an Infra hat other than Greg, but his
> response was maybe.
>

My answers/responses are official Infra replies. Gavin responded earlier in
the thread, and is also part of Infra.

But the most important person on this thread, and who will say whether this
can be allowed, is Roman with his VP Legal Affairs role. In fact, I regard
Infra as Legal's "hands". Much of what we do is rooted in legal concerns of
the Foundation. (the other half is keeping shared tools operating)

>...

> these ideas as well.  Maybe they will, maybe they won't.  But to me, the
> point is that it will be save the community much time and energy if we have
> one shared machine that can crank artifacts.  Most of us change computers
> every few years and have to get set up all over again.


My non-official reply to the above is "sounds brittle". If you plan to set
up a single machine to release artifacts, and the machine blows up ... then
how do you set it up again? Whatever set up is written down (or
containerized, or scripted, or...) would be the same for each RM, no?

Under this  "do more work" logic, builds.a.o doesn't have to exist.
> Communities can do more work and set up their own Jenkins somewhere.
>

Oh, that would be awesome. Jenkins is a headache. We'd love that.

But nah... Infra will maintain our Jenkins installation for the long and
foreseeable future. Very many projects depend upon it, and that's what
we're here for: stand up shared tools, and keep them operating.


> If someone with an Infra hat on tells me to go away,


Nope, I won't say that. But I've tried to share what the official concerns
are with bot accounts. The short answer is that we can't trace their
commits to an ICLA, nor restrict their ability to sneak in bad code.
Reliance upon human review is a non-starter, as there is no audit trail
that a human *performed* that review.

Thus, Infra/Legal leaves it to you/community to find a solution that meets
the Foundation's concerns about provenance. For example, maybe the bot just
creates a PR, for a human to review and merge (thus, we have an ICLA
matched to the merge of the work).

Note that git-at-Apache was initially designed and set up by volunteers.
When the boundaries of what Infra provides needs to be pushed, we like to
turn to volunteers to drive that. We've got a boundless set of work
already, so when a project says "I'd like to do $X", then we say "figure
out how to do it, and then work with us".

Cheers,
Greg
InfraAdmin, ASF

Re: Can we package release artifacts on builds.a.o?

[Justin Mclean <ju...@classsoftware.com>]

Hi,

The risk of a number of things happening may be minimal, however the damage that they could cause could be large, traditionally risk management considers both the the chance of the risk occurring and what the outcome would be if it did. Just because it’s unlikely to occur doesn’t mean it’s a safe thing to do. I often use a risk matrix like this when assessing risk. [1]

Thanks,
Justin

1. https://en.wikipedia.org/wiki/Risk_matrix

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

Here's my current summary:

I think at least 3 projects are interested in sharing a computer to build all or some of their release artifacts.

I don't know who actually wears an Infra hat other than Greg, but his response was maybe.

Other feedback seems to be "if you do more work, then we don't have to ask infra for anything".  I don't know if that is an Infra response or not.  Of course we could do more work.  The 3 RMs that tried and gave up didn't have to give up.  But they did because they don't have the time or desire to keep figuring out why something didn't work for them.  They could explore these ideas as well.  Maybe they will, maybe they won't.  But to me, the point is that it will be save the community much time and energy if we have one shared machine that can crank artifacts.  Most of us change computers every few years and have to get set up all over again.  Under this  "do more work" logic, builds.a.o doesn't have to exist.  Communities can do more work and set up their own Jenkins somewhere.

If someone with an Infra hat on tells me to go away, I will, but all of the responses that amount to "do more work" really seem tangential.  Either there is going to be too much risk to allowing shared machines to commit code from inside or outside of a.o, and either it will be a huge amount of work for Infra or it won't.

Can we get back to discussing the risk and work aspects?  I think the risk is minimal.  Someone would have to time the rogue changes so that it coincides with other desired changes and we would have to miss those changes in the review.  What is the amount of work?  Is it major work to add accounts and keys for a non-person?

Thanks,
-Alex

On 1/8/19, 11:54 PM, "Hervé BOUTEMY" <he...@free.fr> wrote:

    did anybody try the intersting proposal from Christofer (doing like PCL4X)?
    
    looks promising for Royale
    
    Regards,
    
    Hervé
    
    Le mardi 8 janvier 2019, 02:35:07 CET Hervé BOUTEMY a écrit :
    > yes, given Royale issue looks like network connectivity reliability (and I
    > suppose numerous and large artifacts), deploying to a local file:// based
    > repository then having a pure upload step from file:// rlocal repository to
    > network based staging repository could be a solution that would be less
    > intrusive than changing the full process
    > 
    > I don't have immediately a plugin in mind for such a process but I'll
    > investigate
    > 
    > Regards,
    > 
    > Hervé
    > 
    > Le lundi 7 janvier 2019, 22:18:50 CET Christofer Dutz a écrit :
    > > Hi Alex,
    > > 
    > > Ways to do bad stuff with just a pom.xml:
    > > - simply adding a dependency to a vulnerable library, even an
    > > intentionally
    > > staged malicious one.
    > 
    >  - Adding an evec-maven-plugin to execute anything on
    > 
    > > the host machine - Generate code
    > > - Like I introduced into the FlexJS maven build: Patch/Modify source files
    > > Guess the is what I could think of in 5 minutes...
    > > 
    > > Ways to do bad stuff by just changing one-line versions:
    > > And changing the version of a dependency to a known vulnerable version
    > > would be such a one-liner.
    > 
    >  I'm currently introducing vulnerability checks into
    > 
    > > all of my builds, so I'm bumping dependencies to unvulnerable versions ...
    > > doing this the other way around would introduce vulnerabilities.
    > > 
    > > Connectivity problems:
    > > Regarding network problems ... on my way to Montreal I staged the first RC
    > > for Apache PLC4X in a plane ...
    > 
    >  it took about 3 hours to upload cause of
    > 
    > > network problems and latencies. Maven usually works around connectivity
    > > problems quite nicely and reliably.
    > > So all in all I would suggest you sort out the problems in the build with
    > > someone with experience.
    > 
    >  I already told Carlos how he could deploy to a
    > 
    > > local directory during the release itself and then use another plugin to
    > > stage that release independently.
    > > If it aborts, you just re-start the deployment and close the repo as soon
    > > as all passes.
    > > 
    > > Chris
    > > 
    > > Am 07.01.19, 19:39 schrieb "Alex Harui" <ah...@adobe.com.INVALID>:
    > >     Hi Greg,
    > >     
    > >     Thanks for the history.  I agree with the general problem, however,
    > >     for
    > > 
    > > Royale, I think the problem is constrained, but I could be wrong.  I don't
    > > think there are exploits from things like missing semicolons and other
    > > code
    > > exploits that can be executed against pom.xml files, so the Royale
    > > reviewers are first looking to see if bot changed any other files.  Maybe
    > > Maven experts can tell us what kinds of exploit could be hacked into a
    > > pom.xml.
    > > 
    > >     Could you answer another question?  What is the current state of
    > >     SVN/Git
    > > 
    > > integration?  Could we spin up an SVN clone of our Git repos, restrict the
    > > bot via SVN, then sync back from SVN to Git (all from Jenkins)?
    > > 
    > >     Thanks,
    > >     -Alex
    > >     
    > >     On 1/7/19, 10:30 AM, "Greg Stein" <gs...@gmail.com> wrote:
    > >         On Mon, Jan 7, 2019 at 12:23 PM Alex Harui
    > > 
    > > <ah...@adobe.com.invalid> wrote:
    > >         >...
    > >         >
    > >         > I still don't get why allowing a bot to commit to a Git repo
    > >         > isn't
    > >         > auditable.  The changes should all be text and sent to commits@
    > >         > and the
    > >         > RMs job is to verify that those commits are ok before putting
    > >         > the
    > >         > artifacts
    > 
    >  up for vote.  I'd even try to  make an email rule that
    > 
    > >         > checks for commits from buildbot and flags changes to files that
    > >         > are outside of what we expected.
    > >         
    > >         The historic position of the Foundation is "no ability to commit
    > > 
    > > without a
    > 
    >  matched ICLA". That is different from "we'll audit any commits
    > 
    > > made by $bot". The trust meter is rather different between those
    > > positions,
    > > specifically with the "what if nobody reviews? what if a commit is missed?
    > > what if that added semicolon is missed, yet opens a vuln?" ... With the
    > > "matched ICLA" position, the Foundation has the assurance of *that*
    > > committer, that everything is Good. ... Yet a bot cannot make any such
    > > assurances, despite any "best effort" of the PMC to review the bot's work.
    > > 
    > >         It is likely a solvable problem! My comments here are to outline
    > >         history/policy, rather than to say "NO". These are just the
    > > 
    > > parameters of
    > 
    >  the problem space.
    > 
    > >         Cheers,
    > >         -g
    > >         InfraAdmin
    
    
    
    
    

Re: Can we package release artifacts on builds.a.o?

[Hervé BOUTEMY <he...@free.fr>]

did anybody try the intersting proposal from Christofer (doing like PCL4X)?

looks promising for Royale

Regards,

Hervé

Le mardi 8 janvier 2019, 02:35:07 CET Hervé BOUTEMY a écrit :
> yes, given Royale issue looks like network connectivity reliability (and I
> suppose numerous and large artifacts), deploying to a local file:// based
> repository then having a pure upload step from file:// rlocal repository to
> network based staging repository could be a solution that would be less
> intrusive than changing the full process
> 
> I don't have immediately a plugin in mind for such a process but I'll
> investigate
> 
> Regards,
> 
> Hervé
> 
> Le lundi 7 janvier 2019, 22:18:50 CET Christofer Dutz a écrit :
> > Hi Alex,
> > 
> > Ways to do bad stuff with just a pom.xml:
> > - simply adding a dependency to a vulnerable library, even an
> > intentionally
> > staged malicious one.
> 
>  - Adding an evec-maven-plugin to execute anything on
> 
> > the host machine - Generate code
> > - Like I introduced into the FlexJS maven build: Patch/Modify source files
> > Guess the is what I could think of in 5 minutes...
> > 
> > Ways to do bad stuff by just changing one-line versions:
> > And changing the version of a dependency to a known vulnerable version
> > would be such a one-liner.
> 
>  I'm currently introducing vulnerability checks into
> 
> > all of my builds, so I'm bumping dependencies to unvulnerable versions ...
> > doing this the other way around would introduce vulnerabilities.
> > 
> > Connectivity problems:
> > Regarding network problems ... on my way to Montreal I staged the first RC
> > for Apache PLC4X in a plane ...
> 
>  it took about 3 hours to upload cause of
> 
> > network problems and latencies. Maven usually works around connectivity
> > problems quite nicely and reliably.
> > So all in all I would suggest you sort out the problems in the build with
> > someone with experience.
> 
>  I already told Carlos how he could deploy to a
> 
> > local directory during the release itself and then use another plugin to
> > stage that release independently.
> > If it aborts, you just re-start the deployment and close the repo as soon
> > as all passes.
> > 
> > Chris
> > 
> > Am 07.01.19, 19:39 schrieb "Alex Harui" <ah...@adobe.com.INVALID>:
> >     Hi Greg,
> >     
> >     Thanks for the history.  I agree with the general problem, however,
> >     for
> > 
> > Royale, I think the problem is constrained, but I could be wrong.  I don't
> > think there are exploits from things like missing semicolons and other
> > code
> > exploits that can be executed against pom.xml files, so the Royale
> > reviewers are first looking to see if bot changed any other files.  Maybe
> > Maven experts can tell us what kinds of exploit could be hacked into a
> > pom.xml.
> > 
> >     Could you answer another question?  What is the current state of
> >     SVN/Git
> > 
> > integration?  Could we spin up an SVN clone of our Git repos, restrict the
> > bot via SVN, then sync back from SVN to Git (all from Jenkins)?
> > 
> >     Thanks,
> >     -Alex
> >     
> >     On 1/7/19, 10:30 AM, "Greg Stein" <gs...@gmail.com> wrote:
> >         On Mon, Jan 7, 2019 at 12:23 PM Alex Harui
> > 
> > <ah...@adobe.com.invalid> wrote:
> >         >...
> >         >
> >         > I still don't get why allowing a bot to commit to a Git repo
> >         > isn't
> >         > auditable.  The changes should all be text and sent to commits@
> >         > and the
> >         > RMs job is to verify that those commits are ok before putting
> >         > the
> >         > artifacts
> 
>  up for vote.  I'd even try to  make an email rule that
> 
> >         > checks for commits from buildbot and flags changes to files that
> >         > are outside of what we expected.
> >         
> >         The historic position of the Foundation is "no ability to commit
> > 
> > without a
> 
>  matched ICLA". That is different from "we'll audit any commits
> 
> > made by $bot". The trust meter is rather different between those
> > positions,
> > specifically with the "what if nobody reviews? what if a commit is missed?
> > what if that added semicolon is missed, yet opens a vuln?" ... With the
> > "matched ICLA" position, the Foundation has the assurance of *that*
> > committer, that everything is Good. ... Yet a bot cannot make any such
> > assurances, despite any "best effort" of the PMC to review the bot's work.
> > 
> >         It is likely a solvable problem! My comments here are to outline
> >         history/policy, rather than to say "NO". These are just the
> > 
> > parameters of
> 
>  the problem space.
> 
> >         Cheers,
> >         -g
> >         InfraAdmin

Re: Can we package release artifacts on builds.a.o?

[Hervé BOUTEMY <he...@free.fr>]

yes, given Royale issue looks like network connectivity reliability (and I 
suppose numerous and large artifacts), deploying to a local file:// based 
repository then having a pure upload step from file:// rlocal repository to 
network based staging repository could be a solution that would be less 
intrusive than changing the full process

I don't have immediately a plugin in mind for such a process but I'll 
investigate

Regards,

Hervé

Le lundi 7 janvier 2019, 22:18:50 CET Christofer Dutz a écrit :
> Hi Alex,
> 
> Ways to do bad stuff with just a pom.xml:
> - simply adding a dependency to a vulnerable library, even an intentionally
> staged malicious one.
 - Adding an evec-maven-plugin to execute anything on
> the host machine - Generate code
> - Like I introduced into the FlexJS maven build: Patch/Modify source files
> Guess the is what I could think of in 5 minutes...
> 
> Ways to do bad stuff by just changing one-line versions:
> And changing the version of a dependency to a known vulnerable version would
> be such a one-liner.
 I'm currently introducing vulnerability checks into
> all of my builds, so I'm bumping dependencies to unvulnerable versions ...
> doing this the other way around would introduce vulnerabilities.
> 
> Connectivity problems:
> Regarding network problems ... on my way to Montreal I staged the first RC
> for Apache PLC4X in a plane ... 
 it took about 3 hours to upload cause of
> network problems and latencies. Maven usually works around connectivity
> problems quite nicely and reliably. 
> So all in all I would suggest you sort out the problems in the build with
> someone with experience. 
 I already told Carlos how he could deploy to a
> local directory during the release itself and then use another plugin to
> stage that release independently.
> If it aborts, you just re-start the deployment and close the repo as soon as
> all passes.
 
> Chris
> 
> 
> 
> Am 07.01.19, 19:39 schrieb "Alex Harui" <ah...@adobe.com.INVALID>:
> 
>     Hi Greg,
>     
>     Thanks for the history.  I agree with the general problem, however, for
> Royale, I think the problem is constrained, but I could be wrong.  I don't
> think there are exploits from things like missing semicolons and other code
> exploits that can be executed against pom.xml files, so the Royale
> reviewers are first looking to see if bot changed any other files.  Maybe
> Maven experts can tell us what kinds of exploit could be hacked into a
> pom.xml.
 
>     Could you answer another question?  What is the current state of SVN/Git
> integration?  Could we spin up an SVN clone of our Git repos, restrict the
> bot via SVN, then sync back from SVN to Git (all from Jenkins)?
 
>     Thanks,
>     -Alex
>     
>     On 1/7/19, 10:30 AM, "Greg Stein" <gs...@gmail.com> wrote:
>     
>         On Mon, Jan 7, 2019 at 12:23 PM Alex Harui
> <ah...@adobe.com.invalid> wrote:
>         >...
> 
>         
> 
>         > I still don't get why allowing a bot to commit to a Git repo
>         > isn't
>         > auditable.  The changes should all be text and sent to commits@
>         > and the
>         > RMs job is to verify that those commits are ok before putting the
>         > artifacts
 up for vote.  I'd even try to  make an email rule that
>         > checks for commits from buildbot and flags changes to files that
>         > are outside of what we expected.
>         >
>         >
> 
>         
>         The historic position of the Foundation is "no ability to commit
> without a
 matched ICLA". That is different from "we'll audit any commits
> made by $bot". The trust meter is rather different between those positions,
> specifically with the "what if nobody reviews? what if a commit is missed?
> what if that added semicolon is missed, yet opens a vuln?" ... With the
> "matched ICLA" position, the Foundation has the assurance of *that*
> committer, that everything is Good. ... Yet a bot cannot make any such
> assurances, despite any "best effort" of the PMC to review the bot's work. 
>         It is likely a solvable problem! My comments here are to outline
>         history/policy, rather than to say "NO". These are just the
> parameters of
 the problem space.
>         
>         Cheers,
>         -g
>         InfraAdmin
>         
>     
>     
> 

Re: Can we package release artifacts on builds.a.o?

[Mike Jumper <mj...@apache.org>]

On Mon, Jan 7, 2019 at 11:00 AM Alex Harui <ah...@adobe.com.invalid> wrote:

> Hi Mike,
>
> Thanks for the input.  IMO, that exploit would be easily seen.


Indeed, but setting readability of the example aside:

In the context of your question, no - it's not sufficient to verify that
only pom.xml was modified. A change to pom.xml may well introduce something
malicious and the nature of the changes needs to be reviewed, ideally by
someone who would spot such things.

The release plugin should only be changing one-liners with version
> numbers.  Can you think of one-liner attacks?


Off the top of my head, a change in plugin/dependency could cause the build
to inherit a vulnerability from elsewhere, but I don't think we need to go
that far here. Changes to code can't be trusted without human review.

- Mike

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

On 1/7/19, 1:21 PM, "Allen Wittenauer" <aw...@effectivemachines.com.INVALID> wrote:

    
    
    > On Jan 7, 2019, at 11:50 AM, Alex Harui <ah...@adobe.com.INVALID> wrote:
    > 
    > I don't understand.  Who am I "making" do what work?  And why do at least 3 others want something similar?  And what would you propose Royale should do instead?  Always have me cut releases?
    
    	If their computers are broken, they could always spin up a free micro instance on AWS. Cut the release.  Then spin it down.
    
    	But it really makes me wonder how they are developing and testing their changes locally if building is such a burden ...
    
Building the source code generally works for people.  The RM's are having trouble pushing the built and packaged artifacts to Nexus.

Fundamentally, I am trying to make it easier for our future RM's to cut a release.  Adding more manual steps by spinning up AWS instances or manually pushing artifacts doesn't seem to make it easier.  I thought it would be ok to ask Infra if we could let a bot make some minor changes to pom.xml files that are audited before the release is even voted on.  Some helpful folks have given us things to watch out for in those changes.  I would rather invest time in provisioning a shared machine than to try to get people to help debug intermittent network issues.  Folks who are interested in helping us debug those network issues are welcome to engage the Royale community and the RMs who had the problems.

To try to summarize where I think we are, the main concern is that someone will put together a Jenkins task to inject changes to Royale's pom.xml files and time it with the running of our release job and hope that our reviewers miss those changes.  I agree there is risk there, but is the risk there significant?  If you could run anything on Jenkins would you really attack Royale or run something else and attack a much more lucrative target?

Greg pointed out the connection between an ICLA and a commit.  That made me wonder: could we get a RoyalePMC committer account?  The password and keys would be shared on the private@ list.  I could then provision this shared computer on the Azure Windows VM I have as part of the free MSDN subscription from Microsoft.  Builds@ would be better since it isn't subject to whether I continue to get this benefit from Microsoft, but if folks are too concerned that attackers can run jobs on builds@, what would it take to convince folks that nobody can run a job on my Azure VM and more than they could remote in to my laptop and do the same while I'm sleeping?   We do not run PR tests on that VM.  And any commits from this VM would come from a PMC member so there is an ICLA on behind it, although you may not know which PMC member ran the job.

Thanks,
-Alex

Re: Can we package release artifacts on builds.a.o?

[Allen Wittenauer <aw...@effectivemachines.com.INVALID>]

> On Jan 7, 2019, at 11:50 AM, Alex Harui <ah...@adobe.com.INVALID> wrote:
> 
> I don't understand.  Who am I "making" do what work?  And why do at least 3 others want something similar?  And what would you propose Royale should do instead?  Always have me cut releases?

	If their computers are broken, they could always spin up a free micro instance on AWS. Cut the release.  Then spin it down.

	But it really makes me wonder how they are developing and testing their changes locally if building is such a burden ...

Re: Can we package release artifacts on builds.a.o?

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Mon, Jan 7, 2019 at 11:51 AM Alex Harui <ah...@adobe.com.invalid> wrote:
>
>
>
> On 1/7/19, 11:38 AM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
>
>     On Mon, Jan 7, 2019 at 11:33 AM Alex Harui <ah...@adobe.com.invalid> wrote:
>     >
>     >
>     >
>     > On 1/7/19, 11:05 AM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
>     >
>     >     On Mon, Jan 7, 2019 at 11:00 AM Alex Harui <ah...@adobe.com.invalid> wrote:
>     >     >
>     >     > Hi Mike,
>     >     >
>     >     > Thanks for the input.  IMO, that exploit would be easily seen.  The release plugin should only be changing one-liners with version numbers.
>     >
>     >     If this is all it ever does I still don't understand why a human
>     >     operator won't suffice.
>     >
>     > Because our humans have been unable to get their computers to work.
>     > They seem to be having network reliability issues. So if I can provision
>     > a machine to work that others can use to cut releases, it will save us time and encourage others to become RMs.
>
>     With all due respect, the above sounds like a really *weird* reason to
>     make *others*
>     do the work for you.
>
> I don't understand.  Who am I "making" do what work?

INFRA

>  And why do at least 3 others want something similar?

I don't know -- I haven't heard those arguments.

>  And what would you propose Royale should do instead?  Always have me cut releases?

I suggest you educate your community and make sure that RM
responsibility can be shared.
In fact, if you're the only one who *understands* how releases are
done in Royal -- I don't
think that Royal qualifies as a very healthy community, frankly.

Thanks,
Roman.

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

On 1/7/19, 11:38 AM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:

    On Mon, Jan 7, 2019 at 11:33 AM Alex Harui <ah...@adobe.com.invalid> wrote:
    >
    >
    >
    > On 1/7/19, 11:05 AM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
    >
    >     On Mon, Jan 7, 2019 at 11:00 AM Alex Harui <ah...@adobe.com.invalid> wrote:
    >     >
    >     > Hi Mike,
    >     >
    >     > Thanks for the input.  IMO, that exploit would be easily seen.  The release plugin should only be changing one-liners with version numbers.
    >
    >     If this is all it ever does I still don't understand why a human
    >     operator won't suffice.
    >
    > Because our humans have been unable to get their computers to work.
    > They seem to be having network reliability issues. So if I can provision
    > a machine to work that others can use to cut releases, it will save us time and encourage others to become RMs.
    
    With all due respect, the above sounds like a really *weird* reason to
    make *others*
    do the work for you.
    
I don't understand.  Who am I "making" do what work?  And why do at least 3 others want something similar?  And what would you propose Royale should do instead?  Always have me cut releases?

-Alex 

Re: Can we package release artifacts on builds.a.o?

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Mon, Jan 7, 2019 at 11:33 AM Alex Harui <ah...@adobe.com.invalid> wrote:
>
>
>
> On 1/7/19, 11:05 AM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
>
>     On Mon, Jan 7, 2019 at 11:00 AM Alex Harui <ah...@adobe.com.invalid> wrote:
>     >
>     > Hi Mike,
>     >
>     > Thanks for the input.  IMO, that exploit would be easily seen.  The release plugin should only be changing one-liners with version numbers.
>
>     If this is all it ever does I still don't understand why a human
>     operator won't suffice.
>
> Because our humans have been unable to get their computers to work.
> They seem to be having network reliability issues. So if I can provision
> a machine to work that others can use to cut releases, it will save us time and encourage others to become RMs.

With all due respect, the above sounds like a really *weird* reason to
make *others*
do the work for you.

Thanks,
Roman.

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

On 1/7/19, 11:05 AM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:

    On Mon, Jan 7, 2019 at 11:00 AM Alex Harui <ah...@adobe.com.invalid> wrote:
    >
    > Hi Mike,
    >
    > Thanks for the input.  IMO, that exploit would be easily seen.  The release plugin should only be changing one-liners with version numbers.
    
    If this is all it ever does I still don't understand why a human
    operator won't suffice.
    
Because our humans have been unable to get their computers to work.  They seem to be having network reliability issues. So if I can provision a machine to work that others can use to cut releases, it will save us time and encourage others to become RMs.

Thanks,
-Alex 

Re: Can we package release artifacts on builds.a.o?

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Mon, Jan 7, 2019 at 11:00 AM Alex Harui <ah...@adobe.com.invalid> wrote:
>
> Hi Mike,
>
> Thanks for the input.  IMO, that exploit would be easily seen.  The release plugin should only be changing one-liners with version numbers.

If this is all it ever does I still don't understand why a human
operator won't suffice.

Thanks,
Roman.

P.S. I mean come on, just because a thing could be automated doesn't
automatically mean it should.

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

Hi Mike,

Thanks for the input.  IMO, that exploit would be easily seen.  The release plugin should only be changing one-liners with version numbers.  Can you think of one-liner attacks?  The attacker would also have to know when we are running our artifact job, otherwise commits from bot would be seen as unexpected.

I think there is a way from Git to pull just changes from a person and we could write tools to check the changes to further lower risk.

Thanks,
-Alex

On 1/7/19, 10:55 AM, "Mike Jumper" <mj...@apache.org> wrote:

    On Mon, Jan 7, 2019 at 10:39 AM Alex Harui <ah...@adobe.com.invalid> wrote:
    
    > Hi Greg,
    >
    > Thanks for the history.  I agree with the general problem, however, for
    > Royale, I think the problem is constrained, but I could be wrong.  I don't
    > think there are exploits from things like missing semicolons and other code
    > exploits that can be executed against pom.xml files, so the Royale
    > reviewers are first looking to see if bot changed any other files.
    
    
    It's definitely not safe to assume that a change will not introduce
    something malicious purely based on the files it touches.
    
    Maybe Maven experts can tell us what kinds of exploit could be hacked into
    > a pom.xml.
    >
    
    Given the array of plugins available, just about anything is possible:
    
        ...
        <plugin>
            <artifactId>exec-maven-plugin</artifactId>
            <groupId>org.codehaus.mojo</groupId>
            <version>1.6.0</version>
            <executions>
                <execution>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>exec</goal>
                    </goals>
                    <configuration>
                        <executable>echo</executable>
                        <commandlineArgs>something nefarious</commandlineArgs>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        ...
    
    - Mike
    

Re: Can we package release artifacts on builds.a.o?

[Mike Jumper <mj...@apache.org>]

On Mon, Jan 7, 2019 at 10:39 AM Alex Harui <ah...@adobe.com.invalid> wrote:

> Hi Greg,
>
> Thanks for the history.  I agree with the general problem, however, for
> Royale, I think the problem is constrained, but I could be wrong.  I don't
> think there are exploits from things like missing semicolons and other code
> exploits that can be executed against pom.xml files, so the Royale
> reviewers are first looking to see if bot changed any other files.


It's definitely not safe to assume that a change will not introduce
something malicious purely based on the files it touches.

Maybe Maven experts can tell us what kinds of exploit could be hacked into
> a pom.xml.
>

Given the array of plugins available, just about anything is possible:

    ...
    <plugin>
        <artifactId>exec-maven-plugin</artifactId>
        <groupId>org.codehaus.mojo</groupId>
        <version>1.6.0</version>
        <executions>
            <execution>
                <phase>generate-sources</phase>
                <goals>
                    <goal>exec</goal>
                </goals>
                <configuration>
                    <executable>echo</executable>
                    <commandlineArgs>something nefarious</commandlineArgs>
                </configuration>
            </execution>
        </executions>
    </plugin>
    ...

- Mike

Re: Can we package release artifacts on builds.a.o?

[Dave Fisher <wa...@apache.org>]

 Ich Verstehe dich!

I you share with me do you mind if I share on dev@royale?

Sent from my iPhone

> On Jan 7, 2019, at 1:55 PM, Christofer Dutz <ch...@c-ware.de> wrote:
> 
> Hi Dave,
> 
> Well it was naturally off list.
> 
> Chris
> 
> Outlook for Android<https://aka.ms/ghei36> herunterladen
> 
> ________________________________
> From: Dave Fisher <wa...@apache.org>
> Sent: Monday, January 7, 2019 10:32:38 PM
> To: builds@apache.org
> Subject: Re: Can we package release artifacts on builds.a.o?
> 
> Hi Chris,
> 
> Thank you for providing Carlos with instructions. Was that on or off list?
> 
> Regards,
> Dave
> 
> Sent from my iPhone
> 
>> On Jan 7, 2019, at 1:18 PM, Christofer Dutz <ch...@c-ware.de> wrote:
>> 
>> Hi Alex,
>> 
>> Ways to do bad stuff with just a pom.xml:
>> - simply adding a dependency to a vulnerable library, even an intentionally staged malicious one.
>> - Adding an evec-maven-plugin to execute anything on the host machine
>> - Generate code
>> - Like I introduced into the FlexJS maven build: Patch/Modify source files
>> Guess the is what I could think of in 5 minutes...
>> 
>> Ways to do bad stuff by just changing one-line versions:
>> And changing the version of a dependency to a known vulnerable version would be such a one-liner.
>> I'm currently introducing vulnerability checks into all of my builds, so I'm bumping dependencies to unvulnerable versions ...
>> doing this the other way around would introduce vulnerabilities.
>> 
>> Connectivity problems:
>> Regarding network problems ... on my way to Montreal I staged the first RC for Apache PLC4X in a plane ...
>> it took about 3 hours to upload cause of network problems and latencies.
>> Maven usually works around connectivity problems quite nicely and reliably.
>> 
>> So all in all I would suggest you sort out the problems in the build with someone with experience.
>> I already told Carlos how he could deploy to a local directory during the release itself and then use another
>> plugin to stage that release independently.
>> If it aborts, you just re-start the deployment and close the repo as soon as all passes.
>> 
>> Chris
>> 
>> 
>> 
>> Am 07.01.19, 19:39 schrieb "Alex Harui" <ah...@adobe.com.INVALID>:
>> 
>>   Hi Greg,
>> 
>>   Thanks for the history.  I agree with the general problem, however, for Royale, I think the problem is constrained, but I could be wrong.  I don't think there are exploits from things like missing semicolons and other code exploits that can be executed against pom.xml files, so the Royale reviewers are first looking to see if bot changed any other files.  Maybe Maven experts can tell us what kinds of exploit could be hacked into a pom.xml.
>> 
>>   Could you answer another question?  What is the current state of SVN/Git integration?  Could we spin up an SVN clone of our Git repos, restrict the bot via SVN, then sync back from SVN to Git (all from Jenkins)?
>> 
>>   Thanks,
>>   -Alex
>> 
>>   On 1/7/19, 10:30 AM, "Greg Stein" <gs...@gmail.com> wrote:
>> 
>>       On Mon, Jan 7, 2019 at 12:23 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>>> ...
>> 
>>> I still don't get why allowing a bot to commit to a Git repo isn't
>>> auditable.  The changes should all be text and sent to commits@ and the
>>> RMs job is to verify that those commits are ok before putting the artifacts
>>> up for vote.  I'd even try to  make an email rule that checks for commits
>>> from buildbot and flags changes to files that are outside of what we
>>> expected.
>>> 
>> 
>>       The historic position of the Foundation is "no ability to commit without a
>>       matched ICLA". That is different from "we'll audit any commits made by
>>       $bot". The trust meter is rather different between those positions,
>>       specifically with the "what if nobody reviews? what if a commit is missed?
>>       what if that added semicolon is missed, yet opens a vuln?" ... With the
>>       "matched ICLA" position, the Foundation has the assurance of *that*
>>       committer, that everything is Good. ... Yet a bot cannot make any such
>>       assurances, despite any "best effort" of the PMC to review the bot's work.
>> 
>>       It is likely a solvable problem! My comments here are to outline
>>       history/policy, rather than to say "NO". These are just the parameters of
>>       the problem space.
>> 
>>       Cheers,
>>       -g
>>       InfraAdmin
>> 
>> 
>> 
>> 
> 

Re: Can we package release artifacts on builds.a.o?

[Christofer Dutz <ch...@c-ware.de>]

Hi Dave,

Well it was naturally off list.

Chris

Outlook for Android<https://aka.ms/ghei36> herunterladen

________________________________
From: Dave Fisher <wa...@apache.org>
Sent: Monday, January 7, 2019 10:32:38 PM
To: builds@apache.org
Subject: Re: Can we package release artifacts on builds.a.o?

Hi Chris,

Thank you for providing Carlos with instructions. Was that on or off list?

Regards,
Dave

Sent from my iPhone

> On Jan 7, 2019, at 1:18 PM, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Hi Alex,
>
> Ways to do bad stuff with just a pom.xml:
> - simply adding a dependency to a vulnerable library, even an intentionally staged malicious one.
> - Adding an evec-maven-plugin to execute anything on the host machine
> - Generate code
> - Like I introduced into the FlexJS maven build: Patch/Modify source files
> Guess the is what I could think of in 5 minutes...
>
> Ways to do bad stuff by just changing one-line versions:
> And changing the version of a dependency to a known vulnerable version would be such a one-liner.
> I'm currently introducing vulnerability checks into all of my builds, so I'm bumping dependencies to unvulnerable versions ...
> doing this the other way around would introduce vulnerabilities.
>
> Connectivity problems:
> Regarding network problems ... on my way to Montreal I staged the first RC for Apache PLC4X in a plane ...
> it took about 3 hours to upload cause of network problems and latencies.
> Maven usually works around connectivity problems quite nicely and reliably.
>
> So all in all I would suggest you sort out the problems in the build with someone with experience.
> I already told Carlos how he could deploy to a local directory during the release itself and then use another
> plugin to stage that release independently.
> If it aborts, you just re-start the deployment and close the repo as soon as all passes.
>
> Chris
>
>
>
> Am 07.01.19, 19:39 schrieb "Alex Harui" <ah...@adobe.com.INVALID>:
>
>    Hi Greg,
>
>    Thanks for the history.  I agree with the general problem, however, for Royale, I think the problem is constrained, but I could be wrong.  I don't think there are exploits from things like missing semicolons and other code exploits that can be executed against pom.xml files, so the Royale reviewers are first looking to see if bot changed any other files.  Maybe Maven experts can tell us what kinds of exploit could be hacked into a pom.xml.
>
>    Could you answer another question?  What is the current state of SVN/Git integration?  Could we spin up an SVN clone of our Git repos, restrict the bot via SVN, then sync back from SVN to Git (all from Jenkins)?
>
>    Thanks,
>    -Alex
>
>    On 1/7/19, 10:30 AM, "Greg Stein" <gs...@gmail.com> wrote:
>
>        On Mon, Jan 7, 2019 at 12:23 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>> ...
>
>> I still don't get why allowing a bot to commit to a Git repo isn't
>> auditable.  The changes should all be text and sent to commits@ and the
>> RMs job is to verify that those commits are ok before putting the artifacts
>> up for vote.  I'd even try to  make an email rule that checks for commits
>> from buildbot and flags changes to files that are outside of what we
>> expected.
>>
>
>        The historic position of the Foundation is "no ability to commit without a
>        matched ICLA". That is different from "we'll audit any commits made by
>        $bot". The trust meter is rather different between those positions,
>        specifically with the "what if nobody reviews? what if a commit is missed?
>        what if that added semicolon is missed, yet opens a vuln?" ... With the
>        "matched ICLA" position, the Foundation has the assurance of *that*
>        committer, that everything is Good. ... Yet a bot cannot make any such
>        assurances, despite any "best effort" of the PMC to review the bot's work.
>
>        It is likely a solvable problem! My comments here are to outline
>        history/policy, rather than to say "NO". These are just the parameters of
>        the problem space.
>
>        Cheers,
>        -g
>        InfraAdmin
>
>
>
>

Re: Can we package release artifacts on builds.a.o?

[Dave Fisher <wa...@apache.org>]

Hi Chris,

Thank you for providing Carlos with instructions. Was that on or off list?

Regards,
Dave

Sent from my iPhone

> On Jan 7, 2019, at 1:18 PM, Christofer Dutz <ch...@c-ware.de> wrote:
> 
> Hi Alex,
> 
> Ways to do bad stuff with just a pom.xml:
> - simply adding a dependency to a vulnerable library, even an intentionally staged malicious one.
> - Adding an evec-maven-plugin to execute anything on the host machine
> - Generate code
> - Like I introduced into the FlexJS maven build: Patch/Modify source files
> Guess the is what I could think of in 5 minutes...
> 
> Ways to do bad stuff by just changing one-line versions:
> And changing the version of a dependency to a known vulnerable version would be such a one-liner.
> I'm currently introducing vulnerability checks into all of my builds, so I'm bumping dependencies to unvulnerable versions ... 
> doing this the other way around would introduce vulnerabilities.
> 
> Connectivity problems:
> Regarding network problems ... on my way to Montreal I staged the first RC for Apache PLC4X in a plane ... 
> it took about 3 hours to upload cause of network problems and latencies. 
> Maven usually works around connectivity problems quite nicely and reliably.
> 
> So all in all I would suggest you sort out the problems in the build with someone with experience. 
> I already told Carlos how he could deploy to a local directory during the release itself and then use another
> plugin to stage that release independently. 
> If it aborts, you just re-start the deployment and close the repo as soon as all passes.
> 
> Chris
> 
> 
> 
> Am 07.01.19, 19:39 schrieb "Alex Harui" <ah...@adobe.com.INVALID>:
> 
>    Hi Greg,
> 
>    Thanks for the history.  I agree with the general problem, however, for Royale, I think the problem is constrained, but I could be wrong.  I don't think there are exploits from things like missing semicolons and other code exploits that can be executed against pom.xml files, so the Royale reviewers are first looking to see if bot changed any other files.  Maybe Maven experts can tell us what kinds of exploit could be hacked into a pom.xml.
> 
>    Could you answer another question?  What is the current state of SVN/Git integration?  Could we spin up an SVN clone of our Git repos, restrict the bot via SVN, then sync back from SVN to Git (all from Jenkins)?
> 
>    Thanks,
>    -Alex
> 
>    On 1/7/19, 10:30 AM, "Greg Stein" <gs...@gmail.com> wrote:
> 
>        On Mon, Jan 7, 2019 at 12:23 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>> ...
> 
>> I still don't get why allowing a bot to commit to a Git repo isn't
>> auditable.  The changes should all be text and sent to commits@ and the
>> RMs job is to verify that those commits are ok before putting the artifacts
>> up for vote.  I'd even try to  make an email rule that checks for commits
>> from buildbot and flags changes to files that are outside of what we
>> expected.
>> 
> 
>        The historic position of the Foundation is "no ability to commit without a
>        matched ICLA". That is different from "we'll audit any commits made by
>        $bot". The trust meter is rather different between those positions,
>        specifically with the "what if nobody reviews? what if a commit is missed?
>        what if that added semicolon is missed, yet opens a vuln?" ... With the
>        "matched ICLA" position, the Foundation has the assurance of *that*
>        committer, that everything is Good. ... Yet a bot cannot make any such
>        assurances, despite any "best effort" of the PMC to review the bot's work.
> 
>        It is likely a solvable problem! My comments here are to outline
>        history/policy, rather than to say "NO". These are just the parameters of
>        the problem space.
> 
>        Cheers,
>        -g
>        InfraAdmin
> 
> 
> 
> 

Re: Can we package release artifacts on builds.a.o?

[Christofer Dutz <ch...@c-ware.de>]

Hi Alex,

Ways to do bad stuff with just a pom.xml:
- simply adding a dependency to a vulnerable library, even an intentionally staged malicious one.
- Adding an evec-maven-plugin to execute anything on the host machine
- Generate code
- Like I introduced into the FlexJS maven build: Patch/Modify source files
Guess the is what I could think of in 5 minutes...

Ways to do bad stuff by just changing one-line versions:
And changing the version of a dependency to a known vulnerable version would be such a one-liner.
I'm currently introducing vulnerability checks into all of my builds, so I'm bumping dependencies to unvulnerable versions ... 
doing this the other way around would introduce vulnerabilities.

Connectivity problems:
Regarding network problems ... on my way to Montreal I staged the first RC for Apache PLC4X in a plane ... 
it took about 3 hours to upload cause of network problems and latencies. 
Maven usually works around connectivity problems quite nicely and reliably.

So all in all I would suggest you sort out the problems in the build with someone with experience. 
I already told Carlos how he could deploy to a local directory during the release itself and then use another
plugin to stage that release independently. 
If it aborts, you just re-start the deployment and close the repo as soon as all passes.

Chris



Am 07.01.19, 19:39 schrieb "Alex Harui" <ah...@adobe.com.INVALID>:

    Hi Greg,
    
    Thanks for the history.  I agree with the general problem, however, for Royale, I think the problem is constrained, but I could be wrong.  I don't think there are exploits from things like missing semicolons and other code exploits that can be executed against pom.xml files, so the Royale reviewers are first looking to see if bot changed any other files.  Maybe Maven experts can tell us what kinds of exploit could be hacked into a pom.xml.
    
    Could you answer another question?  What is the current state of SVN/Git integration?  Could we spin up an SVN clone of our Git repos, restrict the bot via SVN, then sync back from SVN to Git (all from Jenkins)?
    
    Thanks,
    -Alex
    
    On 1/7/19, 10:30 AM, "Greg Stein" <gs...@gmail.com> wrote:
    
        On Mon, Jan 7, 2019 at 12:23 PM Alex Harui <ah...@adobe.com.invalid> wrote:
        >...
        
        > I still don't get why allowing a bot to commit to a Git repo isn't
        > auditable.  The changes should all be text and sent to commits@ and the
        > RMs job is to verify that those commits are ok before putting the artifacts
        > up for vote.  I'd even try to  make an email rule that checks for commits
        > from buildbot and flags changes to files that are outside of what we
        > expected.
        >
        
        The historic position of the Foundation is "no ability to commit without a
        matched ICLA". That is different from "we'll audit any commits made by
        $bot". The trust meter is rather different between those positions,
        specifically with the "what if nobody reviews? what if a commit is missed?
        what if that added semicolon is missed, yet opens a vuln?" ... With the
        "matched ICLA" position, the Foundation has the assurance of *that*
        committer, that everything is Good. ... Yet a bot cannot make any such
        assurances, despite any "best effort" of the PMC to review the bot's work.
        
        It is likely a solvable problem! My comments here are to outline
        history/policy, rather than to say "NO". These are just the parameters of
        the problem space.
        
        Cheers,
        -g
        InfraAdmin
        
    
    

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

Hi Greg,

Thanks for the history.  I agree with the general problem, however, for Royale, I think the problem is constrained, but I could be wrong.  I don't think there are exploits from things like missing semicolons and other code exploits that can be executed against pom.xml files, so the Royale reviewers are first looking to see if bot changed any other files.  Maybe Maven experts can tell us what kinds of exploit could be hacked into a pom.xml.

Could you answer another question?  What is the current state of SVN/Git integration?  Could we spin up an SVN clone of our Git repos, restrict the bot via SVN, then sync back from SVN to Git (all from Jenkins)?

Thanks,
-Alex

On 1/7/19, 10:30 AM, "Greg Stein" <gs...@gmail.com> wrote:

    On Mon, Jan 7, 2019 at 12:23 PM Alex Harui <ah...@adobe.com.invalid> wrote:
    >...
    
    > I still don't get why allowing a bot to commit to a Git repo isn't
    > auditable.  The changes should all be text and sent to commits@ and the
    > RMs job is to verify that those commits are ok before putting the artifacts
    > up for vote.  I'd even try to  make an email rule that checks for commits
    > from buildbot and flags changes to files that are outside of what we
    > expected.
    >
    
    The historic position of the Foundation is "no ability to commit without a
    matched ICLA". That is different from "we'll audit any commits made by
    $bot". The trust meter is rather different between those positions,
    specifically with the "what if nobody reviews? what if a commit is missed?
    what if that added semicolon is missed, yet opens a vuln?" ... With the
    "matched ICLA" position, the Foundation has the assurance of *that*
    committer, that everything is Good. ... Yet a bot cannot make any such
    assurances, despite any "best effort" of the PMC to review the bot's work.
    
    It is likely a solvable problem! My comments here are to outline
    history/policy, rather than to say "NO". These are just the parameters of
    the problem space.
    
    Cheers,
    -g
    InfraAdmin
    

Re: Can we package release artifacts on builds.a.o?

[Greg Stein <gs...@gmail.com>]

On Mon, Jan 7, 2019 at 12:23 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>...

> I still don't get why allowing a bot to commit to a Git repo isn't
> auditable.  The changes should all be text and sent to commits@ and the
> RMs job is to verify that those commits are ok before putting the artifacts
> up for vote.  I'd even try to  make an email rule that checks for commits
> from buildbot and flags changes to files that are outside of what we
> expected.
>

The historic position of the Foundation is "no ability to commit without a
matched ICLA". That is different from "we'll audit any commits made by
$bot". The trust meter is rather different between those positions,
specifically with the "what if nobody reviews? what if a commit is missed?
what if that added semicolon is missed, yet opens a vuln?" ... With the
"matched ICLA" position, the Foundation has the assurance of *that*
committer, that everything is Good. ... Yet a bot cannot make any such
assurances, despite any "best effort" of the PMC to review the bot's work.

It is likely a solvable problem! My comments here are to outline
history/policy, rather than to say "NO". These are just the parameters of
the problem space.

Cheers,
-g
InfraAdmin

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

Hi Greg, Stephen,

I think I do need what Greg said:  the bot should be able to commit like I can now.  AFAICT, Maven's release plugin does pushes to Git as it is doing its work so there isn't a clean way to stop until the RM verifies.

I still don't get why allowing a bot to commit to a Git repo isn't auditable.  The changes should all be text and sent to commits@ and the RMs job is to verify that those commits are ok before putting the artifacts up for vote.  I'd even try to  make an email rule that checks for commits from buildbot and flags changes to files that are outside of what we expected.

Royale's problem is only partially getting folks up to speed.  The issue behind this request is that the 3 folks we got up to speed still couldn't get Maven to run without intermittent failures.  It doesn't appear to be a Maven issue, it seems to be a network bandwidth issue, so having a "single computer" that is known to have good connectivity to Nexus and dist and Maven Central would be a huge win for us.

Thanks,
-Alex

On 1/7/19, 7:19 AM, "Greg Stein" <gs...@gmail.com> wrote:

    On Mon, Jan 7, 2019 at 8:39 AM stephen.alan.connolly@gmail.com <
    stephen.alan.connolly@gmail.com> wrote:
    
    > On 2019/01/07 14:35:08, Greg Stein <gs...@gmail.com> wrote:
    > > On Sun, Jan 6, 2019 at 10:20 PM Alex Harui <ah...@adobe.com.invalid>
    > wrote:
    > > >...
    > >
    > > > All commits, even PR's from non-commiters accepted by a committer are
    > > > supposed to be reviewed, AIUI.  So if the bot makes a commit to the
    > repo,
    > > > the PMC is responsible for reviewing it.  In Royale's case, the bot
    > should
    > > > only be changing pom.xml files and making tags and branches, so a bad
    > bot
    > > > commit should be easy to spot and detection may even be tool-able.
    > > >
    > >
    > > Git does not have path-based authorization, so there is no way to
    > restrict
    > > a bot from changing *code*. Give it access to pom.xml, and it has access
    > to
    > > the entire repository. "But the bot won't do that" ... Well, the bot is
    > not
    > > auditable by Legal Affairs or Infrastructure, so there is no way to
    > > validate it is committing Properly.
    >
    > The bot doesn't need commit access to gitbox... it needs commit access to
    > *a* git repo. The RM then pulls the tag from that repo and pushes it *after
    > review* back to gitbox.
    >
    > And there's your auditability for you ;-)
    >
    
    Then make that git repo a local clone, hmm?
    
    If you're talking a public one, then what is the "ask" from Infra for this
    repo? Every PMC can self-serve create git repositories as they need them.
    So it would seem "do that", then you'd need to ask for some extra authz to
    enable the bot for that one repository? And what is the mechanism to
    prevent leakage of released code into that repository? Or, say, the bot
    adjusting pom.xml to pull in malware from $bad ?
    
    Cheers,
    -g
    

Re: Can we package release artifacts on builds.a.o?

[Greg Stein <gs...@gmail.com>]

On Mon, Jan 7, 2019 at 8:39 AM stephen.alan.connolly@gmail.com <
stephen.alan.connolly@gmail.com> wrote:

> On 2019/01/07 14:35:08, Greg Stein <gs...@gmail.com> wrote:
> > On Sun, Jan 6, 2019 at 10:20 PM Alex Harui <ah...@adobe.com.invalid>
> wrote:
> > >...
> >
> > > All commits, even PR's from non-commiters accepted by a committer are
> > > supposed to be reviewed, AIUI.  So if the bot makes a commit to the
> repo,
> > > the PMC is responsible for reviewing it.  In Royale's case, the bot
> should
> > > only be changing pom.xml files and making tags and branches, so a bad
> bot
> > > commit should be easy to spot and detection may even be tool-able.
> > >
> >
> > Git does not have path-based authorization, so there is no way to
> restrict
> > a bot from changing *code*. Give it access to pom.xml, and it has access
> to
> > the entire repository. "But the bot won't do that" ... Well, the bot is
> not
> > auditable by Legal Affairs or Infrastructure, so there is no way to
> > validate it is committing Properly.
>
> The bot doesn't need commit access to gitbox... it needs commit access to
> *a* git repo. The RM then pulls the tag from that repo and pushes it *after
> review* back to gitbox.
>
> And there's your auditability for you ;-)
>

Then make that git repo a local clone, hmm?

If you're talking a public one, then what is the "ask" from Infra for this
repo? Every PMC can self-serve create git repositories as they need them.
So it would seem "do that", then you'd need to ask for some extra authz to
enable the bot for that one repository? And what is the mechanism to
prevent leakage of released code into that repository? Or, say, the bot
adjusting pom.xml to pull in malware from $bad ?

Cheers,
-g

Re: Can we package release artifacts on builds.a.o?

On 2019/01/07 14:35:08, Greg Stein <gs...@gmail.com> wrote: 
> On Sun, Jan 6, 2019 at 10:20 PM Alex Harui <ah...@adobe.com.invalid> wrote:
> >...
> 
> > All commits, even PR's from non-commiters accepted by a committer are
> > supposed to be reviewed, AIUI.  So if the bot makes a commit to the repo,
> > the PMC is responsible for reviewing it.  In Royale's case, the bot should
> > only be changing pom.xml files and making tags and branches, so a bad bot
> > commit should be easy to spot and detection may even be tool-able.
> >
> 
> Git does not have path-based authorization, so there is no way to restrict
> a bot from changing *code*. Give it access to pom.xml, and it has access to
> the entire repository. "But the bot won't do that" ... Well, the bot is not
> auditable by Legal Affairs or Infrastructure, so there is no way to
> validate it is committing Properly.

The bot doesn't need commit access to gitbox... it needs commit access to *a* git repo. The RM then pulls the tag from that repo and pushes it *after review* back to gitbox.

And there's your auditability for you ;-)

> 
> This is the basic conundrum behind Legal/Infra's decision to disallow bots
> from commit access to git repositories. We do have a couple running for svn
> repositories, using path-based authz.
> 
> Within the Apache Subversion project, have tooling[1] to assist an RM with
> pretty much all the steps of a release. From reading this thread, it seems
> like Royale's problem is getting RMs up to speed, so maybe it can be solved
> with additional build-side tooling?
> 
> Cheers,
> Greg
> InfraAdmin, ASF
> 
> [1] https://svn.apache.org/repos/asf/subversion/trunk/tools/dist/
> 

Re: Can we package release artifacts on builds.a.o?

[Greg Stein <gs...@gmail.com>]

On Sun, Jan 6, 2019 at 10:20 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>...

> All commits, even PR's from non-commiters accepted by a committer are
> supposed to be reviewed, AIUI.  So if the bot makes a commit to the repo,
> the PMC is responsible for reviewing it.  In Royale's case, the bot should
> only be changing pom.xml files and making tags and branches, so a bad bot
> commit should be easy to spot and detection may even be tool-able.
>

Git does not have path-based authorization, so there is no way to restrict
a bot from changing *code*. Give it access to pom.xml, and it has access to
the entire repository. "But the bot won't do that" ... Well, the bot is not
auditable by Legal Affairs or Infrastructure, so there is no way to
validate it is committing Properly.

This is the basic conundrum behind Legal/Infra's decision to disallow bots
from commit access to git repositories. We do have a couple running for svn
repositories, using path-based authz.

Within the Apache Subversion project, have tooling[1] to assist an RM with
pretty much all the steps of a release. From reading this thread, it seems
like Royale's problem is getting RMs up to speed, so maybe it can be solved
with additional build-side tooling?

Cheers,
Greg
InfraAdmin, ASF

[1] https://svn.apache.org/repos/asf/subversion/trunk/tools/dist/

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

On 1/6/19, 7:54 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:

    On Sun, Jan 6, 2019 at 7:38 PM Alex Harui <ah...@adobe.com.invalid> wrote:
    >
    >
    >
    > On 1/6/19, 6:58 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
    >
    >     On Sun, Jan 6, 2019 at 6:50 PM Alex Harui <ah...@adobe.com.invalid> wrote:
    >     >
    >     > OK, apparently Infra doesn't want to discuss this in a JIRA issue so I will try to continue it here and bug people with emails if the thread stagnates like it did last time.
    >     >
    >     > I'm unclear what questions and problems are of concern here specific to this ask.  IMO:
    >     > 1) ASF Release Policy currently allows artifacts to be packaged on other hardware.  It just has to be verified on RM/PMC-controlled hardware
    >     > 2) There is no packaging specific security risk.  Rogue executions via Jenkins are either possible or not possible and there are plenty of other juicy targets for rogue executions besides release artifacts that are verifiable.
    >
    >     I don't have a strong opinion on the above, but I'm very concerned
    >     about a requirement of a bot pushing to SCM repos.
    >
    > Please explain your concern.
    
    ASF lives and dies by how well it can track IP provenance in what we release.
    That's why any non-committer interactions around SCM will give me pause.
    
All commits, even PR's from non-commiters accepted by a committer are supposed to be reviewed, AIUI.  So if the bot makes a commit to the repo, the PMC is responsible for reviewing it.  In Royale's case, the bot should only be changing pom.xml files and making tags and branches, so a bad bot commit should be easy to spot and detection may even be tool-able.

    > A bot is already allowed to commit to the website repos, AIUI.
    
    Two things:
       1. can you give me real-world examples of that?

See the beginning of this thread.  I posted this link to an old email:

https://lists.apache.org/thread.html/efed1ff44fbfe5770ea1574b2f53a5295ae8326c5a3a5feb9f88cd48@%3Cbuilds.apache.org%3E

And Karl Heinz Marbaise seemed to say that Maven is doing it.

https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven-site/

Also note that in Royale's case, the Jenkins job would not be triggered.  It would be manually started.  So one requirement of allowing packaging jobs could be that artifact packaging jobs cannot be automatically triggered by repo changes or date/time.  That would better ensure that the PMC has reviewed any bot changes.

Thanks,
-Alex

Re: Can we package release artifacts on builds.a.o?

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Sun, Jan 6, 2019 at 7:38 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>
>
>
> On 1/6/19, 6:58 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:
>
>     On Sun, Jan 6, 2019 at 6:50 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>     >
>     > OK, apparently Infra doesn't want to discuss this in a JIRA issue so I will try to continue it here and bug people with emails if the thread stagnates like it did last time.
>     >
>     > I'm unclear what questions and problems are of concern here specific to this ask.  IMO:
>     > 1) ASF Release Policy currently allows artifacts to be packaged on other hardware.  It just has to be verified on RM/PMC-controlled hardware
>     > 2) There is no packaging specific security risk.  Rogue executions via Jenkins are either possible or not possible and there are plenty of other juicy targets for rogue executions besides release artifacts that are verifiable.
>
>     I don't have a strong opinion on the above, but I'm very concerned
>     about a requirement of a bot pushing to SCM repos.
>
> Please explain your concern.

ASF lives and dies by how well it can track IP provenance in what we release.
That's why any non-committer interactions around SCM will give me pause.

> A bot is already allowed to commit to the website repos, AIUI.

Two things:
   1. can you give me real-world examples of that?
   2. website repos are much lower on my list of priorities than code
repos (see above for reasoning)

Thanks,
Roman.

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

On 1/6/19, 6:58 PM, "Roman Shaposhnik" <ro...@shaposhnik.org> wrote:

    On Sun, Jan 6, 2019 at 6:50 PM Alex Harui <ah...@adobe.com.invalid> wrote:
    >
    > OK, apparently Infra doesn't want to discuss this in a JIRA issue so I will try to continue it here and bug people with emails if the thread stagnates like it did last time.
    >
    > I'm unclear what questions and problems are of concern here specific to this ask.  IMO:
    > 1) ASF Release Policy currently allows artifacts to be packaged on other hardware.  It just has to be verified on RM/PMC-controlled hardware
    > 2) There is no packaging specific security risk.  Rogue executions via Jenkins are either possible or not possible and there are plenty of other juicy targets for rogue executions besides release artifacts that are verifiable.
    
    I don't have a strong opinion on the above, but I'm very concerned
    about a requirement of a bot pushing to SCM repos.
    
Please explain your concern.  A bot is already allowed to commit to the website repos, AIUI.

Thanks,
-Alex 

Re: Can we package release artifacts on builds.a.o?

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Sun, Jan 6, 2019 at 6:50 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>
> OK, apparently Infra doesn't want to discuss this in a JIRA issue so I will try to continue it here and bug people with emails if the thread stagnates like it did last time.
>
> I'm unclear what questions and problems are of concern here specific to this ask.  IMO:
> 1) ASF Release Policy currently allows artifacts to be packaged on other hardware.  It just has to be verified on RM/PMC-controlled hardware
> 2) There is no packaging specific security risk.  Rogue executions via Jenkins are either possible or not possible and there are plenty of other juicy targets for rogue executions besides release artifacts that are verifiable.

I don't have a strong opinion on the above, but I'm very concerned
about a requirement of a bot pushing to SCM repos.

Thanks,
Roman.

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

OK, apparently Infra doesn't want to discuss this in a JIRA issue so I will try to continue it here and bug people with emails if the thread stagnates like it did last time.

I'm unclear what questions and problems are of concern here specific to this ask.  IMO:
1) ASF Release Policy currently allows artifacts to be packaged on other hardware.  It just has to be verified on RM/PMC-controlled hardware
2) There is no packaging specific security risk.  Rogue executions via Jenkins are either possible or not possible and there are plenty of other juicy targets for rogue executions besides release artifacts that are verifiable.

Infra, please list questions and problems.

Thanks,
-Alex

On 1/3/19, 2:19 PM, "Alex Harui" <ah...@adobe.com.INVALID> wrote:

    FWIW, I created this JIRA issue to track the decision.
    
    https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FINFRA-17540&amp;data=02%7C01%7Caharui%40adobe.com%7C9bb10afdd1a9466f890d08d671c97555%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636821507409358402&amp;sdata=Qj9VZ8zG%2BnMbZ%2F9%2B%2B9DKh%2B4hpEoKpc%2FJ5JLiksgPABw%3D&amp;reserved=0
    
    Thanks,
    -Alex
    
    On 12/13/18, 12:22 PM, "Zoran Regvart" <zo...@regvart.com> wrote:
    
        Hi Allen, Alex and Builders,
        I must say that I also think like Alex, who's to say that the builds
        done on a CI server are any worse than those done locally by PMC's.
        
        I understand that CI server is far from a clean room environment, but
        take a look at all the software you have installed on your machine and
        call that cleaner with a straight face. If signing is done locally by
        PMC's who very the build, in a yet to be determined fashion, what's
        the real risk here?
        
        I would very much like to have as little friction to releases for
        Apache Camel as possible.
        
        zoran
        
        On Tue, Dec 11, 2018 at 7:58 PM Alex Harui <ah...@adobe.com.invalid> wrote:
        >
        > IMO, we wouldn't publish releases signed by buildbot without being also signed by a PMC RM.  If there is a way to skip buildbot PGP signing that would be even better.  And we don't have to build from clean if we have a way to verify the binaries.  There are new efforts going on towards creating reproducible binaries that allow for such verification.
        >
        > My suggestion is rather simple:
        >
        > 1) Find a way to skip signing before pushing to Nexus release staging or have buildbot sign
        > 2) Have a buildbot account that can push to Git and SVN
        >
        > If we can do that, the PMC's can take care of the rest.
        
        
        
        --
        Zoran Regvart
        
    
    

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

FWIW, I created this JIRA issue to track the decision.

https://issues.apache.org/jira/browse/INFRA-17540

Thanks,
-Alex

On 12/13/18, 12:22 PM, "Zoran Regvart" <zo...@regvart.com> wrote:

    Hi Allen, Alex and Builders,
    I must say that I also think like Alex, who's to say that the builds
    done on a CI server are any worse than those done locally by PMC's.
    
    I understand that CI server is far from a clean room environment, but
    take a look at all the software you have installed on your machine and
    call that cleaner with a straight face. If signing is done locally by
    PMC's who very the build, in a yet to be determined fashion, what's
    the real risk here?
    
    I would very much like to have as little friction to releases for
    Apache Camel as possible.
    
    zoran
    
    On Tue, Dec 11, 2018 at 7:58 PM Alex Harui <ah...@adobe.com.invalid> wrote:
    >
    > IMO, we wouldn't publish releases signed by buildbot without being also signed by a PMC RM.  If there is a way to skip buildbot PGP signing that would be even better.  And we don't have to build from clean if we have a way to verify the binaries.  There are new efforts going on towards creating reproducible binaries that allow for such verification.
    >
    > My suggestion is rather simple:
    >
    > 1) Find a way to skip signing before pushing to Nexus release staging or have buildbot sign
    > 2) Have a buildbot account that can push to Git and SVN
    >
    > If we can do that, the PMC's can take care of the rest.
    
    
    
    --
    Zoran Regvart
    

Re: Can we package release artifacts on builds.a.o?

[Zoran Regvart <zo...@regvart.com>]

Hi Allen, Alex and Builders,
I must say that I also think like Alex, who's to say that the builds
done on a CI server are any worse than those done locally by PMC's.

I understand that CI server is far from a clean room environment, but
take a look at all the software you have installed on your machine and
call that cleaner with a straight face. If signing is done locally by
PMC's who very the build, in a yet to be determined fashion, what's
the real risk here?

I would very much like to have as little friction to releases for
Apache Camel as possible.

zoran

On Tue, Dec 11, 2018 at 7:58 PM Alex Harui <ah...@adobe.com.invalid> wrote:
>
> IMO, we wouldn't publish releases signed by buildbot without being also signed by a PMC RM.  If there is a way to skip buildbot PGP signing that would be even better.  And we don't have to build from clean if we have a way to verify the binaries.  There are new efforts going on towards creating reproducible binaries that allow for such verification.
>
> My suggestion is rather simple:
>
> 1) Find a way to skip signing before pushing to Nexus release staging or have buildbot sign
> 2) Have a buildbot account that can push to Git and SVN
>
> If we can do that, the PMC's can take care of the rest.



--
Zoran Regvart

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

IMO, we wouldn't publish releases signed by buildbot without being also signed by a PMC RM.  If there is a way to skip buildbot PGP signing that would be even better.  And we don't have to build from clean if we have a way to verify the binaries.  There are new efforts going on towards creating reproducible binaries that allow for such verification.

My suggestion is rather simple:

1) Find a way to skip signing before pushing to Nexus release staging or have buildbot sign
2) Have a buildbot account that can push to Git and SVN

If we can do that, the PMC's can take care of the rest.

Thanks,
-Alex

On 12/11/18, 9:54 AM, "Allen Wittenauer" <aw...@effectivemachines.com.INVALID> wrote:

    
    
    > On Dec 11, 2018, at 9:43 AM, Joan Touzet <wo...@apache.org> wrote:
    > Thanks, Allen. So I am still fighting against the system here.
    
    	I view it more as tilting at windmills but tomato, tomato. ;)
    
    > If binaries are conveniences, and they are not official, we should be able to auto-push binaries built on trusted infrastructure out to the world. Why can't that be our (Infra maintained & supported, costly from a non-profit perspective) CI/CD infrastructure?
    
    	Frankly:  given how much dumb stuff I see happening on the ASF Jenkins servers on a regular basis, I know I wouldn’t trust them as far as I could throw them.  [I’m pretty sure those servers are heavy and I’m not very strong, so that wouldn’t be very far. :) ]  All it would take is one person firing off a ‘bad' build that then gets signed by a buildbot account and now ALL of the ASF builds signed by that account are suspect.  That would be super bad.
    
    	From a more philosophical perspective, the current model definitely stresses the idea that the ASF is made up of diverse communities that all have their own (relative) governance.  The binary artifacts I’ve done for Apache Yetus take a few minutes and look very different than binary artifacts from other projects. Meanwhile, people would scream bloody murder if the artifact build server were tied up for the ~2-3 hours it takes to make Apache Hadoop while it downloads fresh copies of the hundreds of Docker and Apache Maven dependencies required to build.  [Because, I mean, you _are_ building _everything_ from scratch when building these, right???]

Re: Can we package release artifacts on builds.a.o?

[Allen Wittenauer <aw...@effectivemachines.com.INVALID>]

> On Dec 11, 2018, at 9:43 AM, Joan Touzet <wo...@apache.org> wrote:
> Thanks, Allen. So I am still fighting against the system here.

	I view it more as tilting at windmills but tomato, tomato. ;)

> If binaries are conveniences, and they are not official, we should be able to auto-push binaries built on trusted infrastructure out to the world. Why can't that be our (Infra maintained & supported, costly from a non-profit perspective) CI/CD infrastructure?

	Frankly:  given how much dumb stuff I see happening on the ASF Jenkins servers on a regular basis, I know I wouldn’t trust them as far as I could throw them.  [I’m pretty sure those servers are heavy and I’m not very strong, so that wouldn’t be very far. :) ]  All it would take is one person firing off a ‘bad' build that then gets signed by a buildbot account and now ALL of the ASF builds signed by that account are suspect.  That would be super bad.

	From a more philosophical perspective, the current model definitely stresses the idea that the ASF is made up of diverse communities that all have their own (relative) governance.  The binary artifacts I’ve done for Apache Yetus take a few minutes and look very different than binary artifacts from other projects. Meanwhile, people would scream bloody murder if the artifact build server were tied up for the ~2-3 hours it takes to make Apache Hadoop while it downloads fresh copies of the hundreds of Docker and Apache Maven dependencies required to build.  [Because, I mean, you _are_ building _everything_ from scratch when building these, right???]

Re: Can we package release artifacts on builds.a.o?

[Joan Touzet <wo...@apache.org>]

----- Original Message -----
Allen Wittenauer <aw...@effectivemachines.com> wrote:
> > On Dec 11, 2018, at 9:09 AM, Joan Touzet <wo...@apache.org> wrote:
> > Perhaps I'm misunderstanding...are the Nexus-published builds not treated the same because they're not on dist.apache.org? Or are they not release versions?
> 	Yes, you are misunderstanding.
> 	1) Officially (legally?), source code distributions are "the release."  Any and all binaries are considered to be convenience binaries so users don’t have to  compile.  They are not official.   [Statements like “verify a release by rebuilding” don’t really parse as a result.]
> 	2) As far as I’m aware/all the projects I’ve ever worked with, the uploads to Nexus are to the snapshot repo, not the release repo.  The release repos are still done manually. 

Thanks, Allen. So I am still fighting against the system here.

If binaries are conveniences, and they are not official, we should be able to auto-push binaries built on trusted infrastructure out to the world. Why can't that be our (Infra maintained & supported, costly from a non-profit perspective) CI/CD infrastructure?

-Joan

Re: Can we package release artifacts on builds.a.o?

[Allen Wittenauer <aw...@effectivemachines.com.INVALID>]

> On Dec 11, 2018, at 9:09 AM, Joan Touzet <wo...@apache.org> wrote:
> 
> Jenkins users are deploying directly to Nexus with builds.
> 
> Isn't that speaking out of both sides of our mouths at the same time, if Java developers can push release builds directly to Nexus but non-Java developers can't?
> 
> Perhaps I'm misunderstanding...are the Nexus-published builds not treated the same because they're not on dist.apache.org? Or are they not release versions?

	Yes, you are misunderstanding.

	1) Officially (legally?), source code distributions are "the release."  Any and all binaries are considered to be convenience binaries so users don’t have to compile.  They are not official.   [Statements like “verify a release by rebuilding” don’t really parse as a result.]  

	2) As far as I’m aware/all the projects I’ve ever worked with, the uploads to Nexus are to the snapshot repo, not the release repo.  The release repos are still done manually. 

Re: Can we package release artifacts on builds.a.o?

[Joan Touzet <wo...@apache.org>]

Back on this topic, the recent post on Jenkins has me thinking again.

Jenkins users are deploying directly to Nexus with builds.

Isn't that speaking out of both sides of our mouths at the same time, if Java developers can push release builds directly to Nexus but non-Java developers can't?

Perhaps I'm misunderstanding...are the Nexus-published builds not treated the same because they're not on dist.apache.org? Or are they not release versions?

I'm just asking for equal treatment here.

-Joan

----- Original Message -----
From: "Alex Harui" <ah...@adobe.com>
To: builds@apache.org, wohali@apache.org
Sent: Saturday, December 8, 2018 1:52:14 PM
Subject: Re: Can we package release artifacts on builds.a.o?

Good to know it isn't just me.

I could be wrong, but I believe the "policy" at Apache is only that a human must verify the packages and PGP sign them.  The packages can be built on another machine.

-Alex

On 12/8/18, 10:48 AM, "Joan Touzet" <wo...@apache.org> wrote:

    I would like to see support for something like this as well, even if it came down to individual VMs/donated HW per project, locked down by project - only project X can use build machine X'.
    
    Automated repeatable builds actually *increases* trust vs. who knows what a release manager has running on their workstation. At this point, I trust Docker builds with published, auditable cryptographic hashes per layer more than I trust some Apache releases.
    
    I don't actually believe that all projects in the Apache world are actually following the strict edict of "human must run the build and push any binary release," but I'm not going to point fingers.
    
    -Joan
    ----- Original Message -----
    From: "Alex Harui" <ah...@adobe.com.INVALID>
    To: builds@apache.org
    Sent: Saturday, December 8, 2018 12:43:37 PM
    Subject: Re: Can we package release artifacts on builds.a.o?
    
    Gavin, Alan, Karl,
    
    Thanks for the information.
    
    This email implies that there is a Jenkins node that can commit something.  What creds are used for that?  Is there a buildbot user?
    https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2Fefed1ff44fbfe5770ea1574b2f53a5295ae8326c5a3a5feb9f88cd48%40%253Cbuilds.apache.org%253E&amp;data=02%7C01%7Caharui%40adobe.com%7C5b7803529e3943bd483808d65d3daf70%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636798916859824875&amp;sdata=vJouQVPXMtHyvxOo%2BZuhhi0TmAdw9nJYPTC3HTaj1%2B0%3D&amp;reserved=0
    
    If so, I was imagining the following workflow:
    
    1) Jenkins runs Maven release.  I forgot about the PGP signing part.  If there is no way to skip it, then can a buildbot "user" PGP sign it?
    2) RM downloads the artifacts and verifies them.  The source package has to match the tag so I think that would detect any injections from other stuff running in Jenkins or elsewhere on the build server.  There's been a recent discussion on reproducible binaries and if this workflow is approved I would make our binaries are reproducible, and that should again detect any injections from the build server.
    3) RM adds his/her PGP signature to the artifacts.  Not sure if there is a Maven way to do that.
    4) Voting and other steps follow from there.
    
    These would not be continuously running jobs.  They would have to be kicked off manually so it shouldn't add significant load, and we would know which commits came from buildbot so we could detect if anything went funky.
    
    Thoughts?
    -Alex
    
    On 12/8/18, 7:54 AM, "Gavin McDonald" <ip...@gmail.com> wrote:
    
        additionally, nobody should have their creds stored anyway other than their
        own machine.
        
        On Sat, Dec 8, 2018 at 3:49 PM Allen Wittenauer
        <aw...@effectivemachines.com.invalid> wrote:
        
        >
        >
        > > On Dec 7, 2018, at 11:56 PM, Alex Harui <ah...@adobe.com.INVALID>
        > wrote:
        > >
        > >
        > >
        > > On 12/7/18, 10:49 PM, "Allen Wittenauer" <aw...@effectivemachines.com.INVALID>
        > wrote:
        > >
        > >
        > >
        > >> On Dec 7, 2018, at 10:22 PM, Alex Harui <ah...@adobe.com.INVALID>
        > wrote:
        > >>
        > >> Maven's release plugins commit and push to Git and upload to
        > repository.a.o.  I saw that some folks have a node that can commit to the
        > a.o website SVN.  Is anyone already doing releases from builds?  What
        > issues are there, if any?
        > >
        > >       It's just flat out not secure enough to do a release on.
        > >
        > > Can you give me an example of how it isn't secure enough?
        >
        >
        >         The primary purpose of these servers is to run untested,
        > unverified code.
        >
        >         Jenkins has some very sharp security corners that makes it
        > trivially un-trustable.  Something easy to understand: when Jenkins is
        > configured to run multiple builds on a node, all builds on that node run in
        > the same user space. Because there is no separation between executors, it's
        > very possible for anyone to execute something that modifies another running
        > build.  For example, probably the biggest bang for the least amount of work
        > would be to replace jars in the shared maven cache.
        >
        >         [... and no, Docker doesn't help.]
        >
        >         There are other, bigger problems, but I'd rather not put that out
        > in the public.
        >
        >
        >
        
        -- 
        Gav...
       
    

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

Good to know it isn't just me.

I could be wrong, but I believe the "policy" at Apache is only that a human must verify the packages and PGP sign them.  The packages can be built on another machine.

-Alex

On 12/8/18, 10:48 AM, "Joan Touzet" <wo...@apache.org> wrote:

    I would like to see support for something like this as well, even if it came down to individual VMs/donated HW per project, locked down by project - only project X can use build machine X'.
    
    Automated repeatable builds actually *increases* trust vs. who knows what a release manager has running on their workstation. At this point, I trust Docker builds with published, auditable cryptographic hashes per layer more than I trust some Apache releases.
    
    I don't actually believe that all projects in the Apache world are actually following the strict edict of "human must run the build and push any binary release," but I'm not going to point fingers.
    
    -Joan
    ----- Original Message -----
    From: "Alex Harui" <ah...@adobe.com.INVALID>
    To: builds@apache.org
    Sent: Saturday, December 8, 2018 12:43:37 PM
    Subject: Re: Can we package release artifacts on builds.a.o?
    
    Gavin, Alan, Karl,
    
    Thanks for the information.
    
    This email implies that there is a Jenkins node that can commit something.  What creds are used for that?  Is there a buildbot user?
    https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2Fefed1ff44fbfe5770ea1574b2f53a5295ae8326c5a3a5feb9f88cd48%40%253Cbuilds.apache.org%253E&amp;data=02%7C01%7Caharui%40adobe.com%7C5b7803529e3943bd483808d65d3daf70%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636798916859824875&amp;sdata=vJouQVPXMtHyvxOo%2BZuhhi0TmAdw9nJYPTC3HTaj1%2B0%3D&amp;reserved=0
    
    If so, I was imagining the following workflow:
    
    1) Jenkins runs Maven release.  I forgot about the PGP signing part.  If there is no way to skip it, then can a buildbot "user" PGP sign it?
    2) RM downloads the artifacts and verifies them.  The source package has to match the tag so I think that would detect any injections from other stuff running in Jenkins or elsewhere on the build server.  There's been a recent discussion on reproducible binaries and if this workflow is approved I would make our binaries are reproducible, and that should again detect any injections from the build server.
    3) RM adds his/her PGP signature to the artifacts.  Not sure if there is a Maven way to do that.
    4) Voting and other steps follow from there.
    
    These would not be continuously running jobs.  They would have to be kicked off manually so it shouldn't add significant load, and we would know which commits came from buildbot so we could detect if anything went funky.
    
    Thoughts?
    -Alex
    
    On 12/8/18, 7:54 AM, "Gavin McDonald" <ip...@gmail.com> wrote:
    
        additionally, nobody should have their creds stored anyway other than their
        own machine.
        
        On Sat, Dec 8, 2018 at 3:49 PM Allen Wittenauer
        <aw...@effectivemachines.com.invalid> wrote:
        
        >
        >
        > > On Dec 7, 2018, at 11:56 PM, Alex Harui <ah...@adobe.com.INVALID>
        > wrote:
        > >
        > >
        > >
        > > On 12/7/18, 10:49 PM, "Allen Wittenauer" <aw...@effectivemachines.com.INVALID>
        > wrote:
        > >
        > >
        > >
        > >> On Dec 7, 2018, at 10:22 PM, Alex Harui <ah...@adobe.com.INVALID>
        > wrote:
        > >>
        > >> Maven's release plugins commit and push to Git and upload to
        > repository.a.o.  I saw that some folks have a node that can commit to the
        > a.o website SVN.  Is anyone already doing releases from builds?  What
        > issues are there, if any?
        > >
        > >       It's just flat out not secure enough to do a release on.
        > >
        > > Can you give me an example of how it isn't secure enough?
        >
        >
        >         The primary purpose of these servers is to run untested,
        > unverified code.
        >
        >         Jenkins has some very sharp security corners that makes it
        > trivially un-trustable.  Something easy to understand: when Jenkins is
        > configured to run multiple builds on a node, all builds on that node run in
        > the same user space. Because there is no separation between executors, it's
        > very possible for anyone to execute something that modifies another running
        > build.  For example, probably the biggest bang for the least amount of work
        > would be to replace jars in the shared maven cache.
        >
        >         [... and no, Docker doesn't help.]
        >
        >         There are other, bigger problems, but I'd rather not put that out
        > in the public.
        >
        >
        >
        
        -- 
        Gav...
       
    

Re: Can we package release artifacts on builds.a.o?

[Joan Touzet <wo...@apache.org>]

I would like to see support for something like this as well, even if it came down to individual VMs/donated HW per project, locked down by project - only project X can use build machine X'.

Automated repeatable builds actually *increases* trust vs. who knows what a release manager has running on their workstation. At this point, I trust Docker builds with published, auditable cryptographic hashes per layer more than I trust some Apache releases.

I don't actually believe that all projects in the Apache world are actually following the strict edict of "human must run the build and push any binary release," but I'm not going to point fingers.

-Joan
----- Original Message -----
From: "Alex Harui" <ah...@adobe.com.INVALID>
To: builds@apache.org
Sent: Saturday, December 8, 2018 12:43:37 PM
Subject: Re: Can we package release artifacts on builds.a.o?

Gavin, Alan, Karl,

Thanks for the information.

This email implies that there is a Jenkins node that can commit something.  What creds are used for that?  Is there a buildbot user?
https://lists.apache.org/thread.html/efed1ff44fbfe5770ea1574b2f53a5295ae8326c5a3a5feb9f88cd48@%3Cbuilds.apache.org%3E

If so, I was imagining the following workflow:

1) Jenkins runs Maven release.  I forgot about the PGP signing part.  If there is no way to skip it, then can a buildbot "user" PGP sign it?
2) RM downloads the artifacts and verifies them.  The source package has to match the tag so I think that would detect any injections from other stuff running in Jenkins or elsewhere on the build server.  There's been a recent discussion on reproducible binaries and if this workflow is approved I would make our binaries are reproducible, and that should again detect any injections from the build server.
3) RM adds his/her PGP signature to the artifacts.  Not sure if there is a Maven way to do that.
4) Voting and other steps follow from there.

These would not be continuously running jobs.  They would have to be kicked off manually so it shouldn't add significant load, and we would know which commits came from buildbot so we could detect if anything went funky.

Thoughts?
-Alex

On 12/8/18, 7:54 AM, "Gavin McDonald" <ip...@gmail.com> wrote:

    additionally, nobody should have their creds stored anyway other than their
    own machine.
    
    On Sat, Dec 8, 2018 at 3:49 PM Allen Wittenauer
    <aw...@effectivemachines.com.invalid> wrote:
    
    >
    >
    > > On Dec 7, 2018, at 11:56 PM, Alex Harui <ah...@adobe.com.INVALID>
    > wrote:
    > >
    > >
    > >
    > > On 12/7/18, 10:49 PM, "Allen Wittenauer" <aw...@effectivemachines.com.INVALID>
    > wrote:
    > >
    > >
    > >
    > >> On Dec 7, 2018, at 10:22 PM, Alex Harui <ah...@adobe.com.INVALID>
    > wrote:
    > >>
    > >> Maven's release plugins commit and push to Git and upload to
    > repository.a.o.  I saw that some folks have a node that can commit to the
    > a.o website SVN.  Is anyone already doing releases from builds?  What
    > issues are there, if any?
    > >
    > >       It's just flat out not secure enough to do a release on.
    > >
    > > Can you give me an example of how it isn't secure enough?
    >
    >
    >         The primary purpose of these servers is to run untested,
    > unverified code.
    >
    >         Jenkins has some very sharp security corners that makes it
    > trivially un-trustable.  Something easy to understand: when Jenkins is
    > configured to run multiple builds on a node, all builds on that node run in
    > the same user space. Because there is no separation between executors, it's
    > very possible for anyone to execute something that modifies another running
    > build.  For example, probably the biggest bang for the least amount of work
    > would be to replace jars in the shared maven cache.
    >
    >         [... and no, Docker doesn't help.]
    >
    >         There are other, bigger problems, but I'd rather not put that out
    > in the public.
    >
    >
    >
    
    -- 
    Gav...
   

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

Gavin, Alan, Karl,

Thanks for the information.

This email implies that there is a Jenkins node that can commit something.  What creds are used for that?  Is there a buildbot user?
https://lists.apache.org/thread.html/efed1ff44fbfe5770ea1574b2f53a5295ae8326c5a3a5feb9f88cd48@%3Cbuilds.apache.org%3E

If so, I was imagining the following workflow:

1) Jenkins runs Maven release.  I forgot about the PGP signing part.  If there is no way to skip it, then can a buildbot "user" PGP sign it?
2) RM downloads the artifacts and verifies them.  The source package has to match the tag so I think that would detect any injections from other stuff running in Jenkins or elsewhere on the build server.  There's been a recent discussion on reproducible binaries and if this workflow is approved I would make our binaries are reproducible, and that should again detect any injections from the build server.
3) RM adds his/her PGP signature to the artifacts.  Not sure if there is a Maven way to do that.
4) Voting and other steps follow from there.

These would not be continuously running jobs.  They would have to be kicked off manually so it shouldn't add significant load, and we would know which commits came from buildbot so we could detect if anything went funky.

Thoughts?
-Alex

On 12/8/18, 7:54 AM, "Gavin McDonald" <ip...@gmail.com> wrote:

    additionally, nobody should have their creds stored anyway other than their
    own machine.
    
    On Sat, Dec 8, 2018 at 3:49 PM Allen Wittenauer
    <aw...@effectivemachines.com.invalid> wrote:
    
    >
    >
    > > On Dec 7, 2018, at 11:56 PM, Alex Harui <ah...@adobe.com.INVALID>
    > wrote:
    > >
    > >
    > >
    > > On 12/7/18, 10:49 PM, "Allen Wittenauer" <aw...@effectivemachines.com.INVALID>
    > wrote:
    > >
    > >
    > >
    > >> On Dec 7, 2018, at 10:22 PM, Alex Harui <ah...@adobe.com.INVALID>
    > wrote:
    > >>
    > >> Maven's release plugins commit and push to Git and upload to
    > repository.a.o.  I saw that some folks have a node that can commit to the
    > a.o website SVN.  Is anyone already doing releases from builds?  What
    > issues are there, if any?
    > >
    > >       It's just flat out not secure enough to do a release on.
    > >
    > > Can you give me an example of how it isn't secure enough?
    >
    >
    >         The primary purpose of these servers is to run untested,
    > unverified code.
    >
    >         Jenkins has some very sharp security corners that makes it
    > trivially un-trustable.  Something easy to understand: when Jenkins is
    > configured to run multiple builds on a node, all builds on that node run in
    > the same user space. Because there is no separation between executors, it's
    > very possible for anyone to execute something that modifies another running
    > build.  For example, probably the biggest bang for the least amount of work
    > would be to replace jars in the shared maven cache.
    >
    >         [... and no, Docker doesn't help.]
    >
    >         There are other, bigger problems, but I'd rather not put that out
    > in the public.
    >
    >
    >
    
    -- 
    Gav...
    

Re: Can we package release artifacts on builds.a.o?

[Gavin McDonald <ip...@gmail.com>]

additionally, nobody should have their creds stored anyway other than their
own machine.

On Sat, Dec 8, 2018 at 3:49 PM Allen Wittenauer
<aw...@effectivemachines.com.invalid> wrote:

>
>
> > On Dec 7, 2018, at 11:56 PM, Alex Harui <ah...@adobe.com.INVALID>
> wrote:
> >
> >
> >
> > On 12/7/18, 10:49 PM, "Allen Wittenauer" <aw...@effectivemachines.com.INVALID>
> wrote:
> >
> >
> >
> >> On Dec 7, 2018, at 10:22 PM, Alex Harui <ah...@adobe.com.INVALID>
> wrote:
> >>
> >> Maven's release plugins commit and push to Git and upload to
> repository.a.o.  I saw that some folks have a node that can commit to the
> a.o website SVN.  Is anyone already doing releases from builds?  What
> issues are there, if any?
> >
> >       It's just flat out not secure enough to do a release on.
> >
> > Can you give me an example of how it isn't secure enough?
>
>
>         The primary purpose of these servers is to run untested,
> unverified code.
>
>         Jenkins has some very sharp security corners that makes it
> trivially un-trustable.  Something easy to understand: when Jenkins is
> configured to run multiple builds on a node, all builds on that node run in
> the same user space. Because there is no separation between executors, it's
> very possible for anyone to execute something that modifies another running
> build.  For example, probably the biggest bang for the least amount of work
> would be to replace jars in the shared maven cache.
>
>         [... and no, Docker doesn't help.]
>
>         There are other, bigger problems, but I'd rather not put that out
> in the public.
>
>
>

-- 
Gav...

Re: Can we package release artifacts on builds.a.o?

[Allen Wittenauer <aw...@effectivemachines.com.INVALID>]

> On Dec 7, 2018, at 11:56 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
> 
> 
> 
> On 12/7/18, 10:49 PM, "Allen Wittenauer" <aw...@effectivemachines.com.INVALID> wrote:
> 
> 
> 
>> On Dec 7, 2018, at 10:22 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>> 
>> Maven's release plugins commit and push to Git and upload to repository.a.o.  I saw that some folks have a node that can commit to the a.o website SVN.  Is anyone already doing releases from builds?  What issues are there, if any?
> 
>    	It's just flat out not secure enough to do a release on.
> 
> Can you give me an example of how it isn't secure enough?


	The primary purpose of these servers is to run untested, unverified code.  

	Jenkins has some very sharp security corners that makes it trivially un-trustable.  Something easy to understand: when Jenkins is configured to run multiple builds on a node, all builds on that node run in the same user space. Because there is no separation between executors, it's very possible for anyone to execute something that modifies another running build.  For example, probably the biggest bang for the least amount of work would be to replace jars in the shared maven cache.

	[... and no, Docker doesn't help.]

	There are other, bigger problems, but I'd rather not put that out in the public.

Re: Can we package release artifacts on builds.a.o?

[Gavin McDonald <gm...@apache.org>]

The build servers are not geared up for doing releases and we don't
recommend it at all.

We have certain build servers for publishing websites to our own tlp
servers.

HTH

Gav...


On Sat, Dec 8, 2018 at 10:50 AM Karl Heinz Marbaise <kh...@gmx.de>
wrote:

> Hi,
>
> On 08/12/18 08:56, Alex Harui wrote:
> >
> >
> > On 12/7/18, 10:49 PM, "Allen Wittenauer" <aw...@effectivemachines.com.INVALID>
> wrote:
> >
> >
> >
> >      > On Dec 7, 2018, at 10:22 PM, Alex Harui <ah...@adobe.com.INVALID>
> wrote:
> >      >
> >      > Maven's release plugins commit and push to Git and upload to
> repository.a.o.  I saw that some folks have a node that can commit to the
> a.o website SVN.  Is anyone already doing releases from builds?  What
> issues are there, if any?
> >
> >       It's just flat out not secure enough to do a release on.
> >
> > Can you give me an example of how it isn't secure enough?
>
>
> I would be interested as well...
>
> Furthermore we (Maven Team) doing the website via SVN/GitPubSub on  a
> node but NOT for the releases of plugins/components only for the
> https://maven.apache.org/ site[1]
>
> The releases are done from the local machine for a reason...We sign the
> artifacts which could only do from my local machine otherwise I had to
> upload my GPG key to a machine ?
>
>
> Kind regards
> Karl Heinz Marbaise
> Apache Maven PMC
>
> [1]:
> https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven-site/
>

Re: Can we package release artifacts on builds.a.o?

[Karl Heinz Marbaise <kh...@gmx.de>]

Hi,

On 08/12/18 08:56, Alex Harui wrote:
> 
> 
> On 12/7/18, 10:49 PM, "Allen Wittenauer" <aw...@effectivemachines.com.INVALID> wrote:
> 
>      
>      
>      > On Dec 7, 2018, at 10:22 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>      >
>      > Maven's release plugins commit and push to Git and upload to repository.a.o.  I saw that some folks have a node that can commit to the a.o website SVN.  Is anyone already doing releases from builds?  What issues are there, if any?
>      
>      	It's just flat out not secure enough to do a release on.
>      
> Can you give me an example of how it isn't secure enough?


I would be interested as well...

Furthermore we (Maven Team) doing the website via SVN/GitPubSub on  a 
node but NOT for the releases of plugins/components only for the 
https://maven.apache.org/ site[1]

The releases are done from the local machine for a reason...We sign the 
artifacts which could only do from my local machine otherwise I had to 
upload my GPG key to a machine ?


Kind regards
Karl Heinz Marbaise
Apache Maven PMC

[1]: 
https://builds.apache.org/view/M-R/view/Maven/job/maven-box/job/maven-site/

Re: Can we package release artifacts on builds.a.o?

[Alex Harui <ah...@adobe.com.INVALID>]

On 12/7/18, 10:49 PM, "Allen Wittenauer" <aw...@effectivemachines.com.INVALID> wrote:

    
    
    > On Dec 7, 2018, at 10:22 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
    > 
    > Maven's release plugins commit and push to Git and upload to repository.a.o.  I saw that some folks have a node that can commit to the a.o website SVN.  Is anyone already doing releases from builds?  What issues are there, if any?
    
    	It's just flat out not secure enough to do a release on.
    
Can you give me an example of how it isn't secure enough?

Thanks,
-Alex

Re: Can we package release artifacts on builds.a.o?

[Allen Wittenauer <aw...@effectivemachines.com.INVALID>]

> On Dec 7, 2018, at 10:22 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
> 
> Maven's release plugins commit and push to Git and upload to repository.a.o.  I saw that some folks have a node that can commit to the a.o website SVN.  Is anyone already doing releases from builds?  What issues are there, if any?

	It's just flat out not secure enough to do a release on.