You are viewing a plain text version of this content. The canonical link for it is here.
Posted to security-discuss@community.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2024/04/02 06:52:01 UTC
[DISCUSS] Should we update our policies to include source provenance check
Following some of the learnings from the CVE-2024-3094 (xz backdoor) and a
few resulting discussions. I would like to start a discussion on that very
specific topic:
TL;DR; I think that we currently do not explicitly state the requirement of
verifying if the release manager has not tampered with the sources when
preparing the source package - and I believe we should be more explicit
about it and require from PMC members to do such verification.
As explained in [1] - there were two important triggers for the CVE to
happen:
a) the attacker was able to gain trust, become a maintainer and release
manager
b) they submitted test binaries to the repository of xv that contained
malicious code
c) acting as release manager - they modified the official source tar-ball
packages of xz to contain a modified Makefile that turn anyone using
official source-tar-ball packages to produce a malicious version of the xz
library (that malicious Makefile had never been part of the source
repository, it's not been reviewed nor approved by anyone).
When I look at requirements explained in our release policy, I think this
kind of scenario (especially point c) is not something our release
policies protect us against, because we have no requirement to check
provenance of the source code in the released package.
Or at least I cannot find it in neither release policy [2] nor distribution
policy [3].
From [2]:
> Before casting +1 binding votes, individuals are REQUIRED to download all
signed source code packages onto their own hardware, verify that they meet
all requirements of ASF policy on releases as described below, validate all
cryptographic signatures, compile as provided, and test the result on their
own platform.
Even if we assume such a check is part of "meet all requirements of ASF
policy on releases" - there is no "check if the sources in the package have
not been modified vs. source repository" anywhere in the policies as far as
I can see.
From earlier discussions - many of us think that verifying whether the
sources in the "source" package contain the same sources as ones stored in
our source repositories is the most important part of such verification,
but - somewhat to my surprise - it has not been explicitly stated in our
policies. And I think it should be an important gate to have PMC members to
be REQUIRED to verify that. That could be done in whatever way is
appropriate for the project - it could be just comparing sources with git,
or having reproducible packages that PMC members can build and compare for
binary identity if the project supports it.
But similarly to comparing cryptographic signatures, possibly we should
explicitly state that this should be a mandatory check. And maybe we have
the chance to use the CVE-2024-3094 as an opportunity to remind/advocate it
and explain what could happen if this step is missing when our source
packages are released?
I think the way it's stated, a malicious release manager could do a similar
package modification as xz release manager did and we could have missed it.
Our policies on release do not have explicit gates protecting against this
- so PMC members explicitly give +1 - following the release policy pretty
rigorously, could have not realise the malicious release manager did such
Just to give an example from the past Airflow releases - when I came to the
project, I've learned how it works, and the release process was very
rigorously followed, including licences, signatures, etc. and we even
pulled a few releases when those were not met. But it's only a few years
later when I became a release manager and realised that I could potentially
act maliciously and modify the packages and .... no-one would notice - we
introduced source provenance check first and reproducible packages after
that realisation.
Or maybe I am exaggerating, and it's "obvious enough" that we do not have
to state it?
I'd love to hear what others think here. And I am happy to provide a
concrete proposal to the policy and do some advocacy for it, if others
think it is needed as well.
J.
[1] https://boehs.org/node/everything-i-know-about-the-xz-backdoor
[2] https://www.apache.org/legal/release-policy.html
[3] https://infra.apache.org/release-distribution.html#sigs-and-sums
Re: [DISCUSS] Should we update our policies to include source provenance check
Posted by "Piotr P. Karwasz" <pi...@gmail.com>.
Hi Jarek,
On Tue, 2 Apr 2024 at 08:52, Jarek Potiuk <ja...@potiuk.com> wrote:
> From earlier discussions - many of us think that verifying whether the
> sources in the "source" package contain the same sources as ones stored in
> our source repositories is the most important part of such verification,
> but - somewhat to my surprise - it has not been explicitly stated in our
> policies. And I think it should be an important gate to have PMC members to
> be REQUIRED to verify that. That could be done in whatever way is
> appropriate for the project - it could be just comparing sources with git,
> or having reproducible packages that PMC members can build and compare for
> binary identity if the project supports it.
Verifying the source archive is not such a trivial task, simply
because the contents of the Git repo and source archive **do** differ.
Most Apache projects that use Maven, use assembly descriptors from the
15 years old `apache-jar-resource-bundle`[1]. These descriptors both:
* remove CI-specific resources that are not useful in the source bundle[1],
* add resources generated at build time[2].
Therefore our source bundles are open to the same attack vector as `liblzma`.
In the Logging Services project we chose to:
1. Create our source archives as **exact** copy of the Git repo,
2. We verify in an indirect way that the source archive is correct:
the published artifacts are generated from the Git repo, while we
verify their reproducibility using the source archive.
Piotr
[1] https://github.com/apache/maven-apache-resources
[2] https://github.com/apache/maven-apache-resources/blob/f609acfd574277be6382ce381f65cab2db895d8d/source-release/src/main/resources/assemblies/source-shared.xml#L58-L63
[3] https://github.com/apache/maven-apache-resources/blob/f609acfd574277be6382ce381f65cab2db895d8d/source-release/src/main/resources/assemblies/source-shared.xml#L76-L80
---------------------------------------------------------------------
To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
For additional commands, e-mail: security-discuss-help@community.apache.org
Re: [DISCUSS] Should we update our policies to include source provenance check
Posted by Jarek Potiuk <ja...@potiuk.com>.
Following up what Mark/ Piotr mentioned and echoing what Sebb wrote:
> Yes, there may be some exceptions where files are needed in the tarball
that are not in the repo.
> However these must be directly derivable from the source repo by
anyone with the appropriate tools.
This is also what I like about the "reproducibility" movement. And not
even bit-to-bit reproducibly. If we are able to have a replicable release
environment and have Release Manager + min 2 other PMC members to actually
build the package from sources + tools - then we do not have to track and
see if some files had been derived from some other files properly.
We could simply compare the three packages generated by 3 different people
using 3 different machines - each of them maintained separately. With
bit-to-bit reproducibility it's easier, but even without it - comparing the
results and reasonable explanation on where the differences come from
(timestamps and the like) is "enough" to announce "Yep, provenance of
these packages has been confirmed as coming from this particular TAG /
commit Hash in the Git repo - because 3 people built it".
Is it wasteful ? Probably - depending how difficult it is to build those
packages and how complicated it is to set-up a build environment and
reproduce it by another person. And those people might not have appropriate
tools and environment, it might take time/effort to have it setup of
course. And the release process might not be "rigid" enough to give similar
results when 3 people follow the "source package build" procedure.
But is it useful? IMHO - immensely. Eventually having the process well
described, reproducible by another person who just follows the instructions
and does not have other undocumented prerequisites is precisely what makes
the release process good, and where "bus factor" is > 1.
I would think about this kind of policy: "PMC member when voting +1 is
REQUIRED to verify the provenance of the package to make sure it comes from
the sources in version control system In whatever way the PMC feels
appropriate: comparing sources, or reproducible builds, or just preparing
the same package and comparing if any differences can be explained". I
would never ever want to mandate HOW to do it, just make sure that
"provenance" is something that is needed for +1 (by default).
That **might** be achievable by virtually all our projects, and might also
eventually drive them to improve and automate more parts of it - eventually
making it possible for the new platform that infra develops - because
basically having a reproducible, fully automated process of building the
package will be prerequisite for using such platform. And maybe some
exceptions might be documented and explained why it can't be achieved with
a goal to achieve it. Even if there will be few projects that won't be able
to achieve it, I'd rather add this expectation by default, and allow for
exceptions.
And then (long term) - we could even do "get the package prepared by the
ASF distribution platform, and have 3 PMC members build it locally to
compare what they built with the one generated by the distribution
platform" - which could allow some of our builds to be
build/signed/published mostly in the ASF, while PMC members just "Verify"
that the packages have the right provenance.
J.
On Tue, Apr 2, 2024 at 1:16 PM sebb <se...@gmail.com> wrote:
> On Tue, 2 Apr 2024 at 09:51, Mark Thomas <ma...@apache.org> wrote:
> >
> > On 02/04/2024 09:12, sebb wrote:
> > > On Tue, 2 Apr 2024 at 08:47, Christofer Dutz <
> christofer.dutz@c-ware.de> wrote:
> > >>
> > >> Hi all,
> > >>
> > >>
> > >>
> > >> I fully agree on this … and adding to sebb’s statement that
> additional files can happen even without malicious intent.
> > >>
> > >> I have seen this several times, if for example maven releases are
> built directly from the main checkout and not from the release-plugin
> checking out the release commit hash and building in a clean directory.
> > >>
> > >
> > > A clean checkout helps, but is no guarantee.
> > >
> > > Spurious files can end up in a source tarball even when it is created
> > > from a clean checkout.
> > >
> > > I saw this in a Maven build where faulty test code left behind some
> > > test artifacts.
> > > Since Maven creates the source archive after the test phase, such
> > > files can end up being included.
> >
> > Nice idea in principle, but it is going to create issues for C based
> > projects in practice.
> >
> > End users expect to be able to build C based projects with configure,
> > make, make install. That only works because the release manager runs a
> > script (typically called buildconf that uses autoconf) to create the
> > support scripts required in the src tarball.
>
> Yes, there may be some exceptions where files are needed in the
> tarball that are not in the repo.
> However these must be directly derivable from the source repo by
> anyone with the appropriate tools.
>
> > There are various possible solutions but I strongly suggest engagement
> > with projects such as httpd before trying to change the current policy.
> >
> > Mark
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> security-discuss-unsubscribe@community.apache.org
> > For additional commands, e-mail:
> security-discuss-help@community.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
> For additional commands, e-mail:
> security-discuss-help@community.apache.org
>
>
Re: [DISCUSS] Should we update our policies to include source provenance check
Posted by sebb <se...@gmail.com>.
On Tue, 2 Apr 2024 at 09:51, Mark Thomas <ma...@apache.org> wrote:
>
> On 02/04/2024 09:12, sebb wrote:
> > On Tue, 2 Apr 2024 at 08:47, Christofer Dutz <ch...@c-ware.de> wrote:
> >>
> >> Hi all,
> >>
> >>
> >>
> >> I fully agree on this … and adding to sebb’s statement that additional files can happen even without malicious intent.
> >>
> >> I have seen this several times, if for example maven releases are built directly from the main checkout and not from the release-plugin checking out the release commit hash and building in a clean directory.
> >>
> >
> > A clean checkout helps, but is no guarantee.
> >
> > Spurious files can end up in a source tarball even when it is created
> > from a clean checkout.
> >
> > I saw this in a Maven build where faulty test code left behind some
> > test artifacts.
> > Since Maven creates the source archive after the test phase, such
> > files can end up being included.
>
> Nice idea in principle, but it is going to create issues for C based
> projects in practice.
>
> End users expect to be able to build C based projects with configure,
> make, make install. That only works because the release manager runs a
> script (typically called buildconf that uses autoconf) to create the
> support scripts required in the src tarball.
Yes, there may be some exceptions where files are needed in the
tarball that are not in the repo.
However these must be directly derivable from the source repo by
anyone with the appropriate tools.
> There are various possible solutions but I strongly suggest engagement
> with projects such as httpd before trying to change the current policy.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
> For additional commands, e-mail: security-discuss-help@community.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
For additional commands, e-mail: security-discuss-help@community.apache.org
Re: [DISCUSS] Should we update our policies to include source provenance check
Posted by Mark Thomas <ma...@apache.org>.
On 02/04/2024 09:12, sebb wrote:
> On Tue, 2 Apr 2024 at 08:47, Christofer Dutz <ch...@c-ware.de> wrote:
>>
>> Hi all,
>>
>>
>>
>> I fully agree on this … and adding to sebb’s statement that additional files can happen even without malicious intent.
>>
>> I have seen this several times, if for example maven releases are built directly from the main checkout and not from the release-plugin checking out the release commit hash and building in a clean directory.
>>
>
> A clean checkout helps, but is no guarantee.
>
> Spurious files can end up in a source tarball even when it is created
> from a clean checkout.
>
> I saw this in a Maven build where faulty test code left behind some
> test artifacts.
> Since Maven creates the source archive after the test phase, such
> files can end up being included.
Nice idea in principle, but it is going to create issues for C based
projects in practice.
End users expect to be able to build C based projects with configure,
make, make install. That only works because the release manager runs a
script (typically called buildconf that uses autoconf) to create the
support scripts required in the src tarball.
There are various possible solutions but I strongly suggest engagement
with projects such as httpd before trying to change the current policy.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
For additional commands, e-mail: security-discuss-help@community.apache.org
Re: [DISCUSS] Should we update our policies to include source provenance check
Posted by sebb <se...@gmail.com>.
On Tue, 2 Apr 2024 at 08:47, Christofer Dutz <ch...@c-ware.de> wrote:
>
> Hi all,
>
>
>
> I fully agree on this … and adding to sebb’s statement that additional files can happen even without malicious intent.
>
> I have seen this several times, if for example maven releases are built directly from the main checkout and not from the release-plugin checking out the release commit hash and building in a clean directory.
>
A clean checkout helps, but is no guarantee.
Spurious files can end up in a source tarball even when it is created
from a clean checkout.
I saw this in a Maven build where faulty test code left behind some
test artifacts.
Since Maven creates the source archive after the test phase, such
files can end up being included.
>
> But indeed, this has also been something worrying me, and I think it could possibly become important with all the CRA and PLD stuff coming our way.
>
>
>
> Chris
>
>
>
>
>
> Von: sebb <se...@gmail.com>
> Datum: Dienstag, 2. April 2024 um 09:14
> An: Jarek Potiuk <ja...@potiuk.com>
> Cc: security-discuss@community.apache.org <se...@community.apache.org>, Users <us...@infra.apache.org>
> Betreff: Re: [DISCUSS] Should we update our policies to include source provenance check
>
> WARNING: this post mixes public and private lists.
>
> In Commons reviewers are supposed to check that the source tarball
> contents all match files from the tag in the vote.
> The reason is mainly for provenance, and licensing, but it is not
> unknown for spurious files to be accidentally added to the source
> tarball.
> Things can go wrong even without malicious intent.
>
> So I agree that this is a vital part of the process, and should be
> made explicit.
>
> On Tue, 2 Apr 2024 at 07:52, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > Following some of the learnings from the CVE-2024-3094 (xz backdoor) and a few resulting discussions. I would like to start a discussion on that very specific topic:
> >
> > TL;DR; I think that we currently do not explicitly state the requirement of verifying if the release manager has not tampered with the sources when preparing the source package - and I believe we should be more explicit about it and require from PMC members to do such verification.
> >
> > As explained in [1] - there were two important triggers for the CVE to happen:
> >
> > a) the attacker was able to gain trust, become a maintainer and release manager
> > b) they submitted test binaries to the repository of xv that contained malicious code
> > c) acting as release manager - they modified the official source tar-ball packages of xz to contain a modified Makefile that turn anyone using official source-tar-ball packages to produce a malicious version of the xz library (that malicious Makefile had never been part of the source repository, it's not been reviewed nor approved by anyone).
> >
> > When I look at requirements explained in our release policy, I think this kind of scenario (especially point c) is not something our release policies protect us against, because we have no requirement to check provenance of the source code in the released package.
> >
> > Or at least I cannot find it in neither release policy [2] nor distribution policy [3].
> >
> > From [2]:
> >
> > > Before casting +1 binding votes, individuals are REQUIRED to download all signed source code packages onto their own hardware, verify that they meet all requirements of ASF policy on releases as described below, validate all cryptographic signatures, compile as provided, and test the result on their own platform.
> >
> > Even if we assume such a check is part of "meet all requirements of ASF policy on releases" - there is no "check if the sources in the package have not been modified vs. source repository" anywhere in the policies as far as I can see.
> >
> > From earlier discussions - many of us think that verifying whether the sources in the "source" package contain the same sources as ones stored in our source repositories is the most important part of such verification, but - somewhat to my surprise - it has not been explicitly stated in our policies. And I think it should be an important gate to have PMC members to be REQUIRED to verify that. That could be done in whatever way is appropriate for the project - it could be just comparing sources with git, or having reproducible packages that PMC members can build and compare for binary identity if the project supports it.
> >
> > But similarly to comparing cryptographic signatures, possibly we should explicitly state that this should be a mandatory check. And maybe we have the chance to use the CVE-2024-3094 as an opportunity to remind/advocate it and explain what could happen if this step is missing when our source packages are released?
> >
> > I think the way it's stated, a malicious release manager could do a similar package modification as xz release manager did and we could have missed it. Our policies on release do not have explicit gates protecting against this - so PMC members explicitly give +1 - following the release policy pretty rigorously, could have not realise the malicious release manager did such
> >
> > Just to give an example from the past Airflow releases - when I came to the project, I've learned how it works, and the release process was very rigorously followed, including licences, signatures, etc. and we even pulled a few releases when those were not met. But it's only a few years later when I became a release manager and realised that I could potentially act maliciously and modify the packages and .... no-one would notice - we introduced source provenance check first and reproducible packages after that realisation.
> >
> > Or maybe I am exaggerating, and it's "obvious enough" that we do not have to state it?
> >
> > I'd love to hear what others think here. And I am happy to provide a concrete proposal to the policy and do some advocacy for it, if others think it is needed as well.
> >
> > J.
> >
> >
> > [1] https://boehs.org/node/everything-i-know-about-the-xz-backdoor
> > [2] https://www.apache.org/legal/release-policy.html
> > [3] https://infra.apache.org/release-distribution.html#sigs-and-sums
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
For additional commands, e-mail: security-discuss-help@community.apache.org
AW: [DISCUSS] Should we update our policies to include source provenance check
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi all,
I fully agree on this … and adding to sebb’s statement that additional files can happen even without malicious intent.
I have seen this several times, if for example maven releases are built directly from the main checkout and not from the release-plugin checking out the release commit hash and building in a clean directory.
But indeed, this has also been something worrying me, and I think it could possibly become important with all the CRA and PLD stuff coming our way.
Chris
Von: sebb <se...@gmail.com>
Datum: Dienstag, 2. April 2024 um 09:14
An: Jarek Potiuk <ja...@potiuk.com>
Cc: security-discuss@community.apache.org <se...@community.apache.org>, Users <us...@infra.apache.org>
Betreff: Re: [DISCUSS] Should we update our policies to include source provenance check
WARNING: this post mixes public and private lists.
In Commons reviewers are supposed to check that the source tarball
contents all match files from the tag in the vote.
The reason is mainly for provenance, and licensing, but it is not
unknown for spurious files to be accidentally added to the source
tarball.
Things can go wrong even without malicious intent.
So I agree that this is a vital part of the process, and should be
made explicit.
On Tue, 2 Apr 2024 at 07:52, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Following some of the learnings from the CVE-2024-3094 (xz backdoor) and a few resulting discussions. I would like to start a discussion on that very specific topic:
>
> TL;DR; I think that we currently do not explicitly state the requirement of verifying if the release manager has not tampered with the sources when preparing the source package - and I believe we should be more explicit about it and require from PMC members to do such verification.
>
> As explained in [1] - there were two important triggers for the CVE to happen:
>
> a) the attacker was able to gain trust, become a maintainer and release manager
> b) they submitted test binaries to the repository of xv that contained malicious code
> c) acting as release manager - they modified the official source tar-ball packages of xz to contain a modified Makefile that turn anyone using official source-tar-ball packages to produce a malicious version of the xz library (that malicious Makefile had never been part of the source repository, it's not been reviewed nor approved by anyone).
>
> When I look at requirements explained in our release policy, I think this kind of scenario (especially point c) is not something our release policies protect us against, because we have no requirement to check provenance of the source code in the released package.
>
> Or at least I cannot find it in neither release policy [2] nor distribution policy [3].
>
> From [2]:
>
> > Before casting +1 binding votes, individuals are REQUIRED to download all signed source code packages onto their own hardware, verify that they meet all requirements of ASF policy on releases as described below, validate all cryptographic signatures, compile as provided, and test the result on their own platform.
>
> Even if we assume such a check is part of "meet all requirements of ASF policy on releases" - there is no "check if the sources in the package have not been modified vs. source repository" anywhere in the policies as far as I can see.
>
> From earlier discussions - many of us think that verifying whether the sources in the "source" package contain the same sources as ones stored in our source repositories is the most important part of such verification, but - somewhat to my surprise - it has not been explicitly stated in our policies. And I think it should be an important gate to have PMC members to be REQUIRED to verify that. That could be done in whatever way is appropriate for the project - it could be just comparing sources with git, or having reproducible packages that PMC members can build and compare for binary identity if the project supports it.
>
> But similarly to comparing cryptographic signatures, possibly we should explicitly state that this should be a mandatory check. And maybe we have the chance to use the CVE-2024-3094 as an opportunity to remind/advocate it and explain what could happen if this step is missing when our source packages are released?
>
> I think the way it's stated, a malicious release manager could do a similar package modification as xz release manager did and we could have missed it. Our policies on release do not have explicit gates protecting against this - so PMC members explicitly give +1 - following the release policy pretty rigorously, could have not realise the malicious release manager did such
>
> Just to give an example from the past Airflow releases - when I came to the project, I've learned how it works, and the release process was very rigorously followed, including licences, signatures, etc. and we even pulled a few releases when those were not met. But it's only a few years later when I became a release manager and realised that I could potentially act maliciously and modify the packages and .... no-one would notice - we introduced source provenance check first and reproducible packages after that realisation.
>
> Or maybe I am exaggerating, and it's "obvious enough" that we do not have to state it?
>
> I'd love to hear what others think here. And I am happy to provide a concrete proposal to the policy and do some advocacy for it, if others think it is needed as well.
>
> J.
>
>
> [1] https://boehs.org/node/everything-i-know-about-the-xz-backdoor
> [2] https://www.apache.org/legal/release-policy.html
> [3] https://infra.apache.org/release-distribution.html#sigs-and-sums
>
Re: [DISCUSS] Should we update our policies to include source provenance check
Posted by Jarek Potiuk <ja...@potiuk.com>.
> WARNING: this post mixes public and private lists.
Yes. Apologies. I keep on forgetting that builds@ is public and users@ is
private. My intention is to make it a public discussion - there is nothing
"private" in this discussion - all the docs/references are public. I moved
"users@" to bcc: - and I guess if someone would like to comment on that can
come here to security-discuss@
J.
On Tue, Apr 2, 2024 at 9:12 AM sebb <se...@gmail.com> wrote:
> WARNING: this post mixes public and private lists.
>
> In Commons reviewers are supposed to check that the source tarball
> contents all match files from the tag in the vote.
> The reason is mainly for provenance, and licensing, but it is not
> unknown for spurious files to be accidentally added to the source
> tarball.
> Things can go wrong even without malicious intent.
>
> So I agree that this is a vital part of the process, and should be
> made explicit.
>
> On Tue, 2 Apr 2024 at 07:52, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > Following some of the learnings from the CVE-2024-3094 (xz backdoor) and
> a few resulting discussions. I would like to start a discussion on that
> very specific topic:
> >
> > TL;DR; I think that we currently do not explicitly state the requirement
> of verifying if the release manager has not tampered with the sources when
> preparing the source package - and I believe we should be more explicit
> about it and require from PMC members to do such verification.
> >
> > As explained in [1] - there were two important triggers for the CVE to
> happen:
> >
> > a) the attacker was able to gain trust, become a maintainer and release
> manager
> > b) they submitted test binaries to the repository of xv that contained
> malicious code
> > c) acting as release manager - they modified the official source
> tar-ball packages of xz to contain a modified Makefile that turn anyone
> using official source-tar-ball packages to produce a malicious version of
> the xz library (that malicious Makefile had never been part of the source
> repository, it's not been reviewed nor approved by anyone).
> >
> > When I look at requirements explained in our release policy, I think
> this kind of scenario (especially point c) is not something our release
> policies protect us against, because we have no requirement to check
> provenance of the source code in the released package.
> >
> > Or at least I cannot find it in neither release policy [2] nor
> distribution policy [3].
> >
> > From [2]:
> >
> > > Before casting +1 binding votes, individuals are REQUIRED to download
> all signed source code packages onto their own hardware, verify that they
> meet all requirements of ASF policy on releases as described below,
> validate all cryptographic signatures, compile as provided, and test the
> result on their own platform.
> >
> > Even if we assume such a check is part of "meet all requirements of ASF
> policy on releases" - there is no "check if the sources in the package have
> not been modified vs. source repository" anywhere in the policies as far as
> I can see.
> >
> > From earlier discussions - many of us think that verifying whether the
> sources in the "source" package contain the same sources as ones stored in
> our source repositories is the most important part of such verification,
> but - somewhat to my surprise - it has not been explicitly stated in our
> policies. And I think it should be an important gate to have PMC members to
> be REQUIRED to verify that. That could be done in whatever way is
> appropriate for the project - it could be just comparing sources with git,
> or having reproducible packages that PMC members can build and compare for
> binary identity if the project supports it.
> >
> > But similarly to comparing cryptographic signatures, possibly we should
> explicitly state that this should be a mandatory check. And maybe we have
> the chance to use the CVE-2024-3094 as an opportunity to remind/advocate it
> and explain what could happen if this step is missing when our source
> packages are released?
> >
> > I think the way it's stated, a malicious release manager could do a
> similar package modification as xz release manager did and we could have
> missed it. Our policies on release do not have explicit gates protecting
> against this - so PMC members explicitly give +1 - following the release
> policy pretty rigorously, could have not realise the malicious release
> manager did such
> >
> > Just to give an example from the past Airflow releases - when I came to
> the project, I've learned how it works, and the release process was very
> rigorously followed, including licences, signatures, etc. and we even
> pulled a few releases when those were not met. But it's only a few years
> later when I became a release manager and realised that I could potentially
> act maliciously and modify the packages and .... no-one would notice - we
> introduced source provenance check first and reproducible packages after
> that realisation.
> >
> > Or maybe I am exaggerating, and it's "obvious enough" that we do not
> have to state it?
> >
> > I'd love to hear what others think here. And I am happy to provide a
> concrete proposal to the policy and do some advocacy for it, if others
> think it is needed as well.
> >
> > J.
> >
> >
> > [1] https://boehs.org/node/everything-i-know-about-the-xz-backdoor
> > [2] https://www.apache.org/legal/release-policy.html
> > [3] https://infra.apache.org/release-distribution.html#sigs-and-sums
> >
>
Re: [DISCUSS] Should we update our policies to include source provenance check
Posted by sebb <se...@gmail.com>.
WARNING: this post mixes public and private lists.
In Commons reviewers are supposed to check that the source tarball
contents all match files from the tag in the vote.
The reason is mainly for provenance, and licensing, but it is not
unknown for spurious files to be accidentally added to the source
tarball.
Things can go wrong even without malicious intent.
So I agree that this is a vital part of the process, and should be
made explicit.
On Tue, 2 Apr 2024 at 07:52, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Following some of the learnings from the CVE-2024-3094 (xz backdoor) and a few resulting discussions. I would like to start a discussion on that very specific topic:
>
> TL;DR; I think that we currently do not explicitly state the requirement of verifying if the release manager has not tampered with the sources when preparing the source package - and I believe we should be more explicit about it and require from PMC members to do such verification.
>
> As explained in [1] - there were two important triggers for the CVE to happen:
>
> a) the attacker was able to gain trust, become a maintainer and release manager
> b) they submitted test binaries to the repository of xv that contained malicious code
> c) acting as release manager - they modified the official source tar-ball packages of xz to contain a modified Makefile that turn anyone using official source-tar-ball packages to produce a malicious version of the xz library (that malicious Makefile had never been part of the source repository, it's not been reviewed nor approved by anyone).
>
> When I look at requirements explained in our release policy, I think this kind of scenario (especially point c) is not something our release policies protect us against, because we have no requirement to check provenance of the source code in the released package.
>
> Or at least I cannot find it in neither release policy [2] nor distribution policy [3].
>
> From [2]:
>
> > Before casting +1 binding votes, individuals are REQUIRED to download all signed source code packages onto their own hardware, verify that they meet all requirements of ASF policy on releases as described below, validate all cryptographic signatures, compile as provided, and test the result on their own platform.
>
> Even if we assume such a check is part of "meet all requirements of ASF policy on releases" - there is no "check if the sources in the package have not been modified vs. source repository" anywhere in the policies as far as I can see.
>
> From earlier discussions - many of us think that verifying whether the sources in the "source" package contain the same sources as ones stored in our source repositories is the most important part of such verification, but - somewhat to my surprise - it has not been explicitly stated in our policies. And I think it should be an important gate to have PMC members to be REQUIRED to verify that. That could be done in whatever way is appropriate for the project - it could be just comparing sources with git, or having reproducible packages that PMC members can build and compare for binary identity if the project supports it.
>
> But similarly to comparing cryptographic signatures, possibly we should explicitly state that this should be a mandatory check. And maybe we have the chance to use the CVE-2024-3094 as an opportunity to remind/advocate it and explain what could happen if this step is missing when our source packages are released?
>
> I think the way it's stated, a malicious release manager could do a similar package modification as xz release manager did and we could have missed it. Our policies on release do not have explicit gates protecting against this - so PMC members explicitly give +1 - following the release policy pretty rigorously, could have not realise the malicious release manager did such
>
> Just to give an example from the past Airflow releases - when I came to the project, I've learned how it works, and the release process was very rigorously followed, including licences, signatures, etc. and we even pulled a few releases when those were not met. But it's only a few years later when I became a release manager and realised that I could potentially act maliciously and modify the packages and .... no-one would notice - we introduced source provenance check first and reproducible packages after that realisation.
>
> Or maybe I am exaggerating, and it's "obvious enough" that we do not have to state it?
>
> I'd love to hear what others think here. And I am happy to provide a concrete proposal to the policy and do some advocacy for it, if others think it is needed as well.
>
> J.
>
>
> [1] https://boehs.org/node/everything-i-know-about-the-xz-backdoor
> [2] https://www.apache.org/legal/release-policy.html
> [3] https://infra.apache.org/release-distribution.html#sigs-and-sums
>
---------------------------------------------------------------------
To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
For additional commands, e-mail: security-discuss-help@community.apache.org
Re: [DISCUSS] Should we update our policies to include source provenance check
Posted by Philippe Ombredanne <po...@nexb.com>.
Hi Jarek:
On Tue, Apr 2, 2024 at 8:53 AM Jarek Potiuk <ja...@potiuk.com> wrote:
[...]
> TL;DR; I think that we currently do not explicitly state the requirement of
> verifying if the release manager has not tampered with the sources when
> preparing the source package - and I believe we should be more explicit
> about it and require from PMC members to do such verification.
FWIW, back2source [1] may help in the near future. This project is a
work in progress with the goal to validate that the code in a VCS tag,
the source archives and the binary archives of a release all match
correctly. It lives in ScanCode.io [2]
[1] https://nlnet.nl/project/Back2source/
[2] https://github.com/nexB/scancode.io
--
Cordially
Philippe Ombredanne
---------------------------------------------------------------------
To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
For additional commands, e-mail: security-discuss-help@community.apache.org