You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Elliotte Rusty Harold <el...@ibiblio.org> on 2025/02/07 13:51:10 UTC
Supply Chain Attacks and Insider Threats
This is an early draft of something I plan to publish for more open
source projects, but I wanted to send this out here first since I'm
more directly involved.
Open source projects, especially widely used ones like Maven, need to
be aware of and protect against the growing risk of threat actors
actively seeking committer privileges on projects as a precursor to
installing backdoors in the source code.
In particular, it seems that one or more nation states are playing the
long game by targeting specific open source projects. Someone (likely
a team of people) begins sending legitimate patches and PRs and
participating in the mailing lists and other fora until they earn
committer privileges. Then they insert the back door using inobvious
code, wait for it to ship, and proceed with the attack against their
targets.
Last year xz utils was very nearly compromised in this way and only
avoided shipping with the backdoor because one engineer at Microsoft
noticed something funky in a beta release:
https://en.wikipedia.org/wiki/XZ_Utils_backdoor
This is not the first time this has happened, and it almost certainly
won't be the last. For all we know there could be active exploits no
one has noticed already shipped by popular open source software.
What should Maven (and everyone else) do about this? There are a
number of things.
Most importantly, we need to make sure that all PRs and commits are
reviewed before merge. Directly committing to master (as many people
including myself currently have the ability to do) is not safe. It
makes it far too easy to hide malicious commits. If I were an attacker
I would seek out projects like Maven that allow direct commits without
review or PRs. Every commit needs to go through code review, and we
need to enforce this at the repo level so it cannot be bypassed.
Second, we need to verify the identities of new committers. If
possible, new committers should be met in person at ApacheCon or other
convenient events. That's probably not feasible for everyone involved
in a widely dispersed project like Maven, but a quick video chat
should at least verify that a real person exists. I'm not sure how
long that's likely to help given the rapid advances in genAI
impersonation. Nonetheless, we should do what we can to establish a
chain of trust between actual human beings.
Third, we should remove commitership from the large numbers of folks
who haven't contributed to the project for 5+ years. They can still
remain official PMC members, and can be reactivated if they one day
decide to participate again. However, right now their Github accounts
and Apache SVN credentials are extra risks we don't need.
Compromising existing committers accounts is another attack vector
that has been used in the past and will be used in the future.
Fourth, we should require all committer Github accounts to turn on two
factor authentication. We might already be doing this. I don't have
access to check. But if we're not, it's an easy fix to reduce the risk
of hacked and stolen Github accounts being used to compromise us.
There's one other technical change I'd like to consider to protect
Maven users more than the Maven project itself. We should eliminate,
or at the very least warn, when version ranges are used in dependency
elements. See https://jlbp.dev/JLBP-14 for the rationale for this one.
I don't know of an attack using this vector in Java (yet) but it has
been used multiple times in other ecosystems to steal bitcoins and
install malware. Java has been lucky so far, but we are by no means
immune to it.
There might be other things we can do to improve our security posture.
This is just what I've seen used elsewhere. Like a lot of security
measures, they're not without inconvenience, but they are necessary
given the evolving threat landscape. We've made more invasive changes
in the past to increase security, most notably requiring https for
downloading artifacts, so I'm confident we can do this too.
--
Elliotte Rusty Harold
elharo@ibiblio.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Supply Chain Attacks and Insider Threats
Posted by Guillaume Nodet <gn...@apache.org>.
Le dim. 9 févr. 2025 à 20:43, Slawomir Jaranowski <s....@gmail.com>
a écrit :
> On Sun, 9 Feb 2025 at 13:34, Elliotte Rusty Harold <el...@ibiblio.org>
> wrote:
> >
> > On Sun, Feb 9, 2025 at 8:00 AM Slawomir Jaranowski
> > <s....@gmail.com> wrote:
> >
> > > We have a simple statistic
> > >
> https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven-dist-tool/job/master/site/dist-tool-committers-stats.html
> > >
> > > To remove somebody we need a procedure for it.
> >
> > Great. I'm glad we already have the information we need. I see 72
> > committers and maybe 20% of those have been active in the last few
> > years. I'm not sure what the technical procedure for removing
> > committer privileges is. I don't have admin access on the github or
> > svn repos. However as policy I propose:
>
> As I know we don't manage access rights separately for each service.
> We can manage assignments to LDAP groups https://whimsy.apache.org/public/
> by whimsy service.
>
> >
> > 1. Once a year, shortly after January 1, an admin manually removes
> > committership from anyone who hasn't committed in the previous 4
> > years. For instance, right now we would revoke commitership from
> > anyone whose last commit was in 2020 or earlier. The size of the task
> > doesn't feel worth automating.
> >
> > 2. If a former committer notices they no longer have permissions and
> > wants them back to do some work, they just have to ask here on dev@
> > and they will be regranted. They don't have to prove themselves worthy
> > of committer privileges again. They've already done that.
> >
> > 3. Other privileges like issue filing and PMC voting remain in effect
> > as these aren't especially risky.
> >
> > There might be other permissions like the ability to push to the
> > website or control the mailing lists we should also lock down. I don't
> > know exactly how that works, but if anyone does please speak up.
> >
> > To be clear, we're not banning anyone. We're simply being cautious
> > about active permissions given the risk of compromised old accounts.
> > With 72 committers some of whom haven't been heard from in over ten
> > years, it's likely some of these accounts are effectively defunct.
> > It's even possible some developers are deceased.
> >
>
> Generally I agree - but we should start separate thread for discussion,
> and finally finish by VOTE.
>
Sounds good to me. And we don't need a vote per committer, just listing
the names in a single vote should be sufficient. If there's any problem
with one of them, we cancel the vote, remove the ones that want to remain
active or whatever the reason is, and restart one.
At the same time (though, that's a slightly different discussion), I think
we should also make PMC members that have not sent a single email or commit
in 4 years, become emeritus. And the same applies, they can be reinstated
with a simple demand.
>
> > --
> > Elliotte Rusty Harold
> > elharo@ibiblio.org
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
>
>
> --
> Sławomir Jaranowski
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
--
------------------------
Guillaume Nodet
Re: Supply Chain Attacks and Insider Threats
Posted by Slawomir Jaranowski <s....@gmail.com>.
On Sun, 9 Feb 2025 at 13:34, Elliotte Rusty Harold <el...@ibiblio.org> wrote:
>
> On Sun, Feb 9, 2025 at 8:00 AM Slawomir Jaranowski
> <s....@gmail.com> wrote:
>
> > We have a simple statistic
> > https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven-dist-tool/job/master/site/dist-tool-committers-stats.html
> >
> > To remove somebody we need a procedure for it.
>
> Great. I'm glad we already have the information we need. I see 72
> committers and maybe 20% of those have been active in the last few
> years. I'm not sure what the technical procedure for removing
> committer privileges is. I don't have admin access on the github or
> svn repos. However as policy I propose:
As I know we don't manage access rights separately for each service.
We can manage assignments to LDAP groups https://whimsy.apache.org/public/
by whimsy service.
>
> 1. Once a year, shortly after January 1, an admin manually removes
> committership from anyone who hasn't committed in the previous 4
> years. For instance, right now we would revoke commitership from
> anyone whose last commit was in 2020 or earlier. The size of the task
> doesn't feel worth automating.
>
> 2. If a former committer notices they no longer have permissions and
> wants them back to do some work, they just have to ask here on dev@
> and they will be regranted. They don't have to prove themselves worthy
> of committer privileges again. They've already done that.
>
> 3. Other privileges like issue filing and PMC voting remain in effect
> as these aren't especially risky.
>
> There might be other permissions like the ability to push to the
> website or control the mailing lists we should also lock down. I don't
> know exactly how that works, but if anyone does please speak up.
>
> To be clear, we're not banning anyone. We're simply being cautious
> about active permissions given the risk of compromised old accounts.
> With 72 committers some of whom haven't been heard from in over ten
> years, it's likely some of these accounts are effectively defunct.
> It's even possible some developers are deceased.
>
Generally I agree - but we should start separate thread for discussion,
and finally finish by VOTE.
> --
> Elliotte Rusty Harold
> elharo@ibiblio.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
--
Sławomir Jaranowski
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Supply Chain Attacks and Insider Threats
Posted by Elliotte Rusty Harold <el...@ibiblio.org>.
On Sun, Feb 9, 2025 at 8:00 AM Slawomir Jaranowski
<s....@gmail.com> wrote:
> We have a simple statistic
> https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven-dist-tool/job/master/site/dist-tool-committers-stats.html
>
> To remove somebody we need a procedure for it.
Great. I'm glad we already have the information we need. I see 72
committers and maybe 20% of those have been active in the last few
years. I'm not sure what the technical procedure for removing
committer privileges is. I don't have admin access on the github or
svn repos. However as policy I propose:
1. Once a year, shortly after January 1, an admin manually removes
committership from anyone who hasn't committed in the previous 4
years. For instance, right now we would revoke commitership from
anyone whose last commit was in 2020 or earlier. The size of the task
doesn't feel worth automating.
2. If a former committer notices they no longer have permissions and
wants them back to do some work, they just have to ask here on dev@
and they will be regranted. They don't have to prove themselves worthy
of committer privileges again. They've already done that.
3. Other privileges like issue filing and PMC voting remain in effect
as these aren't especially risky.
There might be other permissions like the ability to push to the
website or control the mailing lists we should also lock down. I don't
know exactly how that works, but if anyone does please speak up.
To be clear, we're not banning anyone. We're simply being cautious
about active permissions given the risk of compromised old accounts.
With 72 committers some of whom haven't been heard from in over ten
years, it's likely some of these accounts are effectively defunct.
It's even possible some developers are deceased.
--
Elliotte Rusty Harold
elharo@ibiblio.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Supply Chain Attacks and Insider Threats
Posted by Fred Cooke <fr...@gmail.com>.
Excellent write up, Rusty, thank you. Peer review is always a good thing,
period. It's surprising to me that anyone is allowed to push to any
component directly without the opportunity to vet commit quality and other
things that are hard to automate into the build. Even the very best
sometimes make mistakes and even the very best sometimes miss issues during
PR review, but on balance risk of code churn or worse is reduced
dramatically by a thorough review process. Cheers, Fred
On Sun, 9 Feb 2025 at 20:59, Slawomir Jaranowski <s....@gmail.com>
wrote:
> Hi,
>
> On Fri, 7 Feb 2025 at 14:52, Elliotte Rusty Harold <el...@ibiblio.org>
> wrote:
> >
> > This is an early draft of something I plan to publish for more open
> > source projects, but I wanted to send this out here first since I'm
> > more directly involved.
> >
> > Open source projects, especially widely used ones like Maven, need to
> > be aware of and protect against the growing risk of threat actors
> > actively seeking committer privileges on projects as a precursor to
> > installing backdoors in the source code.
> >
> > In particular, it seems that one or more nation states are playing the
> > long game by targeting specific open source projects. Someone (likely
> > a team of people) begins sending legitimate patches and PRs and
> > participating in the mailing lists and other fora until they earn
> > committer privileges. Then they insert the back door using inobvious
> > code, wait for it to ship, and proceed with the attack against their
> > targets.
> >
> > Last year xz utils was very nearly compromised in this way and only
> > avoided shipping with the backdoor because one engineer at Microsoft
> > noticed something funky in a beta release:
> >
> > https://en.wikipedia.org/wiki/XZ_Utils_backdoor
> >
> > This is not the first time this has happened, and it almost certainly
> > won't be the last. For all we know there could be active exploits no
> > one has noticed already shipped by popular open source software.
> >
> > What should Maven (and everyone else) do about this? There are a
> > number of things.
> >
> > Most importantly, we need to make sure that all PRs and commits are
> > reviewed before merge. Directly committing to master (as many people
> > including myself currently have the ability to do) is not safe. It
> > makes it far too easy to hide malicious commits. If I were an attacker
> > I would seek out projects like Maven that allow direct commits without
> > review or PRs. Every commit needs to go through code review, and we
> > need to enforce this at the repo level so it cannot be bypassed.
>
> Even we don't strictly enforce PR requirements we have:
>
> - each commit is reported on ML -
> https://lists.apache.org/list.html?commits@maven.apache.org
> - before release next version of ASF component we have a VOTE, we can
> check what is deliverable
>
>
> >
> > Second, we need to verify the identities of new committers. If
> > possible, new committers should be met in person at ApacheCon or other
> > convenient events. That's probably not feasible for everyone involved
> > in a widely dispersed project like Maven, but a quick video chat
> > should at least verify that a real person exists. I'm not sure how
> > long that's likely to help given the rapid advances in genAI
> > impersonation. Nonetheless, we should do what we can to establish a
> > chain of trust between actual human beings.
> >
> > Third, we should remove commitership from the large numbers of folks
> > who haven't contributed to the project for 5+ years. They can still
> > remain official PMC members, and can be reactivated if they one day
> > decide to participate again. However, right now their Github accounts
> > and Apache SVN credentials are extra risks we don't need.
> > Compromising existing committers accounts is another attack vector
> > that has been used in the past and will be used in the future.
>
> We have a simple statistic
>
> https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven-dist-tool/job/master/site/dist-tool-committers-stats.html
>
> To remove somebody we need a procedure for it.
>
> >
> > Fourth, we should require all committer Github accounts to turn on two
> > factor authentication. We might already be doing this. I don't have
> > access to check. But if we're not, it's an easy fix to reduce the risk
> > of hacked and stolen Github accounts being used to compromise us.
> >
> > There's one other technical change I'd like to consider to protect
> > Maven users more than the Maven project itself. We should eliminate,
> > or at the very least warn, when version ranges are used in dependency
> > elements. See https://jlbp.dev/JLBP-14 for the rationale for this one.
> > I don't know of an attack using this vector in Java (yet) but it has
> > been used multiple times in other ecosystems to steal bitcoins and
> > install malware. Java has been lucky so far, but we are by no means
> > immune to it.
> >
> > There might be other things we can do to improve our security posture.
> > This is just what I've seen used elsewhere. Like a lot of security
> > measures, they're not without inconvenience, but they are necessary
> > given the evolving threat landscape. We've made more invasive changes
> > in the past to increase security, most notably requiring https for
> > downloading artifacts, so I'm confident we can do this too.
> >
> > --
> > Elliotte Rusty Harold
> > elharo@ibiblio.org
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
>
>
> --
> Sławomir Jaranowski
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: Supply Chain Attacks and Insider Threats
Posted by Slawomir Jaranowski <s....@gmail.com>.
Hi,
On Fri, 7 Feb 2025 at 14:52, Elliotte Rusty Harold <el...@ibiblio.org> wrote:
>
> This is an early draft of something I plan to publish for more open
> source projects, but I wanted to send this out here first since I'm
> more directly involved.
>
> Open source projects, especially widely used ones like Maven, need to
> be aware of and protect against the growing risk of threat actors
> actively seeking committer privileges on projects as a precursor to
> installing backdoors in the source code.
>
> In particular, it seems that one or more nation states are playing the
> long game by targeting specific open source projects. Someone (likely
> a team of people) begins sending legitimate patches and PRs and
> participating in the mailing lists and other fora until they earn
> committer privileges. Then they insert the back door using inobvious
> code, wait for it to ship, and proceed with the attack against their
> targets.
>
> Last year xz utils was very nearly compromised in this way and only
> avoided shipping with the backdoor because one engineer at Microsoft
> noticed something funky in a beta release:
>
> https://en.wikipedia.org/wiki/XZ_Utils_backdoor
>
> This is not the first time this has happened, and it almost certainly
> won't be the last. For all we know there could be active exploits no
> one has noticed already shipped by popular open source software.
>
> What should Maven (and everyone else) do about this? There are a
> number of things.
>
> Most importantly, we need to make sure that all PRs and commits are
> reviewed before merge. Directly committing to master (as many people
> including myself currently have the ability to do) is not safe. It
> makes it far too easy to hide malicious commits. If I were an attacker
> I would seek out projects like Maven that allow direct commits without
> review or PRs. Every commit needs to go through code review, and we
> need to enforce this at the repo level so it cannot be bypassed.
Even we don't strictly enforce PR requirements we have:
- each commit is reported on ML -
https://lists.apache.org/list.html?commits@maven.apache.org
- before release next version of ASF component we have a VOTE, we can
check what is deliverable
>
> Second, we need to verify the identities of new committers. If
> possible, new committers should be met in person at ApacheCon or other
> convenient events. That's probably not feasible for everyone involved
> in a widely dispersed project like Maven, but a quick video chat
> should at least verify that a real person exists. I'm not sure how
> long that's likely to help given the rapid advances in genAI
> impersonation. Nonetheless, we should do what we can to establish a
> chain of trust between actual human beings.
>
> Third, we should remove commitership from the large numbers of folks
> who haven't contributed to the project for 5+ years. They can still
> remain official PMC members, and can be reactivated if they one day
> decide to participate again. However, right now their Github accounts
> and Apache SVN credentials are extra risks we don't need.
> Compromising existing committers accounts is another attack vector
> that has been used in the past and will be used in the future.
We have a simple statistic
https://ci-maven.apache.org/job/Maven/job/maven-box/job/maven-dist-tool/job/master/site/dist-tool-committers-stats.html
To remove somebody we need a procedure for it.
>
> Fourth, we should require all committer Github accounts to turn on two
> factor authentication. We might already be doing this. I don't have
> access to check. But if we're not, it's an easy fix to reduce the risk
> of hacked and stolen Github accounts being used to compromise us.
>
> There's one other technical change I'd like to consider to protect
> Maven users more than the Maven project itself. We should eliminate,
> or at the very least warn, when version ranges are used in dependency
> elements. See https://jlbp.dev/JLBP-14 for the rationale for this one.
> I don't know of an attack using this vector in Java (yet) but it has
> been used multiple times in other ecosystems to steal bitcoins and
> install malware. Java has been lucky so far, but we are by no means
> immune to it.
>
> There might be other things we can do to improve our security posture.
> This is just what I've seen used elsewhere. Like a lot of security
> measures, they're not without inconvenience, but they are necessary
> given the evolving threat landscape. We've made more invasive changes
> in the past to increase security, most notably requiring https for
> downloading artifacts, so I'm confident we can do this too.
>
> --
> Elliotte Rusty Harold
> elharo@ibiblio.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
--
Sławomir Jaranowski
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Supply Chain Attacks and Insider Threats
Posted by Elliotte Rusty Harold <el...@ibiblio.org>.
On Sat, Feb 8, 2025 at 1:19 PM Xeno Amess <xe...@gmail.com> wrote:
> BTW Elli yourself have lots of branches at main repo too...well yes master
> branch and release branch is more important, yes. but other branches in the
> same repo shall be...more formal, right? After all, they be central repo.
No, not important. Development branches that aren't released aren't
especially risky. We need to be careful about what we ship, not random
sets of changes that have to be reviewed and merged into master before
they ship.
--
Elliotte Rusty Harold
elharo@ibiblio.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Supply Chain Attacks and Insider Threats
Posted by Xeno Amess <xe...@gmail.com>.
Yes that is why usually it becomes harder and harder to become a new formal
commiter in a mature apache project.
Btw even if you know the commiter be a real human face, can you just decide
if he evil?
Or is the face really him, or a cheap Indian-face-freelancer hired at
network for a 10$ salary?
So maybe the only way is just strengthen the code review, and yes, even
commiters need to be reviewd by 1-2 other commiter, that makes things
safer.(than push to master directlly)
BTW Elli yourself have lots of branches at main repo too...well yes master
branch and release branch is more important, yes. but other branches in the
same repo shall be...more formal, right? After all, they be central repo.
Just, all we senior coders here know the coding world, no matter
open-source, or close-source, is just full of shit, and nobody's hand is
totally clean of it. we have to eat shit others poop (or starve then
struggle), and we poop shit for others to eat(and yes overlook the shit we
poop, think it delicious), and some bad guys, who thinking themselves great
great hackers or some free free hero thing, add poison in their shit to
poison others for nothing(or money? who knows), and that's life...
Maybe 5 years later the malf code review can be done by glm or other llm,
who knows, but now we just...have to take it carefully by human code
review...and yes, behavior standardizes...
Matthias Bünger <ru...@web.de.invalid> 于2025年2月8日周六 14:52写道：
> This is already required. Otherwise Boxer does not link the GH account
>
> Am 07.02.2025 um 14:51 schrieb Elliotte Rusty Harold:
> > Fourth, we should require all committer Github accounts to turn on two
> > factor authentication. We might already be doing this.
Re: Supply Chain Attacks and Insider Threats
Posted by Matthias Bünger <ru...@web.de.INVALID>.
This is already required. Otherwise Boxer does not link the GH account
Am 07.02.2025 um 14:51 schrieb Elliotte Rusty Harold:
> Fourth, we should require all committer Github accounts to turn on two
> factor authentication. We might already be doing this.