You are viewing a plain text version of this content. The canonical link for it is here.
Posted to security-discuss@community.apache.org by Bryan Ellis <er...@apache.org> on 2025/10/31 06:57:48 UTC
Security and Release Question from NPM Project Perspective
I have a security and release-related question from the perspective of an
NPM-based project.
Periodically, we receive Dependabot pull requests (PRs) in GitHub. These
PRs usually appear when a direct or transitive dependency has an update
that resolves a security issue.
In most cases, I've noticed that these issues come from
**sub-dependencies**, not direct dependencies. As a result, the PRs
typically modify the `package-lock.json` file.
The `package-lock.json` file pins the dependency tree for reproducible
installs but is used **only during development**. When creating a release
package for a vote or publishing to the NPM registry, this lock file is
**not bundled or used**. This is part of NPM's standard process, not
something we control directly.
Because of how dependencies are declared in `package.json`, reinstalling
the package generally fetches the **latest compatible versions** of
dependencies, depending on how each is pinned.
---
Question:
If a Dependabot PR appears during the middle of a release vote — and it
updates a sub-dependency to fix a security issue — **does that invalidate
or terminate the vote?**
Things to consider:
What exactly are we voting on?
1. **The "release vote package"** — the tarball uploaded to `dist/dev`,
created using `npm pack`, representing what a consumer would receive when
installing the package.
2. **The Git release tag** — the tag created in the repository, which a
consumer could also install directly (e.g., `npm install
github:apache/project-repo#1.0.0`).
3. **The repository clone** — a developer clones the repo, checks out the
release tag, and runs `npm install` within the repository.
If we are voting on the package that will be consumer-facing and the
general process in how they would consume it, the following two cases is
considered:
**Use Case 1:**
* Download the release vote package from `dist/dev`.
* Open another NPM project where you want to use the package.
* Run: "npm install /path/to/downloaded/release/vote/package.tar.gz"
**Use Case 2:**
* Open another NPM project where you want to use the package.
* Run: "npm install github:apache/project-repo#1.0.0"
Obviously, once the vote has passed and is officially published to the NPM
registry, they would just run "npm install package-name".
In both cases, NPM fetches the **latest compatible dependencies**, so there
should be **no vulnerabilities** present — assuming the dependency
constraints in `package.json` allow the patched versions.
The only scenario where you would still see the vulnerabilities is when you
**clone the repository** directly and run `npm install` inside it, because
that uses the `package-lock.json`. This lock file exists in all branches
and tags but only affects the installation of the project **itself**, not
when it's installed as a dependency to the end-users project.
It's also worth noting that while Use Case 2 installs from a Git tag, where
we know a package-lock.json file exists, NPM will not apply the lock file
as it's being installed as a **dependency**.
---
To summarize: does this terminate a vote if it is discovered that
"package-lock.json" contains a sub-dependency that was flagged with a
vulnerability—discovered in the middle of an ongoing release vote, but not
flagged at the start—and is only seen during development when installing
within the repo, not from the consumer's point of view?
Additionally, this raises a second question: if a release tag "rel/0.0.0"
contains the same vulnerability in "package-lock.json" from a development
perspective, but shows zero vulnerabilities when installed through the
consumer process, would any action need to be taken?
Re: Security and Release Question from NPM Project Perspective
Posted by Jarek Potiuk <ja...@potiuk.com>.
> I'm not sure if this is a valid approach, but if we decide to keep the
package-lock.json, we could include a note in all release blog posts or
announcements suggesting that anyone using the source code package run the
npm upgrade command to resolve any potential issues in the lock file, since
the lock file is intended for development use only and may become stale
over time.
I am a bit less concerned for the "npm package" publishing and it's not
really our case - in our case we don't publish our package.lock as part of
the NPM package and we don't publish it either as part of the .whl files
(or at least we should not). What we publish in our .whl files (Which are
convenience packages built from sources) is literally the transpiled and
minified "dist" javascript code. We do not need, nor use the .lock file in
the final .whl package.
We do - of course - keep the package.lock in the "source" packages though -
both "-sources.tar.gz" which is the official release of ASF, as well as in
.sdist python packages that we publish in PyPI - for those who prefer to
rebuild airflow from sources (mainly distros). But I am very little
concerned about this crowd to be honest. Since those people already have
working build environment and actually "run" the build, it's really the
matter of proper instructions for them how to "refresh" the package.lock
before actual build. That's very simple and straightforward, and I think
it's even GOOD to keep those lock files in the sources as the "source of
truth at release time" - but ONLY for that - and include the refresh of the
lock as an official step in instructions on how to rebuild your packages
from sources.
What I am really after is how to make easier to "upgrade" those npm
dependencies for people who moslty consume the .whl files.
J,.
On Sat, Nov 1, 2025 at 10:55 AM Bryan Ellis <er...@apache.org> wrote:
> Any thoughts on deleting the package-lock.json during the preparation of
> the release package, so that it's not included in the source code package?
>
> I think keeping the package-lock.json is important in the main and other
> long-lived branches for development purposes, but I'm still a bit concerned
> about it remaining in the source code package.
>
> The main concern is that if it stays in the package and contains a pinned
> vulnerable sub-dependency, we'd need to address that issue as soon as
> possible. Meanwhile, for end-users who would install the package through
> the npm registry (the preferred method) would have received the patched
> sub-dependency would already have been patched. In this case, the current
> published package would have zero vulnerabilities as there was no lock
> file.
>
> Just to make note, for end-users that had already installed the package,
> they would need to run npm upgrade or possibly just reinstall the package.
>
> Preparing a new source code package would then require bumping the version
> number. But what's the point of publishing a new version to the npm
> registry if it only includes a version bump and no actual code changes? Not
> saying this is the case all the time but there are cases where packages are
> pretty stable and do not actively receive any code changes to promote a
> release.
>
> I have an idea on how to remove the package-lock.json while keeping it in
> the main and long-lived branches, but it would result in an orphaned
> commit. Might increase steps for release and probably make automation
> harder.
>
> > In which case our response will be
> > "here is the link that explains how you can do it yourself, we never
> > release old versions because dependencies that were "latest" at the
> moment
> > of release have some vulnerabilities
>
> I'm not sure if this is a valid approach, but if we decide to keep the
> package-lock.json, we could include a note in all release blog posts or
> announcements suggesting that anyone using the source code package run the
> npm upgrade command to resolve any potential issues in the lock file, since
> the lock file is intended for development use only and may become stale
> over time.
>
>
> On Sat, Nov 1, 2025 at 2:32 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > That is a very good set of questions - and I am glad we start discussing
> > it,
> >
> > I had a lot of thoughts about that recently - because of Airflow NPM
> > dependencies behaviour when comparing to - for example Python. I think we
> > largely figured good aproach for it in Python
> >
> > *Tl;DR; NPM and lock mechanism is a bit problematic for us and I think
> even
> > for convenience packages we look for ways to give our users ways to
> update
> > their dependencies for released convenience packages.*
> >
> > *First a bit of context - how we do Python (and convenience images) in
> > Airflow:*
> >
> > Let me explain how we do it in Airflow for Python, and how I am thinking
> we
> > can do something similar for NPM.
> >
> > For Airflow Python we almost always use lower-bound but non-upper-bound
> > dependencies - i.e we specify "pytorch>x.y.a" but we very, very rarely
> > (only when we have a good reason do `sqlalchemy>x.y.z,<2` - when we
> > **know** we are not compatible with Sqlalchemy 2. This usually happens
> when
> > our dependencies have a well defined release process, and usually
> > semver-like guarantees, and it also means that 1.x has patchlevel
> releases
> > with security fixes we can depend on. Whenever we upper-bound a
> dependency
> > we make a comment why, we open an issue describing it and add link to the
> > issue to the comment and the issue explains conditions and what is
> blocking
> > us from removing the upper-binding (and we periodically review and
> remove
> > those when conditions are met or even run a project to do so - like it
> > happens at the very moment with sqlalchemy 2 upgrade: [1]
> >
> > This means that our users are usually absolutely free to upgrade the
> > dependencies (after sufficient testing) - we are **not** blocking them.
> > This is a very strong policy we have and the main idea is that when we
> > release our software - applying any "future" fixes to CVEs in our
> > dependencies might heppen in two ways:
> >
> > * our future release will include those fixed versions (automatically as
> we
> > do not upper-bind and we are testing latest versions and highest
> > resolutions of our dependencies in canary build 8 times a day to make
> sure
> > we fix all incompatibilities
> > * USER is responsible to upgrade and test older releases and get those
> > dependencies upgraded
> >
> > Even in CRA - we are steward, not manufacturer - so it's manufacturers
> > responsibility to "put software on the market without known, exploitable
> > vulnerabilities" - so it's on THEM not us to make sure whenever a
> > vulnerability is found in one of our deps, to upgrade those.
> >
> > There are some nuances - of course any **new** version of such
> upper-bound
> > dependency might break airflow - that's why we produce a "golden set" of
> > dependencies that we knew as "working" during the release - and we
> publish
> > them as "constraints" that we recommend our users to use when they want
> to
> > install released airflow version in a reproducible way. Then, after they
> do
> > it, and if they want to upgrade to a new version of dependency that has
> > fixed a vulnerability they can do it "on top of that". The first
> paragraph
> > of our "installing from PyPI" describes this process here [2]
> >
> > Effectively it means that the users of ours have two choices:
> >
> > * if they want to upgrade dependencies in released software - in vast
> > majority of cases we are not blocking them from that and it's on them to
> > make sure it happens
> > * or they can wait until we release a new version of Airflow with (pretty
> > much automatically) those deps upgraded by us
> >
> > There are edge-cases of course that are handled individually but those
> are
> > exceptions rather than a rule even if in Airflow we have in total 750+
> > dependencies - it works nicely with this setup if you ask me. And it not
> > only works for "sources" we release but also for convenience packages
> > (whl/sdist) that we also publish.
> >
> > *How is NPM different ?*
> >
> > Now - NPM is a different beast altogether - because by package.lock and
> > minifying the javascript with it, we are effectively blocking our users
> > from doing similar things - upgrading the npm deps - without actually
> > taking the sources, modifying them and rebuilding the packages. Which
> 95%+
> > of our users will never do. I understand (and I am happy with) the fact
> > that ASF releases sources and the rest is just "convenience packages" -
> yet
> > many of our users (including the vast majority of future manufacturers
> who
> > will release Airflow) are mainly consuming our .whl package. We make them
> > reproducible and we also publish them at "https://downloads.apache.org"
> -
> > similarly as our sources, so users can easily do binary artifact
> > comparison with our signed/checksummed binaries) - but they will have
> > really hard time if we tell them - if you want to upgrade NPM
> dependencies
> > you need to rebuild all packages from the sources and you need to modify
> > the package.lock in those sources before. That's mostly a non-go. They
> will
> > expect ASF to release updated versions with those dependencies upgraded
> and
> > even more - they will likely **demand** us to release patchlevels for old
> > releases (2.7.45, where 3.2.1 is already available) with those
> dependencies
> > bumped. I do not like it. Our policy for releases is that we release
> only
> > the latest MINOR branch with patchlevel fixes. Full stop. If a user wants
> > to update a dependency in an older version we want THEM to do it (and
> > manufacturers will be on a hook for that) - but we do not want us to have
> > to do anything other than tell manufacturers how they can do it and send
> > them to the page which explains how. And it should be easy for end users
> to
> > apply it.
> >
> > I want to figure out if that would be possible with NPM. My initial idea
> is
> > to produce a separate convenience source package with only things that
> npm
> > needs and make it possible to build such a "patch" package by anyone and
> > add a procedure explaining how you can build and test such a patch with
> > latest dependencies. Very similarly as we do with our docker containers.
> We
> > explain our users already - that when they want to update some of the
> > container images they use, we will generally **never** re-build those
> > released images, but they are absolutely free to rebuild the containers
> > themselves and use latest dependencies from debian available - also they
> > can extend such images with updated new version of python packages. We
> > have a dedicated chapter about it in our documentation [3]
> >
> > *Why do I think such a "npm-only rebuild" instructions are needed ?*
> >
> > Now - maybe such a tooling (to easily rebuild and replace packages for
> NPM
> > packages already exists ? Or maybe from our side it's just a matter of
> > writing some custom package builder script to do so? Such a tooling also
> > should include some rudimentary (or even quite comprehensive) regression
> > tests that will allow such a user to be reasonably sure that the newly
> > released dependencies do not break their build.
> >
> > I still have not looked at details - but for me having this is quite a
> game
> > changer that gives you worry-free release mechanism:
> >
> > * make sure that when you release - you bump deps to latest (fairly easy)
> > * when you release a version - freeze the latest "minor" release - and do
> > not worry about 3rd-party released in-between - you explicitly explain
> that
> > you "freeze" the versions (as we do with docker container)
> > * then, put it on the users and make it easy so that they can build and
> use
> > a hassle-free way (replacement of compiled code with NPM) that contains
> the
> > SAME code we released in that version but with updated NPM dependencies.
> >
> > This will give us a very powerful leverage to discuss things with
> > manufacturers who - undoubtedly - will start banging our doors at the day
> > CRA is going to be effective with "Release a new version of Airflow with
> > those dependency issues fixed ASAP:". In which case our response will be
> > "here is the link that explains how you can do it yourself, we never
> > release old versions because dependencies that were "latest" at the
> moment
> > of release have some vulnerabilities - but here is the instruction on how
> > you can do it yourself'. And BTW - We accept any donation to us if you
> > want our security team to work even better as currently it's purely
> > volunteers and your money as manufacturer would be very helpful to
> improve
> > our processes".
> >
> > I hope it might be helpful in the discussion (though probably far too
> long
> > for anyone to get thus far).
> >
> > Links referred to:
> >
> > [1] Issue with sqlalchemy 2 upgrade :
> > https://github.com/apache/airflow/issues/56735
> > [2] Installing from PyPI using "golden set" + upgrading to latest Python
> > dependencies
> >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html
> > [3] What should I do if my security scan shows critical and high
> > vulnerabilities in the image?
> >
> >
> https://airflow.apache.org/docs/docker-stack/index.html#what-should-i-do-if-my-security-scan-shows-critical-and-high-vulnerabilities-in-the-image
> >
> >
> >
> >
> >
> >
> > On Fri, Oct 31, 2025 at 10:48 AM Piotr P. Karwasz <
> > piotr@mailing.copernik.eu>
> > wrote:
> >
> > > Hi Bryan,
> > >
> > > On 31.10.2025 07:57, Bryan Ellis wrote:
> > > > Question:
> > > >
> > > > If a Dependabot PR appears during the middle of a release vote — and
> it
> > > > updates a sub-dependency to fix a security issue — **does that
> > invalidate
> > > > or terminate the vote?**
> > >
> > >
> > > No, only the release manager has the authority to cancel a vote.
> > >
> > > > What exactly are we voting on?
> > > >
> > > > 1. **The "release vote package"** — the tarball uploaded to
> `dist/dev`,
> > > > created using `npm pack`, representing what a consumer would receive
> > when
> > > > installing the package.
> > > > 2. **The Git release tag** — the tag created in the repository,
> which a
> > > > consumer could also install directly (e.g., `npm install
> > > > github:apache/project-repo#1.0.0`).
> > > > 3. **The repository clone** — a developer clones the repo, checks out
> > the
> > > > release tag, and runs `npm install` within the repository.
> > >
> > >
> > > This is consistent across the ASF: votes are cast on the source release
> > > package, which must be sufficient for a user to build and test the
> > > release [1]. Only the source package constitutes an official act of the
> > > Foundation.
> > >
> > > Compiled artifacts (such as those published to NPM) are provided as a
> > > convenience to users, but they are not considered an official ASF
> > release.
> > >
> > > These rules exist because, in compiled languages, it’s difficult to
> > > verify that a compiled artifact precisely matches the source. However,
> > > thanks to reproducible-build efforts, projects like Log4j also verify
> > > the reproducibility of JARs whenever possible. This verification is a
> > > prerequisite for allowing releases to be built via GitHub Actions
> (i.e.,
> > > on infrastructure not owned or directly controlled by the release
> > manager).
> > >
> > > ## Vulnerabilities in dependencies
> > >
> > > If a vulnerability is disclosed in any dependency of the project, I
> > > generally follow this rule of thumb:
> > >
> > > - If you ship the dependency, you effectively own it. You should make a
> > > reasonable effort to release versions without “known exploitable
> > > vulnerabilities”.
> > > - If you do not ship the dependency, it’s sufficient to document its
> > > presence. You don’t need to block or reissue a release solely because
> > > of it.
> > >
> > > If your packages include an `npm-shrinkwrap.json` file, I would pause
> > > the release to assess the potential impact of the vulnerability.
> Without
> > > it, you can assume that users will receive the same Dependabot PR and
> > > handle the update independently.
> > >
> > >
> > > Piotr
> > >
> > > [1] https://www.apache.org/legal/release-policy.html#artifacts
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail:
> > security-discuss-unsubscribe@community.apache.org
> > > For additional commands, e-mail:
> > > security-discuss-help@community.apache.org
> > >
> > >
> >
>
Re: Security and Release Question from NPM Project Perspective
Posted by "Piotr P. Karwasz" <pi...@mailing.copernik.eu>.
Hi Bryan,
*TL;DR*: Don’t cut new ASF releases just to bump dependencies unless we
ship them. Instead, document clearly when dependency updates are safe or
when reported vulnerabilities are non-exploitable.
On 1.11.2025 10:55, Bryan Ellis wrote:
> Any thoughts on deleting the package-lock.json during the preparation of
> the release package, so that it's not included in the source code package?
From what I can see, the Cordova source archives on downloads.apache.org
don’t actually include a `package-lock.json`. Am I missing something?
> The main concern is that if it stays in the package and contains a pinned
> vulnerable sub-dependency, we'd need to address that issue as soon as
> possible. Meanwhile, for end-users who would install the package through
> the npm registry (the preferred method) would have received the patched
> sub-dependency would already have been patched. In this case, the current
> published package would have zero vulnerabilities as there was no lock file.
>
> Just to make note, for end-users that had already installed the package,
> they would need to run npm upgrade or possibly just reinstall the package.
>
> Preparing a new source code package would then require bumping the version
> number. But what's the point of publishing a new version to the npm
> registry if it only includes a version bump and no actual code changes? Not
> saying this is the case all the time but there are cases where packages are
> pretty stable and do not actively receive any code changes to promote a
> release.
I agree it makes little sense to issue a new ASF release *just* because
a dependency was flagged by Dependabot. For npm consumers the fix
arrives automatically, and while Maven users don’t get that convenience,
they can still upgrade the dependency on their side without waiting for us.
In my view, the major package ecosystems (Maven, npm, PyPI) were
designed primarily to resolve *compatible* dependencies, with *security*
considerations added later as an afterthought. As a result, we end up
with two opposite but equally problematic extremes:
- Maven is often criticized for its use of “pinned” (more accurately,
recommended [1]) versions, which can lead to shipping known
vulnerabilities.
- npm and PyPI, on the other hand, prioritize rapid updates, which
enables quick patching, but also allows malicious packages to spread
just as quickly.
That’s why I like the direction Jarek described: have a file/list that
*isn’t* committed to the package repository (but can be audited) and
that you can update *whenever* a dependency CVE appears, without redoing
the whole ASF release. That gives us a place to say:
“we tested dependency X at version Y and it’s OK.”
What I think we’re missing, and what your mail is really pointing at, is
the ability to distinguish three different “versions” for a dependency
(assuming semver-ish deps):
1. Baseline version (API floor)
- This is the oldest version we promise we still work with.
- Some users can’t jump to “latest” because of environment
constraints.
- In Maven this is often what we put for “provided” deps in the POM.
- Raising this in PyPI (`>=x.y.z`) can lock such users out.
2. Recommended / release-tested version
- That’s the version we actually ran the release tests against.
- It’s safe *today*, but it can become “CVE-noisy” with time.
- In Java/Maven this ends up as “the” version if there is no conflict
resolution.
3. Latest tested version
- Every time Dependabot opens a PR, we *could* test the newer dep and
mark it “known good.”
- Right now we have no good, public way to publish that without
cutting a new release, which is overkill.
I had a related discussion this week with the Solr developers [2] about
the misaligned expectations between library and application maintainers
when dealing with vulnerable dependencies:
- My view as a library maintainer: “I won’t cut a new release just to
bump a dependency; application developers can override it themselves”.
- Application maintainers’ view: “Please bump it in the library first so
*you* confirm it doesn’t introduce bugs”.
As a result, both sides often wait for the other to act, and no one
releases an update unless it’s a major incident like Log4Shell, leaving
end users potentially exposed.
Piotr
[1] In Maven they are not really hard-pinned: consumers can still
override them, and ranges exist, they’re just rarely used.
[2] https://the-asf.slack.com/archives/CEKUCUNE9/p1761901493592529
---------------------------------------------------------------------
To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
For additional commands, e-mail: security-discuss-help@community.apache.org
Re: Security and Release Question from NPM Project Perspective
Posted by Bryan Ellis <er...@apache.org>.
Any thoughts on deleting the package-lock.json during the preparation of
the release package, so that it's not included in the source code package?
I think keeping the package-lock.json is important in the main and other
long-lived branches for development purposes, but I'm still a bit concerned
about it remaining in the source code package.
The main concern is that if it stays in the package and contains a pinned
vulnerable sub-dependency, we'd need to address that issue as soon as
possible. Meanwhile, for end-users who would install the package through
the npm registry (the preferred method) would have received the patched
sub-dependency would already have been patched. In this case, the current
published package would have zero vulnerabilities as there was no lock file.
Just to make note, for end-users that had already installed the package,
they would need to run npm upgrade or possibly just reinstall the package.
Preparing a new source code package would then require bumping the version
number. But what's the point of publishing a new version to the npm
registry if it only includes a version bump and no actual code changes? Not
saying this is the case all the time but there are cases where packages are
pretty stable and do not actively receive any code changes to promote a
release.
I have an idea on how to remove the package-lock.json while keeping it in
the main and long-lived branches, but it would result in an orphaned
commit. Might increase steps for release and probably make automation
harder.
> In which case our response will be
> "here is the link that explains how you can do it yourself, we never
> release old versions because dependencies that were "latest" at the moment
> of release have some vulnerabilities
I'm not sure if this is a valid approach, but if we decide to keep the
package-lock.json, we could include a note in all release blog posts or
announcements suggesting that anyone using the source code package run the
npm upgrade command to resolve any potential issues in the lock file, since
the lock file is intended for development use only and may become stale
over time.
On Sat, Nov 1, 2025 at 2:32 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> That is a very good set of questions - and I am glad we start discussing
> it,
>
> I had a lot of thoughts about that recently - because of Airflow NPM
> dependencies behaviour when comparing to - for example Python. I think we
> largely figured good aproach for it in Python
>
> *Tl;DR; NPM and lock mechanism is a bit problematic for us and I think even
> for convenience packages we look for ways to give our users ways to update
> their dependencies for released convenience packages.*
>
> *First a bit of context - how we do Python (and convenience images) in
> Airflow:*
>
> Let me explain how we do it in Airflow for Python, and how I am thinking we
> can do something similar for NPM.
>
> For Airflow Python we almost always use lower-bound but non-upper-bound
> dependencies - i.e we specify "pytorch>x.y.a" but we very, very rarely
> (only when we have a good reason do `sqlalchemy>x.y.z,<2` - when we
> **know** we are not compatible with Sqlalchemy 2. This usually happens when
> our dependencies have a well defined release process, and usually
> semver-like guarantees, and it also means that 1.x has patchlevel releases
> with security fixes we can depend on. Whenever we upper-bound a dependency
> we make a comment why, we open an issue describing it and add link to the
> issue to the comment and the issue explains conditions and what is blocking
> us from removing the upper-binding (and we periodically review and remove
> those when conditions are met or even run a project to do so - like it
> happens at the very moment with sqlalchemy 2 upgrade: [1]
>
> This means that our users are usually absolutely free to upgrade the
> dependencies (after sufficient testing) - we are **not** blocking them.
> This is a very strong policy we have and the main idea is that when we
> release our software - applying any "future" fixes to CVEs in our
> dependencies might heppen in two ways:
>
> * our future release will include those fixed versions (automatically as we
> do not upper-bind and we are testing latest versions and highest
> resolutions of our dependencies in canary build 8 times a day to make sure
> we fix all incompatibilities
> * USER is responsible to upgrade and test older releases and get those
> dependencies upgraded
>
> Even in CRA - we are steward, not manufacturer - so it's manufacturers
> responsibility to "put software on the market without known, exploitable
> vulnerabilities" - so it's on THEM not us to make sure whenever a
> vulnerability is found in one of our deps, to upgrade those.
>
> There are some nuances - of course any **new** version of such upper-bound
> dependency might break airflow - that's why we produce a "golden set" of
> dependencies that we knew as "working" during the release - and we publish
> them as "constraints" that we recommend our users to use when they want to
> install released airflow version in a reproducible way. Then, after they do
> it, and if they want to upgrade to a new version of dependency that has
> fixed a vulnerability they can do it "on top of that". The first paragraph
> of our "installing from PyPI" describes this process here [2]
>
> Effectively it means that the users of ours have two choices:
>
> * if they want to upgrade dependencies in released software - in vast
> majority of cases we are not blocking them from that and it's on them to
> make sure it happens
> * or they can wait until we release a new version of Airflow with (pretty
> much automatically) those deps upgraded by us
>
> There are edge-cases of course that are handled individually but those are
> exceptions rather than a rule even if in Airflow we have in total 750+
> dependencies - it works nicely with this setup if you ask me. And it not
> only works for "sources" we release but also for convenience packages
> (whl/sdist) that we also publish.
>
> *How is NPM different ?*
>
> Now - NPM is a different beast altogether - because by package.lock and
> minifying the javascript with it, we are effectively blocking our users
> from doing similar things - upgrading the npm deps - without actually
> taking the sources, modifying them and rebuilding the packages. Which 95%+
> of our users will never do. I understand (and I am happy with) the fact
> that ASF releases sources and the rest is just "convenience packages" - yet
> many of our users (including the vast majority of future manufacturers who
> will release Airflow) are mainly consuming our .whl package. We make them
> reproducible and we also publish them at "https://downloads.apache.org" -
> similarly as our sources, so users can easily do binary artifact
> comparison with our signed/checksummed binaries) - but they will have
> really hard time if we tell them - if you want to upgrade NPM dependencies
> you need to rebuild all packages from the sources and you need to modify
> the package.lock in those sources before. That's mostly a non-go. They will
> expect ASF to release updated versions with those dependencies upgraded and
> even more - they will likely **demand** us to release patchlevels for old
> releases (2.7.45, where 3.2.1 is already available) with those dependencies
> bumped. I do not like it. Our policy for releases is that we release only
> the latest MINOR branch with patchlevel fixes. Full stop. If a user wants
> to update a dependency in an older version we want THEM to do it (and
> manufacturers will be on a hook for that) - but we do not want us to have
> to do anything other than tell manufacturers how they can do it and send
> them to the page which explains how. And it should be easy for end users to
> apply it.
>
> I want to figure out if that would be possible with NPM. My initial idea is
> to produce a separate convenience source package with only things that npm
> needs and make it possible to build such a "patch" package by anyone and
> add a procedure explaining how you can build and test such a patch with
> latest dependencies. Very similarly as we do with our docker containers. We
> explain our users already - that when they want to update some of the
> container images they use, we will generally **never** re-build those
> released images, but they are absolutely free to rebuild the containers
> themselves and use latest dependencies from debian available - also they
> can extend such images with updated new version of python packages. We
> have a dedicated chapter about it in our documentation [3]
>
> *Why do I think such a "npm-only rebuild" instructions are needed ?*
>
> Now - maybe such a tooling (to easily rebuild and replace packages for NPM
> packages already exists ? Or maybe from our side it's just a matter of
> writing some custom package builder script to do so? Such a tooling also
> should include some rudimentary (or even quite comprehensive) regression
> tests that will allow such a user to be reasonably sure that the newly
> released dependencies do not break their build.
>
> I still have not looked at details - but for me having this is quite a game
> changer that gives you worry-free release mechanism:
>
> * make sure that when you release - you bump deps to latest (fairly easy)
> * when you release a version - freeze the latest "minor" release - and do
> not worry about 3rd-party released in-between - you explicitly explain that
> you "freeze" the versions (as we do with docker container)
> * then, put it on the users and make it easy so that they can build and use
> a hassle-free way (replacement of compiled code with NPM) that contains the
> SAME code we released in that version but with updated NPM dependencies.
>
> This will give us a very powerful leverage to discuss things with
> manufacturers who - undoubtedly - will start banging our doors at the day
> CRA is going to be effective with "Release a new version of Airflow with
> those dependency issues fixed ASAP:". In which case our response will be
> "here is the link that explains how you can do it yourself, we never
> release old versions because dependencies that were "latest" at the moment
> of release have some vulnerabilities - but here is the instruction on how
> you can do it yourself'. And BTW - We accept any donation to us if you
> want our security team to work even better as currently it's purely
> volunteers and your money as manufacturer would be very helpful to improve
> our processes".
>
> I hope it might be helpful in the discussion (though probably far too long
> for anyone to get thus far).
>
> Links referred to:
>
> [1] Issue with sqlalchemy 2 upgrade :
> https://github.com/apache/airflow/issues/56735
> [2] Installing from PyPI using "golden set" + upgrading to latest Python
> dependencies
>
> https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html
> [3] What should I do if my security scan shows critical and high
> vulnerabilities in the image?
>
> https://airflow.apache.org/docs/docker-stack/index.html#what-should-i-do-if-my-security-scan-shows-critical-and-high-vulnerabilities-in-the-image
>
>
>
>
>
>
> On Fri, Oct 31, 2025 at 10:48 AM Piotr P. Karwasz <
> piotr@mailing.copernik.eu>
> wrote:
>
> > Hi Bryan,
> >
> > On 31.10.2025 07:57, Bryan Ellis wrote:
> > > Question:
> > >
> > > If a Dependabot PR appears during the middle of a release vote — and it
> > > updates a sub-dependency to fix a security issue — **does that
> invalidate
> > > or terminate the vote?**
> >
> >
> > No, only the release manager has the authority to cancel a vote.
> >
> > > What exactly are we voting on?
> > >
> > > 1. **The "release vote package"** — the tarball uploaded to `dist/dev`,
> > > created using `npm pack`, representing what a consumer would receive
> when
> > > installing the package.
> > > 2. **The Git release tag** — the tag created in the repository, which a
> > > consumer could also install directly (e.g., `npm install
> > > github:apache/project-repo#1.0.0`).
> > > 3. **The repository clone** — a developer clones the repo, checks out
> the
> > > release tag, and runs `npm install` within the repository.
> >
> >
> > This is consistent across the ASF: votes are cast on the source release
> > package, which must be sufficient for a user to build and test the
> > release [1]. Only the source package constitutes an official act of the
> > Foundation.
> >
> > Compiled artifacts (such as those published to NPM) are provided as a
> > convenience to users, but they are not considered an official ASF
> release.
> >
> > These rules exist because, in compiled languages, it’s difficult to
> > verify that a compiled artifact precisely matches the source. However,
> > thanks to reproducible-build efforts, projects like Log4j also verify
> > the reproducibility of JARs whenever possible. This verification is a
> > prerequisite for allowing releases to be built via GitHub Actions (i.e.,
> > on infrastructure not owned or directly controlled by the release
> manager).
> >
> > ## Vulnerabilities in dependencies
> >
> > If a vulnerability is disclosed in any dependency of the project, I
> > generally follow this rule of thumb:
> >
> > - If you ship the dependency, you effectively own it. You should make a
> > reasonable effort to release versions without “known exploitable
> > vulnerabilities”.
> > - If you do not ship the dependency, it’s sufficient to document its
> > presence. You don’t need to block or reissue a release solely because
> > of it.
> >
> > If your packages include an `npm-shrinkwrap.json` file, I would pause
> > the release to assess the potential impact of the vulnerability. Without
> > it, you can assume that users will receive the same Dependabot PR and
> > handle the update independently.
> >
> >
> > Piotr
> >
> > [1] https://www.apache.org/legal/release-policy.html#artifacts
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> security-discuss-unsubscribe@community.apache.org
> > For additional commands, e-mail:
> > security-discuss-help@community.apache.org
> >
> >
>
Re: Security and Release Question from NPM Project Perspective
Posted by Jarek Potiuk <ja...@potiuk.com>.
That is a very good set of questions - and I am glad we start discussing it,
I had a lot of thoughts about that recently - because of Airflow NPM
dependencies behaviour when comparing to - for example Python. I think we
largely figured good aproach for it in Python
*Tl;DR; NPM and lock mechanism is a bit problematic for us and I think even
for convenience packages we look for ways to give our users ways to update
their dependencies for released convenience packages.*
*First a bit of context - how we do Python (and convenience images) in
Airflow:*
Let me explain how we do it in Airflow for Python, and how I am thinking we
can do something similar for NPM.
For Airflow Python we almost always use lower-bound but non-upper-bound
dependencies - i.e we specify "pytorch>x.y.a" but we very, very rarely
(only when we have a good reason do `sqlalchemy>x.y.z,<2` - when we
**know** we are not compatible with Sqlalchemy 2. This usually happens when
our dependencies have a well defined release process, and usually
semver-like guarantees, and it also means that 1.x has patchlevel releases
with security fixes we can depend on. Whenever we upper-bound a dependency
we make a comment why, we open an issue describing it and add link to the
issue to the comment and the issue explains conditions and what is blocking
us from removing the upper-binding (and we periodically review and remove
those when conditions are met or even run a project to do so - like it
happens at the very moment with sqlalchemy 2 upgrade: [1]
This means that our users are usually absolutely free to upgrade the
dependencies (after sufficient testing) - we are **not** blocking them.
This is a very strong policy we have and the main idea is that when we
release our software - applying any "future" fixes to CVEs in our
dependencies might heppen in two ways:
* our future release will include those fixed versions (automatically as we
do not upper-bind and we are testing latest versions and highest
resolutions of our dependencies in canary build 8 times a day to make sure
we fix all incompatibilities
* USER is responsible to upgrade and test older releases and get those
dependencies upgraded
Even in CRA - we are steward, not manufacturer - so it's manufacturers
responsibility to "put software on the market without known, exploitable
vulnerabilities" - so it's on THEM not us to make sure whenever a
vulnerability is found in one of our deps, to upgrade those.
There are some nuances - of course any **new** version of such upper-bound
dependency might break airflow - that's why we produce a "golden set" of
dependencies that we knew as "working" during the release - and we publish
them as "constraints" that we recommend our users to use when they want to
install released airflow version in a reproducible way. Then, after they do
it, and if they want to upgrade to a new version of dependency that has
fixed a vulnerability they can do it "on top of that". The first paragraph
of our "installing from PyPI" describes this process here [2]
Effectively it means that the users of ours have two choices:
* if they want to upgrade dependencies in released software - in vast
majority of cases we are not blocking them from that and it's on them to
make sure it happens
* or they can wait until we release a new version of Airflow with (pretty
much automatically) those deps upgraded by us
There are edge-cases of course that are handled individually but those are
exceptions rather than a rule even if in Airflow we have in total 750+
dependencies - it works nicely with this setup if you ask me. And it not
only works for "sources" we release but also for convenience packages
(whl/sdist) that we also publish.
*How is NPM different ?*
Now - NPM is a different beast altogether - because by package.lock and
minifying the javascript with it, we are effectively blocking our users
from doing similar things - upgrading the npm deps - without actually
taking the sources, modifying them and rebuilding the packages. Which 95%+
of our users will never do. I understand (and I am happy with) the fact
that ASF releases sources and the rest is just "convenience packages" - yet
many of our users (including the vast majority of future manufacturers who
will release Airflow) are mainly consuming our .whl package. We make them
reproducible and we also publish them at "https://downloads.apache.org" -
similarly as our sources, so users can easily do binary artifact
comparison with our signed/checksummed binaries) - but they will have
really hard time if we tell them - if you want to upgrade NPM dependencies
you need to rebuild all packages from the sources and you need to modify
the package.lock in those sources before. That's mostly a non-go. They will
expect ASF to release updated versions with those dependencies upgraded and
even more - they will likely **demand** us to release patchlevels for old
releases (2.7.45, where 3.2.1 is already available) with those dependencies
bumped. I do not like it. Our policy for releases is that we release only
the latest MINOR branch with patchlevel fixes. Full stop. If a user wants
to update a dependency in an older version we want THEM to do it (and
manufacturers will be on a hook for that) - but we do not want us to have
to do anything other than tell manufacturers how they can do it and send
them to the page which explains how. And it should be easy for end users to
apply it.
I want to figure out if that would be possible with NPM. My initial idea is
to produce a separate convenience source package with only things that npm
needs and make it possible to build such a "patch" package by anyone and
add a procedure explaining how you can build and test such a patch with
latest dependencies. Very similarly as we do with our docker containers. We
explain our users already - that when they want to update some of the
container images they use, we will generally **never** re-build those
released images, but they are absolutely free to rebuild the containers
themselves and use latest dependencies from debian available - also they
can extend such images with updated new version of python packages. We
have a dedicated chapter about it in our documentation [3]
*Why do I think such a "npm-only rebuild" instructions are needed ?*
Now - maybe such a tooling (to easily rebuild and replace packages for NPM
packages already exists ? Or maybe from our side it's just a matter of
writing some custom package builder script to do so? Such a tooling also
should include some rudimentary (or even quite comprehensive) regression
tests that will allow such a user to be reasonably sure that the newly
released dependencies do not break their build.
I still have not looked at details - but for me having this is quite a game
changer that gives you worry-free release mechanism:
* make sure that when you release - you bump deps to latest (fairly easy)
* when you release a version - freeze the latest "minor" release - and do
not worry about 3rd-party released in-between - you explicitly explain that
you "freeze" the versions (as we do with docker container)
* then, put it on the users and make it easy so that they can build and use
a hassle-free way (replacement of compiled code with NPM) that contains the
SAME code we released in that version but with updated NPM dependencies.
This will give us a very powerful leverage to discuss things with
manufacturers who - undoubtedly - will start banging our doors at the day
CRA is going to be effective with "Release a new version of Airflow with
those dependency issues fixed ASAP:". In which case our response will be
"here is the link that explains how you can do it yourself, we never
release old versions because dependencies that were "latest" at the moment
of release have some vulnerabilities - but here is the instruction on how
you can do it yourself'. And BTW - We accept any donation to us if you
want our security team to work even better as currently it's purely
volunteers and your money as manufacturer would be very helpful to improve
our processes".
I hope it might be helpful in the discussion (though probably far too long
for anyone to get thus far).
Links referred to:
[1] Issue with sqlalchemy 2 upgrade :
https://github.com/apache/airflow/issues/56735
[2] Installing from PyPI using "golden set" + upgrading to latest Python
dependencies
https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html
[3] What should I do if my security scan shows critical and high
vulnerabilities in the image?
https://airflow.apache.org/docs/docker-stack/index.html#what-should-i-do-if-my-security-scan-shows-critical-and-high-vulnerabilities-in-the-image
On Fri, Oct 31, 2025 at 10:48 AM Piotr P. Karwasz <pi...@mailing.copernik.eu>
wrote:
> Hi Bryan,
>
> On 31.10.2025 07:57, Bryan Ellis wrote:
> > Question:
> >
> > If a Dependabot PR appears during the middle of a release vote — and it
> > updates a sub-dependency to fix a security issue — **does that invalidate
> > or terminate the vote?**
>
>
> No, only the release manager has the authority to cancel a vote.
>
> > What exactly are we voting on?
> >
> > 1. **The "release vote package"** — the tarball uploaded to `dist/dev`,
> > created using `npm pack`, representing what a consumer would receive when
> > installing the package.
> > 2. **The Git release tag** — the tag created in the repository, which a
> > consumer could also install directly (e.g., `npm install
> > github:apache/project-repo#1.0.0`).
> > 3. **The repository clone** — a developer clones the repo, checks out the
> > release tag, and runs `npm install` within the repository.
>
>
> This is consistent across the ASF: votes are cast on the source release
> package, which must be sufficient for a user to build and test the
> release [1]. Only the source package constitutes an official act of the
> Foundation.
>
> Compiled artifacts (such as those published to NPM) are provided as a
> convenience to users, but they are not considered an official ASF release.
>
> These rules exist because, in compiled languages, it’s difficult to
> verify that a compiled artifact precisely matches the source. However,
> thanks to reproducible-build efforts, projects like Log4j also verify
> the reproducibility of JARs whenever possible. This verification is a
> prerequisite for allowing releases to be built via GitHub Actions (i.e.,
> on infrastructure not owned or directly controlled by the release manager).
>
> ## Vulnerabilities in dependencies
>
> If a vulnerability is disclosed in any dependency of the project, I
> generally follow this rule of thumb:
>
> - If you ship the dependency, you effectively own it. You should make a
> reasonable effort to release versions without “known exploitable
> vulnerabilities”.
> - If you do not ship the dependency, it’s sufficient to document its
> presence. You don’t need to block or reissue a release solely because
> of it.
>
> If your packages include an `npm-shrinkwrap.json` file, I would pause
> the release to assess the potential impact of the vulnerability. Without
> it, you can assume that users will receive the same Dependabot PR and
> handle the update independently.
>
>
> Piotr
>
> [1] https://www.apache.org/legal/release-policy.html#artifacts
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
> For additional commands, e-mail:
> security-discuss-help@community.apache.org
>
>
Re: Security and Release Question from NPM Project Perspective
Posted by "Piotr P. Karwasz" <pi...@mailing.copernik.eu>.
Hi Bryan,
On 31.10.2025 07:57, Bryan Ellis wrote:
> Question:
>
> If a Dependabot PR appears during the middle of a release vote — and it
> updates a sub-dependency to fix a security issue — **does that invalidate
> or terminate the vote?**
No, only the release manager has the authority to cancel a vote.
> What exactly are we voting on?
>
> 1. **The "release vote package"** — the tarball uploaded to `dist/dev`,
> created using `npm pack`, representing what a consumer would receive when
> installing the package.
> 2. **The Git release tag** — the tag created in the repository, which a
> consumer could also install directly (e.g., `npm install
> github:apache/project-repo#1.0.0`).
> 3. **The repository clone** — a developer clones the repo, checks out the
> release tag, and runs `npm install` within the repository.
This is consistent across the ASF: votes are cast on the source release
package, which must be sufficient for a user to build and test the
release [1]. Only the source package constitutes an official act of the
Foundation.
Compiled artifacts (such as those published to NPM) are provided as a
convenience to users, but they are not considered an official ASF release.
These rules exist because, in compiled languages, it’s difficult to
verify that a compiled artifact precisely matches the source. However,
thanks to reproducible-build efforts, projects like Log4j also verify
the reproducibility of JARs whenever possible. This verification is a
prerequisite for allowing releases to be built via GitHub Actions (i.e.,
on infrastructure not owned or directly controlled by the release manager).
## Vulnerabilities in dependencies
If a vulnerability is disclosed in any dependency of the project, I
generally follow this rule of thumb:
- If you ship the dependency, you effectively own it. You should make a
reasonable effort to release versions without “known exploitable
vulnerabilities”.
- If you do not ship the dependency, it’s sufficient to document its
presence. You don’t need to block or reissue a release solely because
of it.
If your packages include an `npm-shrinkwrap.json` file, I would pause
the release to assess the potential impact of the vulnerability. Without
it, you can assume that users will receive the same Dependabot PR and
handle the update independently.
Piotr
[1] https://www.apache.org/legal/release-policy.html#artifacts
---------------------------------------------------------------------
To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
For additional commands, e-mail: security-discuss-help@community.apache.org
Re: Security and Release Question from NPM Project Perspective
Posted by Mark Thomas <ma...@apache.org>.
On 31/10/2025 06:57, Bryan Ellis wrote:
> I have a security and release-related question from the perspective of an
> NPM-based project.
>
> Periodically, we receive Dependabot pull requests (PRs) in GitHub. These
> PRs usually appear when a direct or transitive dependency has an update
> that resolves a security issue.
>
> In most cases, I've noticed that these issues come from
> **sub-dependencies**, not direct dependencies. As a result, the PRs
> typically modify the `package-lock.json` file.
>
> The `package-lock.json` file pins the dependency tree for reproducible
> installs but is used **only during development**. When creating a release
> package for a vote or publishing to the NPM registry, this lock file is
> **not bundled or used**. This is part of NPM's standard process, not
> something we control directly.
>
> Because of how dependencies are declared in `package.json`, reinstalling
> the package generally fetches the **latest compatible versions** of
> dependencies, depending on how each is pinned.
>
> ---
>
> Question:
>
> If a Dependabot PR appears during the middle of a release vote — and it
> updates a sub-dependency to fix a security issue — **does that invalidate
> or terminate the vote?**
>
> Things to consider:
>
> What exactly are we voting on?
>
> 1. **The "release vote package"** — the tarball uploaded to `dist/dev`,
> created using `npm pack`, representing what a consumer would receive when
> installing the package.
>> 2. **The Git release tag** — the tag created in the repository, which a
> consumer could also install directly (e.g., `npm install
> github:apache/project-repo#1.0.0`).
> 3. **The repository clone** — a developer clones the repo, checks out the
> release tag, and runs `npm install` within the repository.
From a strict ASF perspective the answer is always "the source
tarball". That said, having an awareness of how your user community
typically consumes your project and taking that into account is the
right thing to do.
> If we are voting on the package that will be consumer-facing and the
> general process in how they would consume it, the following two cases is
> considered:
>
> **Use Case 1:**
>
> * Download the release vote package from `dist/dev`.
> * Open another NPM project where you want to use the package.
> * Run: "npm install /path/to/downloaded/release/vote/package.tar.gz"
>
> **Use Case 2:**
>
> * Open another NPM project where you want to use the package.
> * Run: "npm install github:apache/project-repo#1.0.0"
>
> Obviously, once the vote has passed and is officially published to the NPM
> registry, they would just run "npm install package-name".
>
> In both cases, NPM fetches the **latest compatible dependencies**, so there
> should be **no vulnerabilities** present — assuming the dependency
> constraints in `package.json` allow the patched versions.
>
> The only scenario where you would still see the vulnerabilities is when you
> **clone the repository** directly and run `npm install` inside it, because
> that uses the `package-lock.json`. This lock file exists in all branches
> and tags but only affects the installation of the project **itself**, not
> when it's installed as a dependency to the end-users project.
>
> It's also worth noting that while Use Case 2 installs from a Git tag, where
> we know a package-lock.json file exists, NPM will not apply the lock file
> as it's being installed as a **dependency**.
>
> ---
>
> To summarize: does this terminate a vote if it is discovered that
> "package-lock.json" contains a sub-dependency that was flagged with a
> vulnerability—discovered in the middle of an ongoing release vote, but not
> flagged at the start—and is only seen during development when installing
> within the repo, not from the consumer's point of view?
Based on what i have read above, I don't see why it should.
> Additionally, this raises a second question: if a release tag "rel/0.0.0"
> contains the same vulnerability in "package-lock.json" from a development
> perspective, but shows zero vulnerabilities when installed through the
> consumer process, would any action need to be taken?
Specifically, I think not.
More generally, what I would suggest that is somewhere where developers
would be expected to read if they were going to clone the repo (maybe in
the building instructions if you have them) that you document that if
they clone the repo and build locally they will get fixed versions that
might include vulnerabilities. You'll want to put that into NPM language
but I hope you get the idea.
My impression is that this should be something that is known and
understood by developers working with NPM but I don't think there is any
harm in explicitly documenting it.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: security-discuss-unsubscribe@community.apache.org
For additional commands, e-mail: security-discuss-help@community.apache.org