You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Björn Höfling <bj...@bjoernhoefling.de> on 2020/11/18 20:29:04 UTC
Talk: Bootstrapping the Java Ecosystem
Dear Maven Developers,
more than 4 years ago I naively asked you on how to build Maven from
sources without using Maven.
If you are interested in a declarative, bootstrappable, reproducible
and effectively executable answer to this question, Julien Lepiller
recorded a video on how he bootstrapped Maven and a maven-build-system
for GNU Guix with only using the ant-build-system. He shows how to
bootstrap Maven only from sources, which difficulties he had and how he
mastered the dependency cycles and other problems.
You can find a link to the video recordings in this announcement:
https://guix.gnu.org/en/blog/2020/online-guix-day-announce-2/
If you have any questions, you can join the discussions on Guix day,
the discussions for this talk will be on Sunday 2020-11-22, 16:00–16:25
UTC.
Happy Hacking,
Björn
Re: Talk: Bootstrapping the Java Ecosystem
Posted by Björn Höfling <bj...@bjoernhoefling.de>.
On Thu, 19 Nov 2020 09:15:07 +0100
Emmanuel Bourg <eb...@apache.org> wrote:
> Nice presentation, the packaging of Maven in Debian followed a similar
> path but we never documented the process. Did you go as far as
> recording the exact steps and build order required to build from
> scratch?
First of all, let my clarify that honor goes to Julien or the Guix
community or the Reproducible-Builds community. I'm merely
communicating the results here.
I'm not sure if this is what you mean by "recording the exact steps",
but you can use "guix graph maven" to get a (huge) dot/GraphViz
dependency tree. That is your record of the work.
> Spoiler for the next part of your quest toward packaging the Android
> SDK: Maven was the easy part, Gradle and Kotlin are many leagues above
> in term of circular dependencies and headache. We have been trying to
> bootstrap Kotlin for 2 years in Debian and haven't found the right
> path yet.
Yes, Android-SDK, Kotlin, Scala is much harder. Scala for example is
written in Scala. Julien started to build a Scala compiler in Guile,
I'm not sure what the current state is:
https://framagit.org/tyreunom/scabo
Björn
Re: Talk: Bootstrapping the Java Ecosystem
Posted by Tamás Cservenák <ta...@cservenak.net>.
Sorry for being chatty :) but was curious, and looked at sources of "GNU
make".
And as I see, "GNU make" NEEDS "GNU make" to have it built :)
So, I guess this is a usual "thing" in the build tools realm...
Hence, same loops Julien had shown in his video stands for GNU make for
example, right?
(which we all know is not quite true: the loop is actually "project verX"
-uses-> "project verX-1" or alike)
And yes, with Maven (or, pretty much with JVM languages) you entered the
"Maven Artifact Universe", where you meet the same, if not deeper
dependency hierarchy as with deb/rpm or whatever other package system has
as well.
(Another joke) By the way, what do you do with npm packages? ;)
Thanks
Tamas
T
On Fri, Nov 20, 2020 at 10:13 AM Tamás Cservenák <ta...@cservenak.net>
wrote:
> Maybe it was not clear from my last statement: so you "hop on bandwagon",
> and you "joined" the branch of Maven releases, you have 3.6.3. Same version
> Maven devs use as well. So, you will be able to use it to rebuild 3.6.4,
> then use 3.6.4 to rebuild 3.6.5, and so on...
>
> Hence, no need to redo all this for EVERY maven version, right?
>
> T
>
> On Fri, Nov 20, 2020 at 10:06 AM Tamás Cservenák <ta...@cservenak.net>
> wrote:
>
>> Thanks for the answers!
>>
>> AFAIK, we in Apache as well "vote for source", while we provide binaries
>> as well.
>>
>> Given the video mentions that Maven `-sources` artifacts are NOT
>> buildable (which is true, they are mainly used by IDEs to display library
>> sources while debug for example), am unsure -- at least for ASF artifacts
>> -- why not using then source release bundles instead? For example:
>> https://dist.apache.org/repos/dist/release/maven/maven-3/3.6.3/source/
>>
>> Also, according to your explanation, the problem is now solved once for
>> all, right? You do have (those distros you mention, like Guix) Maven 3.6.3
>> built now, so you do not have to repeat this anymore?
>>
>> My point is that Maven devs also use Maven 3.6.3 currently, and that
>> version will be used to build any future Maven release as well (ie. 3.6.4
>> or 4.0.0 and so on). So, you just had to "hop on" the bandwagon, do this
>> "dance" once, but from now on, all this work can be scraped, right?
>>
>> Thanks
>> Tamas
>>
>> On Fri, Nov 20, 2020 at 9:49 AM Emmanuel Bourg <eb...@apache.org> wrote:
>>
>>> On 19/11/2020 09:51, Tamás Cservenák wrote:
>>>
>>> > Without starting any flame wars, am really curious: why are you
>>> > repackaging Maven?
>>> >
>>> > I'd understand for OS/distro native packages, but
>>> > why do you rebuild JVM bytecode as well?
>>> >
>>> > Again, am not to start any flame war, am just curious!
>>>
>>> Short answer: why not? This is an Open Source project, not an Open
>>> Binary project. Anyone should be able to rebuild the code, and in an
>>> ideal world where every project is reproducible, get byte identical
>>> binaries.
>>>
>>> Long answer: Debian, Fedora, and I assume Guix are "closed" ecosystems
>>> where you can rebuild every component from sources without needing tools
>>> or libraries outside of the distribution. If you were alone on a desert
>>> island with just a laptop, the sources and no internet connection, you
>>> would be able to rebuild any part of the distribution from scratch.
>>>
>>> This really goes to the roots of the open source philosophy, open source
>>> projects are meant to be built from sources, and if it's not possible
>>> then there is a problem somewhere. Assuming every project becomes
>>> reproducible at some point (see https://reproducible-builds.org for why
>>> it matters) the question of knowing who produced the binaries become
>>> irrelevant, because everyone get the exact same binaries.
>>>
>>>
>>> > 3) What are you really building? As in video, it is said
>>> > several times that you "mutilate" some package to build
>>> > it, then use it to "bootstrap" some other package, and finally
>>> > you rebuild the target package. Given in the process there
>>> > was once a "mutilated" tool, how are you certain, that output
>>> > of the build is correct (I have no doubts about
>>> > reproducibility)? How do you prove that output is what
>>> > it is thought/assumed to be?
>>>
>>> In Debian the Maven package we rebuild from sources is itself used to
>>> build all the other Maven based projects packaged in Debian (that's over
>>> 600 projects currently), so regressions are caught pretty quickly (it's
>>> rare but it happens sometimes when the binary compatibility is broken in
>>> a core library like maven-shared-utils).
>>>
>>>
>>> > 3) (Joker) What is the overall CO2 footprint of distros like
>>> > these? I believe you did not use Apple M1 for this work... :)
>>>
>>> Probably a tiny fraction of what bitcoin mining, Travis CI and
>>> Youtube/Netflix 4K videos generate ;)
>>>
>>> Emmanuel Bourg
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>
>>>
Re: Talk: Bootstrapping the Java Ecosystem
Posted by Tamás Cservenák <ta...@cservenak.net>.
Maybe it was not clear from my last statement: so you "hop on bandwagon",
and you "joined" the branch of Maven releases, you have 3.6.3. Same version
Maven devs use as well. So, you will be able to use it to rebuild 3.6.4,
then use 3.6.4 to rebuild 3.6.5, and so on...
Hence, no need to redo all this for EVERY maven version, right?
T
On Fri, Nov 20, 2020 at 10:06 AM Tamás Cservenák <ta...@cservenak.net>
wrote:
> Thanks for the answers!
>
> AFAIK, we in Apache as well "vote for source", while we provide binaries
> as well.
>
> Given the video mentions that Maven `-sources` artifacts are NOT buildable
> (which is true, they are mainly used by IDEs to display library sources
> while debug for example), am unsure -- at least for ASF artifacts -- why
> not using then source release bundles instead? For example:
> https://dist.apache.org/repos/dist/release/maven/maven-3/3.6.3/source/
>
> Also, according to your explanation, the problem is now solved once for
> all, right? You do have (those distros you mention, like Guix) Maven 3.6.3
> built now, so you do not have to repeat this anymore?
>
> My point is that Maven devs also use Maven 3.6.3 currently, and that
> version will be used to build any future Maven release as well (ie. 3.6.4
> or 4.0.0 and so on). So, you just had to "hop on" the bandwagon, do this
> "dance" once, but from now on, all this work can be scraped, right?
>
> Thanks
> Tamas
>
> On Fri, Nov 20, 2020 at 9:49 AM Emmanuel Bourg <eb...@apache.org> wrote:
>
>> On 19/11/2020 09:51, Tamás Cservenák wrote:
>>
>> > Without starting any flame wars, am really curious: why are you
>> > repackaging Maven?
>> >
>> > I'd understand for OS/distro native packages, but
>> > why do you rebuild JVM bytecode as well?
>> >
>> > Again, am not to start any flame war, am just curious!
>>
>> Short answer: why not? This is an Open Source project, not an Open
>> Binary project. Anyone should be able to rebuild the code, and in an
>> ideal world where every project is reproducible, get byte identical
>> binaries.
>>
>> Long answer: Debian, Fedora, and I assume Guix are "closed" ecosystems
>> where you can rebuild every component from sources without needing tools
>> or libraries outside of the distribution. If you were alone on a desert
>> island with just a laptop, the sources and no internet connection, you
>> would be able to rebuild any part of the distribution from scratch.
>>
>> This really goes to the roots of the open source philosophy, open source
>> projects are meant to be built from sources, and if it's not possible
>> then there is a problem somewhere. Assuming every project becomes
>> reproducible at some point (see https://reproducible-builds.org for why
>> it matters) the question of knowing who produced the binaries become
>> irrelevant, because everyone get the exact same binaries.
>>
>>
>> > 3) What are you really building? As in video, it is said
>> > several times that you "mutilate" some package to build
>> > it, then use it to "bootstrap" some other package, and finally
>> > you rebuild the target package. Given in the process there
>> > was once a "mutilated" tool, how are you certain, that output
>> > of the build is correct (I have no doubts about
>> > reproducibility)? How do you prove that output is what
>> > it is thought/assumed to be?
>>
>> In Debian the Maven package we rebuild from sources is itself used to
>> build all the other Maven based projects packaged in Debian (that's over
>> 600 projects currently), so regressions are caught pretty quickly (it's
>> rare but it happens sometimes when the binary compatibility is broken in
>> a core library like maven-shared-utils).
>>
>>
>> > 3) (Joker) What is the overall CO2 footprint of distros like
>> > these? I believe you did not use Apple M1 for this work... :)
>>
>> Probably a tiny fraction of what bitcoin mining, Travis CI and
>> Youtube/Netflix 4K videos generate ;)
>>
>> Emmanuel Bourg
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
Re: Talk: Bootstrapping the Java Ecosystem
Posted by Emmanuel Bourg <eb...@apache.org>.
Le 20/11/2020 à 10:06, Tamás Cservenák a écrit :
> Also, according to your explanation, the problem is now solved once for
> all, right? You do have (those distros you mention, like Guix) Maven 3.6.3
> built now, so you do not have to repeat this anymore?
That's correct, in Debian this was done only once, future versions of
Maven will be built with the version 3.6.3 currently packaged.
The bootstrapping is really useful to get an initial package in a
distribution, or to provide an irrefutable proof that a set of sources
generate a given binary when you care about reproducibility.
> (Another joke) By the way, what do you do with npm packages?
I leave this mess to the JS folks :)
Emmanuel Bourg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Talk: Bootstrapping the Java Ecosystem
Posted by Björn Höfling <bj...@bjoernhoefling.de>.
On Fri, 20 Nov 2020 10:06:28 +0100
Tamás Cservenák <ta...@cservenak.net> wrote:
> Thanks for the answers!
>
> AFAIK, we in Apache as well "vote for source", while we provide
> binaries as well.
>
> Given the video mentions that Maven `-sources` artifacts are NOT
> buildable (which is true, they are mainly used by IDEs to display
> library sources while debug for example), am unsure -- at least for
> ASF artifacts -- why not using then source release bundles instead?
> For example:
> https://dist.apache.org/repos/dist/release/maven/maven-3/3.6.3/source/
Guix is trying to use these original source code repositories. But
there is no direct connection between the pom.xml and the original
source code. The problem and concern here is that Maven central is a
long-term store for the pom.xml, the binary .jar and the IDE-only
sources.jar. That works very well for rebuilding from binary
dependencies: If my company/project set up an Artifactory cache ten
years ago, I'm still able to rebuild my Maven-based software today by
using the then-downloaded binary dependencies.
But the source code repositories are eliminated quicker. For example,
I'm searching the sources for org.marlin:marlin:0.7.5-Unsafe, a
dependency of the less than 3 years old GeoServer 2.12.2. The source
code is buried with Boundless' death. If I would have at least a hash
sum in the pom.xml at Maven central, I might be able to download it
from softwareheritage.org.
> Also, according to your explanation, the problem is now solved once
> for all, right? You do have (those distros you mention, like Guix)
> Maven 3.6.3 built now, so you do not have to repeat this anymore?
>
> My point is that Maven devs also use Maven 3.6.3 currently, and that
> version will be used to build any future Maven release as well (ie.
> 3.6.4 or 4.0.0 and so on). So, you just had to "hop on" the
> bandwagon, do this "dance" once, but from now on, all this work can
> be scraped, right?
Yes. It is done "once forever". For new versions it might be
necessary to add some additional dependencies or update to newer
versions. This is not always trivial. From a practical point, it would
be nice if you could build Maven 4.0, 4.1, 4.2, etc all from 3.6.3.
Contrary the JDK has a linear dependency chain:
1.9->10->11->12->...
If you now update for example the make-package, you have to rebuild the
full chain.
Concerning make: There is also a bootstrapping-chain starting with
gnu-make-mesboot0 which is built only with tcc, a small C-compiler, which ...
... and at the end, there is some small binary bootstrap kernel, but
this is getting smaller and smaller.
Björn
Re: Talk: Bootstrapping the Java Ecosystem
Posted by Tamás Cservenák <ta...@cservenak.net>.
Thanks for the answers!
AFAIK, we in Apache as well "vote for source", while we provide binaries as
well.
Given the video mentions that Maven `-sources` artifacts are NOT buildable
(which is true, they are mainly used by IDEs to display library sources
while debug for example), am unsure -- at least for ASF artifacts -- why
not using then source release bundles instead? For example:
https://dist.apache.org/repos/dist/release/maven/maven-3/3.6.3/source/
Also, according to your explanation, the problem is now solved once for
all, right? You do have (those distros you mention, like Guix) Maven 3.6.3
built now, so you do not have to repeat this anymore?
My point is that Maven devs also use Maven 3.6.3 currently, and that
version will be used to build any future Maven release as well (ie. 3.6.4
or 4.0.0 and so on). So, you just had to "hop on" the bandwagon, do this
"dance" once, but from now on, all this work can be scraped, right?
Thanks
Tamas
On Fri, Nov 20, 2020 at 9:49 AM Emmanuel Bourg <eb...@apache.org> wrote:
> On 19/11/2020 09:51, Tamás Cservenák wrote:
>
> > Without starting any flame wars, am really curious: why are you
> > repackaging Maven?
> >
> > I'd understand for OS/distro native packages, but
> > why do you rebuild JVM bytecode as well?
> >
> > Again, am not to start any flame war, am just curious!
>
> Short answer: why not? This is an Open Source project, not an Open
> Binary project. Anyone should be able to rebuild the code, and in an
> ideal world where every project is reproducible, get byte identical
> binaries.
>
> Long answer: Debian, Fedora, and I assume Guix are "closed" ecosystems
> where you can rebuild every component from sources without needing tools
> or libraries outside of the distribution. If you were alone on a desert
> island with just a laptop, the sources and no internet connection, you
> would be able to rebuild any part of the distribution from scratch.
>
> This really goes to the roots of the open source philosophy, open source
> projects are meant to be built from sources, and if it's not possible
> then there is a problem somewhere. Assuming every project becomes
> reproducible at some point (see https://reproducible-builds.org for why
> it matters) the question of knowing who produced the binaries become
> irrelevant, because everyone get the exact same binaries.
>
>
> > 3) What are you really building? As in video, it is said
> > several times that you "mutilate" some package to build
> > it, then use it to "bootstrap" some other package, and finally
> > you rebuild the target package. Given in the process there
> > was once a "mutilated" tool, how are you certain, that output
> > of the build is correct (I have no doubts about
> > reproducibility)? How do you prove that output is what
> > it is thought/assumed to be?
>
> In Debian the Maven package we rebuild from sources is itself used to
> build all the other Maven based projects packaged in Debian (that's over
> 600 projects currently), so regressions are caught pretty quickly (it's
> rare but it happens sometimes when the binary compatibility is broken in
> a core library like maven-shared-utils).
>
>
> > 3) (Joker) What is the overall CO2 footprint of distros like
> > these? I believe you did not use Apple M1 for this work... :)
>
> Probably a tiny fraction of what bitcoin mining, Travis CI and
> Youtube/Netflix 4K videos generate ;)
>
> Emmanuel Bourg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: Talk: Bootstrapping the Java Ecosystem
Posted by Emmanuel Bourg <eb...@apache.org>.
On 19/11/2020 09:51, Tamás Cservenák wrote:
> Without starting any flame wars, am really curious: why are you
> repackaging Maven?
>
> I'd understand for OS/distro native packages, but
> why do you rebuild JVM bytecode as well?
>
> Again, am not to start any flame war, am just curious!
Short answer: why not? This is an Open Source project, not an Open
Binary project. Anyone should be able to rebuild the code, and in an
ideal world where every project is reproducible, get byte identical
binaries.
Long answer: Debian, Fedora, and I assume Guix are "closed" ecosystems
where you can rebuild every component from sources without needing tools
or libraries outside of the distribution. If you were alone on a desert
island with just a laptop, the sources and no internet connection, you
would be able to rebuild any part of the distribution from scratch.
This really goes to the roots of the open source philosophy, open source
projects are meant to be built from sources, and if it's not possible
then there is a problem somewhere. Assuming every project becomes
reproducible at some point (see https://reproducible-builds.org for why
it matters) the question of knowing who produced the binaries become
irrelevant, because everyone get the exact same binaries.
> 3) What are you really building? As in video, it is said
> several times that you "mutilate" some package to build
> it, then use it to "bootstrap" some other package, and finally
> you rebuild the target package. Given in the process there
> was once a "mutilated" tool, how are you certain, that output
> of the build is correct (I have no doubts about
> reproducibility)? How do you prove that output is what
> it is thought/assumed to be?
In Debian the Maven package we rebuild from sources is itself used to
build all the other Maven based projects packaged in Debian (that's over
600 projects currently), so regressions are caught pretty quickly (it's
rare but it happens sometimes when the binary compatibility is broken in
a core library like maven-shared-utils).
> 3) (Joker) What is the overall CO2 footprint of distros like
> these? I believe you did not use Apple M1 for this work... :)
Probably a tiny fraction of what bitcoin mining, Travis CI and
Youtube/Netflix 4K videos generate ;)
Emmanuel Bourg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
AW: Talk: Bootstrapping the Java Ecosystem
Posted by Markus KARG <ma...@headcrashing.eu>.
In fact, as along term user of Maven and Linux, I actually even dislike that binary JARs are rebuilt from scratch, as this is not what WORA was invented for. It adds no benefits, it just adds new bugs happening on one distro but no on another. If you ask me, immediately stop doing that and simply use the one and original built from Apache. This is not C++, this is Java.
And if you want to build Maven from scratch, then build it using Maven. That is the intended way.
-Markus
-----Ursprüngliche Nachricht-----
Von: Tamás Cservenák [mailto:tamas@cservenak.net]
Gesendet: Donnerstag, 19. November 2020 09:51
An: Maven Developers List
Betreff: Re: Talk: Bootstrapping the Java Ecosystem
Hi Bjorn and Emmanuel,
Without starting any flame wars, am really curious: why are you
repackaging Maven?
I'd understand for OS/distro native packages, but
why do you rebuild JVM bytecode as well?
Again, am not to start any flame war, am just curious!
Am Linux user since 98 (first worked on S.u.S.E at university
around '96), but was since using "dirty" distros like
openSUSE then Ubuntu and today Mint.
While watching the video, several questions arose in my head:
1) Instead of rebuilding something (let's say LOC of 1x), you
did that several, if not ten times (so LOC of 10x). So, you
had to review the codebase 10x I guess? Otherwise you are not
sure what you built - as I guess you build it from source
to be sure and able to see (among others) what are you
building. How long was the review process?
2) Similarly, in a process like this, how do you track
vulnerabilities or any other outstanding bugs? Or the
"intermediate" bootstrapped dependencies simply "does not
matter"? Just the final "output" is what matters
(let's say Maven)?
3) What are you really building? As in video, it is said
several times that you "mutilate" some package to build
it, then use it to "bootstrap" some other package, and finally
you rebuild the target package. Given in the process there
was once a "mutilated" tool, how are you certain, that output
of the build is correct (I have no doubts about
reproducibility)? How do you prove that output is what
it is thought/assumed to be?
3) (Joker) What is the overall CO2 footprint of distros like
these? I believe you did not use Apple M1 for this work... :)
Thanks in advance,
T
On Thu, Nov 19, 2020 at 9:15 AM Emmanuel Bourg <eb...@apache.org> wrote:
> Hi Björn,
>
> Nice presentation, the packaging of Maven in Debian followed a similar
> path but we never documented the process. Did you go as far as recording
> the exact steps and build order required to build from scratch?
>
> Spoiler for the next part of your quest toward packaging the Android
> SDK: Maven was the easy part, Gradle and Kotlin are many leagues above
> in term of circular dependencies and headache. We have been trying to
> bootstrap Kotlin for 2 years in Debian and haven't found the right path
> yet.
>
> Emmanuel Bourg
>
>
> On 18/11/2020 21:29, Björn Höfling wrote:
> > Dear Maven Developers,
> >
> > more than 4 years ago I naively asked you on how to build Maven from
> > sources without using Maven.
> >
> > If you are interested in a declarative, bootstrappable, reproducible
> > and effectively executable answer to this question, Julien Lepiller
> > recorded a video on how he bootstrapped Maven and a maven-build-system
> > for GNU Guix with only using the ant-build-system. He shows how to
> > bootstrap Maven only from sources, which difficulties he had and how he
> > mastered the dependency cycles and other problems.
> >
> > You can find a link to the video recordings in this announcement:
> >
> > https://guix.gnu.org/en/blog/2020/online-guix-day-announce-2/
> >
> > If you have any questions, you can join the discussions on Guix day,
> > the discussions for this talk will be on Sunday 2020-11-22, 16:00–16:25
> > UTC.
> >
> > Happy Hacking,
> >
> > Björn
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Talk: Bootstrapping the Java Ecosystem
Posted by Björn Höfling <bj...@bjoernhoefling.de>.
On Thu, 19 Nov 2020 09:51:06 +0100
Tamás Cservenák <ta...@cservenak.net> wrote:
> Hi Bjorn and Emmanuel,
>
> Without starting any flame wars, am really curious: why are you
> repackaging Maven?
I also don't want any flame-wars :-) If this discussion goes to far
away from Maven, I would suggest to ask questions on the lists
guix-devel, guix-help or rb-general.
The intention of building Maven (and other programs) from source is the
old idea of software freedom: You as a user have the source code
available and have the rights to share and modify the software as you
like. For that, it is important that people really are able to build
the software from source, and also all dependencies and prerequisites
like compilers, instead of just downloading a binary or container and
run it. When software is packaged for Guix, you can effectively build
it (again) from source, with the exact same result, independent of the
time or machine you are on. You can even slightly modify the sources
and ask Guix to build it again for you.
> I'd understand for OS/distro native packages, but
> why do you rebuild JVM bytecode as well?
First, Guix is a package manager and is the main block of the
distribution "Guix System". For software freedom reasons, Guix is also
building JVM binaries from source.
> 1) Instead of rebuilding something (let's say LOC of 1x), you
> did that several, if not ten times (so LOC of 10x). So, you
> had to review the codebase 10x I guess? Otherwise you are not
> sure what you built - as I guess you build it from source
> to be sure and able to see (among others) what are you
> building. How long was the review process?
>
> 2) Similarly, in a process like this, how do you track
> vulnerabilities or any other outstanding bugs? Or the
> "intermediate" bootstrapped dependencies simply "does not
> matter"? Just the final "output" is what matters
> (let's say Maven)?
I would separate these topics: Guix (and other distros/package
managers) "just" provides a convenient, automated way to build and
package the final binary software from the sources.
The second thing is reviewing the software for security
vulnerabilities. As you said, this is a very time-consuming process
and not part of Guix. But having a reproducible build at hand you can
verify that a binary really was built from the sources someone claims
to stem from.
> 3) What are you really building? As in video, it is said
> several times that you "mutilate" some package to build
> it, then use it to "bootstrap" some other package, and finally
> you rebuild the target package. Given in the process there
> was once a "mutilated" tool, how are you certain, that output
> of the build is correct (I have no doubts about
> reproducibility)? How do you prove that output is what
> it is thought/assumed to be?
For example, to build the newest ant, Guix is first building jikes (an
old Java-compiler written in C for Java <1.6?), then it uses jikes to
build ant 1.8.4. That allows you to build icedtea and that allows you
to build the latest ant.
There is no guarantee that the final/latest ant that pops out of the
process would really work. You have to test that it really works as
expected. On the other hand, that is a which from package managers to
software developers (especially for languages/compilers): That they care
more about the bootstrapping process. That bootstrapping is defined and
as easy as possible. That maybe different ways of bootstrapping are
available and supported.
> 3) (Joker) What is the overall CO2 footprint of distros like
> these? I believe you did not use Apple M1 for this work... :)
As for other distributions, there is a "substitute server" available
that builds the software, you can just download the "official" distro's
binaries and don't need to compile everything for yourself.
Compared to all other computations we do these days, I would say it
doesn't count too much.
Björn
Re: Talk: Bootstrapping the Java Ecosystem
Posted by Tamás Cservenák <ta...@cservenak.net>.
Hi Bjorn and Emmanuel,
Without starting any flame wars, am really curious: why are you
repackaging Maven?
I'd understand for OS/distro native packages, but
why do you rebuild JVM bytecode as well?
Again, am not to start any flame war, am just curious!
Am Linux user since 98 (first worked on S.u.S.E at university
around '96), but was since using "dirty" distros like
openSUSE then Ubuntu and today Mint.
While watching the video, several questions arose in my head:
1) Instead of rebuilding something (let's say LOC of 1x), you
did that several, if not ten times (so LOC of 10x). So, you
had to review the codebase 10x I guess? Otherwise you are not
sure what you built - as I guess you build it from source
to be sure and able to see (among others) what are you
building. How long was the review process?
2) Similarly, in a process like this, how do you track
vulnerabilities or any other outstanding bugs? Or the
"intermediate" bootstrapped dependencies simply "does not
matter"? Just the final "output" is what matters
(let's say Maven)?
3) What are you really building? As in video, it is said
several times that you "mutilate" some package to build
it, then use it to "bootstrap" some other package, and finally
you rebuild the target package. Given in the process there
was once a "mutilated" tool, how are you certain, that output
of the build is correct (I have no doubts about
reproducibility)? How do you prove that output is what
it is thought/assumed to be?
3) (Joker) What is the overall CO2 footprint of distros like
these? I believe you did not use Apple M1 for this work... :)
Thanks in advance,
T
On Thu, Nov 19, 2020 at 9:15 AM Emmanuel Bourg <eb...@apache.org> wrote:
> Hi Björn,
>
> Nice presentation, the packaging of Maven in Debian followed a similar
> path but we never documented the process. Did you go as far as recording
> the exact steps and build order required to build from scratch?
>
> Spoiler for the next part of your quest toward packaging the Android
> SDK: Maven was the easy part, Gradle and Kotlin are many leagues above
> in term of circular dependencies and headache. We have been trying to
> bootstrap Kotlin for 2 years in Debian and haven't found the right path
> yet.
>
> Emmanuel Bourg
>
>
> On 18/11/2020 21:29, Björn Höfling wrote:
> > Dear Maven Developers,
> >
> > more than 4 years ago I naively asked you on how to build Maven from
> > sources without using Maven.
> >
> > If you are interested in a declarative, bootstrappable, reproducible
> > and effectively executable answer to this question, Julien Lepiller
> > recorded a video on how he bootstrapped Maven and a maven-build-system
> > for GNU Guix with only using the ant-build-system. He shows how to
> > bootstrap Maven only from sources, which difficulties he had and how he
> > mastered the dependency cycles and other problems.
> >
> > You can find a link to the video recordings in this announcement:
> >
> > https://guix.gnu.org/en/blog/2020/online-guix-day-announce-2/
> >
> > If you have any questions, you can join the discussions on Guix day,
> > the discussions for this talk will be on Sunday 2020-11-22, 16:00–16:25
> > UTC.
> >
> > Happy Hacking,
> >
> > Björn
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: Talk: Bootstrapping the Java Ecosystem
Posted by Emmanuel Bourg <eb...@apache.org>.
Hi Björn,
Nice presentation, the packaging of Maven in Debian followed a similar
path but we never documented the process. Did you go as far as recording
the exact steps and build order required to build from scratch?
Spoiler for the next part of your quest toward packaging the Android
SDK: Maven was the easy part, Gradle and Kotlin are many leagues above
in term of circular dependencies and headache. We have been trying to
bootstrap Kotlin for 2 years in Debian and haven't found the right path yet.
Emmanuel Bourg
On 18/11/2020 21:29, Björn Höfling wrote:
> Dear Maven Developers,
>
> more than 4 years ago I naively asked you on how to build Maven from
> sources without using Maven.
>
> If you are interested in a declarative, bootstrappable, reproducible
> and effectively executable answer to this question, Julien Lepiller
> recorded a video on how he bootstrapped Maven and a maven-build-system
> for GNU Guix with only using the ant-build-system. He shows how to
> bootstrap Maven only from sources, which difficulties he had and how he
> mastered the dependency cycles and other problems.
>
> You can find a link to the video recordings in this announcement:
>
> https://guix.gnu.org/en/blog/2020/online-guix-day-announce-2/
>
> If you have any questions, you can join the discussions on Guix day,
> the discussions for this talk will be on Sunday 2020-11-22, 16:00–16:25
> UTC.
>
> Happy Hacking,
>
> Björn
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Talk: Bootstrapping the Java Ecosystem
Posted by Olivier Lamy <ol...@apache.org>.
Hi
not sure what is the goal...
not sure neither how you did that and how long it takes you but we used to
have a build.xml file :)
https://github.com/apache/maven/commits/master/build.xml
On Thu, 19 Nov 2020 at 06:29, Björn Höfling <
bjoern.hoefling@bjoernhoefling.de> wrote:
> Dear Maven Developers,
>
> more than 4 years ago I naively asked you on how to build Maven from
> sources without using Maven.
>
> If you are interested in a declarative, bootstrappable, reproducible
> and effectively executable answer to this question, Julien Lepiller
> recorded a video on how he bootstrapped Maven and a maven-build-system
> for GNU Guix with only using the ant-build-system. He shows how to
> bootstrap Maven only from sources, which difficulties he had and how he
> mastered the dependency cycles and other problems.
>
> You can find a link to the video recordings in this announcement:
>
> https://guix.gnu.org/en/blog/2020/online-guix-day-announce-2/
>
> If you have any questions, you can join the discussions on Guix day,
> the discussions for this talk will be on Sunday 2020-11-22, 16:00–16:25
> UTC.
>
> Happy Hacking,
>
> Björn
>
--
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: Talk: Bootstrapping the Java Ecosystem
Posted by Hervé BOUTEMY <he...@free.fr>.
wow, nice results: kudos to Julien
I'll have a deep look, because (as I already discussed with Julien at some
Reproducible Builds events), if we combine his work on this bootstrapping
approach with my own Reproducible Builds for Java / Reproducible Central [1]
to get reproducible binary artifacts in Central Repository directly provided
by upstream projects, we should have the best of both worlds:
- a proof of rebuild capacity from source, which is important for deep safety,
but not easy for normal users
- a proof of content for reference binaries taken from Central Repository, to
keep an easy binary consumption
What a progress in 1 year. It's sad we can't meet and discuss as we did last
year.
I'll try to join Guix day.
Regards,
Hervé
[1] https://repo.maven.apache.org/maven2/
Le mercredi 18 novembre 2020, 21:29:04 CET Björn Höfling a écrit :
> Dear Maven Developers,
>
> more than 4 years ago I naively asked you on how to build Maven from
> sources without using Maven.
>
> If you are interested in a declarative, bootstrappable, reproducible
> and effectively executable answer to this question, Julien Lepiller
> recorded a video on how he bootstrapped Maven and a maven-build-system
> for GNU Guix with only using the ant-build-system. He shows how to
> bootstrap Maven only from sources, which difficulties he had and how he
> mastered the dependency cycles and other problems.
>
> You can find a link to the video recordings in this announcement:
>
> https://guix.gnu.org/en/blog/2020/online-guix-day-announce-2/
>
> If you have any questions, you can join the discussions on Guix day,
> the discussions for this talk will be on Sunday 2020-11-22, 16:00–16:25
> UTC.
>
> Happy Hacking,
>
> Björn
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org