You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Paul Hammant <ha...@apache.org> on 2017/01/24 05:05:07 UTC
Requesting a single Monorepo enhancement for Maven
OK, so I'm a documenter of Google's Monorepo (one biiiig ass trunk) and
it's usage of shell scripts to subset the checkout for speedy development:
http://paulhammant.com/2014/01/06/googlers-subset-their-trunk/
https://trunkbaseddevelopment.com/monorepos/
For Maven to be used with a scripted use of Subversion or Git's
sparse-checkout (or Perforce's client spec), it'd been to be more like
Bazel/Blaze or Buck, in that sub-modules are *not* forward declared, they
are discovered/calculated/inferred somehow.
In pom.xml instead of -
<modules>
<module>one</module>
<module>two</module>
</modules>
We'd need -
<modules>
<search>recursively</search>
</modules>
Or -
<modules>
<defined-in>.full-module-list.txt</defined-in>
<!-- made by
find . -name "pom.xml" | sed 's/\/pom.xml//' >
.full-module-list.txt
after the sparse-checkout modification of working copy -->
</modules>
Thoughts?
Any questions?
- Paul H
PS - I'm a solid Maven user since 2003.
Re: Requesting a single Monorepo enhancement for Maven
Posted by Paul Hammant <pa...@hammant.org>.
100% agree that teams not using this feature should be able to carry on unchanged. If the likes of JetBrains have to do extra code for the short-hand inside <module/> that Robert proposed, then I'm sure they'll agree with me that that is the cost of doing business. And from that they can choose to delay it too, or wait for a PR from someone who has cloned their GH repo.
Sent from my iPhone
> On Jan 24, 2017, at 8:50 AM, Stephen Connolly <st...@gmail.com> wrote:
>
> I don't disagree... but
>
> There is tooling that parses the Pom directly and has made assumptions
> about the modules tag and its structure
>
> We would break such tooling with such a seemingly trivial change (I know of
> at least 5 of my employer's customers who I would be quite upset... any my
> employer is focused on a different product from Maven)
>
>
>> On Tue 24 Jan 2017 at 13:33, Paul Hammant <pa...@hammant.org> wrote:
>>
>> Stephen - I think think Robert's <module>def:.full-module-list.txt</module>
>>
>> is compatible hacking that is POM 4 friendly (and 3 for that matter) until
>>
>> you revisit in 5.
>>
>>
>>
>> I'd be horrified to write more XML than I already write in Maven.
>>
>>
>>
>> I'm able to face Gradle advocates re a particular enterprise app and
>>
>> without feeling I'm lying to myself say "what you're showing me is more of
>>
>> less the same as Maven all things considered". Well if failsafe and tomcat
>>
>> and surefile and coverage aren't in the same solution, that is. With the
>>
>> profile fu, just to get Maven to follow the lead git-sparse-checkout could
>>
>> give for such things, I could not face anyone and say that's what you
>>
>> should do.
>>
>>
>>
>> - Paul
>>
>>
>>
>> On Tue, Jan 24, 2017 at 8:13 AM, Stephen Connolly <
>>
>> stephen.alan.connolly@gmail.com> wrote:
>>
>>
>>
>>> Yep I hear you.
>>
>>>
>>
>>> We cannot change the 4.0.0 schema
>>
>>>
>>
>>> And changing to a new modelVersion requires ensuring that we can evolve
>>
>>> without breaking consumers of the older model.
>>
>>>
>>
>>> Basically we have one chance to make a "breaking" change to something
>> that
>>
>>> allows us to evolve going forward
>>
>>>
>>
>>> So what I provided was the 4.0.0 modelVersion solution... it's ugly but
>>
>>> does not require pre-processing or generation of the pom
>>
>>>
>>
>>>> On Tue 24 Jan 2017 at 12:53, Paul Hammant <pa...@hammant.org> wrote:
>>>
>>>
>>
>>>> Versus profiles, I would rather have pom.xml exhaustively renamed to
>>
>>>>
>>
>>>> pom.xml.template in SCM, and a Python script to generate <modules/> as
>> we
>>
>>>>
>>
>>>> have it today (pom.xml marked as .gitignore) before invoking 'maven
>>
>>>> install'
>>
>>>>
>>
>>>>
>>
>>>>
>>
>>>> - Paul
>>
>>>>
>>
>>>>
>>
>>>>
>>
>>>>
>>
>>>>
>>
>>>> On Tue, Jan 24, 2017 at 7:31 AM, Stephen Connolly <
>>
>>>>
>>
>>>> stephen.alan.connolly@gmail.com> wrote:
>>
>>>>
>>
>>>>
>>
>>>>
>>
>>>>> It's an interesting idea for model version 5.0.0 to consider.
>>
>>>>
>>
>>>>>
>>
>>>>
>>
>>>>> At present, I would handle it by using profiles with activation to
>> pull
>>
>>>> in
>>
>>>>
>>
>>>>> the modules:
>>
>>>>
>>
>>>>>
>>
>>>>
>>
>>>>> if your activation in the root is based on the presence of the
>> module's
>>
>>>>
>>
>>>>> pom.xml then it will only add the module if you checked it out:
>> (approx
>>
>>>>
>>
>>>>> structure and I do not have the XSD to hand)
>>
>>>>
>>
>>>>>
>>
>>>>
>>
>>>>> <profile>
>>
>>>>
>>
>>>>> <id>module-foo</id>
>>
>>>>
>>
>>>>> <activation>
>>
>>>>
>>
>>>>> <file>module-foo/pom.xml</file>
>>
>>>>
>>
>>>>> </activation>
>>
>>>>
>>
>>>>> <modules>
>>
>>>>
>>
>>>>> <module>module-foo</module>
>>
>>>>
>>
>>>>> </modules>
>>
>>>>
>>
>>>>> </profile>
>>
>>>>
>>
>>>>> <profile>
>>
>>>>
>>
>>>>> <id>module-bar</id>
>>
>>>>
>>
>>>>> <activation>
>>
>>>>
>>
>>>>> <file>module-bar/pom.xml</file>
>>
>>>>
>>
>>>>> </activation>
>>
>>>>
>>
>>>>> <modules>
>>
>>>>
>>
>>>>> <module>module-bar</module>
>>
>>>>
>>
>>>>> </modules>
>>
>>>>
>>
>>>>> </profile>
>>
>>>>
>>
>>>>>
>>
>>>>
>>
>>>>> Yes that becomes an ugly pom, but it will do what you want when run
>>
>>> from
>>
>>>>
>>
>>>>> the root and I have used it before
>>
>>>>
>>
>>>>>
>>
>>>>
>>
>>>>> HTH
>>
>>>>
>>
>>>>>
>>
>>>>
>>
>>>>> On 24 January 2017 at 11:14, Paul Hammant <pa...@hammant.org> wrote:
>>
>>>>
>>
>>>>>
>>
>>>>
>>
>>>>>> i thought about that too, except that in a monorepo situation, I
>>
>>> don't
>>
>>>>
>>
>>>>> want
>>
>>>>
>>
>>>>>> the don't want the changed pom to get pushed back in a commit, and
>> I
>>
>>>>
>>
>>>>> don't
>>
>>>>
>>
>>>>>> want one of the those changelists in my IDE labeled "do not commit"
>>
>>> to
>>
>>>>
>>
>>>>>> facilitate that.
>>
>>>>
>>
>>>>>>
>>
>>>>
>>
>>>>>> Rationale: Just because I've subset my checkout/clone doesn't mean
>>
>>> that
>>
>>>>
>>
>>>>> all
>>
>>>>
>>
>>>>>> users of the same repo want to.
>>
>>>>
>>
>>>>>>
>>
>>>>
>>
>>>>>> It was implied, but I'll call it out: .full-module-list.txt is in
>>
>>>>
>>
>>>>>> .gitignore (etc), and that it's easily regenerate per the 'find'
>>
>>>> command.
>>
>>>>
>>
>>>>>>
>>
>>>>
>>
>>>>>> Regards,
>>
>>>>
>>
>>>>>>
>>
>>>>
>>
>>>>>> - Paul
>>
>>>>
>>
>>>>>>
>>
>>>>
>>
>>>>>> On Tue, Jan 24, 2017 at 12:50 AM, Aldrin Leal <al...@leal.eng.br>
>>
>>>>
>>
>>>>> wrote:
>>
>>>>
>>
>>>>>>
>>
>>>>
>>
>>>>>>> Actually, I always wondered if it was interesting to have a tool
>> to
>>
>>>>
>>
>>>>> allow
>>
>>>>
>>
>>>>>>> the modification of POM files from Command Line. Like setting a
>>
>>>>
>>
>>>>> property,
>>
>>>>
>>
>>>>>>> adding a dependency and/or, as you exposed, changing modules.
>>
>>>>
>>
>>>>>>>
>>
>>>>
>>
>>>>>>> --
>>
>>>>
>>
>>>>>>> -- Aldrin Leal, <al...@leal.eng.br> /
>> http://about.me/aldrinleal
>>
>>>>
>>
>>>>>>>
>>
>>>>
>>
>>>>>>> On Tue, Jan 24, 2017 at 12:05 AM, Paul Hammant <
>> hammant@apache.org
>>
>>>>
>>
>>>>
>>
>>>>>> wrote:
>>
>>>>
>>
>>>>>>>
>>
>>>>
>>
>>>>>>>> OK, so I'm a documenter of Google's Monorepo (one biiiig ass
>>
>>> trunk)
>>
>>>>
>>
>>>>> and
>>
>>>>
>>
>>>>>>>> it's usage of shell scripts to subset the checkout for speedy
>>
>>>>
>>
>>>>>>> development:
>>
>>>>
>>
>>>>>>>>
>>
>>>>
>>
>>>>>>>> http://paulhammant.com/2014/01/06/googlers-subset-their-
>>
>>> trunk/
>>
>>>>
>>
>>>>>>>> https://trunkbaseddevelopment.com/monorepos/
>>
>>>>
>>
>>>>>>>>
>>
>>>>
>>
>>>>>>>> For Maven to be used with a scripted use of Subversion or Git's
>>
>>>>
>>
>>>>>>>> sparse-checkout (or Perforce's client spec), it'd been to be
>> more
>>
>>>>
>>
>>>>> like
>>
>>>>
>>
>>>>>>>> Bazel/Blaze or Buck, in that sub-modules are *not* forward
>>
>>>> declared,
>>
>>>>
>>
>>>>>> they
>>
>>>>
>>
>>>>>>>> are discovered/calculated/inferred somehow.
>>
>>>>
>>
>>>>>>>>
>>
>>>>
>>
>>>>>>>> In pom.xml instead of -
>>
>>>>
>>
>>>>>>>>
>>
>>>>
>>
>>>>>>>> <modules>
>>
>>>>
>>
>>>>>>>> <module>one</module>
>>
>>>>
>>
>>>>>>>> <module>two</module>
>>
>>>>
>>
>>>>>>>> </modules>
>>
>>>>
>>
>>>>>>>>
>>
>>>>
>>
>>>>>>>> We'd need -
>>
>>>>
>>
>>>>>>>>
>>
>>>>
>>
>>>>>>>> <modules>
>>
>>>>
>>
>>>>>>>> <search>recursively</search>
>>
>>>>
>>
>>>>>>>> </modules>
>>
>>>>
>>
>>>>>>>>
>>
>>>>
>>
>>>>>>>> Or -
>>
>>>>
>>
>>>>>>>>
>>
>>>>
>>
>>>>>>>> <modules>
>>
>>>>
>>
>>>>>>>> <defined-in>.full-module-list.txt</defined-in>
>>
>>>>
>>
>>>>>>>> <!-- made by
>>
>>>>
>>
>>>>>>>> find . -name "pom.xml" | sed 's/\/pom.xml//' >
>>
>>>>
>>
>>>>>>>> .full-module-list.txt
>>
>>>>
>>
>>>>>>>> after the sparse-checkout modification of working
>>
>>>> copy
>>
>>>>
>>
>>>>>> -->
>>
>>>>
>>
>>>>>>>> </modules>
>>
>>>>
>>
>>>>>>>>
>>
>>>>
>>
>>>>>>>> Thoughts?
>>
>>>>
>>
>>>>>>>>
>>
>>>>
>>
>>>>>>>> Any questions?
>>
>>>>
>>
>>>>>>>>
>>
>>>>
>>
>>>>>>>> - Paul H
>>
>>>>
>>
>>>>>>>>
>>
>>>>
>>
>>>>>>>> PS - I'm a solid Maven user since 2003.
>>
>>>>
>>
>>>>>>>>
>>
>>>>
>>
>>>>>>>
>>
>>>>
>>
>>>>>>
>>
>>>>
>>
>>>>>
>>
>>>>
>>
>>>> --
>>
>>> Sent from my phone
>>
>>>
>>
>> --
> Sent from my phone
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Requesting a single Monorepo enhancement for Maven
Posted by Stephen Connolly <st...@gmail.com>.
I don't disagree... but
There is tooling that parses the Pom directly and has made assumptions
about the modules tag and its structure
We would break such tooling with such a seemingly trivial change (I know of
at least 5 of my employer's customers who I would be quite upset... any my
employer is focused on a different product from Maven)
On Tue 24 Jan 2017 at 13:33, Paul Hammant <pa...@hammant.org> wrote:
> Stephen - I think think Robert's <module>def:.full-module-list.txt</module>
>
> is compatible hacking that is POM 4 friendly (and 3 for that matter) until
>
> you revisit in 5.
>
>
>
> I'd be horrified to write more XML than I already write in Maven.
>
>
>
> I'm able to face Gradle advocates re a particular enterprise app and
>
> without feeling I'm lying to myself say "what you're showing me is more of
>
> less the same as Maven all things considered". Well if failsafe and tomcat
>
> and surefile and coverage aren't in the same solution, that is. With the
>
> profile fu, just to get Maven to follow the lead git-sparse-checkout could
>
> give for such things, I could not face anyone and say that's what you
>
> should do.
>
>
>
> - Paul
>
>
>
> On Tue, Jan 24, 2017 at 8:13 AM, Stephen Connolly <
>
> stephen.alan.connolly@gmail.com> wrote:
>
>
>
> > Yep I hear you.
>
> >
>
> > We cannot change the 4.0.0 schema
>
> >
>
> > And changing to a new modelVersion requires ensuring that we can evolve
>
> > without breaking consumers of the older model.
>
> >
>
> > Basically we have one chance to make a "breaking" change to something
> that
>
> > allows us to evolve going forward
>
> >
>
> > So what I provided was the 4.0.0 modelVersion solution... it's ugly but
>
> > does not require pre-processing or generation of the pom
>
> >
>
> > On Tue 24 Jan 2017 at 12:53, Paul Hammant <pa...@hammant.org> wrote:
>
> >
>
> > > Versus profiles, I would rather have pom.xml exhaustively renamed to
>
> > >
>
> > > pom.xml.template in SCM, and a Python script to generate <modules/> as
> we
>
> > >
>
> > > have it today (pom.xml marked as .gitignore) before invoking 'maven
>
> > > install'
>
> > >
>
> > >
>
> > >
>
> > > - Paul
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > > On Tue, Jan 24, 2017 at 7:31 AM, Stephen Connolly <
>
> > >
>
> > > stephen.alan.connolly@gmail.com> wrote:
>
> > >
>
> > >
>
> > >
>
> > > > It's an interesting idea for model version 5.0.0 to consider.
>
> > >
>
> > > >
>
> > >
>
> > > > At present, I would handle it by using profiles with activation to
> pull
>
> > > in
>
> > >
>
> > > > the modules:
>
> > >
>
> > > >
>
> > >
>
> > > > if your activation in the root is based on the presence of the
> module's
>
> > >
>
> > > > pom.xml then it will only add the module if you checked it out:
> (approx
>
> > >
>
> > > > structure and I do not have the XSD to hand)
>
> > >
>
> > > >
>
> > >
>
> > > > <profile>
>
> > >
>
> > > > <id>module-foo</id>
>
> > >
>
> > > > <activation>
>
> > >
>
> > > > <file>module-foo/pom.xml</file>
>
> > >
>
> > > > </activation>
>
> > >
>
> > > > <modules>
>
> > >
>
> > > > <module>module-foo</module>
>
> > >
>
> > > > </modules>
>
> > >
>
> > > > </profile>
>
> > >
>
> > > > <profile>
>
> > >
>
> > > > <id>module-bar</id>
>
> > >
>
> > > > <activation>
>
> > >
>
> > > > <file>module-bar/pom.xml</file>
>
> > >
>
> > > > </activation>
>
> > >
>
> > > > <modules>
>
> > >
>
> > > > <module>module-bar</module>
>
> > >
>
> > > > </modules>
>
> > >
>
> > > > </profile>
>
> > >
>
> > > >
>
> > >
>
> > > > Yes that becomes an ugly pom, but it will do what you want when run
>
> > from
>
> > >
>
> > > > the root and I have used it before
>
> > >
>
> > > >
>
> > >
>
> > > > HTH
>
> > >
>
> > > >
>
> > >
>
> > > > On 24 January 2017 at 11:14, Paul Hammant <pa...@hammant.org> wrote:
>
> > >
>
> > > >
>
> > >
>
> > > > > i thought about that too, except that in a monorepo situation, I
>
> > don't
>
> > >
>
> > > > want
>
> > >
>
> > > > > the don't want the changed pom to get pushed back in a commit, and
> I
>
> > >
>
> > > > don't
>
> > >
>
> > > > > want one of the those changelists in my IDE labeled "do not commit"
>
> > to
>
> > >
>
> > > > > facilitate that.
>
> > >
>
> > > > >
>
> > >
>
> > > > > Rationale: Just because I've subset my checkout/clone doesn't mean
>
> > that
>
> > >
>
> > > > all
>
> > >
>
> > > > > users of the same repo want to.
>
> > >
>
> > > > >
>
> > >
>
> > > > > It was implied, but I'll call it out: .full-module-list.txt is in
>
> > >
>
> > > > > .gitignore (etc), and that it's easily regenerate per the 'find'
>
> > > command.
>
> > >
>
> > > > >
>
> > >
>
> > > > > Regards,
>
> > >
>
> > > > >
>
> > >
>
> > > > > - Paul
>
> > >
>
> > > > >
>
> > >
>
> > > > > On Tue, Jan 24, 2017 at 12:50 AM, Aldrin Leal <al...@leal.eng.br>
>
> > >
>
> > > > wrote:
>
> > >
>
> > > > >
>
> > >
>
> > > > > > Actually, I always wondered if it was interesting to have a tool
> to
>
> > >
>
> > > > allow
>
> > >
>
> > > > > > the modification of POM files from Command Line. Like setting a
>
> > >
>
> > > > property,
>
> > >
>
> > > > > > adding a dependency and/or, as you exposed, changing modules.
>
> > >
>
> > > > > >
>
> > >
>
> > > > > > --
>
> > >
>
> > > > > > -- Aldrin Leal, <al...@leal.eng.br> /
> http://about.me/aldrinleal
>
> > >
>
> > > > > >
>
> > >
>
> > > > > > On Tue, Jan 24, 2017 at 12:05 AM, Paul Hammant <
> hammant@apache.org
>
> > >
>
> > >
>
> > > > > wrote:
>
> > >
>
> > > > > >
>
> > >
>
> > > > > > > OK, so I'm a documenter of Google's Monorepo (one biiiig ass
>
> > trunk)
>
> > >
>
> > > > and
>
> > >
>
> > > > > > > it's usage of shell scripts to subset the checkout for speedy
>
> > >
>
> > > > > > development:
>
> > >
>
> > > > > > >
>
> > >
>
> > > > > > > http://paulhammant.com/2014/01/06/googlers-subset-their-
>
> > trunk/
>
> > >
>
> > > > > > > https://trunkbaseddevelopment.com/monorepos/
>
> > >
>
> > > > > > >
>
> > >
>
> > > > > > > For Maven to be used with a scripted use of Subversion or Git's
>
> > >
>
> > > > > > > sparse-checkout (or Perforce's client spec), it'd been to be
> more
>
> > >
>
> > > > like
>
> > >
>
> > > > > > > Bazel/Blaze or Buck, in that sub-modules are *not* forward
>
> > > declared,
>
> > >
>
> > > > > they
>
> > >
>
> > > > > > > are discovered/calculated/inferred somehow.
>
> > >
>
> > > > > > >
>
> > >
>
> > > > > > > In pom.xml instead of -
>
> > >
>
> > > > > > >
>
> > >
>
> > > > > > > <modules>
>
> > >
>
> > > > > > > <module>one</module>
>
> > >
>
> > > > > > > <module>two</module>
>
> > >
>
> > > > > > > </modules>
>
> > >
>
> > > > > > >
>
> > >
>
> > > > > > > We'd need -
>
> > >
>
> > > > > > >
>
> > >
>
> > > > > > > <modules>
>
> > >
>
> > > > > > > <search>recursively</search>
>
> > >
>
> > > > > > > </modules>
>
> > >
>
> > > > > > >
>
> > >
>
> > > > > > > Or -
>
> > >
>
> > > > > > >
>
> > >
>
> > > > > > > <modules>
>
> > >
>
> > > > > > > <defined-in>.full-module-list.txt</defined-in>
>
> > >
>
> > > > > > > <!-- made by
>
> > >
>
> > > > > > > find . -name "pom.xml" | sed 's/\/pom.xml//' >
>
> > >
>
> > > > > > > .full-module-list.txt
>
> > >
>
> > > > > > > after the sparse-checkout modification of working
>
> > > copy
>
> > >
>
> > > > > -->
>
> > >
>
> > > > > > > </modules>
>
> > >
>
> > > > > > >
>
> > >
>
> > > > > > > Thoughts?
>
> > >
>
> > > > > > >
>
> > >
>
> > > > > > > Any questions?
>
> > >
>
> > > > > > >
>
> > >
>
> > > > > > > - Paul H
>
> > >
>
> > > > > > >
>
> > >
>
> > > > > > > PS - I'm a solid Maven user since 2003.
>
> > >
>
> > > > > > >
>
> > >
>
> > > > > >
>
> > >
>
> > > > >
>
> > >
>
> > > >
>
> > >
>
> > > --
>
> > Sent from my phone
>
> >
>
> --
Sent from my phone
Re: Requesting a single Monorepo enhancement for Maven
Posted by Paul Hammant <pa...@hammant.org>.
Stephen - I think think Robert's <module>def:.full-module-list.txt</module>
is compatible hacking that is POM 4 friendly (and 3 for that matter) until
you revisit in 5.
I'd be horrified to write more XML than I already write in Maven.
I'm able to face Gradle advocates re a particular enterprise app and
without feeling I'm lying to myself say "what you're showing me is more of
less the same as Maven all things considered". Well if failsafe and tomcat
and surefile and coverage aren't in the same solution, that is. With the
profile fu, just to get Maven to follow the lead git-sparse-checkout could
give for such things, I could not face anyone and say that's what you
should do.
- Paul
On Tue, Jan 24, 2017 at 8:13 AM, Stephen Connolly <
stephen.alan.connolly@gmail.com> wrote:
> Yep I hear you.
>
> We cannot change the 4.0.0 schema
>
> And changing to a new modelVersion requires ensuring that we can evolve
> without breaking consumers of the older model.
>
> Basically we have one chance to make a "breaking" change to something that
> allows us to evolve going forward
>
> So what I provided was the 4.0.0 modelVersion solution... it's ugly but
> does not require pre-processing or generation of the pom
>
> On Tue 24 Jan 2017 at 12:53, Paul Hammant <pa...@hammant.org> wrote:
>
> > Versus profiles, I would rather have pom.xml exhaustively renamed to
> >
> > pom.xml.template in SCM, and a Python script to generate <modules/> as we
> >
> > have it today (pom.xml marked as .gitignore) before invoking 'maven
> > install'
> >
> >
> >
> > - Paul
> >
> >
> >
> >
> >
> > On Tue, Jan 24, 2017 at 7:31 AM, Stephen Connolly <
> >
> > stephen.alan.connolly@gmail.com> wrote:
> >
> >
> >
> > > It's an interesting idea for model version 5.0.0 to consider.
> >
> > >
> >
> > > At present, I would handle it by using profiles with activation to pull
> > in
> >
> > > the modules:
> >
> > >
> >
> > > if your activation in the root is based on the presence of the module's
> >
> > > pom.xml then it will only add the module if you checked it out: (approx
> >
> > > structure and I do not have the XSD to hand)
> >
> > >
> >
> > > <profile>
> >
> > > <id>module-foo</id>
> >
> > > <activation>
> >
> > > <file>module-foo/pom.xml</file>
> >
> > > </activation>
> >
> > > <modules>
> >
> > > <module>module-foo</module>
> >
> > > </modules>
> >
> > > </profile>
> >
> > > <profile>
> >
> > > <id>module-bar</id>
> >
> > > <activation>
> >
> > > <file>module-bar/pom.xml</file>
> >
> > > </activation>
> >
> > > <modules>
> >
> > > <module>module-bar</module>
> >
> > > </modules>
> >
> > > </profile>
> >
> > >
> >
> > > Yes that becomes an ugly pom, but it will do what you want when run
> from
> >
> > > the root and I have used it before
> >
> > >
> >
> > > HTH
> >
> > >
> >
> > > On 24 January 2017 at 11:14, Paul Hammant <pa...@hammant.org> wrote:
> >
> > >
> >
> > > > i thought about that too, except that in a monorepo situation, I
> don't
> >
> > > want
> >
> > > > the don't want the changed pom to get pushed back in a commit, and I
> >
> > > don't
> >
> > > > want one of the those changelists in my IDE labeled "do not commit"
> to
> >
> > > > facilitate that.
> >
> > > >
> >
> > > > Rationale: Just because I've subset my checkout/clone doesn't mean
> that
> >
> > > all
> >
> > > > users of the same repo want to.
> >
> > > >
> >
> > > > It was implied, but I'll call it out: .full-module-list.txt is in
> >
> > > > .gitignore (etc), and that it's easily regenerate per the 'find'
> > command.
> >
> > > >
> >
> > > > Regards,
> >
> > > >
> >
> > > > - Paul
> >
> > > >
> >
> > > > On Tue, Jan 24, 2017 at 12:50 AM, Aldrin Leal <al...@leal.eng.br>
> >
> > > wrote:
> >
> > > >
> >
> > > > > Actually, I always wondered if it was interesting to have a tool to
> >
> > > allow
> >
> > > > > the modification of POM files from Command Line. Like setting a
> >
> > > property,
> >
> > > > > adding a dependency and/or, as you exposed, changing modules.
> >
> > > > >
> >
> > > > > --
> >
> > > > > -- Aldrin Leal, <al...@leal.eng.br> / http://about.me/aldrinleal
> >
> > > > >
> >
> > > > > On Tue, Jan 24, 2017 at 12:05 AM, Paul Hammant <hammant@apache.org
> >
> >
> > > > wrote:
> >
> > > > >
> >
> > > > > > OK, so I'm a documenter of Google's Monorepo (one biiiig ass
> trunk)
> >
> > > and
> >
> > > > > > it's usage of shell scripts to subset the checkout for speedy
> >
> > > > > development:
> >
> > > > > >
> >
> > > > > > http://paulhammant.com/2014/01/06/googlers-subset-their-
> trunk/
> >
> > > > > > https://trunkbaseddevelopment.com/monorepos/
> >
> > > > > >
> >
> > > > > > For Maven to be used with a scripted use of Subversion or Git's
> >
> > > > > > sparse-checkout (or Perforce's client spec), it'd been to be more
> >
> > > like
> >
> > > > > > Bazel/Blaze or Buck, in that sub-modules are *not* forward
> > declared,
> >
> > > > they
> >
> > > > > > are discovered/calculated/inferred somehow.
> >
> > > > > >
> >
> > > > > > In pom.xml instead of -
> >
> > > > > >
> >
> > > > > > <modules>
> >
> > > > > > <module>one</module>
> >
> > > > > > <module>two</module>
> >
> > > > > > </modules>
> >
> > > > > >
> >
> > > > > > We'd need -
> >
> > > > > >
> >
> > > > > > <modules>
> >
> > > > > > <search>recursively</search>
> >
> > > > > > </modules>
> >
> > > > > >
> >
> > > > > > Or -
> >
> > > > > >
> >
> > > > > > <modules>
> >
> > > > > > <defined-in>.full-module-list.txt</defined-in>
> >
> > > > > > <!-- made by
> >
> > > > > > find . -name "pom.xml" | sed 's/\/pom.xml//' >
> >
> > > > > > .full-module-list.txt
> >
> > > > > > after the sparse-checkout modification of working
> > copy
> >
> > > > -->
> >
> > > > > > </modules>
> >
> > > > > >
> >
> > > > > > Thoughts?
> >
> > > > > >
> >
> > > > > > Any questions?
> >
> > > > > >
> >
> > > > > > - Paul H
> >
> > > > > >
> >
> > > > > > PS - I'm a solid Maven user since 2003.
> >
> > > > > >
> >
> > > > >
> >
> > > >
> >
> > >
> >
> > --
> Sent from my phone
>
Re: Requesting a single Monorepo enhancement for Maven
Posted by Stephen Connolly <st...@gmail.com>.
Yep I hear you.
We cannot change the 4.0.0 schema
And changing to a new modelVersion requires ensuring that we can evolve
without breaking consumers of the older model.
Basically we have one chance to make a "breaking" change to something that
allows us to evolve going forward
So what I provided was the 4.0.0 modelVersion solution... it's ugly but
does not require pre-processing or generation of the pom
On Tue 24 Jan 2017 at 12:53, Paul Hammant <pa...@hammant.org> wrote:
> Versus profiles, I would rather have pom.xml exhaustively renamed to
>
> pom.xml.template in SCM, and a Python script to generate <modules/> as we
>
> have it today (pom.xml marked as .gitignore) before invoking 'maven
> install'
>
>
>
> - Paul
>
>
>
>
>
> On Tue, Jan 24, 2017 at 7:31 AM, Stephen Connolly <
>
> stephen.alan.connolly@gmail.com> wrote:
>
>
>
> > It's an interesting idea for model version 5.0.0 to consider.
>
> >
>
> > At present, I would handle it by using profiles with activation to pull
> in
>
> > the modules:
>
> >
>
> > if your activation in the root is based on the presence of the module's
>
> > pom.xml then it will only add the module if you checked it out: (approx
>
> > structure and I do not have the XSD to hand)
>
> >
>
> > <profile>
>
> > <id>module-foo</id>
>
> > <activation>
>
> > <file>module-foo/pom.xml</file>
>
> > </activation>
>
> > <modules>
>
> > <module>module-foo</module>
>
> > </modules>
>
> > </profile>
>
> > <profile>
>
> > <id>module-bar</id>
>
> > <activation>
>
> > <file>module-bar/pom.xml</file>
>
> > </activation>
>
> > <modules>
>
> > <module>module-bar</module>
>
> > </modules>
>
> > </profile>
>
> >
>
> > Yes that becomes an ugly pom, but it will do what you want when run from
>
> > the root and I have used it before
>
> >
>
> > HTH
>
> >
>
> > On 24 January 2017 at 11:14, Paul Hammant <pa...@hammant.org> wrote:
>
> >
>
> > > i thought about that too, except that in a monorepo situation, I don't
>
> > want
>
> > > the don't want the changed pom to get pushed back in a commit, and I
>
> > don't
>
> > > want one of the those changelists in my IDE labeled "do not commit" to
>
> > > facilitate that.
>
> > >
>
> > > Rationale: Just because I've subset my checkout/clone doesn't mean that
>
> > all
>
> > > users of the same repo want to.
>
> > >
>
> > > It was implied, but I'll call it out: .full-module-list.txt is in
>
> > > .gitignore (etc), and that it's easily regenerate per the 'find'
> command.
>
> > >
>
> > > Regards,
>
> > >
>
> > > - Paul
>
> > >
>
> > > On Tue, Jan 24, 2017 at 12:50 AM, Aldrin Leal <al...@leal.eng.br>
>
> > wrote:
>
> > >
>
> > > > Actually, I always wondered if it was interesting to have a tool to
>
> > allow
>
> > > > the modification of POM files from Command Line. Like setting a
>
> > property,
>
> > > > adding a dependency and/or, as you exposed, changing modules.
>
> > > >
>
> > > > --
>
> > > > -- Aldrin Leal, <al...@leal.eng.br> / http://about.me/aldrinleal
>
> > > >
>
> > > > On Tue, Jan 24, 2017 at 12:05 AM, Paul Hammant <ha...@apache.org>
>
> > > wrote:
>
> > > >
>
> > > > > OK, so I'm a documenter of Google's Monorepo (one biiiig ass trunk)
>
> > and
>
> > > > > it's usage of shell scripts to subset the checkout for speedy
>
> > > > development:
>
> > > > >
>
> > > > > http://paulhammant.com/2014/01/06/googlers-subset-their-trunk/
>
> > > > > https://trunkbaseddevelopment.com/monorepos/
>
> > > > >
>
> > > > > For Maven to be used with a scripted use of Subversion or Git's
>
> > > > > sparse-checkout (or Perforce's client spec), it'd been to be more
>
> > like
>
> > > > > Bazel/Blaze or Buck, in that sub-modules are *not* forward
> declared,
>
> > > they
>
> > > > > are discovered/calculated/inferred somehow.
>
> > > > >
>
> > > > > In pom.xml instead of -
>
> > > > >
>
> > > > > <modules>
>
> > > > > <module>one</module>
>
> > > > > <module>two</module>
>
> > > > > </modules>
>
> > > > >
>
> > > > > We'd need -
>
> > > > >
>
> > > > > <modules>
>
> > > > > <search>recursively</search>
>
> > > > > </modules>
>
> > > > >
>
> > > > > Or -
>
> > > > >
>
> > > > > <modules>
>
> > > > > <defined-in>.full-module-list.txt</defined-in>
>
> > > > > <!-- made by
>
> > > > > find . -name "pom.xml" | sed 's/\/pom.xml//' >
>
> > > > > .full-module-list.txt
>
> > > > > after the sparse-checkout modification of working
> copy
>
> > > -->
>
> > > > > </modules>
>
> > > > >
>
> > > > > Thoughts?
>
> > > > >
>
> > > > > Any questions?
>
> > > > >
>
> > > > > - Paul H
>
> > > > >
>
> > > > > PS - I'm a solid Maven user since 2003.
>
> > > > >
>
> > > >
>
> > >
>
> >
>
> --
Sent from my phone
Re: Requesting a single Monorepo enhancement for Maven
Posted by Paul Hammant <pa...@hammant.org>.
Versus profiles, I would rather have pom.xml exhaustively renamed to
pom.xml.template in SCM, and a Python script to generate <modules/> as we
have it today (pom.xml marked as .gitignore) before invoking 'maven install'
- Paul
On Tue, Jan 24, 2017 at 7:31 AM, Stephen Connolly <
stephen.alan.connolly@gmail.com> wrote:
> It's an interesting idea for model version 5.0.0 to consider.
>
> At present, I would handle it by using profiles with activation to pull in
> the modules:
>
> if your activation in the root is based on the presence of the module's
> pom.xml then it will only add the module if you checked it out: (approx
> structure and I do not have the XSD to hand)
>
> <profile>
> <id>module-foo</id>
> <activation>
> <file>module-foo/pom.xml</file>
> </activation>
> <modules>
> <module>module-foo</module>
> </modules>
> </profile>
> <profile>
> <id>module-bar</id>
> <activation>
> <file>module-bar/pom.xml</file>
> </activation>
> <modules>
> <module>module-bar</module>
> </modules>
> </profile>
>
> Yes that becomes an ugly pom, but it will do what you want when run from
> the root and I have used it before
>
> HTH
>
> On 24 January 2017 at 11:14, Paul Hammant <pa...@hammant.org> wrote:
>
> > i thought about that too, except that in a monorepo situation, I don't
> want
> > the don't want the changed pom to get pushed back in a commit, and I
> don't
> > want one of the those changelists in my IDE labeled "do not commit" to
> > facilitate that.
> >
> > Rationale: Just because I've subset my checkout/clone doesn't mean that
> all
> > users of the same repo want to.
> >
> > It was implied, but I'll call it out: .full-module-list.txt is in
> > .gitignore (etc), and that it's easily regenerate per the 'find' command.
> >
> > Regards,
> >
> > - Paul
> >
> > On Tue, Jan 24, 2017 at 12:50 AM, Aldrin Leal <al...@leal.eng.br>
> wrote:
> >
> > > Actually, I always wondered if it was interesting to have a tool to
> allow
> > > the modification of POM files from Command Line. Like setting a
> property,
> > > adding a dependency and/or, as you exposed, changing modules.
> > >
> > > --
> > > -- Aldrin Leal, <al...@leal.eng.br> / http://about.me/aldrinleal
> > >
> > > On Tue, Jan 24, 2017 at 12:05 AM, Paul Hammant <ha...@apache.org>
> > wrote:
> > >
> > > > OK, so I'm a documenter of Google's Monorepo (one biiiig ass trunk)
> and
> > > > it's usage of shell scripts to subset the checkout for speedy
> > > development:
> > > >
> > > > http://paulhammant.com/2014/01/06/googlers-subset-their-trunk/
> > > > https://trunkbaseddevelopment.com/monorepos/
> > > >
> > > > For Maven to be used with a scripted use of Subversion or Git's
> > > > sparse-checkout (or Perforce's client spec), it'd been to be more
> like
> > > > Bazel/Blaze or Buck, in that sub-modules are *not* forward declared,
> > they
> > > > are discovered/calculated/inferred somehow.
> > > >
> > > > In pom.xml instead of -
> > > >
> > > > <modules>
> > > > <module>one</module>
> > > > <module>two</module>
> > > > </modules>
> > > >
> > > > We'd need -
> > > >
> > > > <modules>
> > > > <search>recursively</search>
> > > > </modules>
> > > >
> > > > Or -
> > > >
> > > > <modules>
> > > > <defined-in>.full-module-list.txt</defined-in>
> > > > <!-- made by
> > > > find . -name "pom.xml" | sed 's/\/pom.xml//' >
> > > > .full-module-list.txt
> > > > after the sparse-checkout modification of working copy
> > -->
> > > > </modules>
> > > >
> > > > Thoughts?
> > > >
> > > > Any questions?
> > > >
> > > > - Paul H
> > > >
> > > > PS - I'm a solid Maven user since 2003.
> > > >
> > >
> >
>
Re: Requesting a single Monorepo enhancement for Maven
Posted by Stephen Connolly <st...@gmail.com>.
It's an interesting idea for model version 5.0.0 to consider.
At present, I would handle it by using profiles with activation to pull in
the modules:
if your activation in the root is based on the presence of the module's
pom.xml then it will only add the module if you checked it out: (approx
structure and I do not have the XSD to hand)
<profile>
<id>module-foo</id>
<activation>
<file>module-foo/pom.xml</file>
</activation>
<modules>
<module>module-foo</module>
</modules>
</profile>
<profile>
<id>module-bar</id>
<activation>
<file>module-bar/pom.xml</file>
</activation>
<modules>
<module>module-bar</module>
</modules>
</profile>
Yes that becomes an ugly pom, but it will do what you want when run from
the root and I have used it before
HTH
On 24 January 2017 at 11:14, Paul Hammant <pa...@hammant.org> wrote:
> i thought about that too, except that in a monorepo situation, I don't want
> the don't want the changed pom to get pushed back in a commit, and I don't
> want one of the those changelists in my IDE labeled "do not commit" to
> facilitate that.
>
> Rationale: Just because I've subset my checkout/clone doesn't mean that all
> users of the same repo want to.
>
> It was implied, but I'll call it out: .full-module-list.txt is in
> .gitignore (etc), and that it's easily regenerate per the 'find' command.
>
> Regards,
>
> - Paul
>
> On Tue, Jan 24, 2017 at 12:50 AM, Aldrin Leal <al...@leal.eng.br> wrote:
>
> > Actually, I always wondered if it was interesting to have a tool to allow
> > the modification of POM files from Command Line. Like setting a property,
> > adding a dependency and/or, as you exposed, changing modules.
> >
> > --
> > -- Aldrin Leal, <al...@leal.eng.br> / http://about.me/aldrinleal
> >
> > On Tue, Jan 24, 2017 at 12:05 AM, Paul Hammant <ha...@apache.org>
> wrote:
> >
> > > OK, so I'm a documenter of Google's Monorepo (one biiiig ass trunk) and
> > > it's usage of shell scripts to subset the checkout for speedy
> > development:
> > >
> > > http://paulhammant.com/2014/01/06/googlers-subset-their-trunk/
> > > https://trunkbaseddevelopment.com/monorepos/
> > >
> > > For Maven to be used with a scripted use of Subversion or Git's
> > > sparse-checkout (or Perforce's client spec), it'd been to be more like
> > > Bazel/Blaze or Buck, in that sub-modules are *not* forward declared,
> they
> > > are discovered/calculated/inferred somehow.
> > >
> > > In pom.xml instead of -
> > >
> > > <modules>
> > > <module>one</module>
> > > <module>two</module>
> > > </modules>
> > >
> > > We'd need -
> > >
> > > <modules>
> > > <search>recursively</search>
> > > </modules>
> > >
> > > Or -
> > >
> > > <modules>
> > > <defined-in>.full-module-list.txt</defined-in>
> > > <!-- made by
> > > find . -name "pom.xml" | sed 's/\/pom.xml//' >
> > > .full-module-list.txt
> > > after the sparse-checkout modification of working copy
> -->
> > > </modules>
> > >
> > > Thoughts?
> > >
> > > Any questions?
> > >
> > > - Paul H
> > >
> > > PS - I'm a solid Maven user since 2003.
> > >
> >
>
Re: Requesting a single Monorepo enhancement for Maven
Posted by Paul Hammant <pa...@hammant.org>.
i thought about that too, except that in a monorepo situation, I don't want
the don't want the changed pom to get pushed back in a commit, and I don't
want one of the those changelists in my IDE labeled "do not commit" to
facilitate that.
Rationale: Just because I've subset my checkout/clone doesn't mean that all
users of the same repo want to.
It was implied, but I'll call it out: .full-module-list.txt is in
.gitignore (etc), and that it's easily regenerate per the 'find' command.
Regards,
- Paul
On Tue, Jan 24, 2017 at 12:50 AM, Aldrin Leal <al...@leal.eng.br> wrote:
> Actually, I always wondered if it was interesting to have a tool to allow
> the modification of POM files from Command Line. Like setting a property,
> adding a dependency and/or, as you exposed, changing modules.
>
> --
> -- Aldrin Leal, <al...@leal.eng.br> / http://about.me/aldrinleal
>
> On Tue, Jan 24, 2017 at 12:05 AM, Paul Hammant <ha...@apache.org> wrote:
>
> > OK, so I'm a documenter of Google's Monorepo (one biiiig ass trunk) and
> > it's usage of shell scripts to subset the checkout for speedy
> development:
> >
> > http://paulhammant.com/2014/01/06/googlers-subset-their-trunk/
> > https://trunkbaseddevelopment.com/monorepos/
> >
> > For Maven to be used with a scripted use of Subversion or Git's
> > sparse-checkout (or Perforce's client spec), it'd been to be more like
> > Bazel/Blaze or Buck, in that sub-modules are *not* forward declared, they
> > are discovered/calculated/inferred somehow.
> >
> > In pom.xml instead of -
> >
> > <modules>
> > <module>one</module>
> > <module>two</module>
> > </modules>
> >
> > We'd need -
> >
> > <modules>
> > <search>recursively</search>
> > </modules>
> >
> > Or -
> >
> > <modules>
> > <defined-in>.full-module-list.txt</defined-in>
> > <!-- made by
> > find . -name "pom.xml" | sed 's/\/pom.xml//' >
> > .full-module-list.txt
> > after the sparse-checkout modification of working copy -->
> > </modules>
> >
> > Thoughts?
> >
> > Any questions?
> >
> > - Paul H
> >
> > PS - I'm a solid Maven user since 2003.
> >
>
Re: Requesting a single Monorepo enhancement for Maven
Posted by Aldrin Leal <al...@leal.eng.br>.
Actually, I always wondered if it was interesting to have a tool to allow
the modification of POM files from Command Line. Like setting a property,
adding a dependency and/or, as you exposed, changing modules.
--
-- Aldrin Leal, <al...@leal.eng.br> / http://about.me/aldrinleal
On Tue, Jan 24, 2017 at 12:05 AM, Paul Hammant <ha...@apache.org> wrote:
> OK, so I'm a documenter of Google's Monorepo (one biiiig ass trunk) and
> it's usage of shell scripts to subset the checkout for speedy development:
>
> http://paulhammant.com/2014/01/06/googlers-subset-their-trunk/
> https://trunkbaseddevelopment.com/monorepos/
>
> For Maven to be used with a scripted use of Subversion or Git's
> sparse-checkout (or Perforce's client spec), it'd been to be more like
> Bazel/Blaze or Buck, in that sub-modules are *not* forward declared, they
> are discovered/calculated/inferred somehow.
>
> In pom.xml instead of -
>
> <modules>
> <module>one</module>
> <module>two</module>
> </modules>
>
> We'd need -
>
> <modules>
> <search>recursively</search>
> </modules>
>
> Or -
>
> <modules>
> <defined-in>.full-module-list.txt</defined-in>
> <!-- made by
> find . -name "pom.xml" | sed 's/\/pom.xml//' >
> .full-module-list.txt
> after the sparse-checkout modification of working copy -->
> </modules>
>
> Thoughts?
>
> Any questions?
>
> - Paul H
>
> PS - I'm a solid Maven user since 2003.
>
Re: Requesting a single Monorepo enhancement for Maven
Posted by Robert Scholte <rf...@apache.org>.
Hi Paul,
the problem with the proposals is that it doesn't fit the modelversion
4.0.0 (XSD)
However, what I could think of having a module with a protocol and provide
an extension which can locate the actual module locations. The extension
kicks in at startup while creating the BuildPlan.
e.g.
<modules>
<module>def:.full-module-list.txt</module>
</modules>
One question: suppose there's an issue in production, how can we get
exactly the same setup of the project to reproduce that issue? Or do you
accept that if you can't reproduce it on the master/trunk it is not an
issue anymore?
thanks,
Robert
On Tue, 24 Jan 2017 06:05:07 +0100, Paul Hammant <ha...@apache.org>
wrote:
> OK, so I'm a documenter of Google's Monorepo (one biiiig ass trunk) and
> it's usage of shell scripts to subset the checkout for speedy
> development:
>
> http://paulhammant.com/2014/01/06/googlers-subset-their-trunk/
> https://trunkbaseddevelopment.com/monorepos/
>
> For Maven to be used with a scripted use of Subversion or Git's
> sparse-checkout (or Perforce's client spec), it'd been to be more like
> Bazel/Blaze or Buck, in that sub-modules are *not* forward declared, they
> are discovered/calculated/inferred somehow.
>
> In pom.xml instead of -
>
> <modules>
> <module>one</module>
> <module>two</module>
> </modules>
>
> We'd need -
>
> <modules>
> <search>recursively</search>
> </modules>
>
> Or -
>
> <modules>
> <defined-in>.full-module-list.txt</defined-in>
> <!-- made by
> find . -name "pom.xml" | sed 's/\/pom.xml//' >
> .full-module-list.txt
> after the sparse-checkout modification of working copy -->
> </modules>
>
> Thoughts?
>
> Any questions?
>
> - Paul H
>
> PS - I'm a solid Maven user since 2003.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Requesting a single Monorepo enhancement for Maven
Posted by Paul Hammant <pa...@hammant.org>.
Here it is working with some Python scripting growing/shrinking the checkout.
http://paulhammant.com/2017/01/27/maven-in-a-google-style-monorepo/
A team could use it like that, right now.
On Wed, Jan 25, 2017 at 6:35 AM, Paul Hammant <pa...@hammant.org> wrote:
> OK so I've grappled git's sparse checkout and won (for now). I can use it
> to subset the checkout (not clone) and make Maven3 do a reduced build graph
> without using profiles.
>
> Details:
> https://github.com/paul-hammant/maven-monorepo-experiment/tree/trick-maven-monorepo
> details
>
> Google has 86 TB of history in their trunk, and you could't use Git for that
> for many reasons, but we' are a couple of inches closer for Maven now.
>
> - Paul
>
> On Tue, Jan 24, 2017 at 10:44 PM, Paul Hammant <pa...@hammant.org> wrote:
>>
>> OK, take a look at
>> https://github.com/paul-hammant/maven-monorepo-experiment/compare/trick-maven-monorepo
>>
>> Branch 1 - vanilla-recursive is a branch with HazelCast's core and samples
>> checked in - a 14 minute build IF YOU SKIP TESTS AND YOU ALREADY HAD CACHED
>> DEPS !!.
>>
>> Branch 2 - pom.xml files renamed to pom-template.xml and some shell/python
>> fu to recreate pom.xml files (read-only, .gitignore'd) obeying the directory
>> structure, and excluding <module/> lines where the directory is missing.
>>
>> See
>> https://github.com/paul-hammant/maven-monorepo-experiment/blob/trick-maven-monorepo/README.md
>>
>> That's enough for one night - more tomorrow. I get to find out whether
>> Git's sparse-checkout is elegant or a mess. At least for my use case.
>>
>> - Paul
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Requesting a single Monorepo enhancement for Maven
Posted by Paul Hammant <pa...@hammant.org>.
OK so I've grappled git's sparse checkout and won (for now). I can use it
to subset the checkout (not clone) and make Maven3 do a reduced build graph
without using profiles.
Details:
https://github.com/paul-hammant/maven-monorepo-experiment/tree/trick-maven-monorepo
details
Google has 86 TB of history in their trunk, and you could't use Git for
that for many reasons, but we' are a couple of inches closer for Maven now.
- Paul
On Tue, Jan 24, 2017 at 10:44 PM, Paul Hammant <pa...@hammant.org> wrote:
> OK, take a look at https://github.com/paul-hammant/maven-monorepo-
> experiment/compare/trick-maven-monorepo
>
> Branch 1 - vanilla-recursive
> <https://github.com/paul-hammant/maven-monorepo-experiment> is a branch
> with HazelCast's core and samples checked in - a 14 minute build IF YOU
> SKIP TESTS AND YOU ALREADY HAD CACHED DEPS !!.
>
> Branch 2 - pom.xml files renamed to pom-template.xml and some shell/python
> fu to recreate pom.xml files (read-only, .gitignore'd) obeying the
> directory structure, and excluding <module/> lines where the directory is
> missing.
>
> See https://github.com/paul-hammant/maven-monorepo-
> experiment/blob/trick-maven-monorepo/README.md
>
> That's enough for one night - more tomorrow. I get to find out whether
> Git's sparse-checkout is elegant or a mess. At least for my use case.
>
> - Paul
>
Re: Requesting a single Monorepo enhancement for Maven
Posted by Paul Hammant <pa...@hammant.org>.
OK, take a look at
https://github.com/paul-hammant/maven-monorepo-experiment/compare/trick-maven-monorepo
Branch 1 - vanilla-recursive
<https://github.com/paul-hammant/maven-monorepo-experiment> is a branch
with HazelCast's core and samples checked in - a 14 minute build IF YOU
SKIP TESTS AND YOU ALREADY HAD CACHED DEPS !!.
Branch 2 - pom.xml files renamed to pom-template.xml and some shell/python
fu to recreate pom.xml files (read-only, .gitignore'd) obeying the
directory structure, and excluding <module/> lines where the directory is
missing.
See
https://github.com/paul-hammant/maven-monorepo-experiment/blob/trick-maven-monorepo/README.md
That's enough for one night - more tomorrow. I get to find out whether
Git's sparse-checkout is elegant or a mess. At least for my use case.
- Paul
Re: Requesting a single Monorepo enhancement for Maven
Posted by Paul Hammant <pa...@hammant.org>.
Thanks Jörg - I'll keep that in mind for a fallback.
For a hacky maven3 modification towards Robert's <module>def:.full-module-
list.txt</module> I have found:
https://github.com/apache/maven/blob/master/maven-core/src/main/java/org/apache/maven/project/MavenProject.java#L1480
It looks like that method can modify module/sub-project lists. And that's
where I'm going to play around.
The method is marked as to-delete for Maven 4+, but that's not going to
stop me I don't think.
- Paul
On Tue, Jan 24, 2017 at 11:26 AM, Jörg Schaible <
joerg.schaible@bpm-inspire.com> wrote:
> Hi Paul,
>
> Paul Hammant wrote:
>
> > OK, so I'm a documenter of Google's Monorepo (one biiiig ass trunk) and
> > it's usage of shell scripts to subset the checkout for speedy
> development:
> >
> > http://paulhammant.com/2014/01/06/googlers-subset-their-trunk/
> > https://trunkbaseddevelopment.com/monorepos/
> >
> > For Maven to be used with a scripted use of Subversion or Git's
> > sparse-checkout (or Perforce's client spec), it'd been to be more like
> > Bazel/Blaze or Buck, in that sub-modules are *not* forward declared, they
> > are discovered/calculated/inferred somehow.
> >
> > In pom.xml instead of -
> >
> > <modules>
> > <module>one</module>
> > <module>two</module>
> > </modules>
> >
> > We'd need -
> >
> > <modules>
> > <search>recursively</search>
> > </modules>
> >
> > Or -
> >
> > <modules>
> > <defined-in>.full-module-list.txt</defined-in>
> > <!-- made by
> > find . -name "pom.xml" | sed 's/\/pom.xml//' >
> > .full-module-list.txt
> > after the sparse-checkout modification of working copy -->
> > </modules>
> >
> > Thoughts?
> >
> > Any questions?
> >
> > - Paul H
> >
> > PS - I'm a solid Maven user since 2003.
>
> we did something like that just with bash scripts and it boils down to to a
> simple shell function. In the follwing example it creates module entries
> for
> all direct subdirectories containing a pom.xml file, but you might easily
> adjust this to some expression using globs, find or a list in a file.
>
> ==================== %< ================
>
> function createBuilder()
> {
> local artifactId
> local path
>
> artifactId=${1##*/}
> cat > "$1"/pom.xml <<EOF
> <?xml version="1.0" encoding="UTF-8"?>
> <project
> xmlns="http://maven.apache.org/POM/4.0.0"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/maven-v4_0_0.xsd">
>
> <modelVersion>4.0.0</modelVersion>
> <groupId>builder</groupId>
> <artifactId>$artifactId</artifactId>
> <packaging>pom</packaging>
> <version>x-SNAPSHOT</version>
> <name>Builder $artifactId</name>
>
> <modules>
> EOF
> while read path; do
> if [ -f "$path" ]; then
> continue
> fi
> path=${path##*/}
> cat >> "$1"/pom.xml <<EOF
> <module>$path</module>
> EOF
> done < <(ls -d "$1"/*)
> cat >> "$1"/pom.xml <<EOF
> </modules>
> </project>
> EOF
> }
>
> ==================== %< ================
>
> Cheers,
> Jörg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: Requesting a single Monorepo enhancement for Maven
Posted by Jörg Schaible <jo...@bpm-inspire.com>.
Hi Paul,
Paul Hammant wrote:
> OK, so I'm a documenter of Google's Monorepo (one biiiig ass trunk) and
> it's usage of shell scripts to subset the checkout for speedy development:
>
> http://paulhammant.com/2014/01/06/googlers-subset-their-trunk/
> https://trunkbaseddevelopment.com/monorepos/
>
> For Maven to be used with a scripted use of Subversion or Git's
> sparse-checkout (or Perforce's client spec), it'd been to be more like
> Bazel/Blaze or Buck, in that sub-modules are *not* forward declared, they
> are discovered/calculated/inferred somehow.
>
> In pom.xml instead of -
>
> <modules>
> <module>one</module>
> <module>two</module>
> </modules>
>
> We'd need -
>
> <modules>
> <search>recursively</search>
> </modules>
>
> Or -
>
> <modules>
> <defined-in>.full-module-list.txt</defined-in>
> <!-- made by
> find . -name "pom.xml" | sed 's/\/pom.xml//' >
> .full-module-list.txt
> after the sparse-checkout modification of working copy -->
> </modules>
>
> Thoughts?
>
> Any questions?
>
> - Paul H
>
> PS - I'm a solid Maven user since 2003.
we did something like that just with bash scripts and it boils down to to a
simple shell function. In the follwing example it creates module entries for
all direct subdirectories containing a pom.xml file, but you might easily
adjust this to some expression using globs, find or a list in a file.
==================== %< ================
function createBuilder()
{
local artifactId
local path
artifactId=${1##*/}
cat > "$1"/pom.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<project
xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>builder</groupId>
<artifactId>$artifactId</artifactId>
<packaging>pom</packaging>
<version>x-SNAPSHOT</version>
<name>Builder $artifactId</name>
<modules>
EOF
while read path; do
if [ -f "$path" ]; then
continue
fi
path=${path##*/}
cat >> "$1"/pom.xml <<EOF
<module>$path</module>
EOF
done < <(ls -d "$1"/*)
cat >> "$1"/pom.xml <<EOF
</modules>
</project>
EOF
}
==================== %< ================
Cheers,
J�rg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Requesting a single Monorepo enhancement for Maven
Posted by Paul Hammant <pa...@hammant.org>.
It is the last big problem, yes.
Sent from my iPhone
> On Jan 24, 2017, at 10:10 AM, Igor Fedorenko <ig...@ifedorenko.com> wrote:
>
> Don't mean to derail the discussion, but I am wondering if sparse
> checkout is the last/biggest problem you have to solve to use maven for
> your "one biiiig ass trunk".
>
> We are using maven for a monorepo (maybe not as "biiiig ass" as google's
> but pretty big nonetheless) for few years now and lack of sparse
> checkout was never a big problem for us. Disk space and even network are
> relatively cheap these days, after all. We did have to implement what we
> call "partial build", where we don't build the whole tree, but a subset
> of modules selected by the user, with the required dependencies coming
> as prebuilt artifacts from a repository. This is conceptually what
> --projects and -SNAPSHOTs claim do, only hardened to scale for large
> number of modules and developers.
>
> --
> Regards,
> Igor
>
>> On Tue, Jan 24, 2017, at 12:05 AM, Paul Hammant wrote:
>> OK, so I'm a documenter of Google's Monorepo (one biiiig ass trunk) and
>> it's usage of shell scripts to subset the checkout for speedy
>> development:
>>
>> http://paulhammant.com/2014/01/06/googlers-subset-their-trunk/
>> https://trunkbaseddevelopment.com/monorepos/
>>
>> For Maven to be used with a scripted use of Subversion or Git's
>> sparse-checkout (or Perforce's client spec), it'd been to be more like
>> Bazel/Blaze or Buck, in that sub-modules are *not* forward declared, they
>> are discovered/calculated/inferred somehow.
>>
>> In pom.xml instead of -
>>
>> <modules>
>> <module>one</module>
>> <module>two</module>
>> </modules>
>>
>> We'd need -
>>
>> <modules>
>> <search>recursively</search>
>> </modules>
>>
>> Or -
>>
>> <modules>
>> <defined-in>.full-module-list.txt</defined-in>
>> <!-- made by
>> find . -name "pom.xml" | sed 's/\/pom.xml//' >
>> .full-module-list.txt
>> after the sparse-checkout modification of working copy -->
>> </modules>
>>
>> Thoughts?
>>
>> Any questions?
>>
>> - Paul H
>>
>> PS - I'm a solid Maven user since 2003.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Requesting a single Monorepo enhancement for Maven
Posted by Igor Fedorenko <ig...@ifedorenko.com>.
Don't mean to derail the discussion, but I am wondering if sparse
checkout is the last/biggest problem you have to solve to use maven for
your "one biiiig ass trunk".
We are using maven for a monorepo (maybe not as "biiiig ass" as google's
but pretty big nonetheless) for few years now and lack of sparse
checkout was never a big problem for us. Disk space and even network are
relatively cheap these days, after all. We did have to implement what we
call "partial build", where we don't build the whole tree, but a subset
of modules selected by the user, with the required dependencies coming
as prebuilt artifacts from a repository. This is conceptually what
--projects and -SNAPSHOTs claim do, only hardened to scale for large
number of modules and developers.
--
Regards,
Igor
On Tue, Jan 24, 2017, at 12:05 AM, Paul Hammant wrote:
> OK, so I'm a documenter of Google's Monorepo (one biiiig ass trunk) and
> it's usage of shell scripts to subset the checkout for speedy
> development:
>
> http://paulhammant.com/2014/01/06/googlers-subset-their-trunk/
> https://trunkbaseddevelopment.com/monorepos/
>
> For Maven to be used with a scripted use of Subversion or Git's
> sparse-checkout (or Perforce's client spec), it'd been to be more like
> Bazel/Blaze or Buck, in that sub-modules are *not* forward declared, they
> are discovered/calculated/inferred somehow.
>
> In pom.xml instead of -
>
> <modules>
> <module>one</module>
> <module>two</module>
> </modules>
>
> We'd need -
>
> <modules>
> <search>recursively</search>
> </modules>
>
> Or -
>
> <modules>
> <defined-in>.full-module-list.txt</defined-in>
> <!-- made by
> find . -name "pom.xml" | sed 's/\/pom.xml//' >
> .full-module-list.txt
> after the sparse-checkout modification of working copy -->
> </modules>
>
> Thoughts?
>
> Any questions?
>
> - Paul H
>
> PS - I'm a solid Maven user since 2003.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org