You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@netbeans.apache.org by Geertjan Wielenga <ge...@googlemail.com> on 2016/10/12 11:41:45 UTC
[PROPOSAL] Split the main NetBeans repo
Hello all,
Please note Jan Lahoda, one of the NetBeans gurus in Prague, has published
this on the Wiki:
https://cwiki.apache.org/confluence/display/NETBEANS/Experiment%3A+split+the+main+NetBeans+repository
Comments, thoughts, feedback, welcome.
Gj
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Emilian Bold <em...@gmail.com>.
> As another example: we implemented Terminal in C++, then 'contributed'
into Full IDE by simple re-regestring module in nbbuild/cluster.properties.
For those who don't know NetBeans internals this was basically a 2 line
change in a text file.
> If the main advantage of split is "size", then we need to see new numbers
when history is moved as well.
I guess the biggest cluster will still be 300 - 500MB. We should run a full
git filter, etc. and see.
BTW, since we have commits that modify files in multiple clusters it would
be nice to easily verify this in the future when we will have per cluster
repositories. Perhaps the history split/migration script should be smart
about this and use the same changeset ID across multiple repositories (if
possible); or maybe generate some unique tag?
--emi
On Wed, Oct 12, 2016 at 4:54 PM, Vladimir Voskresensky <
vladimirvv@apache.org> wrote:
> On 2016-10-12 15:30 (+0300), Emilian Bold <em...@gmail.com> wrote:
> > As I mentioned before I don't see this an improvement of the status-quo,
> > technically.
> I would agree with Emilian, that the current structure proved to work and
> it works reliable.
> If the main advantage of split is "size", then we need to see new numbers
> when history is moved as well.
>
> Btw, sometimes Java introduces the new feature, then C++ introduces the
> same functionality, so we agree to make refactorings by generalizing common
> parts and moving them into platform/nb layer, while keeping
> Java-specific/C++-specific code in corresponding places. History is easier
> manageable when it happens in one repo.
>
> As another example: we implemented Terminal in C++, then 'contributed'
> into Full IDE by simple re-regestring module in nbbuild/cluster.properties.
> One repo fits better, than moving modules between different ones
>
> Thanks,
> Vladimir.
>
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Vladimir Voskresensky <vl...@apache.org>.
On 2016-10-12 15:30 (+0300), Emilian Bold <em...@gmail.com> wrote:
> As I mentioned before I don't see this an improvement of the status-quo,
> technically.
I would agree with Emilian, that the current structure proved to work and it works reliable.
If the main advantage of split is "size", then we need to see new numbers when history is moved as well.
Btw, sometimes Java introduces the new feature, then C++ introduces the same functionality, so we agree to make refactorings by generalizing common parts and moving them into platform/nb layer, while keeping Java-specific/C++-specific code in corresponding places. History is easier manageable when it happens in one repo.
As another example: we implemented Terminal in C++, then 'contributed' into Full IDE by simple re-regestring module in nbbuild/cluster.properties. One repo fits better, than moving modules between different ones
Thanks,
Vladimir.
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Emilian Bold <em...@gmail.com>.
I'm glad Jan went ahead and did this. Per-cluster seems to most logical way
to carve the repository while still allowing history to be kept.
By definition clusters don't really map to a library or a build because if
you look at the matrix on the download page
https://netbeans.org/downloads/ you'll
see that there is some overlapping. So, this approach is similar to
Mercurial Forest.
Still, Jan seems to have thought about this:
> in the experiment, there is still a single global
nbbuild/cluster.properties - would be better to split and place parts into
individual repositories
which means that there is nothing stopping up to tweak this a bit and add
per-cluster cluster.properties which might perhaps allow us to build, say
NetBeans C/C++ by just cloning the netbeans-cpp cluster (and
netbeans-platform and netbeans-build(?)).
As I mentioned before I don't see this an improvement of the status-quo,
technically. But it could be a big gain socially, by allowing people to
contribute by, say, using GitHub without worrying about space restrictions
(although a single cluster will not be small).
Also note the final point on the page:
> the main intent here is to experiment with a way to keep repository sizes
smaller while keeping history. One of the risks here is that having (too
many) repositories is also not inconvenient. So, some consolidation of
clusters into repositories seems to be in order. As a data point, some
other big project is proposing to move from multiple smaller repositories
to a single bigger repository (i.e. closer to the current NetBeans state):
http://mail.openjdk.java.net/pipermail/jdk9-dev/2016-October/004987.html
Google is famous for their single repository (
http://programmers.stackexchange.com/a/41451 )
> Google manages to keep the source code of all its projects, over 2000, in
a single code trunk containing hundreds of millions of code lines
as is Facebook, which uses Mercurial (
https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/
> Facebook's main source repository is enormous--many times larger than
even the Linux kernel, which checked in at 17 million lines of code and
44,000 files in 2013
And of course, there are other projects like FreeBSD that use single
repositories for the whole product.
--emi
On Wed, Oct 12, 2016 at 2:41 PM, Geertjan Wielenga <
geertjan.wielenga@googlemail.com> wrote:
> Hello all,
>
> Please note Jan Lahoda, one of the NetBeans gurus in Prague, has published
> this on the Wiki:
>
> https://cwiki.apache.org/confluence/display/NETBEANS/
> Experiment%3A+split+the+main+NetBeans+repository
>
> Comments, thoughts, feedback, welcome.
>
> Gj
>
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Julien Enselme <je...@jujens.eu>.
On Thu, 2016-10-13 at 08:14 -0400, Wade Chandler wrote:
> On Oct 12, 2016, at 15:23, Sven Reimers <sv...@gmail.com>
> wrote:
> >
> >
> > Hi all,
> >
> > what keeps me thinking is that there is actually always a
> > difference
> > between modules that are now part of the main repository and
> > "other"
> > modules (provided by the larger community), e.g. being a friend to
> > "core"
> > modules if your are not part of the main tree always seemed very
> > hard
> > (resulting in a lot of implementation dependencies requiring re-
> > releases
> > for all major/minor NetBeans versions).
> >
> > How will we handle this in the new infrastructure at / in Apache?
> > How would
> > we decide which module ge's in a mega master repository?
> >
> > I have a gut feeling that we have the chance to / should fix this
> > now.. but
> > not yet sure how..
> >
> > Ideas? Comments?
> >
> > Sven
> >
>
> Yes, I think this is problematic, and plays with the below to some
> degree. I will explain below.
>
> >
> >
> > On Oct 12, 2016, at 09:54, Vladimir Voskresensky <vladimirvv@apache
> > .org> wrote:
> >
> > On 2016-10-12 15:30 (+0300), Emilian Bold <em...@gmail.com>
> > wrote:
> > >
> > > As I mentioned before I don't see this an improvement of the
> > > status-quo,
> > > technically.
> > I would agree with Emilian, that the current structure proved to
> > work and it works reliable.
> > If the main advantage of split is "size", then we need to see new
> > numbers when history is moved as well.
> >
> > Btw, sometimes Java introduces the new feature, then C++ introduces
> > the same functionality, so we agree to make refactorings by
> > generalizing common parts and moving them into platform/nb layer,
> > while keeping Java-specific/C++-specific code in corresponding
> > places. History is easier manageable when it happens in one repo.
> >
>
> Perhaps more manageable for the teams working in that single repo,
> but what about other community modules? Is this not perhaps why we
> end up with more friends and less open APIs, and then requests to get
> that support are not able to be handled because of time, and the
> thing not being easy to do after the fact? If those gates currently
> considered easy were not available, but perhaps replaced with a
> better distribution plan, then perhaps quick “friend” fixes would be
> done in a more modular manner from the beginning leading to better
> community module support and requiring less friend dependencies. I
> think this addresses Sven’s point by breaking things up, and
> personally, the more separation which can be worked in the more it
> causes APIs to be flushed out for the sake of the community plus
> those “core” modules.
>
> Too, I think there is practicality in having smaller more modular
> repositories where some smarter system for producing the final
> artifacts exists. Smaller repositories makes it easier for folks to
> contribute to specific problem sets by not having to clone some
> massive repository. If the successes hoped by the move to Apache
> prove out, then should we not see the repository grow significantly
> over time?
>
> So, maybe there is something to be said for bringing things over as
> they are now, for the sake of incubation, progress, and the code
> grant, but I think there has to be a goal and a plan for something
> more nuanced than exists now.
>
> >
> > As another example: we implemented Terminal in C++, then
> > 'contributed' into Full IDE by simple re-regestring module in
> > nbbuild/cluster.properties. One repo fits better, than moving
> > modules between different ones
> >
>
> I’m not sure that had as much to do with it being the single
> repository as the way the artifacts bundles (installs and archives)
> are produced. I think the notion of a core release could be baked
> into a separate repository allowing disjoint development with a
> unified release; a build problem versus a source repository problem.
> This could even allow Python to be more easily integrated into the
> community release of core again, or allow for more “out of the box”
> type installers. What if someone adds Kotlin support as an example?
> Would it then come into the core? What about Attila’s gradle support?
> Should gradle really continue as something only available by way of
> the plugin center yet not Maven and Ant? It is quite popular and
> important these days. Should that have to come to “core” or the main
> repository for that to happen?
>
> Wade
I tend to agree with you Wade.
Both solutions have their pros and cons. But as a contributor NBPython,
I'd prefer not to have the big main repository on my machine. I don't
need most of it and it takes a lot of space. From a private discussion
I had with a user of NBPython in an RCP app, having the whole main repo
is problematic (space, compilation time) So I think that we are more
likely to get external contributors with small repositories.
I like your point about integrating modules currently developed outside
the core (Python, Gradle). Even if by importing the repos in git we
manage to filter stuff and make it a reasonable size, it will be much
easier to integrate these modules from outside in the infrastructure
without loosing any history in a modular approach.
Regards,
--
Julien Enselme
http://www.jujens.eu/
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Wade Chandler <co...@wadechandler.com>.
On Oct 12, 2016, at 15:23, Sven Reimers <sv...@gmail.com> wrote:
>
> Hi all,
>
> what keeps me thinking is that there is actually always a difference
> between modules that are now part of the main repository and "other"
> modules (provided by the larger community), e.g. being a friend to "core"
> modules if your are not part of the main tree always seemed very hard
> (resulting in a lot of implementation dependencies requiring re-releases
> for all major/minor NetBeans versions).
>
> How will we handle this in the new infrastructure at / in Apache? How would
> we decide which module ge's in a mega master repository?
>
> I have a gut feeling that we have the chance to / should fix this now.. but
> not yet sure how..
>
> Ideas? Comments?
>
> Sven
>
Yes, I think this is problematic, and plays with the below to some degree. I will explain below.
>
> On Oct 12, 2016, at 09:54, Vladimir Voskresensky <vl...@apache.org> wrote:
>
> On 2016-10-12 15:30 (+0300), Emilian Bold <em...@gmail.com> wrote:
>> As I mentioned before I don't see this an improvement of the status-quo,
>> technically.
> I would agree with Emilian, that the current structure proved to work and it works reliable.
> If the main advantage of split is "size", then we need to see new numbers when history is moved as well.
>
> Btw, sometimes Java introduces the new feature, then C++ introduces the same functionality, so we agree to make refactorings by generalizing common parts and moving them into platform/nb layer, while keeping Java-specific/C++-specific code in corresponding places. History is easier manageable when it happens in one repo.
>
Perhaps more manageable for the teams working in that single repo, but what about other community modules? Is this not perhaps why we end up with more friends and less open APIs, and then requests to get that support are not able to be handled because of time, and the thing not being easy to do after the fact? If those gates currently considered easy were not available, but perhaps replaced with a better distribution plan, then perhaps quick “friend” fixes would be done in a more modular manner from the beginning leading to better community module support and requiring less friend dependencies. I think this addresses Sven’s point by breaking things up, and personally, the more separation which can be worked in the more it causes APIs to be flushed out for the sake of the community plus those “core” modules.
Too, I think there is practicality in having smaller more modular repositories where some smarter system for producing the final artifacts exists. Smaller repositories makes it easier for folks to contribute to specific problem sets by not having to clone some massive repository. If the successes hoped by the move to Apache prove out, then should we not see the repository grow significantly over time?
So, maybe there is something to be said for bringing things over as they are now, for the sake of incubation, progress, and the code grant, but I think there has to be a goal and a plan for something more nuanced than exists now.
> As another example: we implemented Terminal in C++, then 'contributed' into Full IDE by simple re-regestring module in nbbuild/cluster.properties. One repo fits better, than moving modules between different ones
>
I’m not sure that had as much to do with it being the single repository as the way the artifacts bundles (installs and archives) are produced. I think the notion of a core release could be baked into a separate repository allowing disjoint development with a unified release; a build problem versus a source repository problem. This could even allow Python to be more easily integrated into the community release of core again, or allow for more “out of the box” type installers. What if someone adds Kotlin support as an example? Would it then come into the core? What about Attila’s gradle support? Should gradle really continue as something only available by way of the plugin center yet not Maven and Ant? It is quite popular and important these days. Should that have to come to “core” or the main repository for that to happen?
Wade
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Sven Reimers <sv...@gmail.com>.
Hi all,
what keeps me thinking is that there is actually always a difference
between modules that are now part of the main repository and "other"
modules (provided by the larger community), e.g. being a friend to "core"
modules if your are not part of the main tree always seemed very hard
(resulting in a lot of implementation dependencies requiring re-releases
for all major/minor NetBeans versions).
How will we handle this in the new infrastructure at / in Apache? How would
we decide which module ge's in a mega master repository?
I have a gut feeling that we have the chance to / should fix this now.. but
not yet sure how..
Ideas? Comments?
Sven
Am 12.10.2016 20:31 schrieb "Emilian Bold" <em...@gmail.com>:
> Yes, I think it makes sense to use branches for 'teams'. Maybe even for
> main-silver, main-golden.
>
>
> --emi
>
> On Wed, Oct 12, 2016 at 9:23 PM, Vladimir Voskresensky <
> vladimir.voskresensky@oracle.com> wrote:
>
> > Hi,
> >
> > On 12.10.2016 18:23, Eric Barboni wrote:
> >
> >> With the split in cluster it may possible to get rid of the "team"
> >> repositories (i.e. "core-main","cnd-main","jet-main") but we may need
> >> another repository ( kind of current "main-silver" or "main-golden")
> to
> >> prevent a bad commit leading to compile error in one cluster to be
> >> propagated to the full NetBeans break the build for others who did not
> need
> >> this particular cluster
> >>
> > What if we leave one repository as it is now and move 'teams' into git
> > branches (core-main/cnd-main/...).
> > Teams will work on branches.
> > Then like now [1] the current push-cnd-main job can be replaced by
> > automatic 'push cnd-main into main-silver branch', while push-to-cnd-main
> > job will be automatic 'push main-silver into cnd-main'.
> > Branch main-silver is built and tested as now and on success main-silver
> > is pushed into main-golden for the production build.
> >
> > Thanks,
> > Vladimir.
> > [1] http://wiki.netbeans.org/HgParallelProjectIntegration
> >
>
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Wade Chandler <co...@wadechandler.com>.
Yes, I too am wondering about PRs. Does the Apache git infra support pull
requests? Is it a GitHub Enterprise install? Maybe a better question is:
What is supported by the infrastructure? PRs are massively useful.
Wade
On Oct 18, 2016 6:25 PM, "Leonardo Loch Zanivan" <le...@gmail.com>
wrote:
> I like the idea of having many repos in git (ex: platform, java, php, etc)
> and a parent repo with references to other repos for building all together.
>
> +1 to work with external feature branches.
>
> My only concern is about code reviews and continuous integration, there's a
> possibility to use GitHub for that?
>
> Regards,
> Leonardo
>
> On Tue, Oct 18, 2016 at 4:49 PM Mark Struberg <st...@yahoo.de.invalid>
> wrote:
>
> > The gitflow model is imo only worth it if you have dedicated build
> masters
> > who really are the ‚masters‘ of what goes into a release or not.
> >
> > This works fine for company projects where you have seniors waving
> through
> > the work of juniors. But at the end those build-masters need to have a
> good
> > knowledge about the single features.
> > It’s kind of the Linux model, but all in the same repo instead of pulling
> > from other repos and applying patches.
> > Do we have such a build-master at the ASF? Usually we do not! All
> > committers usually know the code very well. Of course NetBeans is a bit
> > different and of course a few people like Geertjan and Jan do know MUCH
> > more about what is good for a stable code base and what rather needs some
> > more tinkering.
> > In a long term I guess it is one of the goals to broaden the pool of
> > people who have a good overlook over the project.
> > For contributors who are rather new to such a huge code base, I think it
> > might be good to work with external feature branches and rebase instead.
> >
> > just my .02
> >
> > LieGrue,
> > strub
> >
> > > Am 15.10.2016 um 01:05 schrieb Niclas Hedhman <ni...@hedhman.org>:
> > >
> > > http://nvie.com/posts/a-successful-git-branching-model/
> > >
> > > On Fri, Oct 14, 2016 at 10:46 PM, Wade Chandler <
> > consult@wadechandler.com>
> > > wrote:
> > >
> > >> On Oct 14, 2016, at 08:53, Emilian Bold <em...@gmail.com>
> wrote:
> > >>>
> > >>> This incubation is just the beginning of NetBeans under Apache. We
> > don't
> > >>> have to reinvent ourselves at once.
> > >>>
> > >>
> > >> I agree with that.
> > >>
> > >>> Let's first see how people start contributing and if there is
> friction
> > >> due
> > >>> to different work approaches, I'm sure some new consensus will be
> > >> reached.
> > >>>
> > >>
> > >> I think that is feasible as long as everyone agrees if we need to
> shift
> > we
> > >> will do it quickly, and we give it some thought ahead of time so it
> > isn’t
> > >> such a big deal if/when it happens.
> > >>
> > >>> Like I've mentioned before, I don't believe the repository size is an
> > >>> issue. It's a big project with a big history. Even if I could
> checkout
> > >> just
> > >>> the file I want to patch, in order to build and test that I would
> still
> > >>> have to download the NetBeans binaries and the smallest download is
> > >> 110MB.
> > >>>
> > >>
> > >> But not exactly apples to apples…even without history you are talking
> > >> about 110MB to 1GB which is nearly a 1000% difference.
> > >>
> > >>> Friend dependencies are also not a problem of the source code
> > structure,
> > >>> it's a matter of API stability <http://wiki.netbeans.org/API_
> Stability
> > >.
> > >>> It's easy to mark an API as Stable/Official and then the plugins
> > problem
> > >> is
> > >>> solved. But then somebody has to respect this contract and support
> that
> > >> API
> > >>> throughout releases, which is not an easy task.
> > >>>
> > >>
> > >> To me we are talking 2 different contexts though; supporting the
> > framework
> > >> for the larger community versus a narrowed scope of who has to manage
> > >> something once it goes into the wild. Too, I think there are plenty of
> > >> examples of folks not being able to do something because of friend
> deps
> > >> without becoming a friend, and that seems to suggest it was easier to
> > just
> > >> “get there” and stop without going further; resource contentions etc.
> > Sure,
> > >> that same thing could happen even across repositories, and would need
> > to be
> > >> left that way for some time, but I think definitely needs to be part
> of
> > the
> > >> process to address, and to me is some times related to separation of
> > >> development; monolith mindset versus modular even if modular the
> mindset
> > >> can set in with a gigantic blob of code all bound by the build system
> > and
> > >> repository. I’m not saying it is directly related, but I think there
> is
> > >> good correlation as it tends to happen in systems, and the trends seem
> > to
> > >> suggest it happens because of the culture of way things are organized.
> > This
> > >> is called Conway’s Law, and I have an intuition some of that may be at
> > play
> > >> with this topic:
> > >> https://en.wikipedia.org/wiki/Conway%27s_law <
> https://en.wikipedia.org/
> > >> wiki/Conway's_law>
> > >>
> > >> Wade
> > >
> > >
> > >
> > >
> > > --
> > > Niclas Hedhman, Software Developer
> > > http://zest.apache.org - New Energy for Java
> >
> >
>
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Leonardo Loch Zanivan <le...@gmail.com>.
I like the idea of having many repos in git (ex: platform, java, php, etc)
and a parent repo with references to other repos for building all together.
+1 to work with external feature branches.
My only concern is about code reviews and continuous integration, there's a
possibility to use GitHub for that?
Regards,
Leonardo
On Tue, Oct 18, 2016 at 4:49 PM Mark Struberg <st...@yahoo.de.invalid>
wrote:
> The gitflow model is imo only worth it if you have dedicated build masters
> who really are the ‚masters‘ of what goes into a release or not.
>
> This works fine for company projects where you have seniors waving through
> the work of juniors. But at the end those build-masters need to have a good
> knowledge about the single features.
> It’s kind of the Linux model, but all in the same repo instead of pulling
> from other repos and applying patches.
> Do we have such a build-master at the ASF? Usually we do not! All
> committers usually know the code very well. Of course NetBeans is a bit
> different and of course a few people like Geertjan and Jan do know MUCH
> more about what is good for a stable code base and what rather needs some
> more tinkering.
> In a long term I guess it is one of the goals to broaden the pool of
> people who have a good overlook over the project.
> For contributors who are rather new to such a huge code base, I think it
> might be good to work with external feature branches and rebase instead.
>
> just my .02
>
> LieGrue,
> strub
>
> > Am 15.10.2016 um 01:05 schrieb Niclas Hedhman <ni...@hedhman.org>:
> >
> > http://nvie.com/posts/a-successful-git-branching-model/
> >
> > On Fri, Oct 14, 2016 at 10:46 PM, Wade Chandler <
> consult@wadechandler.com>
> > wrote:
> >
> >> On Oct 14, 2016, at 08:53, Emilian Bold <em...@gmail.com> wrote:
> >>>
> >>> This incubation is just the beginning of NetBeans under Apache. We
> don't
> >>> have to reinvent ourselves at once.
> >>>
> >>
> >> I agree with that.
> >>
> >>> Let's first see how people start contributing and if there is friction
> >> due
> >>> to different work approaches, I'm sure some new consensus will be
> >> reached.
> >>>
> >>
> >> I think that is feasible as long as everyone agrees if we need to shift
> we
> >> will do it quickly, and we give it some thought ahead of time so it
> isn’t
> >> such a big deal if/when it happens.
> >>
> >>> Like I've mentioned before, I don't believe the repository size is an
> >>> issue. It's a big project with a big history. Even if I could checkout
> >> just
> >>> the file I want to patch, in order to build and test that I would still
> >>> have to download the NetBeans binaries and the smallest download is
> >> 110MB.
> >>>
> >>
> >> But not exactly apples to apples…even without history you are talking
> >> about 110MB to 1GB which is nearly a 1000% difference.
> >>
> >>> Friend dependencies are also not a problem of the source code
> structure,
> >>> it's a matter of API stability <http://wiki.netbeans.org/API_Stability
> >.
> >>> It's easy to mark an API as Stable/Official and then the plugins
> problem
> >> is
> >>> solved. But then somebody has to respect this contract and support that
> >> API
> >>> throughout releases, which is not an easy task.
> >>>
> >>
> >> To me we are talking 2 different contexts though; supporting the
> framework
> >> for the larger community versus a narrowed scope of who has to manage
> >> something once it goes into the wild. Too, I think there are plenty of
> >> examples of folks not being able to do something because of friend deps
> >> without becoming a friend, and that seems to suggest it was easier to
> just
> >> “get there” and stop without going further; resource contentions etc.
> Sure,
> >> that same thing could happen even across repositories, and would need
> to be
> >> left that way for some time, but I think definitely needs to be part of
> the
> >> process to address, and to me is some times related to separation of
> >> development; monolith mindset versus modular even if modular the mindset
> >> can set in with a gigantic blob of code all bound by the build system
> and
> >> repository. I’m not saying it is directly related, but I think there is
> >> good correlation as it tends to happen in systems, and the trends seem
> to
> >> suggest it happens because of the culture of way things are organized.
> This
> >> is called Conway’s Law, and I have an intuition some of that may be at
> play
> >> with this topic:
> >> https://en.wikipedia.org/wiki/Conway%27s_law <https://en.wikipedia.org/
> >> wiki/Conway's_law>
> >>
> >> Wade
> >
> >
> >
> >
> > --
> > Niclas Hedhman, Software Developer
> > http://zest.apache.org - New Energy for Java
>
>
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Mark Struberg <st...@yahoo.de.INVALID>.
The gitflow model is imo only worth it if you have dedicated build masters who really are the ‚masters‘ of what goes into a release or not.
This works fine for company projects where you have seniors waving through the work of juniors. But at the end those build-masters need to have a good knowledge about the single features.
It’s kind of the Linux model, but all in the same repo instead of pulling from other repos and applying patches.
Do we have such a build-master at the ASF? Usually we do not! All committers usually know the code very well. Of course NetBeans is a bit different and of course a few people like Geertjan and Jan do know MUCH more about what is good for a stable code base and what rather needs some more tinkering.
In a long term I guess it is one of the goals to broaden the pool of people who have a good overlook over the project.
For contributors who are rather new to such a huge code base, I think it might be good to work with external feature branches and rebase instead.
just my .02
LieGrue,
strub
> Am 15.10.2016 um 01:05 schrieb Niclas Hedhman <ni...@hedhman.org>:
>
> http://nvie.com/posts/a-successful-git-branching-model/
>
> On Fri, Oct 14, 2016 at 10:46 PM, Wade Chandler <co...@wadechandler.com>
> wrote:
>
>> On Oct 14, 2016, at 08:53, Emilian Bold <em...@gmail.com> wrote:
>>>
>>> This incubation is just the beginning of NetBeans under Apache. We don't
>>> have to reinvent ourselves at once.
>>>
>>
>> I agree with that.
>>
>>> Let's first see how people start contributing and if there is friction
>> due
>>> to different work approaches, I'm sure some new consensus will be
>> reached.
>>>
>>
>> I think that is feasible as long as everyone agrees if we need to shift we
>> will do it quickly, and we give it some thought ahead of time so it isn’t
>> such a big deal if/when it happens.
>>
>>> Like I've mentioned before, I don't believe the repository size is an
>>> issue. It's a big project with a big history. Even if I could checkout
>> just
>>> the file I want to patch, in order to build and test that I would still
>>> have to download the NetBeans binaries and the smallest download is
>> 110MB.
>>>
>>
>> But not exactly apples to apples…even without history you are talking
>> about 110MB to 1GB which is nearly a 1000% difference.
>>
>>> Friend dependencies are also not a problem of the source code structure,
>>> it's a matter of API stability <http://wiki.netbeans.org/API_Stability>.
>>> It's easy to mark an API as Stable/Official and then the plugins problem
>> is
>>> solved. But then somebody has to respect this contract and support that
>> API
>>> throughout releases, which is not an easy task.
>>>
>>
>> To me we are talking 2 different contexts though; supporting the framework
>> for the larger community versus a narrowed scope of who has to manage
>> something once it goes into the wild. Too, I think there are plenty of
>> examples of folks not being able to do something because of friend deps
>> without becoming a friend, and that seems to suggest it was easier to just
>> “get there” and stop without going further; resource contentions etc. Sure,
>> that same thing could happen even across repositories, and would need to be
>> left that way for some time, but I think definitely needs to be part of the
>> process to address, and to me is some times related to separation of
>> development; monolith mindset versus modular even if modular the mindset
>> can set in with a gigantic blob of code all bound by the build system and
>> repository. I’m not saying it is directly related, but I think there is
>> good correlation as it tends to happen in systems, and the trends seem to
>> suggest it happens because of the culture of way things are organized. This
>> is called Conway’s Law, and I have an intuition some of that may be at play
>> with this topic:
>> https://en.wikipedia.org/wiki/Conway%27s_law <https://en.wikipedia.org/
>> wiki/Conway's_law>
>>
>> Wade
>
>
>
>
> --
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Niclas Hedhman <ni...@hedhman.org>.
http://nvie.com/posts/a-successful-git-branching-model/
On Fri, Oct 14, 2016 at 10:46 PM, Wade Chandler <co...@wadechandler.com>
wrote:
> On Oct 14, 2016, at 08:53, Emilian Bold <em...@gmail.com> wrote:
> >
> > This incubation is just the beginning of NetBeans under Apache. We don't
> > have to reinvent ourselves at once.
> >
>
> I agree with that.
>
> > Let's first see how people start contributing and if there is friction
> due
> > to different work approaches, I'm sure some new consensus will be
> reached.
> >
>
> I think that is feasible as long as everyone agrees if we need to shift we
> will do it quickly, and we give it some thought ahead of time so it isn’t
> such a big deal if/when it happens.
>
> > Like I've mentioned before, I don't believe the repository size is an
> > issue. It's a big project with a big history. Even if I could checkout
> just
> > the file I want to patch, in order to build and test that I would still
> > have to download the NetBeans binaries and the smallest download is
> 110MB.
> >
>
> But not exactly apples to apples…even without history you are talking
> about 110MB to 1GB which is nearly a 1000% difference.
>
> > Friend dependencies are also not a problem of the source code structure,
> > it's a matter of API stability <http://wiki.netbeans.org/API_Stability>.
> > It's easy to mark an API as Stable/Official and then the plugins problem
> is
> > solved. But then somebody has to respect this contract and support that
> API
> > throughout releases, which is not an easy task.
> >
>
> To me we are talking 2 different contexts though; supporting the framework
> for the larger community versus a narrowed scope of who has to manage
> something once it goes into the wild. Too, I think there are plenty of
> examples of folks not being able to do something because of friend deps
> without becoming a friend, and that seems to suggest it was easier to just
> “get there” and stop without going further; resource contentions etc. Sure,
> that same thing could happen even across repositories, and would need to be
> left that way for some time, but I think definitely needs to be part of the
> process to address, and to me is some times related to separation of
> development; monolith mindset versus modular even if modular the mindset
> can set in with a gigantic blob of code all bound by the build system and
> repository. I’m not saying it is directly related, but I think there is
> good correlation as it tends to happen in systems, and the trends seem to
> suggest it happens because of the culture of way things are organized. This
> is called Conway’s Law, and I have an intuition some of that may be at play
> with this topic:
> https://en.wikipedia.org/wiki/Conway%27s_law <https://en.wikipedia.org/
> wiki/Conway's_law>
>
> Wade
--
Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Wade Chandler <co...@wadechandler.com>.
On Oct 14, 2016, at 08:53, Emilian Bold <em...@gmail.com> wrote:
>
> This incubation is just the beginning of NetBeans under Apache. We don't
> have to reinvent ourselves at once.
>
I agree with that.
> Let's first see how people start contributing and if there is friction due
> to different work approaches, I'm sure some new consensus will be reached.
>
I think that is feasible as long as everyone agrees if we need to shift we will do it quickly, and we give it some thought ahead of time so it isn’t such a big deal if/when it happens.
> Like I've mentioned before, I don't believe the repository size is an
> issue. It's a big project with a big history. Even if I could checkout just
> the file I want to patch, in order to build and test that I would still
> have to download the NetBeans binaries and the smallest download is 110MB.
>
But not exactly apples to apples…even without history you are talking about 110MB to 1GB which is nearly a 1000% difference.
> Friend dependencies are also not a problem of the source code structure,
> it's a matter of API stability <http://wiki.netbeans.org/API_Stability>.
> It's easy to mark an API as Stable/Official and then the plugins problem is
> solved. But then somebody has to respect this contract and support that API
> throughout releases, which is not an easy task.
>
To me we are talking 2 different contexts though; supporting the framework for the larger community versus a narrowed scope of who has to manage something once it goes into the wild. Too, I think there are plenty of examples of folks not being able to do something because of friend deps without becoming a friend, and that seems to suggest it was easier to just “get there” and stop without going further; resource contentions etc. Sure, that same thing could happen even across repositories, and would need to be left that way for some time, but I think definitely needs to be part of the process to address, and to me is some times related to separation of development; monolith mindset versus modular even if modular the mindset can set in with a gigantic blob of code all bound by the build system and repository. I’m not saying it is directly related, but I think there is good correlation as it tends to happen in systems, and the trends seem to suggest it happens because of the culture of way things are organized. This is called Conway’s Law, and I have an intuition some of that may be at play with this topic:
https://en.wikipedia.org/wiki/Conway%27s_law <https://en.wikipedia.org/wiki/Conway's_law>
Wade
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Emilian Bold <em...@gmail.com>.
This incubation is just the beginning of NetBeans under Apache. We don't
have to reinvent ourselves at once.
> But, if we bring everything in, just as a dump, and then Oracle folks
continue to work as they had previously, without the bigger context, and
the community tries to work around that, then IMO it won't be as successful.
You have a point here, but if NetBeans developers used team repositories /
branches until now who are we to tell them to stop? It's not inherently a
bad way to work.
Let's first see how people start contributing and if there is friction due
to different work approaches, I'm sure some new consensus will be reached.
Like I've mentioned before, I don't believe the repository size is an
issue. It's a big project with a big history. Even if I could checkout just
the file I want to patch, in order to build and test that I would still
have to download the NetBeans binaries and the smallest download is 110MB.
Friend dependencies are also not a problem of the source code structure,
it's a matter of API stability <http://wiki.netbeans.org/API_Stability>.
It's easy to mark an API as Stable/Official and then the plugins problem is
solved. But then somebody has to respect this contract and support that API
throughout releases, which is not an easy task.
--emi
On Fri, Oct 14, 2016 at 3:28 PM, Wade Chandler <co...@wadechandler.com>
wrote:
> On Oct 14, 2016 6:47 AM, "Emilian Bold" <em...@gmail.com> wrote:
> >
> > When we spoke about "teams" we spoke in the context of how NetBeans
> > developer from Oracle have been working on NetBeans.
> >
> > If you look at http://hg.netbeans.org/ there are multiple repositories
> with
> > the title "xxx team repository". So the profiler guys would commit their
> > (stil-not-finished) stuff in their own branch (which is similar in
> concept
> > to a feature branch) and only later on will their changes be pushed to
> > everybody else. This reduces the chance of a random commit breaking a
> build
> > for everybody.
> >
> > So, by using the team branches it will be easier to continue development.
> >
>
> What about integrating early and often? Especially in the context of a
> single monolith repository such things are dangerous, and in the context of
> a broader community. If a team is making changes to many modules, and some
> other group from the community another, then that becomes problematic. Once
> they are done, then integration is going to be a nightmare if there is more
> isolation. If those things are for just the specifics of their modules,
> then a feature branch works perfectly. Maybe someone is solidifying a
> concept, OK, so a single new branch lives a little longer, but not a life
> time. I definitely think the Oracle teams plus the community needs to give
> this some thought. It is a different context, and external communications
> with others is paramount, and how everyone views integration helps a lot
> with that communication.
>
> > There is nothing stopping new or existing contributors to use feature
> > branches.
> >
>
> I think (IMO) everyone should per reasons stated above. Why not? It should
> be rare, outside of development/main and master/main-golden, for there to
> be long lived branches which will include changes which may break a lot of
> people. That would be difficult to rectify without some extra thought to
> how these problems arise ahead of time.
>
> > Do note that the purpose of this migration is just to be able to start
> > working under Apache. It is not about imposing a technical way of doing
> > software. Every community is allowed to develop software how they see
> fit.
> >
>
> Perhaps you'd have to be more clear on that. Apache is a community as well
> as each sub-community inside of it. Then there is the bigger notion of all
> us who contribute to one.
>
> Within this one which we are contributing, we have to be able to check out
> code, and too, have some notions of how different companies and
> organizations will tread and not step on each other's toes. I imagine if
> the processes don't promote more positive ways to keep things smooth, the
> community will get irritable. Even in smaller contexts that can be painful
> without a plan, and this is the community where we are talking about
> developing software, and where we should be choosing how we see fit to
> develop software; this is the context unless I miss your point.
>
> Too, there is getting the code here and working on it, and then precedents
> we set with how we will work. I think those things are good to talk about
> ahead of time.
>
> If we bring in the whole repo as a single repo now, and have a plan to
> split it, or at least talk through some of the issues many see with the way
> things are done now, and how to address them, see Sven and Julien's
> comments as others examples, then that sounds fine. But, if we bring
> everything in, just as a dump, and then Oracle folks continue to work as
> they had previously, without the bigger context, and the community tries to
> work around that, then IMO it won't be as successful. This is a new
> context.
>
> There were other issues with contributions in the old model, and I think we
> have a good opportunity to look at those. One of them was the repo size IMO
> which is over 1GB even without history. I think another is an over reliance
> on friend dependencies which have made it harder to do more things with
> plugins without modifying the core, or some other big central thing, which
> inevitably meant fixing an issue, and getting to use the fix in the
> released software took a long time in many cases.
>
> Wade
>
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Wade Chandler <co...@wadechandler.com>.
On Oct 14, 2016 6:47 AM, "Emilian Bold" <em...@gmail.com> wrote:
>
> When we spoke about "teams" we spoke in the context of how NetBeans
> developer from Oracle have been working on NetBeans.
>
> If you look at http://hg.netbeans.org/ there are multiple repositories
with
> the title "xxx team repository". So the profiler guys would commit their
> (stil-not-finished) stuff in their own branch (which is similar in concept
> to a feature branch) and only later on will their changes be pushed to
> everybody else. This reduces the chance of a random commit breaking a
build
> for everybody.
>
> So, by using the team branches it will be easier to continue development.
>
What about integrating early and often? Especially in the context of a
single monolith repository such things are dangerous, and in the context of
a broader community. If a team is making changes to many modules, and some
other group from the community another, then that becomes problematic. Once
they are done, then integration is going to be a nightmare if there is more
isolation. If those things are for just the specifics of their modules,
then a feature branch works perfectly. Maybe someone is solidifying a
concept, OK, so a single new branch lives a little longer, but not a life
time. I definitely think the Oracle teams plus the community needs to give
this some thought. It is a different context, and external communications
with others is paramount, and how everyone views integration helps a lot
with that communication.
> There is nothing stopping new or existing contributors to use feature
> branches.
>
I think (IMO) everyone should per reasons stated above. Why not? It should
be rare, outside of development/main and master/main-golden, for there to
be long lived branches which will include changes which may break a lot of
people. That would be difficult to rectify without some extra thought to
how these problems arise ahead of time.
> Do note that the purpose of this migration is just to be able to start
> working under Apache. It is not about imposing a technical way of doing
> software. Every community is allowed to develop software how they see fit.
>
Perhaps you'd have to be more clear on that. Apache is a community as well
as each sub-community inside of it. Then there is the bigger notion of all
us who contribute to one.
Within this one which we are contributing, we have to be able to check out
code, and too, have some notions of how different companies and
organizations will tread and not step on each other's toes. I imagine if
the processes don't promote more positive ways to keep things smooth, the
community will get irritable. Even in smaller contexts that can be painful
without a plan, and this is the community where we are talking about
developing software, and where we should be choosing how we see fit to
develop software; this is the context unless I miss your point.
Too, there is getting the code here and working on it, and then precedents
we set with how we will work. I think those things are good to talk about
ahead of time.
If we bring in the whole repo as a single repo now, and have a plan to
split it, or at least talk through some of the issues many see with the way
things are done now, and how to address them, see Sven and Julien's
comments as others examples, then that sounds fine. But, if we bring
everything in, just as a dump, and then Oracle folks continue to work as
they had previously, without the bigger context, and the community tries to
work around that, then IMO it won't be as successful. This is a new context.
There were other issues with contributions in the old model, and I think we
have a good opportunity to look at those. One of them was the repo size IMO
which is over 1GB even without history. I think another is an over reliance
on friend dependencies which have made it harder to do more things with
plugins without modifying the core, or some other big central thing, which
inevitably meant fixing an issue, and getting to use the fix in the
released software took a long time in many cases.
Wade
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Emilian Bold <em...@gmail.com>.
When we spoke about "teams" we spoke in the context of how NetBeans
developer from Oracle have been working on NetBeans.
If you look at http://hg.netbeans.org/ there are multiple repositories with
the title "xxx team repository". So the profiler guys would commit their
(stil-not-finished) stuff in their own branch (which is similar in concept
to a feature branch) and only later on will their changes be pushed to
everybody else. This reduces the chance of a random commit breaking a build
for everybody.
So, by using the team branches it will be easier to continue development.
There is nothing stopping new or existing contributors to use feature
branches.
Do note that the purpose of this migration is just to be able to start
working under Apache. It is not about imposing a technical way of doing
software. Every community is allowed to develop software how they see fit.
--emi
On Fri, Oct 14, 2016 at 1:19 PM, Julien Enselme <je...@jujens.eu> wrote:
> +1 for feature/issue branches that shouldn't live long and master/devel
> for long term stuff. As far as I know, this is how we are supposed to
> use git.
>
> On Fri, 2016-10-14 at 12:38 +0300, Vladimir Voskresensky wrote:
> >
> > In this context one repo has advantage as well, otherwise
> > feature/issue
> > branches have to be created in multiple repos.
>
> If an issue touches a huge number of different files, one repo makes it
> easier to solve. But as already said, from a contributor's point of
> view, it is easier to deal with smaller repos (fast to clone and don't
> take a lot of disk space). It also seems to be "the git way".
>
> >
> > Thanks,
> > Vladimir.
> >
> > On 13.10.2016 15:18, Bruno Flávio wrote:
> > >
> > > +1 for feature/issue branches.
> > >
> > > Besides being easy to understand (and pretty standard procedure
> > > IMO) it
> > > would also improve the work flow and collaboration between
> > > developers,
> > > reporters.
> > >
> > > Right now I produce a patch and attach it to the issue I'm working
> > > on
> > > prior to it's integration in the repo so that those watching the
> > > issue
> > > can have a say before it's committed. This is a bit cumbersome and
> > > would
> > > be nicely solved by short lived feature branches.
> > >
> > > Regards,
> > >
> > > -Bruno
> > >
> > > On 13/10/16 12:40, Wade Chandler wrote:
> > > >
> > > > Why not feature branches for individual issues? Why long lived
> > > > “team” branches? I think feature branches work for the entire
> > > > community, and they address a particular problem which should be
> > > > fixed, and not a long list of them which is more likely to cause
> > > > late integration; i.e. changes are not just for cnd or groovy,
> > > > but to something more common.
> > > >
> > > > i.e. (without a separate silver)
> > > > master (what used to be main-golden)
> > > > --development (what used to be main and silver (integration and
> > > > main build))
> > > > ----cnd/feature-abc (a cnd specific feature branch)
> > > > ----groovy/feature-abc (a groovy specific feature branch)
> > > > ------wadechandler/bug-or-feature-abc (a specific fix I want to
> > > > merge up to this feature branch with others)
> > > > ----wadechandler/bug-xyz (a specific fix for a bug submitted by
> > > > me to merge to development)
> > > >
> > > > The feature branches then get merged back to development, and
> > > > hopefully those are relatively small and fast lived branches.
> > > > Everyone integrates on development sooner rather than later. If
> > > > one needs to make modifications to something in lookup or some
> > > > other shared thing, they would use a specific branch for that to
> > > > make sure that is seen separately and faster other than at some
> > > > other long lived branch which causes delay. Development goes to
> > > > master when “stable”.
> > > >
> > > > If a group needs a longer lived branch for some reason, they can
> > > > do that here, but the point is to better think about early
> > > > integration as well as the reality these are no longer products
> > > > managed by Oracle, but the entire community, so early integration
> > > > is going to be more important.
> > > >
> > > > If broken out to separate clusters, which I think better supports
> > > > the community working on smaller subsets, and am in favor of
> > > > that, then a similar model would exist. But, it would be in each
> > > > repository.
> > > >
> > > > Thanks,
> > > >
> > > > Wade
> > > >
> > > > ===================
> > > >
> > > > Wade Chandler
> > > > e: consult@wadechandler.com
> > > >
> > > >
> > > >
> > > > >
> > > > > On Oct 12, 2016, at 14:30, Emilian Bold <emilian.bold@gmail.com
> > > > > > wrote:
> > > > >
> > > > > Yes, I think it makes sense to use branches for 'teams'. Maybe
> > > > > even for
> > > > > main-silver, main-golden.
> > > > >
> > > > >
> > > > > --emi
> > > > >
> > > > > On Wed, Oct 12, 2016 at 9:23 PM, Vladimir Voskresensky <
> > > > > vladimir.voskresensky@oracle.com> wrote:
> > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > On 12.10.2016 18:23, Eric Barboni wrote:
> > > > > >
> > > > > > >
> > > > > > > With the split in cluster it may possible to get rid
> > > > > > > of the "team"
> > > > > > > repositories (i.e. "core-main","cnd-main","jet-main") but
> > > > > > > we may need
> > > > > > > another repository ( kind of current "main-silver" or
> > > > > > > "main-golden") to
> > > > > > > prevent a bad commit leading to compile error in one
> > > > > > > cluster to be
> > > > > > > propagated to the full NetBeans break the build for others
> > > > > > > who did not need
> > > > > > > this particular cluster
> > > > > > >
> > > > > > What if we leave one repository as it is now and move 'teams'
> > > > > > into git
> > > > > > branches (core-main/cnd-main/...).
> > > > > > Teams will work on branches.
> > > > > > Then like now [1] the current push-cnd-main job can be
> > > > > > replaced by
> > > > > > automatic 'push cnd-main into main-silver branch', while
> > > > > > push-to-cnd-main
> > > > > > job will be automatic 'push main-silver into cnd-main'.
> > > > > > Branch main-silver is built and tested as now and on success
> > > > > > main-silver
> > > > > > is pushed into main-golden for the production build.
> > > > > >
> > > > > > Thanks,
> > > > > > Vladimir.
> > > > > > [1] http://wiki.netbeans.org/HgParallelProjectIntegration
> > > > > >
> > >
> >
> --
> Julien Enselme
> http://www.jujens.eu/
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Julien Enselme <je...@jujens.eu>.
+1 for feature/issue branches that shouldn't live long and master/devel
for long term stuff. As far as I know, this is how we are supposed to
use git.
On Fri, 2016-10-14 at 12:38 +0300, Vladimir Voskresensky wrote:
>
> In this context one repo has advantage as well, otherwise
> feature/issue
> branches have to be created in multiple repos.
If an issue touches a huge number of different files, one repo makes it
easier to solve. But as already said, from a contributor's point of
view, it is easier to deal with smaller repos (fast to clone and don't
take a lot of disk space). It also seems to be "the git way".
>
> Thanks,
> Vladimir.
>
> On 13.10.2016 15:18, Bruno Flávio wrote:
> >
> > +1 for feature/issue branches.
> >
> > Besides being easy to understand (and pretty standard procedure
> > IMO) it
> > would also improve the work flow and collaboration between
> > developers,
> > reporters.
> >
> > Right now I produce a patch and attach it to the issue I'm working
> > on
> > prior to it's integration in the repo so that those watching the
> > issue
> > can have a say before it's committed. This is a bit cumbersome and
> > would
> > be nicely solved by short lived feature branches.
> >
> > Regards,
> >
> > -Bruno
> >
> > On 13/10/16 12:40, Wade Chandler wrote:
> > >
> > > Why not feature branches for individual issues? Why long lived
> > > “team” branches? I think feature branches work for the entire
> > > community, and they address a particular problem which should be
> > > fixed, and not a long list of them which is more likely to cause
> > > late integration; i.e. changes are not just for cnd or groovy,
> > > but to something more common.
> > >
> > > i.e. (without a separate silver)
> > > master (what used to be main-golden)
> > > --development (what used to be main and silver (integration and
> > > main build))
> > > ----cnd/feature-abc (a cnd specific feature branch)
> > > ----groovy/feature-abc (a groovy specific feature branch)
> > > ------wadechandler/bug-or-feature-abc (a specific fix I want to
> > > merge up to this feature branch with others)
> > > ----wadechandler/bug-xyz (a specific fix for a bug submitted by
> > > me to merge to development)
> > >
> > > The feature branches then get merged back to development, and
> > > hopefully those are relatively small and fast lived branches.
> > > Everyone integrates on development sooner rather than later. If
> > > one needs to make modifications to something in lookup or some
> > > other shared thing, they would use a specific branch for that to
> > > make sure that is seen separately and faster other than at some
> > > other long lived branch which causes delay. Development goes to
> > > master when “stable”.
> > >
> > > If a group needs a longer lived branch for some reason, they can
> > > do that here, but the point is to better think about early
> > > integration as well as the reality these are no longer products
> > > managed by Oracle, but the entire community, so early integration
> > > is going to be more important.
> > >
> > > If broken out to separate clusters, which I think better supports
> > > the community working on smaller subsets, and am in favor of
> > > that, then a similar model would exist. But, it would be in each
> > > repository.
> > >
> > > Thanks,
> > >
> > > Wade
> > >
> > > ===================
> > >
> > > Wade Chandler
> > > e: consult@wadechandler.com
> > >
> > >
> > >
> > > >
> > > > On Oct 12, 2016, at 14:30, Emilian Bold <emilian.bold@gmail.com
> > > > > wrote:
> > > >
> > > > Yes, I think it makes sense to use branches for 'teams'. Maybe
> > > > even for
> > > > main-silver, main-golden.
> > > >
> > > >
> > > > --emi
> > > >
> > > > On Wed, Oct 12, 2016 at 9:23 PM, Vladimir Voskresensky <
> > > > vladimir.voskresensky@oracle.com> wrote:
> > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > On 12.10.2016 18:23, Eric Barboni wrote:
> > > > >
> > > > > >
> > > > > > With the split in cluster it may possible to get rid
> > > > > > of the "team"
> > > > > > repositories (i.e. "core-main","cnd-main","jet-main") but
> > > > > > we may need
> > > > > > another repository ( kind of current "main-silver" or
> > > > > > "main-golden") to
> > > > > > prevent a bad commit leading to compile error in one
> > > > > > cluster to be
> > > > > > propagated to the full NetBeans break the build for others
> > > > > > who did not need
> > > > > > this particular cluster
> > > > > >
> > > > > What if we leave one repository as it is now and move 'teams'
> > > > > into git
> > > > > branches (core-main/cnd-main/...).
> > > > > Teams will work on branches.
> > > > > Then like now [1] the current push-cnd-main job can be
> > > > > replaced by
> > > > > automatic 'push cnd-main into main-silver branch', while
> > > > > push-to-cnd-main
> > > > > job will be automatic 'push main-silver into cnd-main'.
> > > > > Branch main-silver is built and tested as now and on success
> > > > > main-silver
> > > > > is pushed into main-golden for the production build.
> > > > >
> > > > > Thanks,
> > > > > Vladimir.
> > > > > [1] http://wiki.netbeans.org/HgParallelProjectIntegration
> > > > >
> >
>
--
Julien Enselme
http://www.jujens.eu/
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Wade Chandler <co...@wadechandler.com>.
On Oct 14, 2016 5:38 AM, "Vladimir Voskresensky" <
vladimir.voskresensky@oracle.com> wrote:
>.
>
> In this context one repo has advantage as well, otherwise feature/issue
branches have to be created in multiple repos.
>
I don't see it a bad thing to have to create multiple branches across
repos. We do that, and we name them the same thing. In almost all cases
that will be fairly minimal, like 2 repos, some or many, 1 repo, and then
every now and then a few, with all being some very massive refactor of
something impacting every module, but that being an extreme rarity. That
all sounds manageable to me. Say groovy support needs a new feature, then
perhaps Java, Core, and Groovy get a branch. As Groovy support gets better
cleaned up, along with Java and others, to have less friend and awkward
dependencies, then fewer branches will be needed most of the time.
The devs working on it before then need to merge Core, then Java, then
Groovy. To me it forces everyone to better think out changes they make at a
modular level, and keeps the repos much smaller. Say it is a simple fix or
addition to Groovy, or the modularity of things is fixed, then only Groovy
would get the branch.
Keeping repos as small as possible also makes them work with other tools,
like CLI status tools, quite faster. I have some bash tools setup in my
bashrc, and NB is the only repo out of all OSS and commercial sources I
have which causes a pause after every action I take on the CLI as it
inquires about the hg repo status, and that includes Tomcat, the entire
Spring Framework, and Qt on a beast of a machine with a blazing SSD. It is
also the only repo I work with which times out from time to time. So, I
think there are various reasons massive repositories are not practical.
I do see your point on some things being easier in a single repository
however. I suggest we start a section on the wiki page with pros and cons
of both approaches, then we drill down through them, and we mark off or
annotate the ones which sound good but have negatives or are not good/easy
within the new infrastructure. Either way, we may find a single repository
improbable or unlikely. Some comments on infra, plus GitHub limits, already
suggest it is unlikely to be feasible.
https://help.github.com/articles/what-is-my-disk-quota/
I have written them. I have not yet responded to their response which
didn't say one couldn't get a bigger repo sign off, but did ask us to trim
out as much as possible as well as old history to see what the size then
looks like. I haven't played with that yet, so I don't have enough info to
follow up.
Wade
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Vladimir Voskresensky <vl...@oracle.com>.
+1 for feature/issue branches style as well, btw it's a part of git
philosophy.
It could simplify API/patch reviews as well.
In this context one repo has advantage as well, otherwise feature/issue
branches have to be created in multiple repos.
Thanks,
Vladimir.
On 13.10.2016 15:18, Bruno Fl�vio wrote:
> +1 for feature/issue branches.
>
> Besides being easy to understand (and pretty standard procedure IMO) it
> would also improve the work flow and collaboration between developers,
> reporters.
>
> Right now I produce a patch and attach it to the issue I'm working on
> prior to it's integration in the repo so that those watching the issue
> can have a say before it's committed. This is a bit cumbersome and would
> be nicely solved by short lived feature branches.
>
> Regards,
>
> -Bruno
>
> On 13/10/16 12:40, Wade Chandler wrote:
>> Why not feature branches for individual issues? Why long lived \u201cteam\u201d branches? I think feature branches work for the entire community, and they address a particular problem which should be fixed, and not a long list of them which is more likely to cause late integration; i.e. changes are not just for cnd or groovy, but to something more common.
>>
>> i.e. (without a separate silver)
>> master (what used to be main-golden)
>> --development (what used to be main and silver (integration and main build))
>> ----cnd/feature-abc (a cnd specific feature branch)
>> ----groovy/feature-abc (a groovy specific feature branch)
>> ------wadechandler/bug-or-feature-abc (a specific fix I want to merge up to this feature branch with others)
>> ----wadechandler/bug-xyz (a specific fix for a bug submitted by me to merge to development)
>>
>> The feature branches then get merged back to development, and hopefully those are relatively small and fast lived branches. Everyone integrates on development sooner rather than later. If one needs to make modifications to something in lookup or some other shared thing, they would use a specific branch for that to make sure that is seen separately and faster other than at some other long lived branch which causes delay. Development goes to master when \u201cstable\u201d.
>>
>> If a group needs a longer lived branch for some reason, they can do that here, but the point is to better think about early integration as well as the reality these are no longer products managed by Oracle, but the entire community, so early integration is going to be more important.
>>
>> If broken out to separate clusters, which I think better supports the community working on smaller subsets, and am in favor of that, then a similar model would exist. But, it would be in each repository.
>>
>> Thanks,
>>
>> Wade
>>
>> ===================
>>
>> Wade Chandler
>> e: consult@wadechandler.com
>>
>>
>>
>>> On Oct 12, 2016, at 14:30, Emilian Bold <em...@gmail.com> wrote:
>>>
>>> Yes, I think it makes sense to use branches for 'teams'. Maybe even for
>>> main-silver, main-golden.
>>>
>>>
>>> --emi
>>>
>>> On Wed, Oct 12, 2016 at 9:23 PM, Vladimir Voskresensky <
>>> vladimir.voskresensky@oracle.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> On 12.10.2016 18:23, Eric Barboni wrote:
>>>>
>>>>> With the split in cluster it may possible to get rid of the "team"
>>>>> repositories (i.e. "core-main","cnd-main","jet-main") but we may need
>>>>> another repository ( kind of current "main-silver" or "main-golden") to
>>>>> prevent a bad commit leading to compile error in one cluster to be
>>>>> propagated to the full NetBeans break the build for others who did not need
>>>>> this particular cluster
>>>>>
>>>> What if we leave one repository as it is now and move 'teams' into git
>>>> branches (core-main/cnd-main/...).
>>>> Teams will work on branches.
>>>> Then like now [1] the current push-cnd-main job can be replaced by
>>>> automatic 'push cnd-main into main-silver branch', while push-to-cnd-main
>>>> job will be automatic 'push main-silver into cnd-main'.
>>>> Branch main-silver is built and tested as now and on success main-silver
>>>> is pushed into main-golden for the production build.
>>>>
>>>> Thanks,
>>>> Vladimir.
>>>> [1] http://wiki.netbeans.org/HgParallelProjectIntegration
>>>>
>
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Bruno Flávio <br...@apache.org>.
+1 for feature/issue branches.
Besides being easy to understand (and pretty standard procedure IMO) it
would also improve the work flow and collaboration between developers,
reporters.
Right now I produce a patch and attach it to the issue I'm working on
prior to it's integration in the repo so that those watching the issue
can have a say before it's committed. This is a bit cumbersome and would
be nicely solved by short lived feature branches.
Regards,
-Bruno
On 13/10/16 12:40, Wade Chandler wrote:
> Why not feature branches for individual issues? Why long lived “team” branches? I think feature branches work for the entire community, and they address a particular problem which should be fixed, and not a long list of them which is more likely to cause late integration; i.e. changes are not just for cnd or groovy, but to something more common.
>
> i.e. (without a separate silver)
> master (what used to be main-golden)
> --development (what used to be main and silver (integration and main build))
> ----cnd/feature-abc (a cnd specific feature branch)
> ----groovy/feature-abc (a groovy specific feature branch)
> ------wadechandler/bug-or-feature-abc (a specific fix I want to merge up to this feature branch with others)
> ----wadechandler/bug-xyz (a specific fix for a bug submitted by me to merge to development)
>
> The feature branches then get merged back to development, and hopefully those are relatively small and fast lived branches. Everyone integrates on development sooner rather than later. If one needs to make modifications to something in lookup or some other shared thing, they would use a specific branch for that to make sure that is seen separately and faster other than at some other long lived branch which causes delay. Development goes to master when “stable”.
>
> If a group needs a longer lived branch for some reason, they can do that here, but the point is to better think about early integration as well as the reality these are no longer products managed by Oracle, but the entire community, so early integration is going to be more important.
>
> If broken out to separate clusters, which I think better supports the community working on smaller subsets, and am in favor of that, then a similar model would exist. But, it would be in each repository.
>
> Thanks,
>
> Wade
>
> ===================
>
> Wade Chandler
> e: consult@wadechandler.com
>
>
>
>> On Oct 12, 2016, at 14:30, Emilian Bold <em...@gmail.com> wrote:
>>
>> Yes, I think it makes sense to use branches for 'teams'. Maybe even for
>> main-silver, main-golden.
>>
>>
>> --emi
>>
>> On Wed, Oct 12, 2016 at 9:23 PM, Vladimir Voskresensky <
>> vladimir.voskresensky@oracle.com> wrote:
>>
>>> Hi,
>>>
>>> On 12.10.2016 18:23, Eric Barboni wrote:
>>>
>>>> With the split in cluster it may possible to get rid of the "team"
>>>> repositories (i.e. "core-main","cnd-main","jet-main") but we may need
>>>> another repository ( kind of current "main-silver" or "main-golden") to
>>>> prevent a bad commit leading to compile error in one cluster to be
>>>> propagated to the full NetBeans break the build for others who did not need
>>>> this particular cluster
>>>>
>>> What if we leave one repository as it is now and move 'teams' into git
>>> branches (core-main/cnd-main/...).
>>> Teams will work on branches.
>>> Then like now [1] the current push-cnd-main job can be replaced by
>>> automatic 'push cnd-main into main-silver branch', while push-to-cnd-main
>>> job will be automatic 'push main-silver into cnd-main'.
>>> Branch main-silver is built and tested as now and on success main-silver
>>> is pushed into main-golden for the production build.
>>>
>>> Thanks,
>>> Vladimir.
>>> [1] http://wiki.netbeans.org/HgParallelProjectIntegration
>>>
>
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Wade Chandler <co...@wadechandler.com>.
Why not feature branches for individual issues? Why long lived “team” branches? I think feature branches work for the entire community, and they address a particular problem which should be fixed, and not a long list of them which is more likely to cause late integration; i.e. changes are not just for cnd or groovy, but to something more common.
i.e. (without a separate silver)
master (what used to be main-golden)
--development (what used to be main and silver (integration and main build))
----cnd/feature-abc (a cnd specific feature branch)
----groovy/feature-abc (a groovy specific feature branch)
------wadechandler/bug-or-feature-abc (a specific fix I want to merge up to this feature branch with others)
----wadechandler/bug-xyz (a specific fix for a bug submitted by me to merge to development)
The feature branches then get merged back to development, and hopefully those are relatively small and fast lived branches. Everyone integrates on development sooner rather than later. If one needs to make modifications to something in lookup or some other shared thing, they would use a specific branch for that to make sure that is seen separately and faster other than at some other long lived branch which causes delay. Development goes to master when “stable”.
If a group needs a longer lived branch for some reason, they can do that here, but the point is to better think about early integration as well as the reality these are no longer products managed by Oracle, but the entire community, so early integration is going to be more important.
If broken out to separate clusters, which I think better supports the community working on smaller subsets, and am in favor of that, then a similar model would exist. But, it would be in each repository.
Thanks,
Wade
===================
Wade Chandler
e: consult@wadechandler.com
> On Oct 12, 2016, at 14:30, Emilian Bold <em...@gmail.com> wrote:
>
> Yes, I think it makes sense to use branches for 'teams'. Maybe even for
> main-silver, main-golden.
>
>
> --emi
>
> On Wed, Oct 12, 2016 at 9:23 PM, Vladimir Voskresensky <
> vladimir.voskresensky@oracle.com> wrote:
>
>> Hi,
>>
>> On 12.10.2016 18:23, Eric Barboni wrote:
>>
>>> With the split in cluster it may possible to get rid of the "team"
>>> repositories (i.e. "core-main","cnd-main","jet-main") but we may need
>>> another repository ( kind of current "main-silver" or "main-golden") to
>>> prevent a bad commit leading to compile error in one cluster to be
>>> propagated to the full NetBeans break the build for others who did not need
>>> this particular cluster
>>>
>> What if we leave one repository as it is now and move 'teams' into git
>> branches (core-main/cnd-main/...).
>> Teams will work on branches.
>> Then like now [1] the current push-cnd-main job can be replaced by
>> automatic 'push cnd-main into main-silver branch', while push-to-cnd-main
>> job will be automatic 'push main-silver into cnd-main'.
>> Branch main-silver is built and tested as now and on success main-silver
>> is pushed into main-golden for the production build.
>>
>> Thanks,
>> Vladimir.
>> [1] http://wiki.netbeans.org/HgParallelProjectIntegration
>>
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Emilian Bold <em...@gmail.com>.
Yes, I think it makes sense to use branches for 'teams'. Maybe even for
main-silver, main-golden.
--emi
On Wed, Oct 12, 2016 at 9:23 PM, Vladimir Voskresensky <
vladimir.voskresensky@oracle.com> wrote:
> Hi,
>
> On 12.10.2016 18:23, Eric Barboni wrote:
>
>> With the split in cluster it may possible to get rid of the "team"
>> repositories (i.e. "core-main","cnd-main","jet-main") but we may need
>> another repository ( kind of current "main-silver" or "main-golden") to
>> prevent a bad commit leading to compile error in one cluster to be
>> propagated to the full NetBeans break the build for others who did not need
>> this particular cluster
>>
> What if we leave one repository as it is now and move 'teams' into git
> branches (core-main/cnd-main/...).
> Teams will work on branches.
> Then like now [1] the current push-cnd-main job can be replaced by
> automatic 'push cnd-main into main-silver branch', while push-to-cnd-main
> job will be automatic 'push main-silver into cnd-main'.
> Branch main-silver is built and tested as now and on success main-silver
> is pushed into main-golden for the production build.
>
> Thanks,
> Vladimir.
> [1] http://wiki.netbeans.org/HgParallelProjectIntegration
>
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Vladimir Voskresensky <vl...@oracle.com>.
Hi,
On 12.10.2016 18:23, Eric Barboni wrote:
>
> With the split in cluster it may possible to get rid of the "team" repositories (i.e. "core-main","cnd-main","jet-main") but we may need another repository ( kind of current "main-silver" or "main-golden") to prevent a bad commit leading to compile error in one cluster to be propagated to the full NetBeans break the build for others who did not need this particular cluster
What if we leave one repository as it is now and move 'teams' into git
branches (core-main/cnd-main/...).
Teams will work on branches.
Then like now [1] the current push-cnd-main job can be replaced by
automatic 'push cnd-main into main-silver branch', while
push-to-cnd-main job will be automatic 'push main-silver into cnd-main'.
Branch main-silver is built and tested as now and on success main-silver
is pushed into main-golden for the production build.
Thanks,
Vladimir.
[1] http://wiki.netbeans.org/HgParallelProjectIntegration
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Wade Chandler <co...@wadechandler.com>.
> On Oct 12, 2016, at 11:23, Eric Barboni <sk...@apache.org> wrote:
>
> Hi,
> Cluster separation seems very logical to me.
>
> Would it be possible to put build infrastructure alongside with source management ?
>
> With the split in cluster it may possible to get rid of the "team" repositories (i.e. "core-main","cnd-main","jet-main") but we may need another repository ( kind of current "main-silver" or "main-golden") to prevent a bad commit leading to compile error in one cluster to be propagated to the full NetBeans break the build for others who did not need this particular cluster
>
I think extra community repositories, outside of user clones and repositories for PRs and patches, unless they are specifically to independently hold a cluster for the purpose of better and smaller granularity, are bad ideas. That causes a lot more sync than I personally feel is needed. Feature branches and simplicity for the committers sounds better IMO. Too, looking for ways to reduce integration points sounds like a good idea; i.e. don’t have too many long lived branches, but instead focus on features as the unit you are trying to solve with long lived branches only existing for some really heavy thing such as a massive refactor, and as such the branches become strategic versus big artifacts in themselves.
In another email I mentioned something like:
i.e. (without a separate silver)
master (what used to be main-golden)
--development (what used to be main and silver (integration and main build))
----cnd/feature-abc (a cnd specific feature branch)
----groovy/feature-abc (a groovy specific feature branch)
------wadechandler/bug-or-feature-abc (a specific fix I want to merge up to this feature branch with others)
----wadechandler/bug-xyz (a specific fix for a bug submitted by me to merge to development)
Too, I think if one is not a committer, their PRs would come from their own individual repositories. I believe that is how we should accept patches going forward. That allows us to then do code reviews right there in the PRs with inline comments and other nice features plus there is a trail of all that data. Merges between committers should also use PRs IMO. My teams use them, and they work wonderfully. We also use branching, and that works out too.
https://help.github.com/articles/about-pull-requests/ <https://help.github.com/articles/about-pull-requests/>
These sound like good times to rethink some of the NB processes IMO considering new infra and other concerns.
Wade
===================
Wade Chandler
e: consult@wadechandler.com
RE: [PROPOSAL] Split the main NetBeans repo
Posted by Eric Barboni <sk...@apache.org>.
Hi,
Cluster separation seems very logical to me.
Would it be possible to put build infrastructure alongside with source management ?
With the split in cluster it may possible to get rid of the "team" repositories (i.e. "core-main","cnd-main","jet-main") but we may need another repository ( kind of current "main-silver" or "main-golden") to prevent a bad commit leading to compile error in one cluster to be propagated to the full NetBeans break the build for others who did not need this particular cluster
Eric
-----Message d'origine-----
De : Geertjan Wielenga [mailto:geertjan.wielenga@googlemail.com]
Envoyé : mercredi 12 octobre 2016 13:42
À : dev@netbeans.incubator.apache.org
Objet : [PROPOSAL] Split the main NetBeans repo
Hello all,
Please note Jan Lahoda, one of the NetBeans gurus in Prague, has published this on the Wiki:
https://cwiki.apache.org/confluence/display/NETBEANS/Experiment%3A+split+the+main+NetBeans+repository
Comments, thoughts, feedback, welcome.
Gj
Re: [PROPOSAL] Split the main NetBeans repo
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Wed, Oct 12, 2016 at 1:41 PM, Geertjan Wielenga
<ge...@googlemail.com> wrote:
...
> https://cwiki.apache.org/confluence/display/NETBEANS/Experiment%3A+split+the+main+NetBeans+repository
....
I cannot comment on the NetBeans code but in general my impression is
that Git works best with small focused repositories.
Tools like https://code.google.com/p/git-repo/ and
http://gitslave.sourceforge.net/ help work with multiple repositories.
-Bertrand