You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Ryan Schmitt <rs...@apache.org> on 2019/12/13 20:49:13 UTC
Repo consolidation
I would like to propose the following changes:
1. The `parent`, `core`, `client`, `website`, and `stylecheck` source trees
shall be consolidated into a single Git repository for all of Apache
HttpComponents 5.
2. All artifacts contained in this repository shall be co-versioned (i.e.
released in concert, with aligned version numbers [1]), similar to the way
`httpcore5-reactive`, `httpcore5-h2`, and `httpcore5` are co-versioned
within the `core` repository today.
Under this scheme, httpcore5 would still be a standalone product subject to
the same compatibility constraints as today, but friction would be
significantly reduced for developers, as changes between `core` and
`client`--not breaking changes (at least not after GA), but additive
changes for new features, and bugfixes affecting both packages--could be
coordinated in terms of development, code review, building, and testing.
Currently `httpclient5` depends on the latest Maven release of `httpcore5`,
which makes for a very awkward development experience: in order to test
changes, I have to:
1. Make changes to core and install them locally as a SNAPSHOT release
2. Change the client to depend on this SNAPSHOT release
3. Change the client codebase, sometimes extensively, to work with this
SNAPSHOT release of core
4. Install the client SNAPSHOT locally
5. Change whatever I'm working on to consume the latest SNAPSHOT releases
of both core and client
I think building (and versioning) core and client together would
drastically simplify this procedure, making it less error-prone, and easier
to develop and review overarching changes to all of httpcomponents. In the
long term, I also think it would simplify things for users to have a single
set of versions for core and client, similar to the way Jackson versions
are aligned [1].
It's worth pointing out that these repositories can easily be consolidated
in such a way that their histories will all be retained, so that operations
like `git blame` and `git log` will continue to give useful results. It's
also interesting to note that the OpenJDK project recently consolidated
their repository tree into a single Mercurial repository [2], with the next
goal being migration from Mercurial to Git [3].
If the other committers are open to considering this idea, I'd be glad to
put together a proof-of-concept.
[1]
https://docs.gradle.org/current/userguide/dependency_version_alignment.html
[2] https://openjdk.java.net/jeps/296
[3] https://openjdk.java.net/jeps/357
Re: Repo consolidation
Posted by Gary Gregory <ga...@gmail.com>.
I have a few thoughts here, with the TL;DR that I like the idea of one repo
for core and client.
My background is that I spent a long time working with HttpCore5 to build a
proprietary reverse proxy (which has since been rewritten on top of Jetty
but that's a different and unrelated story and has nothing to do with the
merits or quality of HttpComponents.)
The pro of the current set up is that it _seems_ simpler for me to have
used new releases of HC5 because there is less code in that repo and
therefore less to manage for releases. For example, no need to wait for a
bug fix in HttpClient to get a new Core out.
The con is that I still use HttpClient for testing what I built with
HttpCore, so those two sets of jars MUST be in sync to at least make
testing simplest so that they all can live in the same class loader. From
an outside POV, I am never 100% sure if versions of Client and Core are
guaranteed to match up. This is the same kind of issue I have with Apache
Commons DBCP sitting on top of Commons Pool. In our current scheme here, I
would want a Client to be released immediately after a Core release to mark
the fact that one is made to work with the other. Having both Core and
Client in the same repo would make releasing matching jars easier I think,
All of this to say that I like one repo and releasing a set of matching
jars. It would be even be OK with me if an edge case would be to fix a bug
in Core and release both Core and Client, even if nothing has changed in
Client. It would tell folks, yep, this guaranteed to match up and designed
to work together.
Gary
On Wed, Dec 18, 2019 at 2:32 AM Ryan Schmitt <rs...@apache.org> wrote:
> It sounds like these changes aren't going to happen, but I'm going to press
> the point anyway because I think this is important.
>
> > I am not sure I can agree with that. Usually one should care about the
> > top level library such as HttpClient or HttpAsyncClient only. One needs
> > to manually override their HttpCore dependency only in case of some
> > severe bug in core components.
>
> HttpCore includes both :httpcore5-h2 and :httpcore5-reactive, which are
> brand new, highly complex components that are much less stable than
> :httpcore5 (which is not that stable in the first place). By your own
> admission, I could have written my entire SDK integration using just these
> three components. That's a guarantee that HttpCore will *not* be a mature,
> slow-moving product any time soon, and that users will need to explicitly
> track both versions if we persist in this scheme.
>
> Let's not kid ourselves: HttpCore is going to be plagued with serious bugs
> for years, and we will regularly put out unusably defective releases. The
> complexity of HTTP/2 is all but unmanageable. I've seen it myself working
> with Netty, which *still* hasn't stabilized its high-level HTTP/2 API after
> all this time and effort.
>
> > Parent and website and changed rarely, why force releases? So are core
> > and client logically decoupled.
>
> On reviewing the repositories, I agree with you about parent and website
> (and stylecheck), since these repositories don't produce Maven artifacts
> for public consumption. However, I see the version decoupling of core and
> client as purely a source of friction for everyone, adding no value
> whatsoever.
>
> You object to the idea of "forcing releases" of other components. Why? What
> resource exactly do we manage to conserve by releasing three jar files at a
> time, instead of six? The contradiction, by the way, is quite glaring here:
> why are :httpcore5, :httpcore5-h2, and :httpcore5-reactive unified and
> version aligned with each other, while HttpCore and HttpClient are
> decoupled? It makes no sense.
>
> > Also consider the more modules you have in a reactor, the longer it
> > takes to build and test everything.
>
> First of all, even this claim is not true if you consider how much faster
> Gradle could build everything in a single repository, by parallelizing
> across all modules. More importantly, the time it takes to *iterate* on
> coordinated changes between Core and Client is literally orders of
> magnitude longer because they are developed totally separately.
>
> > The moving targets you have described will slow down when version 5 has
> > gone GA.
>
> Right now, to debug my sporadically failing integration tests in
> HttpClient, I want to add debug logging to various places in HttpCore, and
> add other types of debugging code as well. Every time I change the
> debugging code, I have to locally simulate *an entire release cycle* of
> HttpCore, instead of just making all of these changes through my IDE in a
> single workspace.
>
Re: Repo consolidation
Posted by Michael Osipov <mi...@apache.org>.
Am 2019-12-21 um 08:14 schrieb Ryan Schmitt:
>>
>> I do not know what IDE you use but IntelliJ IDEA makes it completely
>> effortless to run SNAPSHOT versions of multiple modules within the same
>> project.
>
>
> I don't know how to develop separate Maven projects as if they were a
> single project. I do know how to do this with Gradle, which has a feature
> called composite builds:
> https://docs.gradle.org/current/userguide/composite_builds.html
Use aggergator projects/parents. They should mimic what you are looking for.
M
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Repo consolidation
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2019-12-20 at 23:14 -0800, Ryan Schmitt wrote:
> >
> > I do not know what IDE you use but IntelliJ IDEA makes it
> > completely
> > effortless to run SNAPSHOT versions of multiple modules within the
> > same
> > project.
>
>
> I don't know how to develop separate Maven projects as if they were a
> single project. I do know how to do this with Gradle, which has a
> feature
> called composite builds:
> https://docs.gradle.org/current/userguide/composite_builds.html
Hi Ryan
Maven does not support composite builds natively but IntelliJ does.
What it usually taken for me is to import HttpCore as modules of
HttpClient project and change HttpCore version to the corresponding
snapshot version.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Repo consolidation
Posted by Ryan Schmitt <rs...@apache.org>.
>
> I do not know what IDE you use but IntelliJ IDEA makes it completely
> effortless to run SNAPSHOT versions of multiple modules within the same
> project.
I don't know how to develop separate Maven projects as if they were a
single project. I do know how to do this with Gradle, which has a feature
called composite builds:
https://docs.gradle.org/current/userguide/composite_builds.html
Re: Repo consolidation
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2019-12-17 at 23:32 -0800, Ryan Schmitt wrote:
> It sounds like these changes aren't going to happen, but I'm going to
> press
> the point anyway because I think this is important.
>
> > I am not sure I can agree with that. Usually one should care about
> > the
> > top level library such as HttpClient or HttpAsyncClient only. One
> > needs
> > to manually override their HttpCore dependency only in case of some
> > severe bug in core components.
>
> HttpCore includes both :httpcore5-h2 and :httpcore5-reactive, which
> are
> brand new, highly complex components that are much less stable than
> :httpcore5 (which is not that stable in the first place). By your own
> admission, I could have written my entire SDK integration using just
> these
> three components. That's a guarantee that HttpCore will *not* be a
> mature,
> slow-moving product any time soon, and that users will need to
> explicitly
> track both versions if we persist in this scheme.
>
> Let's not kid ourselves: HttpCore is going to be plagued with serious
> bugs
> for years, and we will regularly put out unusably defective releases.
> The
> complexity of HTTP/2 is all but unmanageable. I've seen it myself
> working
> with Netty, which *still* hasn't stabilized its high-level HTTP/2 API
> after
> all this time and effort.
>
Since you mentioned Netty I will have to say that. Not being a big fan
of Netty in general for all sorts of reasons I think what they do with
their unified component versioning is pretty close to cheating. They
throw in a completely new codec such as HTTP/2 into their stable branch
and once of a sudden that codec looks stable and GA to a casual
bystander, though in fact it is not. I am not quite comfortable doing
the same.
> What
> resource exactly do we manage to conserve by releasing three jar
> files at a
> time, instead of six? The contradiction, by the way, is quite glaring
> here:
> why are :httpcore5, :httpcore5-h2, and :httpcore5-reactive unified
> and
> version aligned with each other, while HttpCore and HttpClient are
> decoupled? It makes no sense.
>
The primary reason for core and client being separate projects is a
different scope and target audience most of all.
By the way I would be open to having a separate release cycle for
httpcore5-h2. We can still release httpcore5 as GA and keep httpcore5-
h2 in BETA longer.
> >
> Right now, to debug my sporadically failing integration tests in
> HttpClient, I want to add debug logging to various places in
> HttpCore, and
> add other types of debugging code as well. Every time I change the
> debugging code, I have to locally simulate *an entire release cycle*
> of
> HttpCore, instead of just making all of these changes through my IDE
> in a
> single workspace.
I do not know what IDE you use but IntelliJ IDEA makes it completely
effortless to run SNAPSHOT versions of multiple modules within the same
project.
Again, this sounds like a blonde vs brunette type of debate.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Repo consolidation
Posted by Ryan Schmitt <rs...@apache.org>.
It sounds like these changes aren't going to happen, but I'm going to press
the point anyway because I think this is important.
> I am not sure I can agree with that. Usually one should care about the
> top level library such as HttpClient or HttpAsyncClient only. One needs
> to manually override their HttpCore dependency only in case of some
> severe bug in core components.
HttpCore includes both :httpcore5-h2 and :httpcore5-reactive, which are
brand new, highly complex components that are much less stable than
:httpcore5 (which is not that stable in the first place). By your own
admission, I could have written my entire SDK integration using just these
three components. That's a guarantee that HttpCore will *not* be a mature,
slow-moving product any time soon, and that users will need to explicitly
track both versions if we persist in this scheme.
Let's not kid ourselves: HttpCore is going to be plagued with serious bugs
for years, and we will regularly put out unusably defective releases. The
complexity of HTTP/2 is all but unmanageable. I've seen it myself working
with Netty, which *still* hasn't stabilized its high-level HTTP/2 API after
all this time and effort.
> Parent and website and changed rarely, why force releases? So are core
> and client logically decoupled.
On reviewing the repositories, I agree with you about parent and website
(and stylecheck), since these repositories don't produce Maven artifacts
for public consumption. However, I see the version decoupling of core and
client as purely a source of friction for everyone, adding no value
whatsoever.
You object to the idea of "forcing releases" of other components. Why? What
resource exactly do we manage to conserve by releasing three jar files at a
time, instead of six? The contradiction, by the way, is quite glaring here:
why are :httpcore5, :httpcore5-h2, and :httpcore5-reactive unified and
version aligned with each other, while HttpCore and HttpClient are
decoupled? It makes no sense.
> Also consider the more modules you have in a reactor, the longer it
> takes to build and test everything.
First of all, even this claim is not true if you consider how much faster
Gradle could build everything in a single repository, by parallelizing
across all modules. More importantly, the time it takes to *iterate* on
coordinated changes between Core and Client is literally orders of
magnitude longer because they are developed totally separately.
> The moving targets you have described will slow down when version 5 has
> gone GA.
Right now, to debug my sporadically failing integration tests in
HttpClient, I want to add debug logging to various places in HttpCore, and
add other types of debugging code as well. Every time I change the
debugging code, I have to locally simulate *an entire release cycle* of
HttpCore, instead of just making all of these changes through my IDE in a
single workspace.
Re: Repo consolidation
Posted by Michael Osipov <mi...@apache.org>.
Am 2019-12-13 um 22:57 schrieb Oleg Kalnichevski:
> On Fri, 2019-12-13 at 12:49 -0800, Ryan Schmitt wrote:
>> I would like to propose the following changes:
>>
>> 1. The `parent`, `core`, `client`, `website`, and `stylecheck` source
>> trees
>> shall be consolidated into a single Git repository for all of Apache
>> HttpComponents 5.
>> 2. All artifacts contained in this repository shall be co-versioned
>> (i.e.
>> released in concert, with aligned version numbers [1]), similar to
>> the way
>> `httpcore5-reactive`, `httpcore5-h2`, and `httpcore5` are co-
>> versioned
>> within the `core` repository today.
>>
>> Under this scheme, httpcore5 would still be a standalone product
>> subject to
>> the same compatibility constraints as today, but friction would be
>> significantly reduced for developers, as changes between `core` and
>> `client`--not breaking changes (at least not after GA), but additive
>> changes for new features, and bugfixes affecting both packages--could
>> be
>> coordinated in terms of development, code review, building, and
>> testing.
>> Currently `httpclient5` depends on the latest Maven release of
>> `httpcore5`,
>> which makes for a very awkward development experience: in order to
>> test
>> changes, I have to:
>>
>> 1. Make changes to core and install them locally as a SNAPSHOT
>> release
>> 2. Change the client to depend on this SNAPSHOT release
>> 3. Change the client codebase, sometimes extensively, to work with
>> this
>> SNAPSHOT release of core
>> 4. Install the client SNAPSHOT locally
>> 5. Change whatever I'm working on to consume the latest SNAPSHOT
>> releases
>> of both core and client
>>
>> I think building (and versioning) core and client together would
>> drastically simplify this procedure, making it less error-prone, and
>> easier
>> to develop and review overarching changes to all of httpcomponents.
>> In the
>> long term, I also think it would simplify things for users to have a
>> single
>> set of versions for core and client, similar to the way Jackson
>> versions
>> are aligned [1].
Jackson does not use a mono repo. They have distinct repos (subprojects)
on GitHub.
The moving targets you have described will slow down when version 5 has
gone GA.
> I understand it can be frustrating while core APIs are not stable
> but core and client components tend to have different maturity cycles
> and therefore having distinct release cycles for core and client
> components is beneficial. In HC 4.x we can make fixes to core
> components and release them fast while some client features are still
> being worked on and are not ready for a release.
>
> We might also have a reverse-proxy or a more feature rich server sub-
> projects at some point. Coupling core with client would make it much
> harder and would necessitate a separate core project again at that
> point.
I agree with Oleg's views. All subprojects move at different speeds.
Parent and website and changed rarely, why force releases? So are core
and client logically decoupled.
Also consider the more modules you have in a reactor, the longer it
takes to build and test everything.
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Repo consolidation
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2019-12-13 at 15:04 -0800, Ryan Schmitt wrote:
> I don't think core is special in this regard, but I *do* think it is
> special in having a separate release cycle. Most Java projects these
> days
> use version alignment (and usually a unified repository as well),
> despite
> featuring many distinct components that all move at different speeds.
> Examples include:
>
> - Log4j2 (api, core, four logging bridges, many many appenders...)
> - JUnit 5 (platform commons, Jupiter API, Jupiter engine, vintage
> engine,
> Jupiter params support, platform runner...)
> - Jackson (annotations, core, databind, dataformat, datatype, jaxrs,
> Kotlin/scala support...)
> - Netty (common, buffer, codec, resolver, transport, native libs,
> numerous
> protocols and data formats...)
> - AWS SDK for Java (*over a hundred* separate service SDKs, multiple
> HTTP
> clients, SPI, code generators, SDK core...)
>
> Under this approach, there is a clear benefit to the user of only
> having to
> track one version number, no matter how many components are pulled
> in.
> Currently, httpclient5 works quite the other way: users must add
> dependencies on both httpcore5 and httpclient5 as a matter of course,
> and
> must model both versions explicitly: this is the only way to be sure
> that
> they are getting the latest versions of both components.
>
I am not sure I can agree with that. Usually one should care about the
top level library such as HttpClient or HttpAsyncClient only. One needs
to manually override their HttpCore dependency only in case of some
severe bug in core components.
Ultimately this is a blond vs brunette type of debate. While a lot of
fun I doubt it can have an ultimate resolution.
Oleg
PS: One thing I cannot fully understand is why you chose to use
httpclient5 for AWS SDK in the initial iteration. I suppose
HttpAsyncRequester shipped with httpcore5 would be perfectly adequate
initially. Once all low level transport issues are worked out you could
upgrade to httpclient5 and replace HttpAsyncRequester with
HttpAsyncClient with little effort.
> I also think that the projects listed above make it clear that
> dependency
> alignment scales to a much larger number of components than we have
> currently, and adopting it does not foreclose the possibility of
> developing
> new httpcomponents in the future.
>
> On Fri, Dec 13, 2019 at 1:57 PM Oleg Kalnichevski <ol...@apache.org>
> wrote:
>
> > I understand it can be frustrating while core APIs are not stable
> > but core and client components tend to have different maturity
> > cycles
> > and therefore having distinct release cycles for core and client
> > components is beneficial. In HC 4.x we can make fixes to core
> > components and release them fast while some client features are
> > still
> > being worked on and are not ready for a release.
> >
> > We might also have a reverse-proxy or a more feature rich server
> > sub-
> > projects at some point. Coupling core with client would make it
> > much
> > harder and would necessitate a separate core project again at that
> > point.
> >
> > Unless we decide we only develop HttpClient and nothing else, which
> > is
> > fine by me as long as this is what we all want.
> >
> > Oleg
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Repo consolidation
Posted by Ryan Schmitt <rs...@apache.org>.
I don't think core is special in this regard, but I *do* think it is
special in having a separate release cycle. Most Java projects these days
use version alignment (and usually a unified repository as well), despite
featuring many distinct components that all move at different speeds.
Examples include:
- Log4j2 (api, core, four logging bridges, many many appenders...)
- JUnit 5 (platform commons, Jupiter API, Jupiter engine, vintage engine,
Jupiter params support, platform runner...)
- Jackson (annotations, core, databind, dataformat, datatype, jaxrs,
Kotlin/scala support...)
- Netty (common, buffer, codec, resolver, transport, native libs, numerous
protocols and data formats...)
- AWS SDK for Java (*over a hundred* separate service SDKs, multiple HTTP
clients, SPI, code generators, SDK core...)
Under this approach, there is a clear benefit to the user of only having to
track one version number, no matter how many components are pulled in.
Currently, httpclient5 works quite the other way: users must add
dependencies on both httpcore5 and httpclient5 as a matter of course, and
must model both versions explicitly: this is the only way to be sure that
they are getting the latest versions of both components.
I also think that the projects listed above make it clear that dependency
alignment scales to a much larger number of components than we have
currently, and adopting it does not foreclose the possibility of developing
new httpcomponents in the future.
On Fri, Dec 13, 2019 at 1:57 PM Oleg Kalnichevski <ol...@apache.org> wrote:
> I understand it can be frustrating while core APIs are not stable
> but core and client components tend to have different maturity cycles
> and therefore having distinct release cycles for core and client
> components is beneficial. In HC 4.x we can make fixes to core
> components and release them fast while some client features are still
> being worked on and are not ready for a release.
>
> We might also have a reverse-proxy or a more feature rich server sub-
> projects at some point. Coupling core with client would make it much
> harder and would necessitate a separate core project again at that
> point.
>
> Unless we decide we only develop HttpClient and nothing else, which is
> fine by me as long as this is what we all want.
>
> Oleg
>
>
Re: Repo consolidation
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2019-12-13 at 12:49 -0800, Ryan Schmitt wrote:
> I would like to propose the following changes:
>
> 1. The `parent`, `core`, `client`, `website`, and `stylecheck` source
> trees
> shall be consolidated into a single Git repository for all of Apache
> HttpComponents 5.
> 2. All artifacts contained in this repository shall be co-versioned
> (i.e.
> released in concert, with aligned version numbers [1]), similar to
> the way
> `httpcore5-reactive`, `httpcore5-h2`, and `httpcore5` are co-
> versioned
> within the `core` repository today.
>
> Under this scheme, httpcore5 would still be a standalone product
> subject to
> the same compatibility constraints as today, but friction would be
> significantly reduced for developers, as changes between `core` and
> `client`--not breaking changes (at least not after GA), but additive
> changes for new features, and bugfixes affecting both packages--could
> be
> coordinated in terms of development, code review, building, and
> testing.
> Currently `httpclient5` depends on the latest Maven release of
> `httpcore5`,
> which makes for a very awkward development experience: in order to
> test
> changes, I have to:
>
> 1. Make changes to core and install them locally as a SNAPSHOT
> release
> 2. Change the client to depend on this SNAPSHOT release
> 3. Change the client codebase, sometimes extensively, to work with
> this
> SNAPSHOT release of core
> 4. Install the client SNAPSHOT locally
> 5. Change whatever I'm working on to consume the latest SNAPSHOT
> releases
> of both core and client
>
> I think building (and versioning) core and client together would
> drastically simplify this procedure, making it less error-prone, and
> easier
> to develop and review overarching changes to all of httpcomponents.
> In the
> long term, I also think it would simplify things for users to have a
> single
> set of versions for core and client, similar to the way Jackson
> versions
> are aligned [1].
>
I understand it can be frustrating while core APIs are not stable
but core and client components tend to have different maturity cycles
and therefore having distinct release cycles for core and client
components is beneficial. In HC 4.x we can make fixes to core
components and release them fast while some client features are still
being worked on and are not ready for a release.
We might also have a reverse-proxy or a more feature rich server sub-
projects at some point. Coupling core with client would make it much
harder and would necessitate a separate core project again at that
point.
Unless we decide we only develop HttpClient and nothing else, which is
fine by me as long as this is what we all want.
Oleg
> It's worth pointing out that these repositories can easily be
> consolidated
> in such a way that their histories will all be retained, so that
> operations
> like `git blame` and `git log` will continue to give useful results.
> It's
> also interesting to note that the OpenJDK project recently
> consolidated
> their repository tree into a single Mercurial repository [2], with
> the next
> goal being migration from Mercurial to Git [3].
>
> If the other committers are open to considering this idea, I'd be
> glad to
> put together a proof-of-concept.
>
> [1]
>
https://docs.gradle.org/current/userguide/dependency_version_alignment.html
> [2] https://openjdk.java.net/jeps/296
> [3] https://openjdk.java.net/jeps/357
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org