You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jclouds.apache.org by Everett Toews <ev...@RACKSPACE.COM> on 2014/07/29 03:52:41 UTC
Labs repos rejoining the main repo
One of the things that quickly becomes apparent when doing a release is that having so many repos makes it excruciatingly complicated to do a release. On a separate note, it also makes it very difficult to refactor jclouds core code.
Seems like every time we chat about it in IRC everybody agrees that having so many repos is a serious pain point. Which is to say, no one is excited about having so many repos.
I know there were reasons for splitting the main repo up in the first place but I think it’s time to revisit this. Here are the pros and cons as I see them.
Pros:
* Sane Maven parenting
* Easier releases
* Better visibility into all jclouds code when making changes to jclouds main repo code
* Easier refactoring of jclouds main repo code that labs code depends on
* Less chance of breaking a labs repo when upgrading a core dependency
* Less CloudBees jobs
* Less confusion from users about why the labs are separate repos
Cons:
* Longer build times
What have I missed?
Is it time for the labs repos to rejoin?
We can figure out the mechanics of it after we’ve reached some kind of consensus.
Thanks,
Everett
Re: Labs repos rejoining the main repo
Posted by Everett Toews <ev...@RACKSPACE.COM>.
On Aug 1, 2014, at 9:55 AM, Andrew Phillips <ap...@qrmedia.com> wrote:
>> I ask about this because we need to be able to provide value to our users quickly in actual releases. The users we work with don’t care to take snapshot releases and many will outright refuse snapshot releases. These are the same users who appreciate the predictability of things like semver.
>
> This is really a different discussion that I'd like to pursue elsewhere, but who are "the users we work with" here? Do we have any details that we can share?
Sigh…no.
> It's certainly not that I want to cast any doubt on the above statement. I just feel that getting a better idea of our *actual* user population and the different requirements within it would really help in terms of determining priorities and direction.
I can only sum it up as these things are important to a portion of our user base. Many of our enterprise customers are not willing to publicly share what tools they use in their applications. This is the default position of many (most?) enterprise shops. In the near future I’m going to be making a concerted effort to change minds on this but it’s an uphill climb.
Apologies for the distracting comment. Let’s focus on the labs repos here.
Everett
Re: Labs repos rejoining the main repo
Posted by Andrew Phillips <ap...@qrmedia.com>.
> I ask about this because we need to be able to provide value to our
> users quickly in actual releases. The users we work with don’t care
> to take snapshot releases and many will outright refuse snapshot
> releases. These are the same users who appreciate the predictability
> of things like semver.
This is really a different discussion that I'd like to pursue
elsewhere, but who are "the users we work with" here? Do we have any
details that we can share?
It's certainly not that I want to cast any doubt on the above
statement. I just feel that getting a better idea of our *actual* user
population and the different requirements within it would really help
in terms of determining priorities and direction.
I think it would be great to get more numbers like the ones Andrew G
extracted from JIRA about which versions of Java our users seem to be
using when the Java 6/7 discussion was going on.
Thoughts on how to go about this, assuming we feel it's possible?
Worth bringin up the survey idea [1] again here?
ap
[1] http://markmail.org/message/or7xmsqry6qqwfpx
Re: Labs repos rejoining the main repo
Posted by Everett Toews <ev...@RACKSPACE.COM>.
On Jul 30, 2014, at 6:22 PM, Andrew Gaul <ga...@apache.org> wrote:
> What if we continue to use the labs repository as a staging area but
> stop releasing it?
I think this could be a viable approach but I have some questions below.
> This will allow development of new providers to
> proceed and user testing via snapshots but not impact the release
> process. Obviously, this will require migrating the mature providers to
> core, which we seem to agree should happen anyway.
Agreed.
I’m assuming this includes apis/providers that aren’t associated with an abstraction, since that seems to be our de facto position anyway. Was that your intention as well?
There’s one scenario that I would like to know is okay with everybody before going down this path because “mature” is not well defined.
When it comes to a trusted committer adding an api/provider, we would need to be able to move it into the main repo as soon as it has a minimal amount of features that make the api useful (this will vary from api to api).
By trusted committer, I mean those who are known to deliver quality apis and maintain them. The work could start in a labs repository or maybe even the master branch of the main repo, if we’re not too close to a major point release. As soon as we’ve reached a minimal amount of useful features, we move it to the main repo or back port it into a release branch (if it started in the master branch of the main repo).
I ask about this because we need to be able to provide value to our users quickly in actual releases. The users we work with don’t care to take snapshot releases and many will outright refuse snapshot releases. These are the same users who appreciate the predictability of things like semver. Right now we can help these users because the labs repos are being released.
If we are not releasing the labs repos, it is extremely important to me that we be able to move code into the main repo as soon as it has a minimal amount of useful features and has met the jclouds quality standard (i.e. has been reviewed throughly in our PR process and has live and mock tests).
What is everyone’s thoughts on that?
>> As regards karaf and cli: would they belong in core, or in tools or
>> elsewhere? They're really more downstream consumers of jclouds than
>> *parts* of jclouds themselves.
>
> Honestly I do not understand karaf and see it as something outside of
> the main jclouds project. As for cli, we should replace it with
> something friendlier and lightweight such as:
Are we reaching a consensus on removing karaf and cli from our release process? Am I reading this right?
Seems we’re all agreed they're more downstream users of jclouds than part of jclouds itself. Does that mean not releasing them along with jclouds?
It would be really great to hear from somebody with a vested interest in karaf and cli on this.
Thanks,
Everett
Re: Labs repos rejoining the main repo
Posted by Andrew Gaul <ga...@apache.org>.
On Tue, Jul 29, 2014 at 04:16:52AM +0200, Andrew Phillips wrote:
> One of the main "cons", certainly w.r.t. labs, was:
>
> * code of vastly different degrees of maturity/up-to-dateness in the same repo
>
> I certainly wouldn't want to include a whole bunch of the current
> labs providers in core, since they're either simply broken and/or
> hopelessly out of date.
>
> If I had to make a choice, I would probably lean towards removing
> them and merging the rest, but there have been discussions every now
> and them of people perhaps being willing to pick them up.
What if we continue to use the labs repository as a staging area but
stop releasing it? This will allow development of new providers to
proceed and user testing via snapshots but not impact the release
process. Obviously, this will require migrating the mature providers to
core, which we seem to agree should happen anyway.
> As regards karaf and cli: would they belong in core, or in tools or
> elsewhere? They're really more downstream consumers of jclouds than
> *parts* of jclouds themselves.
Honestly I do not understand karaf and see it as something outside of
the main jclouds project. As for cli, we should replace it with
something friendlier and lightweight such as:
https://github.com/andrewgaul/jclouds/tree/blobstore-cli
--
Andrew Gaul
http://gaul.org/
Re: Labs repos rejoining the main repo
Posted by Ignasi Barrera <na...@apache.org>.
Here I am with Andrew P. and believe that karat and CLI are more
downstream "clients" of jclouds than actual parts of the project.
I would keep them in their own repositories. Furthermore, the karat
tests take about 17 min. to complete and often present indeterministic
failures that are difficult to diagnose/reproduce. We don't have that
expertise on Karaf (hopefully this is something we can fix soon), and
having it in the main repo IMHO would cause now more trouble than
benefits.
Regarding the labs repos, there is a mix:
* There are providers that work quite well: gee, digitalocean,
cloudsigma2, the rackspace ones.
* There are providers that are obsolete (work with old versions of the
apis) but are being used by some users: abiquo (is being used by NEC).
* There are providers that are directly broken and are unusable: joyentcloud
* There are providers that are more like "experiments" that never got
the maturity level to be merged to the main repo.
Given this scenario, I think we should take a different approach
depending on the providers:
* I'm OK to merge to the main repo the stable ones, and the ones that
are obsolete but used, as we know they're working as expected. We want
to keep releasing them, and can expect bug fixes and contributions.
* Keep all the broken and experimental providers in the labs repo, and
poll in the user/dev mewling lists, and other channels such as
twitter, etc, to see if there are real users of those providers. Then
decide if we drop them or not.
* Those providers that the community is using but are in labs could be
moved to the main repo.
This is the "conceptual approach". Talking about the implementation,
I'd suggest:
* Move the providers to the main repo, but keep them in a "labs"
folder, just as we did in early jclouds versions. We still want to
keep the semantics that they're not mature or are in beta stages.
Moving them out to the apis/providers folders is trivial and should
keep the git history, so that shouldn't be a problem. It will also
make it easier to include/exclude the labs projects from the releases,
or the creation of a "labs" maven profile that gives us more
flexibility on what has to be built.
Finally, regarding the "level of maturity" question, I don't think we
should accept contributions that are not mature enough to join
directly the main repo as a top level provider. We should gather live
test output feedback for every new provider and work with the
contributors to reach the desired quality. It makes no sense to merge
a contribution that does not meet our quality standards just because
"it goes to the labs repo and eventually the provider will get
improved". Experience shows that this won't happen.
I.
On 29 July 2014 05:06, Everett Toews <ev...@rackspace.com> wrote:
> On Jul 28, 2014, at 9:16 PM, Andrew Phillips <ap...@qrmedia.com> wrote:
>
>> As regards karaf and cli: would they belong in core, or in tools or elsewhere? They're really more downstream consumers of jclouds than *parts* of jclouds themselves.
>
> Originally I thought of the tools dir as just being a collection of scripts but it could be more than that. If we did put karaf and cli and the BlobStore stuff in there it’s clear that it would become something of a catch-all for jclouds related stuff that we want to keep close. I’m not necessarily against that but we need to be clear about the intention of the tools dir.
>
> Personally, I’ve never used karaf or the cli so I don’t feel strongly about where they go or even if they stay within our repo. I hope the maintainers of these things speak up here.
>
> Everett
>
Re: Labs repos rejoining the main repo
Posted by Chris Custine <ch...@gmail.com>.
I agree that the Karaf based tooling is a little heavy just to provide a CLI. If jclouds was already using osgi for other things it would be ok, but its a bit painful if it is not a daily use type of thing. I haven’t tried out blobstore-cli but it looks really nice. There could also be some room for improvement in the Karaf bits so if it helps at all, I will give it some attention.
Chris
--
Chris Custine
On July 30, 2014 at 5:30:03 PM, Andrew Gaul (gaul@apache.org) wrote:
> On Tue, Jul 29, 2014 at 03:06:40AM +0000, Everett Toews wrote:
> > On Jul 28, 2014, at 9:16 PM, Andrew Phillips wrote:
> >
> > > As regards karaf and cli: would they belong in core, or in tools or elsewhere? They're
> really more downstream consumers of jclouds than *parts* of jclouds themselves.
> >
> > Originally I thought of the tools dir as just being a collection of scripts but it could
> be more than that. If we did put karaf and cli and the BlobStore stuff in there it’s clear
> that it would become something of a catch-all for jclouds related stuff that we want to
> keep close. I’m not necessarily against that but we need to be clear about the intention
> of the tools dir.
> >
> > Personally, I’ve never used karaf or the cli so I don’t feel strongly about where they
> go or even if they stay within our repo. I hope the maintainers of these things speak up
> here.
>
> The frequently-suggested blobstore tools are much smaller than karaf and
> potentially a much larger user base; these could help users diagnose
> connectivity, authentication, proxy, and other configuration issues in a
> dependable way instead of the zoo of user application problems,
> including logging. Further there is a legitimate administrative use and
> potentially a way to reach new users. Check out how clean BlobStoreCli
> is compared to anything in karaf/cli:
>
> https://github.com/andrewgaul/jclouds/blob/blobstore-cli/blobstore-cli/src/main/java/org/jclouds/blobstore/cli/BlobStoreCli.java
>
> I would really like to find a way forward on these tools.
>
> --
> Andrew Gaul
> http://gaul.org/
>
Re: Labs repos rejoining the main repo
Posted by Andrew Gaul <ga...@apache.org>.
On Tue, Jul 29, 2014 at 03:06:40AM +0000, Everett Toews wrote:
> On Jul 28, 2014, at 9:16 PM, Andrew Phillips <ap...@qrmedia.com> wrote:
>
> > As regards karaf and cli: would they belong in core, or in tools or elsewhere? They're really more downstream consumers of jclouds than *parts* of jclouds themselves.
>
> Originally I thought of the tools dir as just being a collection of scripts but it could be more than that. If we did put karaf and cli and the BlobStore stuff in there it’s clear that it would become something of a catch-all for jclouds related stuff that we want to keep close. I’m not necessarily against that but we need to be clear about the intention of the tools dir.
>
> Personally, I’ve never used karaf or the cli so I don’t feel strongly about where they go or even if they stay within our repo. I hope the maintainers of these things speak up here.
The frequently-suggested blobstore tools are much smaller than karaf and
potentially a much larger user base; these could help users diagnose
connectivity, authentication, proxy, and other configuration issues in a
dependable way instead of the zoo of user application problems,
including logging. Further there is a legitimate administrative use and
potentially a way to reach new users. Check out how clean BlobStoreCli
is compared to anything in karaf/cli:
https://github.com/andrewgaul/jclouds/blob/blobstore-cli/blobstore-cli/src/main/java/org/jclouds/blobstore/cli/BlobStoreCli.java
I would really like to find a way forward on these tools.
--
Andrew Gaul
http://gaul.org/
Re: Labs repos rejoining the main repo
Posted by Everett Toews <ev...@RACKSPACE.COM>.
On Jul 28, 2014, at 9:16 PM, Andrew Phillips <ap...@qrmedia.com> wrote:
> As regards karaf and cli: would they belong in core, or in tools or elsewhere? They're really more downstream consumers of jclouds than *parts* of jclouds themselves.
Originally I thought of the tools dir as just being a collection of scripts but it could be more than that. If we did put karaf and cli and the BlobStore stuff in there it’s clear that it would become something of a catch-all for jclouds related stuff that we want to keep close. I’m not necessarily against that but we need to be clear about the intention of the tools dir.
Personally, I’ve never used karaf or the cli so I don’t feel strongly about where they go or even if they stay within our repo. I hope the maintainers of these things speak up here.
Everett
Re: Labs repos rejoining the main repo
Posted by Andrew Phillips <ap...@qrmedia.com>.
> Cons:
>
> * Longer build times
>
> What have I missed?
One of the main "cons", certainly w.r.t. labs, was:
* code of vastly different degrees of maturity/up-to-dateness in the same repo
I certainly wouldn't want to include a whole bunch of the current labs
providers in core, since they're either simply broken and/or
hopelessly out of date.
If I had to make a choice, I would probably lean towards removing them
and merging the rest, but there have been discussions every now and
them of people perhaps being willing to pick them up.
As regards karaf and cli: would they belong in core, or in tools or
elsewhere? They're really more downstream consumers of jclouds than
*parts* of jclouds themselves.
Big +1 on finally (whether for this release or soon after) merging
jclouds-chef in, by the way.
ap
PS: I think getting rid of all the manual answers parts in the
releases will be a big step in the right direction of reducing the
pain level, and we should be able to do that pretty easily.
Re: Labs repos rejoining the main repo
Posted by Everett Toews <ev...@RACKSPACE.COM>.
On Jul 28, 2014, at 8:59 PM, Andrew Gaul <ga...@apache.org> wrote:
> Agree with all these points, especially the last one. Do we clearly
> communicate to users what labs means today?
I attempted to do so for the labs-openstack repo here [1] and in all the sub-dirs. Mixed success…
Everett
[1] https://github.com/jclouds/jclouds-labs-openstack/blob/master/README.md
Re: Labs repos rejoining the main repo
Posted by Andrew Gaul <ga...@apache.org>.
On Tue, Jul 29, 2014 at 01:52:41AM +0000, Everett Toews wrote:
> One of the things that quickly becomes apparent when doing a release is that having so many repos makes it excruciatingly complicated to do a release. On a separate note, it also makes it very difficult to refactor jclouds core code.
>
> Seems like every time we chat about it in IRC everybody agrees that having so many repos is a serious pain point. Which is to say, no one is excited about having so many repos.
>
> I know there were reasons for splitting the main repo up in the first place but I think it’s time to revisit this. Here are the pros and cons as I see them.
Sorry that this is so painful, but thanks for describing the pros and
cons!
> Pros:
>
> * Sane Maven parenting
> * Easier releases
> * Better visibility into all jclouds code when making changes to jclouds main repo code
> * Easier refactoring of jclouds main repo code that labs code depends on
> * Less chance of breaking a labs repo when upgrading a core dependency
> * Less CloudBees jobs
> * Less confusion from users about why the labs are separate repos
Agree with all these points, especially the last one. Do we clearly
communicate to users what labs means today?
> Cons:
>
> * Longer build times
>
> What have I missed?
We will lose git history if we migrate repositories; people will have to
consult the old repositories.
> Is it time for the labs repos to rejoin?
>
> We can figure out the mechanics of it after we’ve reached some kind of consensus.
+1, I strongly support bringing labs back to the main repo. We can
continue to have a separate labs directory if we want to continue to
have some distinction.
--
Andrew Gaul
http://gaul.org/