Contrib for new services

[Andrei Savu <sa...@gmail.com>]

Hi,

Each new service we add to Whirr makes the product harder to maintain
and develop. At this stage, as we've discussed in the past, it's
really important to focus on evolving the core and improving the
underlying abstractions. Unfortunately at the same time we should keep
in mind that Whirr is more useful only if it supports more services.

How should we move forward? Refuse to add new services while we
improve the core (doesn't sound like a good idea)? Create something
like contrib/services for partially supported services so that we can
have a clear distinction and decrease the maintenance workload (I
would go for this one)?

Cheers,

-- Andrei Savu / andreisavu.ro

Re: Contrib for new services

[Andrei Savu <sa...@gmail.com>]

On Wed, Jun 1, 2011 at 9:31 PM, Patrick Hunt <ph...@cloudera.com> wrote:
> Is something like thrift a good example here for comparison purposes?
> http://svn.apache.org/viewvc/thrift/trunk/lib/

I like this page on the Thrift wiki:
http://wiki.apache.org/thrift/LibraryFeatures?action=show&redirect=LanguageSupport

I guess we will need something similar.

>
> As suggested earlier I think streamlining the release process and
> making more frequent releases would be a plus.

+1

>
> Patrick

Thanks,
Andrei

Re: Contrib for new services

[Tom White <to...@gmail.com>]

I agree that we should keep the services in the same release. At the
time of release we could run all the service integration tests and
record which ones passed or failed. Failed tests wouldn't necessarily
block a release, but if a service kept failing this would alert us to
where more attention was needed.

Cheers,
Tom

On Wed, Jun 1, 2011 at 11:31 AM, Patrick Hunt <ph...@cloudera.com> wrote:
> Is something like thrift a good example here for comparison purposes?
> http://svn.apache.org/viewvc/thrift/trunk/lib/
> They have a large number of different language implementations, a few
> which have all the features, and many which support a subset (also in
> terms of quality). As was suggested earlier in this thread we might
> include some specific details in the docs for each of the services
> detailing their level of support for various features. I think this
> makes more sense than adding complexity to the build/runtime, also my
> experience with the contrib approach is not favorable either. At some
> later time we could work on separating things out, but for now it
> would be more of a hindrance than a help.
>
> As suggested earlier I think streamlining the release process and
> making more frequent releases would be a plus.
>
> Patrick
>
> On Wed, Jun 1, 2011 at 11:14 AM, Andrei Savu <sa...@gmail.com> wrote:
>> I understand your concerns and also I see that's not trivial to test even a
>> single version of a service released together with the whirr core.
>>
>> I guess it's reasonable to postpone this decision as long as we can still
>> maintain all the services.
>>  On Jun 1, 2011 8:04 PM, "Adrian Cole" <ad...@jclouds.org> wrote:
>>> I like the *idea* of having separate releases for services and core,
>>> but while sands are shifting it might be error-prone maintenance and
>>> build dependencies.
>>>
>>> How about if we work to increase the frequency of releases, including
>>> a contrib build, until the core apis solidify? I think there's still
>>> more to gain from minds on coding at this point vs managing dozens of
>>> individual release projects. Moreover, there will be less "my
>>> classpath doesn't work" issues from onset if versions are consistent,
>>> especially knowing the classpath will near certainly be incompatible
>>> between core and services for another release or so.
>>>
>>> I don't mean to discourage, rather to highlight that this is a lot of
>>> effort and will likely increase the errors users will encounter for
>>> the perception of more scalable deployment. I'm of the opinion that
>>> this will be scalable once core is a bit less shakey.
>>>
>>> my 2p.
>>> -a
>>>
>>> On Wed, Jun 1, 2011 at 8:25 AM, Andrei Savu <sa...@gmail.com> wrote:
>>>> On Wed, Jun 1, 2011 at 5:55 PM, Tom White <to...@gmail.com> wrote:
>>>>> Andrei, what would the impact be of your proposal on releases? Do we
>>>>> only ship services that compile (and pass tests) at the time of
>>>>> release? Or are you suggesting a separate release of services?
>>>>
>>>> I really like this idea of having different release cycles for
>>>> services and core. It should give us enough flexibility to evolve the
>>>> core while maintaining just a subset of services in sync. We could
>>>> create a version numbering convention in order to avoid confusions
>>>> (e.g. Whirr Hadoop 0.5.x should always work with Whirr Core 0.5.y).
>>>>
>>>> It sounds like we will need some sort of tool for installing supported
>>>> services while using the CLI (elasticsearch is using something like
>>>> this). When using Whirr as library maven should be enough to handle
>>>> multiple versions and releases.
>>>>
>>>> +1 for having different releases for core and services as part of the
>>>> same project. This should help as build a larger community, be more
>>>> dynamic and increase the number of supported services.
>>>>
>>
>

Re: Contrib for new services

[Patrick Hunt <ph...@cloudera.com>]

Is something like thrift a good example here for comparison purposes?
http://svn.apache.org/viewvc/thrift/trunk/lib/
They have a large number of different language implementations, a few
which have all the features, and many which support a subset (also in
terms of quality). As was suggested earlier in this thread we might
include some specific details in the docs for each of the services
detailing their level of support for various features. I think this
makes more sense than adding complexity to the build/runtime, also my
experience with the contrib approach is not favorable either. At some
later time we could work on separating things out, but for now it
would be more of a hindrance than a help.

As suggested earlier I think streamlining the release process and
making more frequent releases would be a plus.

Patrick

On Wed, Jun 1, 2011 at 11:14 AM, Andrei Savu <sa...@gmail.com> wrote:
> I understand your concerns and also I see that's not trivial to test even a
> single version of a service released together with the whirr core.
>
> I guess it's reasonable to postpone this decision as long as we can still
> maintain all the services.
>  On Jun 1, 2011 8:04 PM, "Adrian Cole" <ad...@jclouds.org> wrote:
>> I like the *idea* of having separate releases for services and core,
>> but while sands are shifting it might be error-prone maintenance and
>> build dependencies.
>>
>> How about if we work to increase the frequency of releases, including
>> a contrib build, until the core apis solidify? I think there's still
>> more to gain from minds on coding at this point vs managing dozens of
>> individual release projects. Moreover, there will be less "my
>> classpath doesn't work" issues from onset if versions are consistent,
>> especially knowing the classpath will near certainly be incompatible
>> between core and services for another release or so.
>>
>> I don't mean to discourage, rather to highlight that this is a lot of
>> effort and will likely increase the errors users will encounter for
>> the perception of more scalable deployment. I'm of the opinion that
>> this will be scalable once core is a bit less shakey.
>>
>> my 2p.
>> -a
>>
>> On Wed, Jun 1, 2011 at 8:25 AM, Andrei Savu <sa...@gmail.com> wrote:
>>> On Wed, Jun 1, 2011 at 5:55 PM, Tom White <to...@gmail.com> wrote:
>>>> Andrei, what would the impact be of your proposal on releases? Do we
>>>> only ship services that compile (and pass tests) at the time of
>>>> release? Or are you suggesting a separate release of services?
>>>
>>> I really like this idea of having different release cycles for
>>> services and core. It should give us enough flexibility to evolve the
>>> core while maintaining just a subset of services in sync. We could
>>> create a version numbering convention in order to avoid confusions
>>> (e.g. Whirr Hadoop 0.5.x should always work with Whirr Core 0.5.y).
>>>
>>> It sounds like we will need some sort of tool for installing supported
>>> services while using the CLI (elasticsearch is using something like
>>> this). When using Whirr as library maven should be enough to handle
>>> multiple versions and releases.
>>>
>>> +1 for having different releases for core and services as part of the
>>> same project. This should help as build a larger community, be more
>>> dynamic and increase the number of supported services.
>>>
>

Re: Contrib for new services

[Andrei Savu <sa...@gmail.com>]

I understand your concerns and also I see that's not trivial to test even a
single version of a service released together with the whirr core.

I guess it's reasonable to postpone this decision as long as we can still
maintain all the services.
 On Jun 1, 2011 8:04 PM, "Adrian Cole" <ad...@jclouds.org> wrote:
> I like the *idea* of having separate releases for services and core,
> but while sands are shifting it might be error-prone maintenance and
> build dependencies.
>
> How about if we work to increase the frequency of releases, including
> a contrib build, until the core apis solidify? I think there's still
> more to gain from minds on coding at this point vs managing dozens of
> individual release projects. Moreover, there will be less "my
> classpath doesn't work" issues from onset if versions are consistent,
> especially knowing the classpath will near certainly be incompatible
> between core and services for another release or so.
>
> I don't mean to discourage, rather to highlight that this is a lot of
> effort and will likely increase the errors users will encounter for
> the perception of more scalable deployment. I'm of the opinion that
> this will be scalable once core is a bit less shakey.
>
> my 2p.
> -a
>
> On Wed, Jun 1, 2011 at 8:25 AM, Andrei Savu <sa...@gmail.com> wrote:
>> On Wed, Jun 1, 2011 at 5:55 PM, Tom White <to...@gmail.com> wrote:
>>> Andrei, what would the impact be of your proposal on releases? Do we
>>> only ship services that compile (and pass tests) at the time of
>>> release? Or are you suggesting a separate release of services?
>>
>> I really like this idea of having different release cycles for
>> services and core. It should give us enough flexibility to evolve the
>> core while maintaining just a subset of services in sync. We could
>> create a version numbering convention in order to avoid confusions
>> (e.g. Whirr Hadoop 0.5.x should always work with Whirr Core 0.5.y).
>>
>> It sounds like we will need some sort of tool for installing supported
>> services while using the CLI (elasticsearch is using something like
>> this). When using Whirr as library maven should be enough to handle
>> multiple versions and releases.
>>
>> +1 for having different releases for core and services as part of the
>> same project. This should help as build a larger community, be more
>> dynamic and increase the number of supported services.
>>

Re: Contrib for new services

[Adrian Cole <ad...@jclouds.org>]

I like the *idea* of having separate releases for services and core,
but while sands are shifting it might be error-prone maintenance and
build dependencies.

How about if we work to increase the frequency of releases, including
a contrib build, until the core apis solidify?   I think there's still
more to gain from minds on coding at this point vs managing dozens of
individual release projects.  Moreover, there will be less "my
classpath doesn't work" issues from onset if versions are consistent,
especially knowing the classpath will near certainly be incompatible
between core and services for another release or so.

I don't mean to discourage, rather to highlight that this is a lot of
effort and will likely increase the errors users will encounter for
the perception of more scalable deployment.  I'm of the opinion that
this will be scalable once core is a bit less shakey.

my 2p.
-a

On Wed, Jun 1, 2011 at 8:25 AM, Andrei Savu <sa...@gmail.com> wrote:
> On Wed, Jun 1, 2011 at 5:55 PM, Tom White <to...@gmail.com> wrote:
>> Andrei, what would the impact be of your proposal on releases? Do we
>> only ship services that compile (and pass tests) at the time of
>> release? Or are you suggesting a separate release of services?
>
> I really like this idea of having different release cycles for
> services and core. It should give us enough flexibility to evolve the
> core while maintaining just a subset of services in sync. We could
> create a version numbering convention in order to avoid confusions
> (e.g. Whirr Hadoop 0.5.x should always work with Whirr Core 0.5.y).
>
> It sounds like we will need some sort of tool for installing supported
> services while using the CLI (elasticsearch is using something like
> this). When using Whirr as library maven should be enough to handle
> multiple versions and releases.
>
> +1 for having different releases for core and services as part of the
> same project. This should help as build a larger community, be more
> dynamic and increase the number of supported services.
>

Re: Contrib for new services

[Andrei Savu <sa...@gmail.com>]

On Wed, Jun 1, 2011 at 5:55 PM, Tom White <to...@gmail.com> wrote:
> Andrei, what would the impact be of your proposal on releases? Do we
> only ship services that compile (and pass tests) at the time of
> release? Or are you suggesting a separate release of services?

I really like this idea of having different release cycles for
services and core. It should give us enough flexibility to evolve the
core while maintaining just a subset of services in sync. We could
create a version numbering convention in order to avoid confusions
(e.g. Whirr Hadoop 0.5.x should always work with Whirr Core 0.5.y).

It sounds like we will need some sort of tool for installing supported
services while using the CLI (elasticsearch is using something like
this). When using Whirr as library maven should be enough to handle
multiple versions and releases.

+1 for having different releases for core and services as part of the
same project. This should help as build a larger community, be more
dynamic and increase the number of supported services.

Re: Contrib for new services

[Tom White <to...@gmail.com>]

There's a balance between compatibility and introducing new features.
As a young project, my feeling is that it is OK for Whirr to break
compatibility between releases - until we reach version 1.0 (then it
won't be allowed except between major version numbers). The fact that
most (all?) of the Whirr service implementations live in the same
tree, and that there are not many of them, makes this reasonable for
users, since we make sure they get updated and tested. This is how
we've been operating so far. However, this doesn't scale well so it's
good to think about how we'd like to operate when there are more
services.

Andrei, what would the impact be of your proposal on releases? Do we
only ship services that compile (and pass tests) at the time of
release? Or are you suggesting a separate release of services?

Tom

On Mon, May 30, 2011 at 1:59 PM, Andrei Savu <sa...@gmail.com> wrote:
> Tom, what do you think? Would it be useful to have some sort of
> Manifest file for each service we add so that we can have the freedom
> to evolve the core without being forced to update all the services?
>
> -- Andrei
>
> On Mon, May 30, 2011 at 7:36 PM, Adrian Cole <ad...@jclouds.org> wrote:
>> On Mon, May 30, 2011 at 9:28 AM, Andrei Savu <sa...@gmail.com> wrote:
>>> On Mon, May 30, 2011 at 7:23 PM, Adrian Cole <ad...@jclouds.org> wrote:
>>>> I like the sandbox concept, as this is more about them being
>>>> unfinished then something that we wouldn't pull in.
>>>
>>> Yes and we should be able to change the core without having to update
>>> all the services to support the new features (e.g rolling restarts,
>>> reconfiguration).
>>>
>>> Should we add something like a supported list of Whirr capabilities
>>> for each service?
>> +1
>>
>

Re: Contrib for new services

[Andrei Savu <sa...@gmail.com>]

Tom, what do you think? Would it be useful to have some sort of
Manifest file for each service we add so that we can have the freedom
to evolve the core without being forced to update all the services?

-- Andrei

On Mon, May 30, 2011 at 7:36 PM, Adrian Cole <ad...@jclouds.org> wrote:
> On Mon, May 30, 2011 at 9:28 AM, Andrei Savu <sa...@gmail.com> wrote:
>> On Mon, May 30, 2011 at 7:23 PM, Adrian Cole <ad...@jclouds.org> wrote:
>>> I like the sandbox concept, as this is more about them being
>>> unfinished then something that we wouldn't pull in.
>>
>> Yes and we should be able to change the core without having to update
>> all the services to support the new features (e.g rolling restarts,
>> reconfiguration).
>>
>> Should we add something like a supported list of Whirr capabilities
>> for each service?
> +1
>

Re: Contrib for new services

[Adrian Cole <ad...@jclouds.org>]

On Mon, May 30, 2011 at 9:28 AM, Andrei Savu <sa...@gmail.com> wrote:
> On Mon, May 30, 2011 at 7:23 PM, Adrian Cole <ad...@jclouds.org> wrote:
>> I like the sandbox concept, as this is more about them being
>> unfinished then something that we wouldn't pull in.
>
> Yes and we should be able to change the core without having to update
> all the services to support the new features (e.g rolling restarts,
> reconfiguration).
>
> Should we add something like a supported list of Whirr capabilities
> for each service?
+1

Re: Contrib for new services

["Edward J. Yoon" <ed...@apache.org>]

+1

On Tue, May 31, 2011 at 1:28 AM, Andrei Savu <sa...@gmail.com> wrote:
> On Mon, May 30, 2011 at 7:23 PM, Adrian Cole <ad...@jclouds.org> wrote:
>> I like the sandbox concept, as this is more about them being
>> unfinished then something that we wouldn't pull in.
>
> Yes and we should be able to change the core without having to update
> all the services to support the new features (e.g rolling restarts,
> reconfiguration).
>
> Should we add something like a supported list of Whirr capabilities
> for each service?
>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Re: Contrib for new services

[Andrei Savu <sa...@gmail.com>]

On Mon, May 30, 2011 at 7:23 PM, Adrian Cole <ad...@jclouds.org> wrote:
> I like the sandbox concept, as this is more about them being
> unfinished then something that we wouldn't pull in.

Yes and we should be able to change the core without having to update
all the services to support the new features (e.g rolling restarts,
reconfiguration).

Should we add something like a supported list of Whirr capabilities
for each service?

Re: Contrib for new services

[Adrian Cole <ad...@jclouds.org>]

I like the sandbox concept, as this is more about them being
unfinished then something that we wouldn't pull in.

-a

On Mon, May 30, 2011 at 9:20 AM, Andrei Savu <sa...@gmail.com> wrote:
> Hi,
>
> Each new service we add to Whirr makes the product harder to maintain
> and develop. At this stage, as we've discussed in the past, it's
> really important to focus on evolving the core and improving the
> underlying abstractions. Unfortunately at the same time we should keep
> in mind that Whirr is more useful only if it supports more services.
>
> How should we move forward? Refuse to add new services while we
> improve the core (doesn't sound like a good idea)? Create something
> like contrib/services for partially supported services so that we can
> have a clear distinction and decrease the maintenance workload (I
> would go for this one)?
>
> Cheers,
>
> -- Andrei Savu / andreisavu.ro
>