Running 0.19.2 branch in production before release

[Nathan Marz <na...@rapleaf.com>]

I would like to get the community's opinion on this. Do you think it's  
safe to run the unreleased 0.19.2 branch in production? Or do you  
recommend sticking with 0.19.1 for production use? There are some bug  
fixes in 0.19.2 which we would like to take advantage of although they  
are not blocking issues for us. 

Re: Running 0.19.2 branch in production before release

[Jimmy Wan <ji...@indeed.com>]

Has anyone setup this dual cluster approach and come up with a
complete scheme for being able to test a new hadoop revision and
migrate data from one datanode revision to the next? It would be
useful if someone could share their list of gotchas that have been
experienced..

In the past, pretty much every attempt to upgrade Hadoop DataNodes via
the documented (but outdated?) approaches mentioned in the FAQs and
wikis has not worked well for me. That's why I'm still on 0.17.2.1.

Jimmy Wan

On Fri, Mar 6, 2009 at 00:17, Aaron Kimball <aa...@cloudera.com> wrote:
> Right, there's no sense in freezing your Hadoop version forever :)
>
> But if you're an ops team tasked with keeping a production cluster running
> 24/7, running on 0.19 (or even more daringly, TRUNK) is not something that I
> would consider a Best Practice. Ideally you'll be able to carve out some
> spare capacity (maybe 3--5 nodes) to use as a staging cluster that runs on
> 0.19 or TRUNK that you can use to evaluate the next version. Then when you
> are convinced that it's stable, and your staging cluster passes your
> internal tests (e.g., running test versions of your critical nightly jobs
> successfully), you can move that to production.
>
> - Aaron
>
>
> On Thu, Mar 5, 2009 at 2:33 AM, Steve Loughran <st...@apache.org> wrote:
>
>> Aaron Kimball wrote:
>>
>>> I recommend 0.18.3 for production use and avoid the 19 branch entirely. If
>>> your priority is stability, then stay a full minor version behind, not
>>> just
>>> a revision.
>>>
>>
>> Of course, if everyone stays that far behind, they don't get to find the
>> bugs for other people.
>>
>> * If you play with the latest releases early, while they are in the beta
>> phase -you will encounter the problems specific to your
>> applications/datacentres, and get them fixed fast.
>>
>> * If you work with stuff further back you get stability, but not only are
>> you behind on features, you can't be sure that all "fixes" that matter to
>> you get pushed back.
>>
>> * If you plan on making changes, of adding features, get onto SVN_HEAD
>>
>> * If you want to catch changes being made that break your site, SVN_HEAD.
>> Better yet, have a private Hudson server checking out SVN_HEAD hadoop *then*
>> building and testing your app against it.
>>
>> Normally I work with stable releases of things I dont depend on, and
>> SVN_HEAD of OSS stuff whose code I have any intent to change; there is a
>> price -merge time, the odd change breaking your code- but you get to make
>> changes that help you long term.
>>
>> Where Hadoop is different is that it is a filesystem, and you don't want to
>> hit bugs that delete files that matter. I'm only bringing up transient
>> clusters on VMs, pulling in data from elsewhere, so this isn't an issue. All
>> that remains is changing APIs.
>>
>> -Steve
>>
>

Re: Running 0.19.2 branch in production before release

[Aaron Kimball <aa...@cloudera.com>]

Right, there's no sense in freezing your Hadoop version forever :)

But if you're an ops team tasked with keeping a production cluster running
24/7, running on 0.19 (or even more daringly, TRUNK) is not something that I
would consider a Best Practice. Ideally you'll be able to carve out some
spare capacity (maybe 3--5 nodes) to use as a staging cluster that runs on
0.19 or TRUNK that you can use to evaluate the next version. Then when you
are convinced that it's stable, and your staging cluster passes your
internal tests (e.g., running test versions of your critical nightly jobs
successfully), you can move that to production.

- Aaron


On Thu, Mar 5, 2009 at 2:33 AM, Steve Loughran <st...@apache.org> wrote:

> Aaron Kimball wrote:
>
>> I recommend 0.18.3 for production use and avoid the 19 branch entirely. If
>> your priority is stability, then stay a full minor version behind, not
>> just
>> a revision.
>>
>
> Of course, if everyone stays that far behind, they don't get to find the
> bugs for other people.
>
> * If you play with the latest releases early, while they are in the beta
> phase -you will encounter the problems specific to your
> applications/datacentres, and get them fixed fast.
>
> * If you work with stuff further back you get stability, but not only are
> you behind on features, you can't be sure that all "fixes" that matter to
> you get pushed back.
>
> * If you plan on making changes, of adding features, get onto SVN_HEAD
>
> * If you want to catch changes being made that break your site, SVN_HEAD.
> Better yet, have a private Hudson server checking out SVN_HEAD hadoop *then*
> building and testing your app against it.
>
> Normally I work with stable releases of things I dont depend on, and
> SVN_HEAD of OSS stuff whose code I have any intent to change; there is a
> price -merge time, the odd change breaking your code- but you get to make
> changes that help you long term.
>
> Where Hadoop is different is that it is a filesystem, and you don't want to
> hit bugs that delete files that matter. I'm only bringing up transient
> clusters on VMs, pulling in data from elsewhere, so this isn't an issue. All
> that remains is changing APIs.
>
> -Steve
>

Re: Running 0.19.2 branch in production before release

[Steve Loughran <st...@apache.org>]

Aaron Kimball wrote:
> I recommend 0.18.3 for production use and avoid the 19 branch entirely. If
> your priority is stability, then stay a full minor version behind, not just
> a revision.

Of course, if everyone stays that far behind, they don't get to find the 
bugs for other people.

* If you play with the latest releases early, while they are in the beta 
phase -you will encounter the problems specific to your 
applications/datacentres, and get them fixed fast.

* If you work with stuff further back you get stability, but not only 
are you behind on features, you can't be sure that all "fixes" that 
matter to you get pushed back.

* If you plan on making changes, of adding features, get onto SVN_HEAD

* If you want to catch changes being made that break your site, 
SVN_HEAD. Better yet, have a private Hudson server checking out SVN_HEAD 
hadoop *then* building and testing your app against it.

Normally I work with stable releases of things I dont depend on, and 
SVN_HEAD of OSS stuff whose code I have any intent to change; there is a 
price -merge time, the odd change breaking your code- but you get to 
make changes that help you long term.

Where Hadoop is different is that it is a filesystem, and you don't want 
to hit bugs that delete files that matter. I'm only bringing up 
transient clusters on VMs, pulling in data from elsewhere, so this isn't 
an issue. All that remains is changing APIs.

-Steve

Re: Running 0.19.2 branch in production before release

[Aaron Kimball <aa...@cloudera.com>]

I recommend 0.18.3 for production use and avoid the 19 branch entirely. If
your priority is stability, then stay a full minor version behind, not just
a revision.

- Aaron


On Tue, Mar 3, 2009 at 5:28 PM, Nathan Marz <na...@rapleaf.com> wrote:

> I would like to get the community's opinion on this. Do you think it's safe
> to run the unreleased 0.19.2 branch in production? Or do you recommend
> sticking with 0.19.1 for production use? There are some bug fixes in 0.19.2
> which we would like to take advantage of although they are not blocking
> issues for us.
>