You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Brian Jeltema <bd...@gmail.com> on 2015/12/20 13:35:40 UTC
how to roll back HDP 2.1 -> HDP 2.2 upgrade
Running Ambari 2.0.2, I tried to do an upgrade to the HDP 2.2 stack. That upgrade
has been a disaster, and the cluster is totally screwed up. At this point I think my best
bet is to rollback to HDP 2.1 and try again another time.
I didn’t see a discussion of how to do a rollback. Is this documented somewhere?
If not, can someone provide guidance?
Thanks.
Brian
Re: how to roll back HDP 2.1 -> HDP 2.2 upgrade
Posted by Brian Jeltema <bd...@gmail.com>.
> On Dec 20, 2015, at 11:14 PM, Sumit Mohanty <sm...@hortonworks.com> wrote:
>
> Can you share me the link you used for the upgrade?
I used the Ambari/HDP upgrade doc at http://docs.hortonworks.com/HDPDocuments/Ambari-2.0.2.0/bk_upgrading_Ambari/content/index.html
>
> Also, is it possible for you to share the dump of the DB - feel free to mail me using my personal address?
its’ attached
>
> I want to understand what state the cluster is in from Ambari perspective. Also, it will give me an idea of how you can restore old configurations.
>
> Can you check on a node what packages are installed? That will tell you if both 2.1 and 2.2 are installed.
a ‘yum list’ shows lots of HDP-2.2.8.0 packages, but no HDP-2.1 packages other than bigtop-jsvc.x86_64
Some additional points:
I updated to 2.2.8.0 rather than the current 2.2.9.0 to be consistent with a previous dev cluster upgrade. However,
the repository setting in ambari was set to 2.2.9.0, which I didn’t notice immediately. I fixed this, but amber
is still trying to for some package installs for 2.2.9.0, I presume because of command pending in the
execution_command table.
I used the info at http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.0/bk_ambari_reference_guide/content/_how_to_customize_the_name_of_a_host.html <http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.0/bk_ambari_reference_guide/content/_how_to_customize_the_name_of_a_host.html>
to set customized names for some nodes, as you suggested in an earlier email. I only did the first 5 steps since I didn’t think
I needed to generate a different public name for each host.
>
> ________________________________________
> From: Brian Jeltema <bd...@gmail.com>
> Sent: Sunday, December 20, 2015 5:07 PM
> To: user@ambari.apache.org
> Subject: Re: how to roll back HDP 2.1 -> HDP 2.2 upgrade
>
>>
>> How large is the cluster?
>
> About 25 service nodes and 5 client nodes
>
>>
>> Do you still have the HDP-2.1 binaries deployed? HDP-2.2 uses a different packaging structure through versioned rpms - so the layout should not have overwritten the earlier installations.
>
> I followed the upgrade cookbook; I’m not clear on the state of the 2.1 binaries, or how to restore
> the original config files.
>
>>
>> At a very high level you need to
>> * Stop any running services
>> * Uninstall 2.2
>> * set the cluster version back to HDP-2.1
>> * revert the configs back to what you have for HDP-2.1 (by applying older versions)
>
> These details are what I was hoping to get information about (other than stopping the services).
>
>> * Start all services
>>
>> What errors do you see with HDP-2.2. We can also look at the errors and see if configs can be modified so that 2.2 works.
>
> Lots of different problems. Hive shell hangs. Ambari start/stop operations don’t work. HBase problems finding rows in the keystone.
> Clients that keep trying to load from repositories other than the one I have configured. So many issues that reverting
> seems like the only sensible option.
>
> Brian
>
>> ________________________________________
>> From: Brian Jeltema <bd...@gmail.com>
>> Sent: Sunday, December 20, 2015 4:35 AM
>> To: user@ambari.apache.org
>> Subject: how to roll back HDP 2.1 -> HDP 2.2 upgrade
>>
>> Running Ambari 2.0.2, I tried to do an upgrade to the HDP 2.2 stack. That upgrade
>> has been a disaster, and the cluster is totally screwed up. At this point I think my best
>> bet is to rollback to HDP 2.1 and try again another time.
>>
>> I didn’t see a discussion of how to do a rollback. Is this documented somewhere?
>> If not, can someone provide guidance?
>>
>> Thanks.
>> Brian
>
>
Re: how to roll back HDP 2.1 -> HDP 2.2 upgrade
Posted by Sumit Mohanty <sm...@hortonworks.com>.
Can you share me the link you used for the upgrade?
Also, is it possible for you to share the dump of the DB - feel free to mail me using my personal address?
I want to understand what state the cluster is in from Ambari perspective. Also, it will give me an idea of how you can restore old configurations.
Can you check on a node what packages are installed? That will tell you if both 2.1 and 2.2 are installed.
________________________________________
From: Brian Jeltema <bd...@gmail.com>
Sent: Sunday, December 20, 2015 5:07 PM
To: user@ambari.apache.org
Subject: Re: how to roll back HDP 2.1 -> HDP 2.2 upgrade
>
> How large is the cluster?
About 25 service nodes and 5 client nodes
>
> Do you still have the HDP-2.1 binaries deployed? HDP-2.2 uses a different packaging structure through versioned rpms - so the layout should not have overwritten the earlier installations.
I followed the upgrade cookbook; I’m not clear on the state of the 2.1 binaries, or how to restore
the original config files.
>
> At a very high level you need to
> * Stop any running services
> * Uninstall 2.2
> * set the cluster version back to HDP-2.1
> * revert the configs back to what you have for HDP-2.1 (by applying older versions)
These details are what I was hoping to get information about (other than stopping the services).
> * Start all services
>
> What errors do you see with HDP-2.2. We can also look at the errors and see if configs can be modified so that 2.2 works.
Lots of different problems. Hive shell hangs. Ambari start/stop operations don’t work. HBase problems finding rows in the keystone.
Clients that keep trying to load from repositories other than the one I have configured. So many issues that reverting
seems like the only sensible option.
Brian
> ________________________________________
> From: Brian Jeltema <bd...@gmail.com>
> Sent: Sunday, December 20, 2015 4:35 AM
> To: user@ambari.apache.org
> Subject: how to roll back HDP 2.1 -> HDP 2.2 upgrade
>
> Running Ambari 2.0.2, I tried to do an upgrade to the HDP 2.2 stack. That upgrade
> has been a disaster, and the cluster is totally screwed up. At this point I think my best
> bet is to rollback to HDP 2.1 and try again another time.
>
> I didn’t see a discussion of how to do a rollback. Is this documented somewhere?
> If not, can someone provide guidance?
>
> Thanks.
> Brian
Re: how to roll back HDP 2.1 -> HDP 2.2 upgrade
Posted by Brian Jeltema <bd...@gmail.com>.
>
> How large is the cluster?
About 25 service nodes and 5 client nodes
>
> Do you still have the HDP-2.1 binaries deployed? HDP-2.2 uses a different packaging structure through versioned rpms - so the layout should not have overwritten the earlier installations.
I followed the upgrade cookbook; I’m not clear on the state of the 2.1 binaries, or how to restore
the original config files.
>
> At a very high level you need to
> * Stop any running services
> * Uninstall 2.2
> * set the cluster version back to HDP-2.1
> * revert the configs back to what you have for HDP-2.1 (by applying older versions)
These details are what I was hoping to get information about (other than stopping the services).
> * Start all services
>
> What errors do you see with HDP-2.2. We can also look at the errors and see if configs can be modified so that 2.2 works.
Lots of different problems. Hive shell hangs. Ambari start/stop operations don’t work. HBase problems finding rows in the keystone.
Clients that keep trying to load from repositories other than the one I have configured. So many issues that reverting
seems like the only sensible option.
Brian
> ________________________________________
> From: Brian Jeltema <bd...@gmail.com>
> Sent: Sunday, December 20, 2015 4:35 AM
> To: user@ambari.apache.org
> Subject: how to roll back HDP 2.1 -> HDP 2.2 upgrade
>
> Running Ambari 2.0.2, I tried to do an upgrade to the HDP 2.2 stack. That upgrade
> has been a disaster, and the cluster is totally screwed up. At this point I think my best
> bet is to rollback to HDP 2.1 and try again another time.
>
> I didn’t see a discussion of how to do a rollback. Is this documented somewhere?
> If not, can someone provide guidance?
>
> Thanks.
> Brian
Re: how to roll back HDP 2.1 -> HDP 2.2 upgrade
Posted by Sumit Mohanty <sm...@hortonworks.com>.
How large is the cluster?
Do you still have the HDP-2.1 binaries deployed? HDP-2.2 uses a different packaging structure through versioned rpms - so the layout should not have overwritten the earlier installations.
At a very high level you need to
* Stop any running services
* Uninstall 2.2
* set the cluster version back to HDP-2.1
* revert the configs back to what you have for HDP-2.1 (by applying older versions)
* Start all services
What errors do you see with HDP-2.2. We can also look at the errors and see if configs can be modified so that 2.2 works.
________________________________________
From: Brian Jeltema <bd...@gmail.com>
Sent: Sunday, December 20, 2015 4:35 AM
To: user@ambari.apache.org
Subject: how to roll back HDP 2.1 -> HDP 2.2 upgrade
Running Ambari 2.0.2, I tried to do an upgrade to the HDP 2.2 stack. That upgrade
has been a disaster, and the cluster is totally screwed up. At this point I think my best
bet is to rollback to HDP 2.1 and try again another time.
I didn’t see a discussion of how to do a rollback. Is this documented somewhere?
If not, can someone provide guidance?
Thanks.
Brian