You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Di Li <di...@ca.ibm.com> on 2015/08/13 15:13:03 UTC
Review Request 37437: AMBARI-12751: MapRed history server is shown as
stopped after downgrade
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37437/
-----------------------------------------------------------
Review request for Ambari and Alejandro Fernandez.
Bugs: AMBARI-12751
https://issues.apache.org/jira/browse/AMBARI-12751
Repository: ambari
Description
-------
RU precheck requires "mapreduce.jobhistory.recovery.store.class" property to be set in the mapred-site.xml with value "org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService"
With this value set, the HDP 2.2 level MapRed history server can not start. It shows an err about class org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService does not exist.
When downgrading, the mapred properties added to pass the RU precheck needs to be removed.
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml f6823c8
Diff: https://reviews.apache.org/r/37437/diff/
Testing
-------
manually patch cluster to run rolling upgrade, after it passes the Core Master step (which includes upgrading the MapRed2 History server), run downgrade, verify the MapRed2 history server is still running once the downgrade is done, the three mapred properties added to satisfy the ru precheck are removed.
Thanks,
Di Li
Re: Review Request 37437: AMBARI-12751: MapRed history server is
shown as stopped after downgrade
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On Aug. 13, 2015, 5:31 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml, line 580
> > <https://reviews.apache.org/r/37437/diff/1/?file=1039334#file1039334line580>
> >
> > Di, can you explain this some more?
> > This is performing an RU from HDP 2.2 -> 2.3, and that property exists in HDP 2.3, so due to config merging, it means that if the property does not exist or the user has not modified it, it will be set to *...HistoryServerLeveldbStateStoreService.
> > Note: if the user does modify it, their value will be preserved.
> >
> > From your description, it sounds like the property does not work in HDP 2.2, so why should there be a PreCheck for it at all?
>
> Di Li wrote:
> hello Alejandro,
>
> AMBARI-12276 added "MapReduce2JobHistoryStatePreservingCheck" class, a RU precheck class that checks whether "mapreduce.jobhistory.recovery.enable", "mapreduce.jobhistory.recovery.store.class", and "mapreduce.jobhistory.recovery.store.leveldb.path" are set in the mapred-site.xml.
>
> In my tests, I updated an HDP 2.2 cluster to HDP 2.3, then I clicked downgrade instead of Finalize. The downgrade kept the three properties in the configuration for the MapRed2 history server. Because the HDP 2.2 version of the MapRed history server does not have the class, it failed to start and shown as stopped post downgrade.
>
> So my thinking was that I would modify the upgrade-2.3.xml to remove the three mapred properties during the downgrade. If users want to perform another round of rolling upgrade, they will need to manually add the three properties back to the mapred-site.xml
So on the brand new HDP 2.2 cluster, the "mapreduce.jobhistory.recovery.store.class" property did not exist by default, so you had to add it before starting RU to get the PreCheck to pass?
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37437/#review95291
-----------------------------------------------------------
On Aug. 13, 2015, 1:13 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37437/
> -----------------------------------------------------------
>
> (Updated Aug. 13, 2015, 1:13 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Bugs: AMBARI-12751
> https://issues.apache.org/jira/browse/AMBARI-12751
>
>
> Repository: ambari
>
>
> Description
> -------
>
> RU precheck requires "mapreduce.jobhistory.recovery.store.class" property to be set in the mapred-site.xml with value "org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService"
>
> With this value set, the HDP 2.2 level MapRed history server can not start. It shows an err about class org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService does not exist.
>
> When downgrading, the mapred properties added to pass the RU precheck needs to be removed.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml f6823c8
>
> Diff: https://reviews.apache.org/r/37437/diff/
>
>
> Testing
> -------
>
> manually patch cluster to run rolling upgrade, after it passes the Core Master step (which includes upgrading the MapRed2 History server), run downgrade, verify the MapRed2 history server is still running once the downgrade is done, the three mapred properties added to satisfy the ru precheck are removed.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 37437: AMBARI-12751: MapRed history server is
shown as stopped after downgrade
Posted by Di Li <di...@ca.ibm.com>.
> On Aug. 13, 2015, 5:31 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml, line 580
> > <https://reviews.apache.org/r/37437/diff/1/?file=1039334#file1039334line580>
> >
> > Di, can you explain this some more?
> > This is performing an RU from HDP 2.2 -> 2.3, and that property exists in HDP 2.3, so due to config merging, it means that if the property does not exist or the user has not modified it, it will be set to *...HistoryServerLeveldbStateStoreService.
> > Note: if the user does modify it, their value will be preserved.
> >
> > From your description, it sounds like the property does not work in HDP 2.2, so why should there be a PreCheck for it at all?
hello Alejandro,
AMBARI-12276 added "MapReduce2JobHistoryStatePreservingCheck" class, a RU precheck class that checks whether "mapreduce.jobhistory.recovery.enable", "mapreduce.jobhistory.recovery.store.class", and "mapreduce.jobhistory.recovery.store.leveldb.path" are set in the mapred-site.xml.
In my tests, I updated an HDP 2.2 cluster to HDP 2.3, then I clicked downgrade instead of Finalize. The downgrade kept the three properties in the configuration for the MapRed2 history server. Because the HDP 2.2 version of the MapRed history server does not have the class, it failed to start and shown as stopped post downgrade.
So my thinking was that I would modify the upgrade-2.3.xml to remove the three mapred properties during the downgrade. If users want to perform another round of rolling upgrade, they will need to manually add the three properties back to the mapred-site.xml
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37437/#review95291
-----------------------------------------------------------
On Aug. 13, 2015, 1:13 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37437/
> -----------------------------------------------------------
>
> (Updated Aug. 13, 2015, 1:13 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Bugs: AMBARI-12751
> https://issues.apache.org/jira/browse/AMBARI-12751
>
>
> Repository: ambari
>
>
> Description
> -------
>
> RU precheck requires "mapreduce.jobhistory.recovery.store.class" property to be set in the mapred-site.xml with value "org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService"
>
> With this value set, the HDP 2.2 level MapRed history server can not start. It shows an err about class org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService does not exist.
>
> When downgrading, the mapred properties added to pass the RU precheck needs to be removed.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml f6823c8
>
> Diff: https://reviews.apache.org/r/37437/diff/
>
>
> Testing
> -------
>
> manually patch cluster to run rolling upgrade, after it passes the Core Master step (which includes upgrading the MapRed2 History server), run downgrade, verify the MapRed2 history server is still running once the downgrade is done, the three mapred properties added to satisfy the ru precheck are removed.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 37437: AMBARI-12751: MapRed history server is
shown as stopped after downgrade
Posted by Di Li <di...@ca.ibm.com>.
> On Aug. 13, 2015, 5:31 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml, line 580
> > <https://reviews.apache.org/r/37437/diff/1/?file=1039334#file1039334line580>
> >
> > Di, can you explain this some more?
> > This is performing an RU from HDP 2.2 -> 2.3, and that property exists in HDP 2.3, so due to config merging, it means that if the property does not exist or the user has not modified it, it will be set to *...HistoryServerLeveldbStateStoreService.
> > Note: if the user does modify it, their value will be preserved.
> >
> > From your description, it sounds like the property does not work in HDP 2.2, so why should there be a PreCheck for it at all?
>
> Di Li wrote:
> hello Alejandro,
>
> AMBARI-12276 added "MapReduce2JobHistoryStatePreservingCheck" class, a RU precheck class that checks whether "mapreduce.jobhistory.recovery.enable", "mapreduce.jobhistory.recovery.store.class", and "mapreduce.jobhistory.recovery.store.leveldb.path" are set in the mapred-site.xml.
>
> In my tests, I updated an HDP 2.2 cluster to HDP 2.3, then I clicked downgrade instead of Finalize. The downgrade kept the three properties in the configuration for the MapRed2 history server. Because the HDP 2.2 version of the MapRed history server does not have the class, it failed to start and shown as stopped post downgrade.
>
> So my thinking was that I would modify the upgrade-2.3.xml to remove the three mapred properties during the downgrade. If users want to perform another round of rolling upgrade, they will need to manually add the three properties back to the mapred-site.xml
>
> Alejandro Fernandez wrote:
> So on the brand new HDP 2.2 cluster, the "mapreduce.jobhistory.recovery.store.class" property did not exist by default, so you had to add it before starting RU to get the PreCheck to pass?
hello Alenjandro,
yes, in my test, I had to manually added the three mapred properties to the "Custom mapred-site" section before kicking off the rolling upgrade. Otherwise, RU precheck fails with err about the mapred props do not exist.
I am not sure if this should have been done differently as in using the upgrade-2.3.xml to insert the properties on the fly during the rolling upgrade.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37437/#review95291
-----------------------------------------------------------
On Aug. 13, 2015, 1:13 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37437/
> -----------------------------------------------------------
>
> (Updated Aug. 13, 2015, 1:13 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Bugs: AMBARI-12751
> https://issues.apache.org/jira/browse/AMBARI-12751
>
>
> Repository: ambari
>
>
> Description
> -------
>
> RU precheck requires "mapreduce.jobhistory.recovery.store.class" property to be set in the mapred-site.xml with value "org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService"
>
> With this value set, the HDP 2.2 level MapRed history server can not start. It shows an err about class org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService does not exist.
>
> When downgrading, the mapred properties added to pass the RU precheck needs to be removed.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml f6823c8
>
> Diff: https://reviews.apache.org/r/37437/diff/
>
>
> Testing
> -------
>
> manually patch cluster to run rolling upgrade, after it passes the Core Master step (which includes upgrading the MapRed2 History server), run downgrade, verify the MapRed2 history server is still running once the downgrade is done, the three mapred properties added to satisfy the ru precheck are removed.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 37437: AMBARI-12751: MapRed history server is
shown as stopped after downgrade
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On Aug. 13, 2015, 5:31 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml, line 580
> > <https://reviews.apache.org/r/37437/diff/1/?file=1039334#file1039334line580>
> >
> > Di, can you explain this some more?
> > This is performing an RU from HDP 2.2 -> 2.3, and that property exists in HDP 2.3, so due to config merging, it means that if the property does not exist or the user has not modified it, it will be set to *...HistoryServerLeveldbStateStoreService.
> > Note: if the user does modify it, their value will be preserved.
> >
> > From your description, it sounds like the property does not work in HDP 2.2, so why should there be a PreCheck for it at all?
>
> Di Li wrote:
> hello Alejandro,
>
> AMBARI-12276 added "MapReduce2JobHistoryStatePreservingCheck" class, a RU precheck class that checks whether "mapreduce.jobhistory.recovery.enable", "mapreduce.jobhistory.recovery.store.class", and "mapreduce.jobhistory.recovery.store.leveldb.path" are set in the mapred-site.xml.
>
> In my tests, I updated an HDP 2.2 cluster to HDP 2.3, then I clicked downgrade instead of Finalize. The downgrade kept the three properties in the configuration for the MapRed2 history server. Because the HDP 2.2 version of the MapRed history server does not have the class, it failed to start and shown as stopped post downgrade.
>
> So my thinking was that I would modify the upgrade-2.3.xml to remove the three mapred properties during the downgrade. If users want to perform another round of rolling upgrade, they will need to manually add the three properties back to the mapred-site.xml
>
> Alejandro Fernandez wrote:
> So on the brand new HDP 2.2 cluster, the "mapreduce.jobhistory.recovery.store.class" property did not exist by default, so you had to add it before starting RU to get the PreCheck to pass?
>
> Di Li wrote:
> hello Alenjandro,
>
> yes, in my test, I had to manually added the three mapred properties to the "Custom mapred-site" section before kicking off the rolling upgrade. Otherwise, RU precheck fails with err about the mapred props do not exist.
>
> I am not sure if this should have been done differently as in using the upgrade-2.3.xml to insert the properties on the fly during the rolling upgrade.
Sounds like we shouldn't have that Pre-Check at all, since it has to be valid in the current stack, which is still HDP 2.2. What should be done, if not already, is that those properties need to be added as part of performing an RU from HDP 2.2 -> 2.3. Since they are already in the definition of HDP 2.3, I believe that they should already be added during RU, and then reverted during a Downgrade.
If not, you can try configuring them in the configurations section for HISTORYSERVER.
Please include Nate Cole in the code review.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37437/#review95291
-----------------------------------------------------------
On Aug. 13, 2015, 1:13 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37437/
> -----------------------------------------------------------
>
> (Updated Aug. 13, 2015, 1:13 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Bugs: AMBARI-12751
> https://issues.apache.org/jira/browse/AMBARI-12751
>
>
> Repository: ambari
>
>
> Description
> -------
>
> RU precheck requires "mapreduce.jobhistory.recovery.store.class" property to be set in the mapred-site.xml with value "org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService"
>
> With this value set, the HDP 2.2 level MapRed history server can not start. It shows an err about class org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService does not exist.
>
> When downgrading, the mapred properties added to pass the RU precheck needs to be removed.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml f6823c8
>
> Diff: https://reviews.apache.org/r/37437/diff/
>
>
> Testing
> -------
>
> manually patch cluster to run rolling upgrade, after it passes the Core Master step (which includes upgrading the MapRed2 History server), run downgrade, verify the MapRed2 history server is still running once the downgrade is done, the three mapred properties added to satisfy the ru precheck are removed.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 37437: AMBARI-12751: MapRed history server is
shown as stopped after downgrade
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37437/#review95291
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml (line 580)
<https://reviews.apache.org/r/37437/#comment150154>
Di, can you explain this some more?
This is performing an RU from HDP 2.2 -> 2.3, and that property exists in HDP 2.3, so due to config merging, it means that if the property does not exist or the user has not modified it, it will be set to *...HistoryServerLeveldbStateStoreService.
Note: if the user does modify it, their value will be preserved.
From your description, it sounds like the property does not work in HDP 2.2, so why should there be a PreCheck for it at all?
- Alejandro Fernandez
On Aug. 13, 2015, 1:13 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37437/
> -----------------------------------------------------------
>
> (Updated Aug. 13, 2015, 1:13 p.m.)
>
>
> Review request for Ambari and Alejandro Fernandez.
>
>
> Bugs: AMBARI-12751
> https://issues.apache.org/jira/browse/AMBARI-12751
>
>
> Repository: ambari
>
>
> Description
> -------
>
> RU precheck requires "mapreduce.jobhistory.recovery.store.class" property to be set in the mapred-site.xml with value "org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService"
>
> With this value set, the HDP 2.2 level MapRed history server can not start. It shows an err about class org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService does not exist.
>
> When downgrading, the mapred properties added to pass the RU precheck needs to be removed.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.2/upgrades/upgrade-2.3.xml f6823c8
>
> Diff: https://reviews.apache.org/r/37437/diff/
>
>
> Testing
> -------
>
> manually patch cluster to run rolling upgrade, after it passes the Core Master step (which includes upgrading the MapRed2 History server), run downgrade, verify the MapRed2 history server is still running once the downgrade is done, the three mapred properties added to satisfy the ru precheck are removed.
>
>
> Thanks,
>
> Di Li
>
>