You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Di Li <di...@ca.ibm.com> on 2017/07/06 21:08:11 UTC
Review Request 60691: AMBARI-21420 HBase master crashed during/post
EU on an IOP4.2.5/HDP2.6 migrated cluster
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/
-----------------------------------------------------------
Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
Bugs: AMBARI-21420
https://issues.apache.org/jira/browse/AMBARI-21420
Repository: ambari
Description
-------
HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apache.
hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
Attach one or more files to this issue
Attachments
Diffs
-----
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 419e5dd
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml c4737c9
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml 419e5dd
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml c4737c9
Diff: https://reviews.apache.org/r/60691/diff/1/
Testing
-------
will be tested in the integration tests.
Thanks,
Di Li
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/#review180063
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
Lines 445 (patched)
<https://reviews.apache.org/r/60691/#comment255074>
Lines 445 and 446 are no longer needed now.
- Alejandro Fernandez
On July 10, 2017, 3:54 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60691/
> -----------------------------------------------------------
>
> (Updated July 10, 2017, 3:54 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21420
> https://issues.apache.org/jira/browse/AMBARI-21420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
> 2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apach
e.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
> Attach one or more files to this issue
> Attachments
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml bff6672
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml e9c8541
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml d931093
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml e9c8541
>
>
> Diff: https://reviews.apache.org/r/60691/diff/4/
>
>
> Testing
> -------
>
> will be tested in the integration tests.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Tim Thorpe <tt...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/#review180181
-----------------------------------------------------------
Ship it!
Ship It!
- Tim Thorpe
On July 10, 2017, 7:55 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60691/
> -----------------------------------------------------------
>
> (Updated July 10, 2017, 7:55 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21420
> https://issues.apache.org/jira/browse/AMBARI-21420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
> 2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apach
e.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
> Attach one or more files to this issue
> Attachments
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 42999b2
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml e9c8541
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml f9e3e15
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml e9c8541
>
>
> Diff: https://reviews.apache.org/r/60691/diff/5/
>
>
> Testing
> -------
>
> will be tested in the integration tests.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/#review180186
-----------------------------------------------------------
Ship it!
Ship It!
- Jonathan Hurley
On July 10, 2017, 3:55 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60691/
> -----------------------------------------------------------
>
> (Updated July 10, 2017, 3:55 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21420
> https://issues.apache.org/jira/browse/AMBARI-21420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
> 2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apach
e.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
> Attach one or more files to this issue
> Attachments
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 42999b2
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml e9c8541
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml f9e3e15
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml e9c8541
>
>
> Diff: https://reviews.apache.org/r/60691/diff/5/
>
>
> Testing
> -------
>
> will be tested in the integration tests.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/
-----------------------------------------------------------
(Updated July 10, 2017, 7:55 p.m.)
Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
Changes
-------
backup the old wal dir instead of removing it.
Bugs: AMBARI-21420
https://issues.apache.org/jira/browse/AMBARI-21420
Repository: ambari
Description
-------
HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apache.
hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
Attach one or more files to this issue
Attachments
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 42999b2
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml e9c8541
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml f9e3e15
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml e9c8541
Diff: https://reviews.apache.org/r/60691/diff/5/
Changes: https://reviews.apache.org/r/60691/diff/4-5/
Testing
-------
will be tested in the integration tests.
Thanks,
Di Li
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/
-----------------------------------------------------------
(Updated July 10, 2017, 3:54 p.m.)
Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
Bugs: AMBARI-21420
https://issues.apache.org/jira/browse/AMBARI-21420
Repository: ambari
Description
-------
HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apache.
hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
Attach one or more files to this issue
Attachments
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml bff6672
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml e9c8541
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml d931093
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml e9c8541
Diff: https://reviews.apache.org/r/60691/diff/4/
Changes: https://reviews.apache.org/r/60691/diff/3-4/
Testing
-------
will be tested in the integration tests.
Thanks,
Di Li
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/#review179939
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
Lines 444 (patched)
<https://reviews.apache.org/r/60691/#comment254877>
Just do this,
remove_wal_dir = default('/configurations/hbase-env/remove_wal_dir', False)
- Alejandro Fernandez
On July 7, 2017, 7:34 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60691/
> -----------------------------------------------------------
>
> (Updated July 7, 2017, 7:34 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21420
> https://issues.apache.org/jira/browse/AMBARI-21420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
> 2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apach
e.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
> Attach one or more files to this issue
> Attachments
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
>
>
> Diff: https://reviews.apache.org/r/60691/diff/3/
>
>
> Testing
> -------
>
> will be tested in the integration tests.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/
-----------------------------------------------------------
(Updated July 7, 2017, 7:34 p.m.)
Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
Bugs: AMBARI-21420
https://issues.apache.org/jira/browse/AMBARI-21420
Repository: ambari
Description
-------
HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apache.
hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
Attach one or more files to this issue
Attachments
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 6597055
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml 6597055
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
Diff: https://reviews.apache.org/r/60691/diff/3/
Changes: https://reviews.apache.org/r/60691/diff/2-3/
Testing
-------
will be tested in the integration tests.
Thanks,
Di Li
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/#review179890
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
Lines 446 (patched)
<https://reviews.apache.org/r/60691/#comment254794>
This still has a bug, I believe default("someconfig") will actually return the boolean, but if not defined you're setting the default to the string "false", so the type will be different.
Once you do convert to str() in the next line, the case will be different, and finally, you cannot compare the string to the boolean true in the following line.
- Alejandro Fernandez
On July 7, 2017, 1:15 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60691/
> -----------------------------------------------------------
>
> (Updated July 7, 2017, 1:15 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21420
> https://issues.apache.org/jira/browse/AMBARI-21420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
> 2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apach
e.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
> Attach one or more files to this issue
> Attachments
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
>
>
> Diff: https://reviews.apache.org/r/60691/diff/2/
>
>
> Testing
> -------
>
> will be tested in the integration tests.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Di Li <di...@ca.ibm.com>.
> On July 7, 2017, 5 p.m., Jonathan Hurley wrote:
> > I don't know enough about HBase internals - is removing the write-ahead log safe - especially during a complex upgrade? Isn't it like a last line of defense in case something goes very wrong?
>
> Di Li wrote:
> yes, this is indeed the last resort as it's either removing the wal dir or having HBase master dead in the water. HBase JIRA https://issues.apache.org/jira/browse/HBASE-14783 is for the issue, but it is still open. IOP uses this remove wal dir approach to deal with downgrading from 4.2.5 to 4.1. The patch here is a port of the internal logic for the migration use case.
>
> It's either this patch (automated during EU) or a manual step post EU.
Forgot to mention that I am OK either way A) automated during EU or B) a manual step post EU.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/#review179888
-----------------------------------------------------------
On July 7, 2017, 7:34 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60691/
> -----------------------------------------------------------
>
> (Updated July 7, 2017, 7:34 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21420
> https://issues.apache.org/jira/browse/AMBARI-21420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
> 2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apach
e.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
> Attach one or more files to this issue
> Attachments
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
>
>
> Diff: https://reviews.apache.org/r/60691/diff/3/
>
>
> Testing
> -------
>
> will be tested in the integration tests.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On July 7, 2017, 1 p.m., Jonathan Hurley wrote:
> > I don't know enough about HBase internals - is removing the write-ahead log safe - especially during a complex upgrade? Isn't it like a last line of defense in case something goes very wrong?
>
> Di Li wrote:
> yes, this is indeed the last resort as it's either removing the wal dir or having HBase master dead in the water. HBase JIRA https://issues.apache.org/jira/browse/HBASE-14783 is for the issue, but it is still open. IOP uses this remove wal dir approach to deal with downgrading from 4.2.5 to 4.1. The patch here is a port of the internal logic for the migration use case.
>
> It's either this patch (automated during EU) or a manual step post EU.
>
> Di Li wrote:
> Forgot to mention that I am OK either way A) automated during EU or B) a manual step post EU.
Yeah - I'm just wondering if we should open an item for fixing this correctly? Is there any other way aside from deleting it? Will Hbase provide a script to run on startup which fixes it?
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/#review179888
-----------------------------------------------------------
On July 7, 2017, 3:34 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60691/
> -----------------------------------------------------------
>
> (Updated July 7, 2017, 3:34 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21420
> https://issues.apache.org/jira/browse/AMBARI-21420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
> 2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apach
e.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
> Attach one or more files to this issue
> Attachments
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
>
>
> Diff: https://reviews.apache.org/r/60691/diff/3/
>
>
> Testing
> -------
>
> will be tested in the integration tests.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Di Li <di...@ca.ibm.com>.
> On July 7, 2017, 5 p.m., Jonathan Hurley wrote:
> > I don't know enough about HBase internals - is removing the write-ahead log safe - especially during a complex upgrade? Isn't it like a last line of defense in case something goes very wrong?
>
> Di Li wrote:
> yes, this is indeed the last resort as it's either removing the wal dir or having HBase master dead in the water. HBase JIRA https://issues.apache.org/jira/browse/HBASE-14783 is for the issue, but it is still open. IOP uses this remove wal dir approach to deal with downgrading from 4.2.5 to 4.1. The patch here is a port of the internal logic for the migration use case.
>
> It's either this patch (automated during EU) or a manual step post EU.
>
> Di Li wrote:
> Forgot to mention that I am OK either way A) automated during EU or B) a manual step post EU.
>
> Jonathan Hurley wrote:
> Yeah - I'm just wondering if we should open an item for fixing this correctly? Is there any other way aside from deleting it? Will Hbase provide a script to run on startup which fixes it?
I don't really know any other ways, I am afraid. The feedback from our HBase team was the removal of the wal dir thus the logic shown in the patch. I would also prefer a fix on the HBase side to deal with the issue.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/#review179888
-----------------------------------------------------------
On July 7, 2017, 7:34 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60691/
> -----------------------------------------------------------
>
> (Updated July 7, 2017, 7:34 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21420
> https://issues.apache.org/jira/browse/AMBARI-21420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
> 2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apach
e.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
> Attach one or more files to this issue
> Attachments
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
>
>
> Diff: https://reviews.apache.org/r/60691/diff/3/
>
>
> Testing
> -------
>
> will be tested in the integration tests.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Di Li <di...@ca.ibm.com>.
> On July 7, 2017, 5 p.m., Jonathan Hurley wrote:
> > I don't know enough about HBase internals - is removing the write-ahead log safe - especially during a complex upgrade? Isn't it like a last line of defense in case something goes very wrong?
yes, this is indeed the last resort as it's either removing the wal dir or having HBase master dead in the water. HBase JIRA https://issues.apache.org/jira/browse/HBASE-14783 is for the issue, but it is still open. IOP uses this remove wal dir approach to deal with downgrading from 4.2.5 to 4.1. The patch here is a port of the internal logic for the migration use case.
It's either this patch (automated during EU) or a manual step post EU.
> On July 7, 2017, 5 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py
> > Lines 41-43 (patched)
> > <https://reviews.apache.org/r/60691/diff/2/?file=1771175#file1771175line41>
> >
> > Do we need to worry about running this with kinit if Kerberized? If not, you can drop this issue.
My tests showed that it worked on a secured cluster, dropping the issue for now.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/#review179888
-----------------------------------------------------------
On July 7, 2017, 1:15 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60691/
> -----------------------------------------------------------
>
> (Updated July 7, 2017, 1:15 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21420
> https://issues.apache.org/jira/browse/AMBARI-21420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
> 2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apach
e.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
> Attach one or more files to this issue
> Attachments
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
>
>
> Diff: https://reviews.apache.org/r/60691/diff/2/
>
>
> Testing
> -------
>
> will be tested in the integration tests.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/#review179888
-----------------------------------------------------------
Fix it, then Ship it!
I don't know enough about HBase internals - is removing the write-ahead log safe - especially during a complex upgrade? Isn't it like a last line of defense in case something goes very wrong?
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py
Lines 41-43 (patched)
<https://reviews.apache.org/r/60691/#comment254788>
Do we need to worry about running this with kinit if Kerberized? If not, you can drop this issue.
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
Lines 442 (patched)
<https://reviews.apache.org/r/60691/#comment254785>
What's being formatted here?
- Jonathan Hurley
On July 7, 2017, 9:15 a.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60691/
> -----------------------------------------------------------
>
> (Updated July 7, 2017, 9:15 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21420
> https://issues.apache.org/jira/browse/AMBARI-21420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
> 2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apach
e.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
> Attach one or more files to this issue
> Attachments
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml 6597055
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
>
>
> Diff: https://reviews.apache.org/r/60691/diff/2/
>
>
> Testing
> -------
>
> will be tested in the integration tests.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/
-----------------------------------------------------------
(Updated July 7, 2017, 1:15 p.m.)
Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
Bugs: AMBARI-21420
https://issues.apache.org/jira/browse/AMBARI-21420
Repository: ambari
Description
-------
HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apache.
hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
Attach one or more files to this issue
Attachments
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 6597055
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml 6597055
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml af87794
Diff: https://reviews.apache.org/r/60691/diff/2/
Changes: https://reviews.apache.org/r/60691/diff/1-2/
Testing
-------
will be tested in the integration tests.
Thanks,
Di Li
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Di Li <di...@ca.ibm.com>.
> On July 6, 2017, 9:44 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
> > Lines 446 (patched)
> > <https://reviews.apache.org/r/60691/diff/1/?file=1770775#file1770775line446>
> >
> > I believe default() will already convert the string "true" or "false" into the boolean True or False.
> >
> > If not, then the 3rd line here has a bug since it should compare it to "true"
This is to make sure the script checks value true/false (case insentive). Otherwise the check becomes falsy vs truthy as people can set value to something like "yes/no", or even just a non-empty string.
Yes, the logic has a bug, I missed the quotes.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/#review179830
-----------------------------------------------------------
On July 6, 2017, 9:08 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60691/
> -----------------------------------------------------------
>
> (Updated July 6, 2017, 9:08 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21420
> https://issues.apache.org/jira/browse/AMBARI-21420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
> 2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apach
e.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
> Attach one or more files to this issue
> Attachments
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 419e5dd
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml c4737c9
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml 419e5dd
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml c4737c9
>
>
> Diff: https://reviews.apache.org/r/60691/diff/1/
>
>
> Testing
> -------
>
> will be tested in the integration tests.
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 60691: AMBARI-21420 HBase master crashed
during/post EU on an IOP4.2.5/HDP2.6 migrated cluster
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60691/#review179830
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py
Lines 37 (patched)
<https://reviews.apache.org/r/60691/#comment254708>
dir may also be a reserved variable name, let's call it something else
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py
Lines 43 (patched)
<https://reviews.apache.org/r/60691/#comment254709>
print the exception as well.
from resource_management.core.logger import Logger
...
except Exception, e:
Logger.error("Failed to remove HBase WAL directory, command: {0} . Exception: {1}".format(rm_cmd, e.message))
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
Lines 446 (patched)
<https://reviews.apache.org/r/60691/#comment254710>
I believe default() will already convert the string "true" or "false" into the boolean True or False.
If not, then the 3rd line here has a bug since it should compare it to "true"
- Alejandro Fernandez
On July 6, 2017, 9:08 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60691/
> -----------------------------------------------------------
>
> (Updated July 6, 2017, 9:08 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21420
> https://issues.apache.org/jira/browse/AMBARI-21420
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HBase in IOP 4.2.5 is of higher version than the one shipped in HDP 2.6. HBase master appeared as if it had started according to the Ambari action log but later crashed with error:
> 2017-07-05 09:28:08,952 FATAL [hats1:16000.activeMasterManager] master.HMaster: Failed to become active masterjava.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructorat org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:663)at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:789)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:111)at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:81)at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:374)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:297)at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:464)at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1297)at org.apach
e.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1196)
> Attach one or more files to this issue
> Attachments
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_service.py a1003dc
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 43c6036
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 419e5dd
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml c4737c9
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml 419e5dd
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml c4737c9
>
>
> Diff: https://reviews.apache.org/r/60691/diff/1/
>
>
> Testing
> -------
>
> will be tested in the integration tests.
>
>
> Thanks,
>
> Di Li
>
>