[jira] [Comment Edited] (HBASE-14783) Proc-V2: Master aborts when downgrading from 1.3 to 1.1

["Matteo Bertozzi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14994605#comment-14994605 ] 

Matteo Bertozzi edited comment on HBASE-14783 at 11/6/15 11:03 PM:
-------------------------------------------------------------------

there are 3 problems here
 - on a not clean shutdown your cluster is in a non consistent state, starting the master and ignoring the exception is a bad idea. you have run hbck or you can restart the cluster and do a clean shutdown before the downgrade.
 - on a clean shutdown and restart with lower version you will not have procedure running. so nothing to load and you'll not hit this problem

the 3rd problem is related to how we load, completed procedures (but not yet deleted) are loaded using convert() which creates a Procedure instance, but we don't really need that. in fact we remove that instance later when we realize that we just need the result. so even in case of a clean shutdown we may get the exception because we are trying to call convert().


was (Author: mbertozzi):
there are 3 problems here
 - on a not clean shutdown your cluster is in a non consistent state, starting the master and ignoring the exception is a bad idea. you have run hbck or you can restart the cluster and do a clean shutdown before the downgrade.
 - on a clean shutdown and restart with lower version you will not have procedure running. so nothing to load and you'll not hit this problem

the 3rd is how do we load, completed procedures (but not yet deleted) are loaded using convert() which creates a Procedure instance, but we don't really need that. in fact we remove that instance later when we realize that we just need the result. 

> Proc-V2: Master aborts when downgrading from 1.3 to 1.1
> -------------------------------------------------------
>
>                 Key: HBASE-14783
>                 URL: https://issues.apache.org/jira/browse/HBASE-14783
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Ted Yu
>            Assignee: Stephen Yuan Jiang
>
> I was running ITBLL with 1.3 deployed on a 6 node cluster.
> Then I stopped the cluster, deployed 1.1 release and tried to start cluster.
> However, master failed to start due to:
> {code}
> 2015-11-06 00:58:40,351 FATAL [eval-test-2:20000.activeMasterManager] master.HMaster: Failed to become active master
> java.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructor
>   at org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:548)
>   at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:640)
>   at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:105)
>   at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:82)
>   at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:298)
>   at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:275)
>   at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:434)
>   at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1208)
>   at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1107)
>   at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:694)
>   at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:186)
>   at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1713)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:191)
>   at org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:536)
>   ... 12 more
> {code}
> The cause was that ServerCrashProcedure, written in some WAL file under MasterProcWALs from first run, was absent in 1.1 release.
> After a brief discussion with Stephen, I am logging this JIRA to solicit discussion on how customer experience can be improved if downgrade of hbase is performed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)