You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@trafodion.apache.org by "D. Markt" <dm...@gmail.com> on 2016/03/09 08:23:08 UTC
A failed Trafodion installation can lead to the hbase:meta table staying in the FAILED_OPEN state.
Hi,
I ran into this situation during a recent installation and thought it
might be useful if others were to hit a similar situation in the future.
This isn't the only way to recover from the situation but it is one option
and was proven to work as expected.
Regards,
Dennis
During a recent Trafodion cluster install the daily build was broken in
such a way that much of the installation proceeded, but the Trafodion files
were not copied to each node. This system was using CDH but I assume the
following would happen for HDP as well. After HBase was restarted as part
of the installation I noticed the HBase icon was red. I know this will
likely not look the best in plain text, but the hbase:meta showed (in a red
box):
Region State RIT time (ms)
1588230740 hbase:meta,,1.1588230740 state=FAILED_OPEN, ts=Mon Mar 07
07:19:00 UTC 2016 (1289s ago),
server=perf-sles-2.novalocal,60020,1457335120507 1289706
Looking at the Region Server's log file that was assigned the hbase:meta
table there was this output:
2016-03-07 16:45:27,243 INFO
org.apache.hadoop.hbase.regionserver.RSRpcServices: Open
hbase:meta,,1.1588230740
2016-03-07 16:45:27,249 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
of region=hbase:meta,,1.1588230740, starting to roll back the global
memstore size.
java.lang.IllegalStateException: Could not instantiate a region instance.
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5486)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5793)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(Op
enRegionHandler.java:356)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenR
egionHandler.java:126)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5475)
... 10 more
Caused by: java.lang.ClassNotFoundException: Class
org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion not
found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
... 11 more
2016-03-07 16:45:27,250 INFO
org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of
region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '',
ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK,
expecting version 115
After consulting with our installer expert, the issue was in fact that the
needed files had not been copied to each node. At that point one option
would be to re-install the previous build or at least undo the changes made
to point to the new build. I did not try that and I'll leave that fallback
option as a separate topic.
Instead, I took the path to see if I could get HBase to successfully come
up without getting the new Trafodion installation properly completed. To do
that there are two HBase properties that have to be reset:
. hbase.coprocessor.region.classes
. hbase.hregion.impl
I actually deleted all of the properties listed under the hbase-site.xml
that showed as non-default values by Cloudera Manager but I assume only the
hbase.hregion.impl property had to be removed. Remember to save the
configuration and remove both sets of properties. I forgot to do both of
those and each time the restart hit the same basic error.
Once the configuration is properly updated the restart will be successful
and after the hbase:meta table can be opened by the Region Server, all the
other regions will also be able to be opened. However, without Trafodion
running I would assume none of the Trafodion tables should be acted upon.
This exercise was to prove HBase could be restarted and running so that when
the Trafodion installation was started it would have a viable
Cloudera/HBase/HDFS environment to act on.
RE: 答复: A failed Trafodion installation can lead to the hbase:meta table staying in the FAILED_OPEN state.
Posted by "D. Markt" <dm...@gmail.com>.
Hi,
Yes Ming that’s an excellent point. Though I didn’t mention it, my first attempt at recovery centered on trying to verify the hbase:meta table was okay using the HBase OfflineMetaRepair utility. Even after that tool said the table was fine, I still tried another restart because the obvious symptom leads you to believe it is the file that is causing the problem. It is very unusual to get into this situation but when you do, you have a tendency to overreact because HBase was working fine and after the restart no regions can be accessed. So it’s important to examine all of the log files looking for the root cause of the problem. The Master log file gave one view, but the Region Server’s log file made it very obvious what had to be resolved.
Thanks,
Dennis
From: Amanda Moran [mailto:amanda.moran@esgyn.com]
Sent: Wednesday, March 09, 2016 12:17 PM
To: user@trafodion.incubator.apache.org
Subject: Re: 答复: A failed Trafodion installation can lead to the hbase:meta table staying in the FAILED_OPEN state.
HI there All-
I have made a jira for the installer, based on this issue.
https://issues.apache.org/jira/browse/TRAFODION-1884
Thanks!
On Wed, Mar 9, 2016 at 8:41 AM, Liu, Ming (Ming) <ming.liu@esgyn.cn <ma...@esgyn.cn> > wrote:
Thanks Denies to share this. We saw this issue during an expansion of Trafodion from 4 nodes to 5 nodes, since newly add node is empty, META region should not be there, so it does no harm. But the problem is similar, the newly added RS cannot work until we update Trafodion into that RS node.
There are two related JIRAs: TRAFODION-1729 and TRAFODION-1730.
we are working on them to solve the issue. Since Trafodion currently modify the HBase server's hbase-site.xml to add coprocessor, it affect *ALL* regions in the hbase, including META region. This is no need and not good. META region definitely no need to load Trafodion coprocessors. It is system region, Trafodion never need to access it directly, and once its open fail, the whole hbase system cannot work.
So with that JIRA fully addressed, we can remove hbase-site.xml modification from Trafodion installer, and no need to restart HBase. And as a proper installation, Trafodion should be installed on all RS node, so coprocessor jar files should be copied to all RS nodes. If Trafodion is not installed on all RS node, there may still be issues, I assume Installer still need to consider this. A better approach is to save coprocessor jar file on HDFS, but that is just a theory, need to study further.
Thanks,
Ming
-----邮件原件-----
发件人: D. Markt [mailto:dmarkt7370@gmail.com <ma...@gmail.com> ]
发送时间: 2016年3月9日 15:23
收件人: user@trafodion.incubator.apache.org <ma...@trafodion.incubator.apache.org>
主题: A failed Trafodion installation can lead to the hbase:meta table staying in the FAILED_OPEN state.
Hi,
I ran into this situation during a recent installation and thought it might be useful if others were to hit a similar situation in the future.
This isn't the only way to recover from the situation but it is one option and was proven to work as expected.
Regards,
Dennis
During a recent Trafodion cluster install the daily build was broken in such a way that much of the installation proceeded, but the Trafodion files were not copied to each node. This system was using CDH but I assume the following would happen for HDP as well. After HBase was restarted as part of the installation I noticed the HBase icon was red. I know this will likely not look the best in plain text, but the hbase:meta showed (in a red
box):
Region State RIT time (ms)
1588230740 hbase:meta,,1.1588230740 state=FAILED_OPEN, ts=Mon Mar 07
07:19:00 UTC 2016 (1289s ago),
server=perf-sles-2.novalocal,60020,1457335120507 1289706
Looking at the Region Server's log file that was assigned the hbase:meta table there was this output:
2016-03-07 16:45:27,243 INFO
org.apache.hadoop.hbase.regionserver.RSRpcServices: Open
hbase:meta,,1.1588230740
2016-03-07 16:45:27,249 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=hbase:meta,,1.1588230740, starting to roll back the global memstore size.
java.lang.IllegalStateException: Could not instantiate a region instance.
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5486)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5793)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(Op
enRegionHandler.java:356)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenR
egionHandler.java:126)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5475)
... 10 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
... 11 more
2016-03-07 16:45:27,250 INFO
org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 115
After consulting with our installer expert, the issue was in fact that the needed files had not been copied to each node. At that point one option would be to re-install the previous build or at least undo the changes made to point to the new build. I did not try that and I'll leave that fallback option as a separate topic.
Instead, I took the path to see if I could get HBase to successfully come up without getting the new Trafodion installation properly completed. To do that there are two HBase properties that have to be reset:
. hbase.coprocessor.region.classes
. hbase.hregion.impl
I actually deleted all of the properties listed under the hbase-site.xml that showed as non-default values by Cloudera Manager but I assume only the hbase.hregion.impl property had to be removed. Remember to save the configuration and remove both sets of properties. I forgot to do both of those and each time the restart hit the same basic error.
Once the configuration is properly updated the restart will be successful and after the hbase:meta table can be opened by the Region Server, all the other regions will also be able to be opened. However, without Trafodion running I would assume none of the Trafodion tables should be acted upon.
This exercise was to prove HBase could be restarted and running so that when the Trafodion installation was started it would have a viable Cloudera/HBase/HDFS environment to act on.
--
Thanks,
Amanda Moran
Re: 答复: A failed Trafodion installation can lead to the hbase:meta table staying in the FAILED_OPEN state.
Posted by Amanda Moran <am...@esgyn.com>.
HI there All-
I have made a jira for the installer, based on this issue.
https://issues.apache.org/jira/browse/TRAFODION-1884
Thanks!
On Wed, Mar 9, 2016 at 8:41 AM, Liu, Ming (Ming) <mi...@esgyn.cn> wrote:
> Thanks Denies to share this. We saw this issue during an expansion of
> Trafodion from 4 nodes to 5 nodes, since newly add node is empty, META
> region should not be there, so it does no harm. But the problem is similar,
> the newly added RS cannot work until we update Trafodion into that RS node.
>
> There are two related JIRAs: TRAFODION-1729 and TRAFODION-1730.
> we are working on them to solve the issue. Since Trafodion currently
> modify the HBase server's hbase-site.xml to add coprocessor, it affect
> *ALL* regions in the hbase, including META region. This is no need and not
> good. META region definitely no need to load Trafodion coprocessors. It is
> system region, Trafodion never need to access it directly, and once its
> open fail, the whole hbase system cannot work.
> So with that JIRA fully addressed, we can remove hbase-site.xml
> modification from Trafodion installer, and no need to restart HBase. And as
> a proper installation, Trafodion should be installed on all RS node, so
> coprocessor jar files should be copied to all RS nodes. If Trafodion is not
> installed on all RS node, there may still be issues, I assume Installer
> still need to consider this. A better approach is to save coprocessor jar
> file on HDFS, but that is just a theory, need to study further.
>
> Thanks,
> Ming
>
> -----邮件原件-----
> 发件人: D. Markt [mailto:dmarkt7370@gmail.com]
> 发送时间: 2016年3月9日 15:23
> 收件人: user@trafodion.incubator.apache.org
> 主题: A failed Trafodion installation can lead to the hbase:meta table
> staying in the FAILED_OPEN state.
>
> Hi,
>
> I ran into this situation during a recent installation and thought it
> might be useful if others were to hit a similar situation in the future.
> This isn't the only way to recover from the situation but it is one option
> and was proven to work as expected.
>
> Regards,
> Dennis
>
> During a recent Trafodion cluster install the daily build was broken in
> such a way that much of the installation proceeded, but the Trafodion files
> were not copied to each node. This system was using CDH but I assume the
> following would happen for HDP as well. After HBase was restarted as part
> of the installation I noticed the HBase icon was red. I know this will
> likely not look the best in plain text, but the hbase:meta showed (in a red
> box):
>
> Region State RIT time (ms)
> 1588230740 hbase:meta,,1.1588230740 state=FAILED_OPEN, ts=Mon Mar 07
> 07:19:00 UTC 2016 (1289s ago),
> server=perf-sles-2.novalocal,60020,1457335120507 1289706
>
> Looking at the Region Server's log file that was assigned the hbase:meta
> table there was this output:
>
> 2016-03-07 16:45:27,243 INFO
> org.apache.hadoop.hbase.regionserver.RSRpcServices: Open
> hbase:meta,,1.1588230740
> 2016-03-07 16:45:27,249 ERROR
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
> open of region=hbase:meta,,1.1588230740, starting to roll back the global
> memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5486)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5793)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672)
> at
>
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(Op
> enRegionHandler.java:356)
> at
>
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenR
> egionHandler.java:126)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
> 45)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
> 15)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> Class
> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
> not found
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5475)
> ... 10 more
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion not
> found
> at
>
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
> ... 11 more
> 2016-03-07 16:45:27,250 INFO
> org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of
> region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '',
> ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK,
> expecting version 115
>
> After consulting with our installer expert, the issue was in fact that the
> needed files had not been copied to each node. At that point one option
> would be to re-install the previous build or at least undo the changes made
> to point to the new build. I did not try that and I'll leave that fallback
> option as a separate topic.
>
> Instead, I took the path to see if I could get HBase to successfully
> come up without getting the new Trafodion installation properly completed.
> To do that there are two HBase properties that have to be reset:
>
> . hbase.coprocessor.region.classes
> . hbase.hregion.impl
>
> I actually deleted all of the properties listed under the hbase-site.xml
> that showed as non-default values by Cloudera Manager but I assume only the
> hbase.hregion.impl property had to be removed. Remember to save the
> configuration and remove both sets of properties. I forgot to do both of
> those and each time the restart hit the same basic error.
>
> Once the configuration is properly updated the restart will be
> successful and after the hbase:meta table can be opened by the Region
> Server, all the other regions will also be able to be opened. However,
> without Trafodion running I would assume none of the Trafodion tables
> should be acted upon.
> This exercise was to prove HBase could be restarted and running so that
> when the Trafodion installation was started it would have a viable
> Cloudera/HBase/HDFS environment to act on.
>
>
--
Thanks,
Amanda Moran
答复: A failed Trafodion installation can lead to the hbase:meta table staying in the FAILED_OPEN state.
Posted by "Liu, Ming (Ming)" <mi...@esgyn.cn>.
Thanks Denies to share this. We saw this issue during an expansion of Trafodion from 4 nodes to 5 nodes, since newly add node is empty, META region should not be there, so it does no harm. But the problem is similar, the newly added RS cannot work until we update Trafodion into that RS node.
There are two related JIRAs: TRAFODION-1729 and TRAFODION-1730.
we are working on them to solve the issue. Since Trafodion currently modify the HBase server's hbase-site.xml to add coprocessor, it affect *ALL* regions in the hbase, including META region. This is no need and not good. META region definitely no need to load Trafodion coprocessors. It is system region, Trafodion never need to access it directly, and once its open fail, the whole hbase system cannot work.
So with that JIRA fully addressed, we can remove hbase-site.xml modification from Trafodion installer, and no need to restart HBase. And as a proper installation, Trafodion should be installed on all RS node, so coprocessor jar files should be copied to all RS nodes. If Trafodion is not installed on all RS node, there may still be issues, I assume Installer still need to consider this. A better approach is to save coprocessor jar file on HDFS, but that is just a theory, need to study further.
Thanks,
Ming
-----邮件原件-----
发件人: D. Markt [mailto:dmarkt7370@gmail.com]
发送时间: 2016年3月9日 15:23
收件人: user@trafodion.incubator.apache.org
主题: A failed Trafodion installation can lead to the hbase:meta table staying in the FAILED_OPEN state.
Hi,
I ran into this situation during a recent installation and thought it might be useful if others were to hit a similar situation in the future.
This isn't the only way to recover from the situation but it is one option and was proven to work as expected.
Regards,
Dennis
During a recent Trafodion cluster install the daily build was broken in such a way that much of the installation proceeded, but the Trafodion files were not copied to each node. This system was using CDH but I assume the following would happen for HDP as well. After HBase was restarted as part of the installation I noticed the HBase icon was red. I know this will likely not look the best in plain text, but the hbase:meta showed (in a red
box):
Region State RIT time (ms)
1588230740 hbase:meta,,1.1588230740 state=FAILED_OPEN, ts=Mon Mar 07
07:19:00 UTC 2016 (1289s ago),
server=perf-sles-2.novalocal,60020,1457335120507 1289706
Looking at the Region Server's log file that was assigned the hbase:meta table there was this output:
2016-03-07 16:45:27,243 INFO
org.apache.hadoop.hbase.regionserver.RSRpcServices: Open
hbase:meta,,1.1588230740
2016-03-07 16:45:27,249 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=hbase:meta,,1.1588230740, starting to roll back the global memstore size.
java.lang.IllegalStateException: Could not instantiate a region instance.
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5486)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5793)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(Op
enRegionHandler.java:356)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenR
egionHandler.java:126)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
Class org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
not found
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112)
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:5475)
... 10 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2018)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110)
... 11 more
2016-03-07 16:45:27,250 INFO
org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 115
After consulting with our installer expert, the issue was in fact that the needed files had not been copied to each node. At that point one option would be to re-install the previous build or at least undo the changes made to point to the new build. I did not try that and I'll leave that fallback option as a separate topic.
Instead, I took the path to see if I could get HBase to successfully come up without getting the new Trafodion installation properly completed. To do that there are two HBase properties that have to be reset:
. hbase.coprocessor.region.classes
. hbase.hregion.impl
I actually deleted all of the properties listed under the hbase-site.xml that showed as non-default values by Cloudera Manager but I assume only the hbase.hregion.impl property had to be removed. Remember to save the configuration and remove both sets of properties. I forgot to do both of those and each time the restart hit the same basic error.
Once the configuration is properly updated the restart will be successful and after the hbase:meta table can be opened by the Region Server, all the other regions will also be able to be opened. However, without Trafodion running I would assume none of the Trafodion tables should be acted upon.
This exercise was to prove HBase could be restarted and running so that when the Trafodion installation was started it would have a viable Cloudera/HBase/HDFS environment to act on.