You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafodion.apache.org by "Suresh Subbiah (JIRA)" <ji...@apache.org> on 2015/10/08 07:10:26 UTC
[jira] [Closed] (TRAFODION-885) LP Bug: 1410355 - Hbase goes down
during gate regression tests
[ https://issues.apache.org/jira/browse/TRAFODION-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Subbiah closed TRAFODION-885.
------------------------------------
Resolution: Cannot Reproduce
> LP Bug: 1410355 - Hbase goes down during gate regression tests
> --------------------------------------------------------------
>
> Key: TRAFODION-885
> URL: https://issues.apache.org/jira/browse/TRAFODION-885
> Project: Apache Trafodion
> Issue Type: Bug
> Components: foundation
> Reporter: Suresh Subbiah
> Assignee: Joanie Cooper
> Fix For: 2.0-incubating
>
>
> During gate tests, while on the seabase test suite, HBase went down right at the start of the first test. It seems to have been OK during suite setup when some rows are inserted into the defaults table. However for the first line in the the test (seabase/test010) there are errors reported. The delivery that failed is https://review.trafodion.org/#/c/937/
> Here is snippet from the log
> ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress/tools ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress
> Trafodion Conversational Interface 0.9.1
> (c) Copyright 2014 Hewlett-Packard Development Company, LP.
> >>-- Used to perform initialization of the SQL database during seabase regr run
> >>--
> >>-- @@@ START COPYRIGHT @@@
> >>--
> >>-- (C) Copyright 2013-2014 Hewlett-Packard Development Company, L.P.
> >>--
> >>-- Licensed under the Apache License, Version 2.0 (the "License");
> >>-- you may not use this file except in compliance with the License.
> >>-- You may obtain a copy of the License at
> >>--
> >>-- http://www.apache.org/licenses/LICENSE-2.0
> >>--
> >>-- Unless required by applicable law or agreed to in writing, software
> >>-- distributed under the License is distributed on an "AS IS" BASIS,
> >>-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> >>-- See the License for the specific language governing permissions and
> >>-- limitations under the License.
> >>--
> >>-- @@@ END COPYRIGHT @@@
> >>
> >>#ifdef SEABASE_REGRESS
> >>
> >>cqd mode_seabase 'ON';
> --- SQL operation complete.
> >>
> >>delete from TRAFODION."_MD_".DEFAULTS
> +> where ATTRIBUTE in('MODE_SEABASE', 'MODE_SEAHIVE', 'SCHEMA',
> +> 'SEABASE_VOLATILE_TABLES');
> --- 0 row(s) deleted.
> >>
> >>insert into TRAFODION."_MD_".DEFAULTS
> +> values
> +> ('MODE_SEABASE ', 'ON', 'inserted during seabase regressions run'),
> +> ('SCHEMA ', 'TRAFODION.SCH ', 'inserted during seabase regressions run'),
> +> ('SEABASE_VOLATILE_TABLES ', 'ON', 'insert during seabase regressions run');
> --- 3 row(s) inserted.
> >>
> >>insert into TRAFODION."_MD_".DEFAULTS
> +> values
> +> ('MODE_SEAHIVE ', 'ON', 'inserted during seabase regressions run');
> --- 1 row(s) inserted.
> >>
> >>create shared schema trafodion.sch;
> --- SQL operation complete.
> >>
> >>#endif
> >>
> >>exit;
> End of MXCI Session
> ***********
> Regression test env vars are now set - cd to /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress and run!
> ***********
> ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress
> Running regressions from seabase
> Checking if processes are up.
> Checking attempt: 1; user specified max: 2. Execution time in seconds: 0.
> The SQ environment is up!
> Process Configured Actual Down
> ------- ---------- ------ ----
> DTM 2 2
> RMS 4 4
> MXOSRVR 0 0
> Making /home/jenkins/workspace/core-regress-seabase-cdh5.1/rundir/seabase - continuing with tests
> ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress/seabase ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress
> Running regressions against MX format objects
> ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/tools ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress/seabase
> Trafodion Conversational Interface 0.9.1
> (c) Copyright 2014 Hewlett-Packard Development Company, LP.
> >>-- Used to perform initialization of the SQL database during seabase regr run
> >>--
> >>-- @@@ START COPYRIGHT @@@
> >>--
> >>-- (C) Copyright 2013-2014 Hewlett-Packard Development Company, L.P.
> >>--
> >>-- Licensed under the Apache License, Version 2.0 (the "License");
> >>-- you may not use this file except in compliance with the License.
> >>-- You may obtain a copy of the License at
> >>--
> >>-- http://www.apache.org/licenses/LICENSE-2.0
> >>--
> >>-- Unless required by applicable law or agreed to in writing, software
> >>-- distributed under the License is distributed on an "AS IS" BASIS,
> >>-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> >>-- See the License for the specific language governing permissions and
> >>-- limitations under the License.
> >>--
> >>-- @@@ END COPYRIGHT @@@
> >>
> >>#ifdef SEABASE_REGRESS
> >>
> >>cqd mode_seabase 'ON';
> --- SQL operation complete.
> >>
> >>delete from TRAFODION."_MD_".DEFAULTS
> +> where ATTRIBUTE in('MODE_SEABASE', 'MODE_SEAHIVE', 'SCHEMA',
> +> 'SEABASE_VOLATILE_TABLES');
> --- 4 row(s) deleted.
> >>
> >>insert into TRAFODION."_MD_".DEFAULTS
> +> values
> +> ('MODE_SEABASE ', 'ON', 'inserted during seabase regressions run'),
> +> ('SCHEMA ', 'TRAFODION.SCH ', 'inserted during seabase regressions run'),
> +> ('SEABASE_VOLATILE_TABLES ', 'ON', 'insert during seabase regressions run');
> --- 3 row(s) inserted.
> >>
> >>insert into TRAFODION."_MD_".DEFAULTS
> +> values
> +> ('MODE_SEAHIVE ', 'ON', 'inserted during seabase regressions run');
> --- 1 row(s) inserted.
> >>
> >>create shared schema trafodion.sch;
> *** ERROR[1022] Schema TRAFODION.SCH already exists.
> --- SQL operation failed with errors.
> >>
> >>#endif
> >>
> >>exit;
> End of MXCI Session
> ***********
> Regression test env vars are now set - cd to /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress and run!
> ***********
> ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress/seabase
> ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/tools/logsort_src ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress/seabase
> /usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG -c line.c -o line.o >> line.is 2>&1
> /usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG -c logsort.c -o logsort.o >> logsort.is 2>&1
> /usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG -c row.c -o row.o >> row.is 2>&1
> /usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG -c rowlst.c -o rowlst.o >> rowlst.is 2>&1
> /usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG -c symtab.c -o symtab.o >> symtab.is 2>&1
> /usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG -c tokstr.c -o tokstr.o >> tokstr.is 2>&1
> /usr/local/bin/g++ -fPIC -w -g -O0 -g -Wno-unknown-pragmas -fshort-wchar -DNA_LINUX -DNGG -D_M_DG -D_NSKFS_ -D_FULL_BUILD -DARKFS_GENERATOR -D_DP2NT_ -D_LITTLE_ENDIAN_ -DARG_PRESENT_OMIT -DNSK_USE_MSGSYS_SHELLS -D_GLIBCXX_DEPRECATED -D_TNS_R_TARGET -D_X86_ -DWIN32_LEAN_AND_MEAN -D_WIN32_WINNT=0x402 -DNA_ITOA_NOT_SUPPORTED -DMPI_ -D_DEBUG line.o logsort.o row.o rowlst.o symtab.o tokstr.o -o logsort.exe >> .is 2>&1
> ~/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/regress/seabase
> /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/tools/runregr_seabase.ksh
> (RELEASE build)
> Creating directory /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/FilteredLogs
> Creating directory /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/FilteredLogs/win
> Creating directory /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/FilteredLogs/win/seabase
> Creating directory /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/FilteredLogs/win/seabase/logfiles
> Creating directory /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/FilteredLogs/win/seabase/expfiles
> FilteredLogs: /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/FilteredLogs/win/seabase
> Running tool with
> /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/tools/runregr_seabase.ksh
> ------------------------------------------------------------
> ls: cannot access test???*: No such file or directory
> ls: cannot access expected???*: No such file or directory
> ls: cannot access filter???*: No such file or directory
> ls: cannot access *.known*: No such file or directory
> -- Skipped testfiles:
> TEST021
> -- Testfiles:
> TEST010 TEST011 TEST012 TEST013 TEST014
> TEST016 TEST018 TEST020 TEST022 TEST024
> TEST025
> --
> -- Executables:
> lrwxrwxrwx. 1 jenkins jenkins 100 Jan 13 00:18 /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/export/bin64/sqlci -> /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/lib/linux/64bit/release/sqlci
> lrwxrwxrwx. 1 jenkins jenkins 105 Jan 13 00:18 /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/export/bin64/tdm_arkcmp -> /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sql/lib/linux/64bit/release/tdm_arkcmp
> --
> -- Current work directory:
> /home/jenkins/workspace/core-regress-seabase-cdh5.1/rundir/seabase
> ------------------------------------------------------------
> ------------------------------------------------------------
> -- Starting test TEST010:
> ------------------------------------------------------------
> cat: /home/jenkins/workspace/core-regress-seabase-cdh5.1/trafodion/core/sqf/sql/regress/seabase/cidefs: No such file or directory
> Trafodion Conversational Interface 0.9.1
> (c) Copyright 2014 Hewlett-Packard Development Company, LP.
> >>-- @@@ START COPYRIGHT @@@
> >>--
> >>-- (C) Copyright 2013-2014 Hewlett-Packard Development Company, L.P.
> >>--
> >>-- Licensed under the Apache License, Version 2.0 (the "License");
> >>-- you may not use this file except in compliance with the License.
> >>-- You may obtain a copy of the License at
> >>--
> >>-- http://www.apache.org/licenses/LICENSE-2.0
> >>--
> >>-- Unless required by applicable law or agreed to in writing, software
> >>-- distributed under the License is distributed on an "AS IS" BASIS,
> >>-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> >>-- See the License for the specific language governing permissions and
> >>-- limitations under the License.
> >>--
> >>-- @@@ END COPYRIGHT @@@
> >>
> >>-- defs included during seabase regr run
> >>
> >>#ifdef SEABASE_REGRESS
> >>cqd mode_seabase 'ON';
> --- SQL operation complete.
> >>cqd seabase_volatile_tables 'ON';
> --- SQL operation complete.
> >>cqd hbase_async_drop_table 'OFF';
> --- SQL operation complete.
> >>cqd hbase_serialization 'ON';
> --- SQL operation complete.
> >>--cqd hbase_coprocessors 'OFF';
> >>cqd hbase_filter_preds 'ON';
> --- SQL operation complete.
> >>--cqd traf_default_aligned_format 'ON';
> >>--cqd traf_default_col_charset 'UTF8';
> >>set SCHEMA $$TEST_SCHEMA$$;
> --- SQL operation complete.
> >>#endif
> >>
> >>-- Tests for SeaBase
> >>-- Added June 2013
> >>--
> >>-- @@@ START COPYRIGHT @@@
> >>--
> >>-- (C) Copyright 2013-2014 Hewlett-Packard Development Company, L.P.
> >>--
> >>-- Licensed under the Apache License, Version 2.0 (the "License");
> >>-- you may not use this file except in compliance with the License.
> >>-- You may obtain a copy of the License at
> >>--
> >>-- http://www.apache.org/licenses/LICENSE-2.0
> >>--
> >>-- Unless required by applicable law or agreed to in writing, software
> >>-- distributed under the License is distributed on an "AS IS" BASIS,
> >>-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> >>-- See the License for the specific language governing permissions and
> >>-- limitations under the License.
> >>--
> >>-- @@@ END COPYRIGHT @@@
> >>
> >>obey TEST010(clean_up);
> >>?section clean_up
> >>--------------------------------------------------------------------------
> >>drop table if exists t010t1;
> *** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::getRowOpen returned error HBASE_OPEN_ERROR(-703). Cause:
> org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for TRAFODION _MD_ OBJECTS BT after 35 tries.
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1194)
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1110)
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1067)
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:908)
> org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:485)
> org.apache.hadoop.hbase.client.transactional.RMInterface.registerTransaction(RMInterface.java:73)
> org.apache.hadoop.hbase.client.transactional.RMInterface.get(RMInterface.java:134)
> org.trafodion.sql.HBaseAccess.HTableClient.startGet(HTableClient.java:333)
> .
> *** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::getRowOpen returned error HBASE_OPEN_ERROR(-703). Cause:
> org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for TRAFODION _MD_ OBJECTS BT after 35 tries.
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1194)
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1110)
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1067)
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:908)
> org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:485)
> org.apache.hadoop.hbase.client.transactional.RMInterface.registerTransaction(RMInterface.java:73)
> org.apache.hadoop.hbase.client.transactional.RMInterface.get(RMInterface.java:134)
> org.trafodion.sql.HBaseAccess.HTableClient.startGet(HTableClient.java:333)
> .
> *** ERROR[1398] Error -705 occured while accessing the hbase subsystem. Fix that error and make sure hbase is up and running. Error Details:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
> Tue Jan 13 00:54:29 GMT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@702da30a, java.net.ConnectException: Connection refused
> Tue Jan 13 00:54:29 GMT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@702da30a, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: slave-cdh51-05.trafodion.org/172.16.0.16:45314
> Tue Jan 13 00:54:30 GMT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@702da30a, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: slave-cdh51-05.trafodion.org/172.16.0.16:45314
> Tue Jan 13 00:54:31 GMT 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@702da30a, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: slave-cdh51-05.trafodion.org/172.16.0.16:45314
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)