You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2013/07/09 02:58:47 UTC
IntegrationTestRecoveryRSLossBetweenInserts output
Here is the tail of output from IntegrationTestRecoveryRSLossBetweenInserts
2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client
environment:java.library.path=/opt/HDP/hadoop-2.1.0-SNAPSHOT/lib/native
2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client
environment:os.name=Linux
2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-279.el6.x86_64
2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client
environment:user.name=root
2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client
environment:user.home=/root
2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client
environment:user.dir=/opt/HDP/hbase-0.95.2-SNAPSHOT
2013-07-08 17:43:01,172 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 sessionTimeout=30000
watcher=hconnection-0x3ea86d12
2013-07-08 17:43:01,199 INFO [main] zookeeper.RecoverableZooKeeper:
Process identifier=hconnection-0x3ea86d12 connecting to ZooKeeper ensemble=
lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181
2013-07-08 17:43:01,206 INFO [main-SendThread(
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Opening
socket connection to server
lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181. Will not attempt
to authenticate using SASL (Unable to locate a login configuration)
2013-07-08 17:43:01,212 INFO [main-SendThread(
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Socket
connection established to
lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181, initiating session
2013-07-08 17:43:01,225 INFO [main-SendThread(
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Session
establishment complete on server
lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181, sessionid =
0x13fbfcea423068e, negotiated timeout = 30000
2013-07-08 17:43:01,401 INFO [main] mttr.AbstractIntegrationTestRecovery:
Creating table myTableName_1373330579708 with 10 splits.
2013-07-08 17:43:01,423 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:02,387 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:02,496 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:02,605 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:02,714 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:02,924 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:06,324 INFO [main] mttr.AbstractIntegrationTestRecovery:
Pre-split table created successfully in 4923ms
2013-07-08 17:43:06,326 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 sessionTimeout=30000
watcher=hconnection-0x244e5056
2013-07-08 17:43:06,327 INFO [main] zookeeper.RecoverableZooKeeper:
Process identifier=hconnection-0x244e5056 connecting to ZooKeeper ensemble=
lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181
2013-07-08 17:43:06,328 INFO [main-SendThread(
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Opening
socket connection to server
lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181. Will not attempt
to authenticate using SASL (Unable to locate a login configuration)
2013-07-08 17:43:06,330 INFO [main-SendThread(
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Socket
connection established to
lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181, initiating session
2013-07-08 17:43:06,339 INFO [main-SendThread(
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Session
establishment complete on server
lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181, sessionid =
0x33fbfcea77e069a, negotiated timeout = 30000
2013-07-08 17:43:06,354 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:06,401 INFO [main] mttr.AbstractIntegrationTestRecovery:
Moving regions of table myTableName_1373330579708 away from
lvsdpehdc25dn0406.stratus.lvs.ebay.com,60020,1373316676163
2013-07-08 17:43:06,405 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 sessionTimeout=30000
watcher=catalogtracker-on-hconnection-0x3ea86d12
2013-07-08 17:43:06,406 INFO [main] zookeeper.RecoverableZooKeeper:
Process identifier=catalogtracker-on-hconnection-0x3ea86d12 connecting to
ZooKeeper ensemble=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181
2013-07-08 17:43:06,406 DEBUG [main] catalog.CatalogTracker: Starting
catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6af37a62
2013-07-08 17:43:06,407 INFO [main-SendThread(
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Opening
socket connection to server
lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181. Will not attempt
to authenticate using SASL (Unable to locate a login configuration)
2013-07-08 17:43:06,408 INFO [main-SendThread(
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Socket
connection established to
lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181, initiating session
2013-07-08 17:43:06,416 INFO [main-SendThread(
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Session
establishment complete on server
lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181, sessionid =
0x33fbfcea77e069b, negotiated timeout = 30000
2013-07-08 17:43:06,423 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,
2013-07-08 17:43:06,432 DEBUG [main] catalog.CatalogTracker: Stopping
catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6af37a62
2013-07-08 17:43:06,441 INFO [main] zookeeper.ZooKeeper: Session:
0x33fbfcea77e069b closed
2013-07-08 17:43:06,442 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
2013-07-08 17:43:06,734 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 sessionTimeout=30000
watcher=catalogtracker-on-hconnection-0x3ea86d12
2013-07-08 17:43:06,736 INFO [main] zookeeper.RecoverableZooKeeper:
Process identifier=catalogtracker-on-hconnection-0x3ea86d12 connecting to
ZooKeeper ensemble=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181,
lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181
2013-07-08 17:43:06,736 DEBUG [main] catalog.CatalogTracker: Starting
catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@1e406b09
2013-07-08 17:43:06,736 INFO [main-SendThread(
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Opening
socket connection to server
lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181. Will not attempt
to authenticate using SASL (Unable to locate a login configuration)
2013-07-08 17:43:06,737 INFO [main-SendThread(
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Socket
connection established to
lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181, initiating session
2013-07-08 17:43:06,744 INFO [main-SendThread(
lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Session
establishment complete on server
lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181, sessionid =
0x13fbfcea423068f, negotiated timeout = 30000
2013-07-08 17:43:06,753 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,
2013-07-08 17:43:06,764 DEBUG [main] catalog.CatalogTracker: Stopping
catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@1e406b09
2013-07-08 17:43:06,772 INFO [main] zookeeper.ZooKeeper: Session:
0x13fbfcea423068f closed
2013-07-08 17:43:06,772 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
2013-07-08 17:43:06,803 INFO [main] mttr.AbstractIntegrationTestRecovery:
Moved 0 regions of table myTableName_1373330579708 away from
lvsdpehdc25dn0406.stratus.lvs.ebay.com,60020,1373316676163
2013-07-08 17:43:06,806 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:43:07,184 INFO [main] hbase.ClusterManager: Executing remote
command: sudo /sbin/iptables -A INPUT -i lo -j ACCEPT;sudo /sbin/iptables
-A OUTPUT -o lo -j ACCEPT;sudo /sbin/iptables -A INPUT -p tcp -s 0/0 -d
lvsdpehdc25dn0615.stratus.lvs.ebay.com --sport 513:65535 --dport 22 -m
state --state NEW,ESTABLISHED -j ACCEPT;sudo /sbin/iptables -A OUTPUT -p
tcp -s lvsdpehdc25dn0615.stratus.lvs.ebay.com -d 0/0 --sport 22 --dport
513:65535 -m state --state ESTABLISHED -j ACCEPT;sudo /sbin/iptables -P
INPUT DROP;sudo /sbin/iptables -P OUTPUT DROP;sudo /sbin/iptables -P
FORWARD DROP; , hostname:lvsdpehdc25dn0615.stratus.lvs.ebay.com
2013-07-08 17:43:07,185 INFO [main] util.Shell: Executing full command
[/usr/bin/ssh lvsdpehdc25dn0615.stratus.lvs.ebay.com "sudo /sbin/iptables
-A INPUT -i lo -j ACCEPT;sudo /sbin/iptables -A OUTPUT -o lo -j ACCEPT;sudo
/sbin/iptables -A INPUT -p tcp -s 0/0 -d
lvsdpehdc25dn0615.stratus.lvs.ebay.com --sport 513:65535 --dport 22 -m
state --state NEW,ESTABLISHED -j ACCEPT;sudo /sbin/iptables -A OUTPUT -p
tcp -s lvsdpehdc25dn0615.stratus.lvs.ebay.com -d 0/0 --sport 22 --dport
513:65535 -m state --state ESTABLISHED -j ACCEPT;sudo /sbin/iptables -P
INPUT DROP;sudo /sbin/iptables -P OUTPUT DROP;sudo /sbin/iptables -P
FORWARD DROP;"]
2013-07-08 17:50:38,229 INFO [main] hbase.ClusterManager: Executed remote
command, exit code:0 , output:
2013-07-08 17:50:38,662 INFO [main] mttr.AbstractIntegrationTestRecovery:
Detection took: 29
2013-07-08 17:50:38,907 INFO [main] mttr.AbstractIntegrationTestRecovery:
Regions available after: 245
2013-07-08 17:50:38,909 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:50:38,985 INFO [main] mttr.AbstractIntegrationTestRecovery:
Total failure fix took: 323
2013-07-08 17:50:38,986 DEBUG [main] client.ClientScanner: Scan
table=.META., startRow=myTableName_1373330579708,,00000000000000
2013-07-08 17:50:39,037 INFO [main]
client.HConnectionManager$HConnectionImplementation: Closing zookeeper
sessionid=0x33fbfcea77e069a
2013-07-08 17:50:39,053 INFO [main] zookeeper.ZooKeeper: Session:
0x33fbfcea77e069a closed
2013-07-08 17:50:39,053 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
2013-07-08 17:50:39,055 INFO [main] hbase.ClusterManager: Executing remote
command: sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT
ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush ,
hostname:lvsdpehdc25dn0614.stratus.lvs.ebay.com
2013-07-08 17:50:39,056 INFO [main] util.Shell: Executing full command
[/usr/bin/ssh lvsdpehdc25dn0614.stratus.lvs.ebay.com "sudo /sbin/iptables
-P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P
FORWARD ACCEPT;sudo /sbin/iptables --flush"]
2013-07-08 17:50:39,176 INFO [main] hbase.ClusterManager: Executed remote
command, exit code:0 , output:
2013-07-08 17:50:39,177 INFO [main] hbase.ClusterManager: Executing remote
command: sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT
ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush ,
hostname:lvsdpehdc25dn0615.stratus.lvs.ebay.com
2013-07-08 17:50:39,177 INFO [main] util.Shell: Executing full command
[/usr/bin/ssh lvsdpehdc25dn0615.stratus.lvs.ebay.com "sudo /sbin/iptables
-P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P
FORWARD ACCEPT;sudo /sbin/iptables --flush"]
2013-07-08 17:50:39,293 INFO [main] hbase.ClusterManager: Executed remote
command, exit code:0 , output:
2013-07-08 17:50:39,293 INFO [main] hbase.ClusterManager: Executing remote
command: sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT
ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush ,
hostname:lvsdpehdc25dn0616.stratus.lvs.ebay.com
2013-07-08 17:50:39,293 INFO [main] util.Shell: Executing full command
[/usr/bin/ssh lvsdpehdc25dn0616.stratus.lvs.ebay.com "sudo /sbin/iptables
-P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P
FORWARD ACCEPT;sudo /sbin/iptables --flush"]
2013-07-08 17:50:39,416 INFO [main] hbase.ClusterManager: Executed remote
command, exit code:0 , output:
2013-07-08 17:50:39,416 INFO [main] hbase.ClusterManager: Executing remote
command: sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT
ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush ,
hostname:lvsdpehdc25dn0617.stratus.lvs.ebay.com
2013-07-08 17:50:39,416 INFO [main] util.Shell: Executing full command
[/usr/bin/ssh lvsdpehdc25dn0617.stratus.lvs.ebay.com "sudo /sbin/iptables
-P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P
FORWARD ACCEPT;sudo /sbin/iptables --flush"]
2013-07-08 17:50:39,538 INFO [main] hbase.ClusterManager: Executed remote
command, exit code:0 , output:
2013-07-08 17:50:39,539 INFO [main] util.PerformanceChecker: max
expected=45000, actual=29, ok=true
2013-07-08 17:50:39,539 INFO [main] util.PerformanceChecker: max
expected=15000, actual=323, ok=true
2013-07-08 17:50:39,541 INFO [main] client.HBaseAdmin: Started disable of
myTableName_1373330579708
2013-07-08 17:50:39,568 DEBUG [main] client.HBaseAdmin: Sleeping= 100ms,
waiting for all regions to be disabled in myTableName_1373330579708
2013-07-08 17:50:39,669 DEBUG [main] client.HBaseAdmin: Sleeping= 100ms,
waiting for all regions to be disabled in myTableName_1373330579708
2013-07-08 17:50:39,771 DEBUG [main] client.HBaseAdmin: Sleeping= 100ms,
waiting for all regions to be disabled in myTableName_1373330579708
2013-07-08 17:50:39,872 DEBUG [main] client.HBaseAdmin: Sleeping= 200ms,
waiting for all regions to be disabled in myTableName_1373330579708
2013-07-08 17:50:40,074 DEBUG [main] client.HBaseAdmin: Sleeping= 200ms,
waiting for all regions to be disabled in myTableName_1373330579708
2013-07-08 17:50:40,275 DEBUG [main] client.HBaseAdmin: Sleeping= 400ms,
waiting for all regions to be disabled in myTableName_1373330579708
2013-07-08 17:50:40,677 INFO [main] client.HBaseAdmin: Disabled
myTableName_1373330579708
2013-07-08 17:50:41,232 INFO [main] client.HBaseAdmin: Deleted
myTableName_1373330579708
Time: 461.624