You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2015/08/31 06:02:46 UTC
[jira] [Resolved] (HBASE-13161) ITBLL needs love; verify spins w/o
end at 10B rows
[ https://issues.apache.org/jira/browse/HBASE-13161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-13161.
---------------------------
Resolution: Fixed
Assignee: stack
Fix Version/s: 1.2.0
Resolving. Did an ITBLL 10B run w/ monkeys. Passed. See below. The change to use securerandom seems to have gotten us over the observed hang.
Running to 10B trying to save all hfiles and WALs made in case I find an issue so I can dig in if verify fails is beyond the disk capacity I have on my small cluster; I had to remove a load of archived material to run verify.
Verify took 18 hours.
Will come back to this to go larger when I have more capacity. Need to also make verify run faster.
{code}
15/08/29 14:58:54 INFO mapreduce.Job: Job job_1440806964191_0001 completed successfully
15/08/29 14:58:54 INFO mapreduce.Job: Counters: 51
File System Counters
FILE: Number of bytes read=3119022047178
FILE: Number of bytes written=3814321075643
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2484
HDFS: Number of bytes written=480
HDFS: Number of read operations=55
HDFS: Number of large read operations=0
HDFS: Number of write operations=10
Job Counters
Launched map tasks=20
Launched reduce tasks=5
Data-local map tasks=19
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=405431651
Total time spent by all reduces in occupied slots (ms)=463831492
Total time spent by all map tasks (ms)=405431651
Total time spent by all reduce tasks (ms)=231915746
Total vcore-seconds taken by all map tasks=405431651
Total vcore-seconds taken by all reduce tasks=231915746
Total megabyte-seconds taken by all map tasks=1621726604000
Total megabyte-seconds taken by all reduce tasks=1855325968000
Map-Reduce Framework
Map input records=10000000000
Map output records=20000000000
Map output bytes=650000000000
Map output materialized bytes=690000000600
Input split bytes=2484
Combine input records=0
Combine output records=0
Reduce input groups=10000000000
Reduce shuffle bytes=690000000600
Reduce input records=20000000000
Reduce output records=0
Spilled Records=110403900880
Shuffled Maps =100
Failed Shuffles=0
Merged Map outputs=100
GC time elapsed (ms)=2193043
CPU time spent (ms)=390646120
Physical memory (bytes) snapshot=23430848512
Virtual memory (bytes) snapshot=183185506304
Total committed heap usage (bytes)=21181235200
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Verify$Counts
REFERENCED=10000000000
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=480
{code}
> ITBLL needs love; verify spins w/o end at 10B rows
> --------------------------------------------------
>
> Key: HBASE-13161
> URL: https://issues.apache.org/jira/browse/HBASE-13161
> Project: HBase
> Issue Type: Bug
> Components: integration tests
> Reporter: stack
> Assignee: stack
> Fix For: 1.2.0
>
>
> Making a note of my recent experience with ITBLL; here is stuff that needs fixing.
> + I was able to run generate for about 24 hours with monkey going and loaded 10B rows. When I ran the verify (a few times), it got stuck at about 90odd percent and never made progress beyond that (this was about 20 hours in IIRC). I spent no time trying to figure why (ran out of cluster and time).
> + Verify at least takes too long. Need to make it run faster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)