You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2009/06/09 01:02:07 UTC
[jira] Created: (HBASE-1500) ArrayIndexOutOfBoundsException out of
KeyValue$KeyComparator.compareRows from
HConnectionManager$TableServers.getCachedLocation
ArrayIndexOutOfBoundsException out of KeyValue$KeyComparator.compareRows from HConnectionManager$TableServers.getCachedLocation
-------------------------------------------------------------------------------------------------------------------------------
Key: HBASE-1500
URL: https://issues.apache.org/jira/browse/HBASE-1500
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.20.0
Reporter: Andrew Purtell
{code}
09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
false
09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
04/172.20.3.231:2181
09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
:2181]
09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
e arguments. Applications should implement Tool for the same.
09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
not be found. See JobConf(Class) or JobConf#setJar(String).
09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
03/172.20.3.230:2181
09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
:2181]
09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
om:,01e33c601a7a9dd0ddb5c8427438f2f1
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
ava:1760)
at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
1696)
at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
1755)
at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
1687)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
hedLocation(HConnectionManager.java:697)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
RegionInMeta(HConnectionManager.java:541)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
Region(HConnectionManager.java:525)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
Region(HConnectionManager.java:488)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
ionLocation(HConnectionManager.java:342)
at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
91)
at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
putFormatBase.java:296)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by Andrew Purtell <ap...@apache.org>.
I put up a Ruby script on the issue which reliably reproduces the problem. You can build latest trunk, run in local mode out of the build, and reproduce in three easy steps:
Terminal 1:
% ./hbase master start
Terminal 2:
% ./hbase shell test.rb
% ./hbase shell
hbase> count 'test'
- Andy
________________________________
From: Ryan Rawson <ry...@gmail.com>
To: hbase-dev@hadoop.apache.org
Sent: Thursday, June 11, 2009 12:24:20 AM
Subject: Re: [jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Can you send me the source to the test program?
-ryan
On Wed, Jun 10, 2009 at 10:43 PM, Andrew Purtell (JIRA) <ji...@apache.org>wrote:
>
> [
> https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718307#action_12718307]
>
> Andrew Purtell commented on HBASE-1500:
> ---------------------------------------
>
> This is pretty basic breakage. I inserted a few 1000 rows of data keyed
> with MD5 sums, then tried to run a count from the shell:
>
> {code}
> count 'content'
> 09/06/10 22:36:15 DEBUG client.HConnectionManager$TableServers: Cache hit
> for ro
> w <> in tableName content: location server 192.168.56.1:39961, location
> region n
> ame content,,1244694091453
> 09/06/10 22:36:15 DEBUG client.HConnectionManager$TableServers: Cache hit
> for ro
> w <> in tableName .META.: location server 192.168.56.1:39961, location
> region na
> me .META.,,1
> 09/06/10 22:36:15 DEBUG client.HTable$ClientScanner: Creating scanner over
> conte
> nt starting at key ''
> 09/06/10 22:36:15 DEBUG client.HTable$ClientScanner: Advancing internal
> scanner
> to startKey at ''
> 09/06/10 22:36:15 DEBUG client.HConnectionManager$TableServers: Cache hit
> for ro
> w <> in tableName content: location server 192.168.56.1:39961, location
> region n
> ame content,,1244694091453
> 09/06/10 22:36:17 DEBUG client.HConnectionManager$TableServers: address:
> 192.168
> .56.1:39961, regioninfo: REGION => {NAME => 'content,,1244698495875',
> STARTKEY =
> > '', ENDKEY => '7f47a51d4f7a8c882288d86a4c5cdc3d', ENCODED => 1221125875,
> TABLE
> => {{NAME => 'content', FAMILIES => [{NAME => 'content', VERSIONS => '1',
> COMPR
> ESSION => 'GZ', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY =>
> 'false',
> BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '1', COMPRESSION =>
> 'NONE'
> , TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> BLOCKCACHE =>
> 'false'}, {NAME => 'url', VERSIONS => '1', COMPRESSION => 'NONE', TTL =>
> '21474
> 83647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> 'false'}]}}
> Current count: 1000, row: 0a917eadf27f4917f1a823f085d1e4bd
>
> Current count: 2000, row: 156d46ad656a85f47e4e8d50c23167e9
>
> Current count: 3000, row: 1fef03d4744e5252dee5f44c36f74a00
>
> Current count: 4000, row: 2a8376e2718993eab5a7aa3cfc769e42
>
> Current count: 5000, row: 34e566f2cd7eb35ea2d7ce6bc2312638
>
> Current count: 6000, row: 3f68045f0aae18f432e0de665909f202
>
> Current count: 7000, row: 49c514099448cfd9609445b37f5b161f
>
> Current count: 8000, row: 5476db74043ccf5e40cb7974e7c07c83
>
> Current count: 9000, row: 5f3998658a477b7cbf806f4082643f2a
>
> Current count: 10000, row: 69fd32fcf76f64116602def18248938b
>
> Current count: 11000, row: 74be3000056b58b8b487c5243aadf02b
>
> Current count: 12000, row: 7ef67ca4c45a28d90847288300f05d4a
>
> Current count: 13000, row: 89783606fa73bcbf10a7b850bfa57877
>
> Current count: 14000, row: 945c830a5eecd63e948e736deada42f3
>
> Current count: 15000, row: 9f41e1454905fd7416f89aa4380a65e1
>
> Current count: 16000, row: aa54517d0450127c90e4dd4631eaa010
>
> Current count: 17000, row: b57b55373bf1302135dcefb044a58528
>
> Current count: 18000, row: c06d66ca04ab273cc96f050ded843160
>
> Current count: 19000, row: cb36f29c8d12098e25f007d7f658f495
>
> Current count: 20000, row: d59d9e495fd2f983b302e67d94aa5962
>
> Current count: 21000, row: e10430e1fda5ad363c72bb5ab7e7f8dd
>
> Current count: 22000, row: ec1d2d9cc89cd8ec67ffa1db1f3a7474
>
> Current count: 23000, row: f73d7cb9759f556b94aad178e6a6e0d5
>
> 09/06/10 22:37:24 DEBUG client.HTable$ClientScanner: Advancing forward from
> regi
> on REGION => {NAME => 'content,,1244698495875', STARTKEY => '', ENDKEY =>
> '7f47a
> 51d4f7a8c882288d86a4c5cdc3d', ENCODED => 1221125875, TABLE => {{NAME =>
> 'content
> ', FAMILIES => [{NAME => 'content', VERSIONS => '1', COMPRESSION => 'GZ',
> TTL =>
> '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> 'false'
> }, {NAME => 'info', VERSIONS => '1', COMPRESSION => 'NONE', TTL =>
> '2147483647',
> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME
> => 'u
> rl', VERSIONS => '1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE
> => '
> 65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
> 09/06/10 22:37:24 DEBUG client.HTable$ClientScanner: Advancing internal
> scanner
> to startKey at '7f47a51d4f7a8c882288d86a4c5cdc3d'
> NativeException: java.lang.RuntimeException:
> org.apache.hadoop.hbase.client.Retr
> iesExhaustedException: Trying to contact region server null for region ,
> row '7f
> 47a51d4f7a8c882288d86a4c5cdc3d', but failed after 5 attempts.
> Exceptions:
> java.lang.ArrayIndexOutOfBoundsException: 32
> java.lang.ArrayIndexOutOfBoundsException: 32
> java.lang.ArrayIndexOutOfBoundsException: 32
> java.lang.ArrayIndexOutOfBoundsException: 32
> java.lang.ArrayIndexOutOfBoundsException: 32
> from org/apache/hadoop/hbase/client/HTable.java:2002:in `hasNext'
> from sun.reflect.GeneratedMethodAccessor2:-1:in `invoke'
> from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> from java/lang/reflect/Method.java:597:in `invoke'
> from org/jruby/javasupport/JavaMethod.java:298:in
> `invokeWithExceptionHandling'
> from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> from org/jruby/runtime/callsite/CachingCallSite.java:70:in `call'
> from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
> from org/jruby/ast/WhileNode.java:127:in `interpret'
> from org/jruby/ast/NewlineNode.java:104:in `interpret'
> from org/jruby/ast/BlockNode.java:71:in `interpret'
> from
> org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in `call'
> from org/jruby/internal/runtime/methods/DefaultMethod.java:144:in
> `call'
> from org/jruby/runtime/callsite/CachingCallSite.java:110:in `call'
> from C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb.rb:369:in
> `method_
> _25$RUBY$count'
> ... 106 levels...
> from C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb#start:-1:in
> `call'
> from org/jruby/internal/runtime/methods/DynamicMethod.java:226:in
> `call'
> from org/jruby/internal/runtime/methods/CompiledMethod.java:211:in
> `call'
> from org/jruby/internal/runtime/methods/CompiledMethod.java:71:in
> `call'
> from org/jruby/runtime/callsite/CachingCallSite.java:253:in
> `cacheAndCall'
> from org/jruby/runtime/callsite/CachingCallSite.java:72:in `call'
> from C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb.rb:450:in
> `__file__'
> from C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb.rb:-1:in
> `load'
> from org/jruby/Ruby.java:577:in `runScript'
> from org/jruby/Ruby.java:480:in `runNormally'
> from org/jruby/Ruby.java:354:in `runFromMain'
> from org/jruby/Main.java:229:in `run'
> from org/jruby/Main.java:110:in `run'
> from org/jruby/Main.java:94:in `main'
> from C:\cygwin\opt\HBASE-~1.3\/bin/hirb.rb:369:in `count'
> from (hbase):4
> {code}
>
> > KeyValue$KeyComparator array overrun
> > ------------------------------------
> >
> > Key: HBASE-1500
> > URL: https://issues.apache.org/jira/browse/HBASE-1500
> > Project: Hadoop HBase
> > Issue Type: Bug
> > Affects Versions: 0.20.0
> > Reporter: Andrew Purtell
> > Priority: Blocker
> > Fix For: 0.20.0
> >
> >
> > {code}
> > 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection,
> host=B
> > OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000
> watcher=org.apac
> > he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn:
> zookeeper.disableAutoWatchReset is
> > false
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to
> server BOA
> > 04/172.20.3.231:2181
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to
> java.nio.chan
> > nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/
> 172.20.3.231
> > :2181]
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> > 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing th
> > e arguments. Applications should implement Tool for the same.
> > 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User
> classes may
> > not be found. See JobConf(Class) or JobConf#setJar(String).
> > 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection,
> host=B
> > OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000
> watcher=org.apac
> > he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to
> server BOA
> > 03/172.20.3.230:2181
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to
> java.nio.chan
> > nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/
> 172.20.3.230
> > :2181]
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> > 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split:
> 0->BOA04.trendmicro.c
> > om:,01e33c601a7a9dd0ddb5c8427438f2f1
> > Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> > at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> > at
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> > ava:1760)
> > at
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> > 1696)
> > at
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> > 1755)
> > at
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> > 1687)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> > hedLocation(HConnectionManager.java:697)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> > RegionInMeta(HConnectionManager.java:541)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> > Region(HConnectionManager.java:525)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> > Region(HConnectionManager.java:488)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> > ionLocation(HConnectionManager.java:342)
> > at
> org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> > 91)
> > at
> org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> > putFormatBase.java:296)
> > at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> > at net.iridiant.crawler.mapred.DocumentParser.main(Unknown
> Source)
> > {code}
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
RE: [jira] Commented: (HBASE-1500) KeyValue$KeyComparator array
overrun
Posted by "Jim Kellerman (POWERSET)" <Ji...@microsoft.com>.
Or attach it to the Jira issue?
---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
> -----Original Message-----
> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> Sent: Thursday, June 11, 2009 12:24 AM
> To: hbase-dev@hadoop.apache.org
> Subject: Re: [jira] Commented: (HBASE-1500) KeyValue$KeyComparator
> array overrun
>
> Can you send me the source to the test program?
>
> -ryan
>
> On Wed, Jun 10, 2009 at 10:43 PM, Andrew Purtell (JIRA)
> <ji...@apache.org>wrote:
>
> >
> > [
> > https://issues.apache.org/jira/browse/HBASE-
> 1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=12718307#action_12718307]
> >
> > Andrew Purtell commented on HBASE-1500:
> > ---------------------------------------
> >
> > This is pretty basic breakage. I inserted a few 1000 rows of data
> keyed
> > with MD5 sums, then tried to run a count from the shell:
> >
> > {code}
> > count 'content'
> > 09/06/10 22:36:15 DEBUG client.HConnectionManager$TableServers:
> Cache hit
> > for ro
> > w <> in tableName content: location server 192.168.56.1:39961,
> location
> > region n
> > ame content,,1244694091453
> > 09/06/10 22:36:15 DEBUG client.HConnectionManager$TableServers:
> Cache hit
> > for ro
> > w <> in tableName .META.: location server 192.168.56.1:39961,
> location
> > region na
> > me .META.,,1
> > 09/06/10 22:36:15 DEBUG client.HTable$ClientScanner: Creating
> scanner over
> > conte
> > nt starting at key ''
> > 09/06/10 22:36:15 DEBUG client.HTable$ClientScanner: Advancing
> internal
> > scanner
> > to startKey at ''
> > 09/06/10 22:36:15 DEBUG client.HConnectionManager$TableServers:
> Cache hit
> > for ro
> > w <> in tableName content: location server 192.168.56.1:39961,
> location
> > region n
> > ame content,,1244694091453
> > 09/06/10 22:36:17 DEBUG client.HConnectionManager$TableServers:
> address:
> > 192.168
> > .56.1:39961, regioninfo: REGION => {NAME =>
> 'content,,1244698495875',
> > STARTKEY =
> > > '', ENDKEY => '7f47a51d4f7a8c882288d86a4c5cdc3d', ENCODED =>
> 1221125875,
> > TABLE
> > => {{NAME => 'content', FAMILIES => [{NAME => 'content', VERSIONS
> => '1',
> > COMPR
> > ESSION => 'GZ', TTL => '2147483647', BLOCKSIZE => '65536',
> IN_MEMORY =>
> > 'false',
> > BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '1',
> COMPRESSION =>
> > 'NONE'
> > , TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> > BLOCKCACHE =>
> > 'false'}, {NAME => 'url', VERSIONS => '1', COMPRESSION => 'NONE',
> TTL =>
> > '21474
> > 83647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> > 'false'}]}}
> > Current count: 1000, row: 0a917eadf27f4917f1a823f085d1e4bd
> >
> > Current count: 2000, row: 156d46ad656a85f47e4e8d50c23167e9
> >
> > Current count: 3000, row: 1fef03d4744e5252dee5f44c36f74a00
> >
> > Current count: 4000, row: 2a8376e2718993eab5a7aa3cfc769e42
> >
> > Current count: 5000, row: 34e566f2cd7eb35ea2d7ce6bc2312638
> >
> > Current count: 6000, row: 3f68045f0aae18f432e0de665909f202
> >
> > Current count: 7000, row: 49c514099448cfd9609445b37f5b161f
> >
> > Current count: 8000, row: 5476db74043ccf5e40cb7974e7c07c83
> >
> > Current count: 9000, row: 5f3998658a477b7cbf806f4082643f2a
> >
> > Current count: 10000, row: 69fd32fcf76f64116602def18248938b
> >
> > Current count: 11000, row: 74be3000056b58b8b487c5243aadf02b
> >
> > Current count: 12000, row: 7ef67ca4c45a28d90847288300f05d4a
> >
> > Current count: 13000, row: 89783606fa73bcbf10a7b850bfa57877
> >
> > Current count: 14000, row: 945c830a5eecd63e948e736deada42f3
> >
> > Current count: 15000, row: 9f41e1454905fd7416f89aa4380a65e1
> >
> > Current count: 16000, row: aa54517d0450127c90e4dd4631eaa010
> >
> > Current count: 17000, row: b57b55373bf1302135dcefb044a58528
> >
> > Current count: 18000, row: c06d66ca04ab273cc96f050ded843160
> >
> > Current count: 19000, row: cb36f29c8d12098e25f007d7f658f495
> >
> > Current count: 20000, row: d59d9e495fd2f983b302e67d94aa5962
> >
> > Current count: 21000, row: e10430e1fda5ad363c72bb5ab7e7f8dd
> >
> > Current count: 22000, row: ec1d2d9cc89cd8ec67ffa1db1f3a7474
> >
> > Current count: 23000, row: f73d7cb9759f556b94aad178e6a6e0d5
> >
> > 09/06/10 22:37:24 DEBUG client.HTable$ClientScanner: Advancing
> forward from
> > regi
> > on REGION => {NAME => 'content,,1244698495875', STARTKEY => '',
> ENDKEY =>
> > '7f47a
> > 51d4f7a8c882288d86a4c5cdc3d', ENCODED => 1221125875, TABLE =>
> {{NAME =>
> > 'content
> > ', FAMILIES => [{NAME => 'content', VERSIONS => '1', COMPRESSION
> => 'GZ',
> > TTL =>
> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> BLOCKCACHE =>
> > 'false'
> > }, {NAME => 'info', VERSIONS => '1', COMPRESSION => 'NONE', TTL =>
> > '2147483647',
> > BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> 'false'}, {NAME
> > => 'u
> > rl', VERSIONS => '1', COMPRESSION => 'NONE', TTL => '2147483647',
> BLOCKSIZE
> > => '
> > 65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
> > 09/06/10 22:37:24 DEBUG client.HTable$ClientScanner: Advancing
> internal
> > scanner
> > to startKey at '7f47a51d4f7a8c882288d86a4c5cdc3d'
> > NativeException: java.lang.RuntimeException:
> > org.apache.hadoop.hbase.client.Retr
> > iesExhaustedException: Trying to contact region server null for
> region ,
> > row '7f
> > 47a51d4f7a8c882288d86a4c5cdc3d', but failed after 5 attempts.
> > Exceptions:
> > java.lang.ArrayIndexOutOfBoundsException: 32
> > java.lang.ArrayIndexOutOfBoundsException: 32
> > java.lang.ArrayIndexOutOfBoundsException: 32
> > java.lang.ArrayIndexOutOfBoundsException: 32
> > java.lang.ArrayIndexOutOfBoundsException: 32
> > from org/apache/hadoop/hbase/client/HTable.java:2002:in
> `hasNext'
> > from sun.reflect.GeneratedMethodAccessor2:-1:in `invoke'
> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in
> `invoke'
> > from java/lang/reflect/Method.java:597:in `invoke'
> > from org/jruby/javasupport/JavaMethod.java:298:in
> > `invokeWithExceptionHandling'
> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> > from
> org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> > from org/jruby/runtime/callsite/CachingCallSite.java:70:in
> `call'
> > from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
> > from org/jruby/ast/WhileNode.java:127:in `interpret'
> > from org/jruby/ast/NewlineNode.java:104:in `interpret'
> > from org/jruby/ast/BlockNode.java:71:in `interpret'
> > from
> > org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in
> `call'
> > from
> org/jruby/internal/runtime/methods/DefaultMethod.java:144:in
> > `call'
> > from org/jruby/runtime/callsite/CachingCallSite.java:110:in
> `call'
> > from
> C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb.rb:369:in
> > `method_
> > _25$RUBY$count'
> > ... 106 levels...
> > from
> C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb#start:-1:in
> > `call'
> > from
> org/jruby/internal/runtime/methods/DynamicMethod.java:226:in
> > `call'
> > from
> org/jruby/internal/runtime/methods/CompiledMethod.java:211:in
> > `call'
> > from
> org/jruby/internal/runtime/methods/CompiledMethod.java:71:in
> > `call'
> > from org/jruby/runtime/callsite/CachingCallSite.java:253:in
> > `cacheAndCall'
> > from org/jruby/runtime/callsite/CachingCallSite.java:72:in
> `call'
> > from
> C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb.rb:450:in
> > `__file__'
> > from C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb.rb:-
> 1:in
> > `load'
> > from org/jruby/Ruby.java:577:in `runScript'
> > from org/jruby/Ruby.java:480:in `runNormally'
> > from org/jruby/Ruby.java:354:in `runFromMain'
> > from org/jruby/Main.java:229:in `run'
> > from org/jruby/Main.java:110:in `run'
> > from org/jruby/Main.java:94:in `main'
> > from C:\cygwin\opt\HBASE-~1.3\/bin/hirb.rb:369:in `count'
> > from (hbase):4
> > {code}
> >
> > > KeyValue$KeyComparator array overrun
> > > ------------------------------------
> > >
> > > Key: HBASE-1500
> > > URL:
> https://issues.apache.org/jira/browse/HBASE-1500
> > > Project: Hadoop HBase
> > > Issue Type: Bug
> > > Affects Versions: 0.20.0
> > > Reporter: Andrew Purtell
> > > Priority: Blocker
> > > Fix For: 0.20.0
> > >
> > >
> > > {code}
> > > 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client
> connection,
> > host=B
> > > OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000
> > watcher=org.apac
> > > he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> > > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn:
> > zookeeper.disableAutoWatchReset is
> > > false
> > > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting
> connection to
> > server BOA
> > > 04/172.20.3.231:2181
> > > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection
> to
> > java.nio.chan
> > > nels.SocketChannel[connected local=/172.20.3.232:40296
> remote=BOA04/
> > 172.20.3.231
> > > :2181]
> > > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection
> successful
> > > 09/06/08 22:58:47 WARN mapred.JobClient: Use
> GenericOptionsParser for
> > parsing th
> > > e arguments. Applications should implement Tool for the same.
> > > 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set.
> User
> > classes may
> > > not be found. See JobConf(Class) or JobConf#setJar(String).
> > > 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client
> connection,
> > host=B
> > > OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000
> > watcher=org.apac
> > > he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> > > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting
> connection to
> > server BOA
> > > 03/172.20.3.230:2181
> > > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection
> to
> > java.nio.chan
> > > nels.SocketChannel[connected local=/172.20.3.232:42792
> remote=BOA03/
> > 172.20.3.230
> > > :2181]
> > > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection
> successful
> > > 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split:
> > 0->BOA04.trendmicro.c
> > > om:,01e33c601a7a9dd0ddb5c8427438f2f1
> > > Exception in thread "main"
> java.lang.ArrayIndexOutOfBoundsException: 32
> > > at
> org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> > > at
> >
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.
> j
> > > ava:1760)
> > > at
> >
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java
> :
> > > 1696)
> > > at
> >
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java
> :
> > > 1755)
> > > at
> >
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java
> :
> > > 1687)
> > > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCa
> c
> > > hedLocation(HConnectionManager.java:697)
> > > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locat
> e
> > > RegionInMeta(HConnectionManager.java:541)
> > > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locat
> e
> > > Region(HConnectionManager.java:525)
> > > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locat
> e
> > > Region(HConnectionManager.java:488)
> > > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRe
> g
> > > ionLocation(HConnectionManager.java:342)
> > > at
> >
> org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:
> 1
> > > 91)
> > > at
> >
> org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableI
> n
> > > putFormatBase.java:296)
> > > at
> > org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> > > at
> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> > > at
> net.iridiant.crawler.mapred.DocumentParser.main(Unknown
> > Source)
> > > {code}
> >
> > --
> > This message is automatically generated by JIRA.
> > -
> > You can reply to this email to add a comment to the issue online.
> >
> >
Re: [jira] Commented: (HBASE-1500) KeyValue$KeyComparator array
overrun
Posted by Ryan Rawson <ry...@gmail.com>.
Can you send me the source to the test program?
-ryan
On Wed, Jun 10, 2009 at 10:43 PM, Andrew Purtell (JIRA) <ji...@apache.org>wrote:
>
> [
> https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718307#action_12718307]
>
> Andrew Purtell commented on HBASE-1500:
> ---------------------------------------
>
> This is pretty basic breakage. I inserted a few 1000 rows of data keyed
> with MD5 sums, then tried to run a count from the shell:
>
> {code}
> count 'content'
> 09/06/10 22:36:15 DEBUG client.HConnectionManager$TableServers: Cache hit
> for ro
> w <> in tableName content: location server 192.168.56.1:39961, location
> region n
> ame content,,1244694091453
> 09/06/10 22:36:15 DEBUG client.HConnectionManager$TableServers: Cache hit
> for ro
> w <> in tableName .META.: location server 192.168.56.1:39961, location
> region na
> me .META.,,1
> 09/06/10 22:36:15 DEBUG client.HTable$ClientScanner: Creating scanner over
> conte
> nt starting at key ''
> 09/06/10 22:36:15 DEBUG client.HTable$ClientScanner: Advancing internal
> scanner
> to startKey at ''
> 09/06/10 22:36:15 DEBUG client.HConnectionManager$TableServers: Cache hit
> for ro
> w <> in tableName content: location server 192.168.56.1:39961, location
> region n
> ame content,,1244694091453
> 09/06/10 22:36:17 DEBUG client.HConnectionManager$TableServers: address:
> 192.168
> .56.1:39961, regioninfo: REGION => {NAME => 'content,,1244698495875',
> STARTKEY =
> > '', ENDKEY => '7f47a51d4f7a8c882288d86a4c5cdc3d', ENCODED => 1221125875,
> TABLE
> => {{NAME => 'content', FAMILIES => [{NAME => 'content', VERSIONS => '1',
> COMPR
> ESSION => 'GZ', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY =>
> 'false',
> BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '1', COMPRESSION =>
> 'NONE'
> , TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> BLOCKCACHE =>
> 'false'}, {NAME => 'url', VERSIONS => '1', COMPRESSION => 'NONE', TTL =>
> '21474
> 83647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> 'false'}]}}
> Current count: 1000, row: 0a917eadf27f4917f1a823f085d1e4bd
>
> Current count: 2000, row: 156d46ad656a85f47e4e8d50c23167e9
>
> Current count: 3000, row: 1fef03d4744e5252dee5f44c36f74a00
>
> Current count: 4000, row: 2a8376e2718993eab5a7aa3cfc769e42
>
> Current count: 5000, row: 34e566f2cd7eb35ea2d7ce6bc2312638
>
> Current count: 6000, row: 3f68045f0aae18f432e0de665909f202
>
> Current count: 7000, row: 49c514099448cfd9609445b37f5b161f
>
> Current count: 8000, row: 5476db74043ccf5e40cb7974e7c07c83
>
> Current count: 9000, row: 5f3998658a477b7cbf806f4082643f2a
>
> Current count: 10000, row: 69fd32fcf76f64116602def18248938b
>
> Current count: 11000, row: 74be3000056b58b8b487c5243aadf02b
>
> Current count: 12000, row: 7ef67ca4c45a28d90847288300f05d4a
>
> Current count: 13000, row: 89783606fa73bcbf10a7b850bfa57877
>
> Current count: 14000, row: 945c830a5eecd63e948e736deada42f3
>
> Current count: 15000, row: 9f41e1454905fd7416f89aa4380a65e1
>
> Current count: 16000, row: aa54517d0450127c90e4dd4631eaa010
>
> Current count: 17000, row: b57b55373bf1302135dcefb044a58528
>
> Current count: 18000, row: c06d66ca04ab273cc96f050ded843160
>
> Current count: 19000, row: cb36f29c8d12098e25f007d7f658f495
>
> Current count: 20000, row: d59d9e495fd2f983b302e67d94aa5962
>
> Current count: 21000, row: e10430e1fda5ad363c72bb5ab7e7f8dd
>
> Current count: 22000, row: ec1d2d9cc89cd8ec67ffa1db1f3a7474
>
> Current count: 23000, row: f73d7cb9759f556b94aad178e6a6e0d5
>
> 09/06/10 22:37:24 DEBUG client.HTable$ClientScanner: Advancing forward from
> regi
> on REGION => {NAME => 'content,,1244698495875', STARTKEY => '', ENDKEY =>
> '7f47a
> 51d4f7a8c882288d86a4c5cdc3d', ENCODED => 1221125875, TABLE => {{NAME =>
> 'content
> ', FAMILIES => [{NAME => 'content', VERSIONS => '1', COMPRESSION => 'GZ',
> TTL =>
> '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> 'false'
> }, {NAME => 'info', VERSIONS => '1', COMPRESSION => 'NONE', TTL =>
> '2147483647',
> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME
> => 'u
> rl', VERSIONS => '1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE
> => '
> 65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
> 09/06/10 22:37:24 DEBUG client.HTable$ClientScanner: Advancing internal
> scanner
> to startKey at '7f47a51d4f7a8c882288d86a4c5cdc3d'
> NativeException: java.lang.RuntimeException:
> org.apache.hadoop.hbase.client.Retr
> iesExhaustedException: Trying to contact region server null for region ,
> row '7f
> 47a51d4f7a8c882288d86a4c5cdc3d', but failed after 5 attempts.
> Exceptions:
> java.lang.ArrayIndexOutOfBoundsException: 32
> java.lang.ArrayIndexOutOfBoundsException: 32
> java.lang.ArrayIndexOutOfBoundsException: 32
> java.lang.ArrayIndexOutOfBoundsException: 32
> java.lang.ArrayIndexOutOfBoundsException: 32
> from org/apache/hadoop/hbase/client/HTable.java:2002:in `hasNext'
> from sun.reflect.GeneratedMethodAccessor2:-1:in `invoke'
> from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> from java/lang/reflect/Method.java:597:in `invoke'
> from org/jruby/javasupport/JavaMethod.java:298:in
> `invokeWithExceptionHandling'
> from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> from org/jruby/runtime/callsite/CachingCallSite.java:70:in `call'
> from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
> from org/jruby/ast/WhileNode.java:127:in `interpret'
> from org/jruby/ast/NewlineNode.java:104:in `interpret'
> from org/jruby/ast/BlockNode.java:71:in `interpret'
> from
> org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in `call'
> from org/jruby/internal/runtime/methods/DefaultMethod.java:144:in
> `call'
> from org/jruby/runtime/callsite/CachingCallSite.java:110:in `call'
> from C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb.rb:369:in
> `method_
> _25$RUBY$count'
> ... 106 levels...
> from C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb#start:-1:in
> `call'
> from org/jruby/internal/runtime/methods/DynamicMethod.java:226:in
> `call'
> from org/jruby/internal/runtime/methods/CompiledMethod.java:211:in
> `call'
> from org/jruby/internal/runtime/methods/CompiledMethod.java:71:in
> `call'
> from org/jruby/runtime/callsite/CachingCallSite.java:253:in
> `cacheAndCall'
> from org/jruby/runtime/callsite/CachingCallSite.java:72:in `call'
> from C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb.rb:450:in
> `__file__'
> from C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb.rb:-1:in
> `load'
> from org/jruby/Ruby.java:577:in `runScript'
> from org/jruby/Ruby.java:480:in `runNormally'
> from org/jruby/Ruby.java:354:in `runFromMain'
> from org/jruby/Main.java:229:in `run'
> from org/jruby/Main.java:110:in `run'
> from org/jruby/Main.java:94:in `main'
> from C:\cygwin\opt\HBASE-~1.3\/bin/hirb.rb:369:in `count'
> from (hbase):4
> {code}
>
> > KeyValue$KeyComparator array overrun
> > ------------------------------------
> >
> > Key: HBASE-1500
> > URL: https://issues.apache.org/jira/browse/HBASE-1500
> > Project: Hadoop HBase
> > Issue Type: Bug
> > Affects Versions: 0.20.0
> > Reporter: Andrew Purtell
> > Priority: Blocker
> > Fix For: 0.20.0
> >
> >
> > {code}
> > 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection,
> host=B
> > OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000
> watcher=org.apac
> > he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn:
> zookeeper.disableAutoWatchReset is
> > false
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to
> server BOA
> > 04/172.20.3.231:2181
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to
> java.nio.chan
> > nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/
> 172.20.3.231
> > :2181]
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> > 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing th
> > e arguments. Applications should implement Tool for the same.
> > 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User
> classes may
> > not be found. See JobConf(Class) or JobConf#setJar(String).
> > 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection,
> host=B
> > OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000
> watcher=org.apac
> > he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to
> server BOA
> > 03/172.20.3.230:2181
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to
> java.nio.chan
> > nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/
> 172.20.3.230
> > :2181]
> > 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> > 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split:
> 0->BOA04.trendmicro.c
> > om:,01e33c601a7a9dd0ddb5c8427438f2f1
> > Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> > at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> > at
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> > ava:1760)
> > at
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> > 1696)
> > at
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> > 1755)
> > at
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> > 1687)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> > hedLocation(HConnectionManager.java:697)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> > RegionInMeta(HConnectionManager.java:541)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> > Region(HConnectionManager.java:525)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> > Region(HConnectionManager.java:488)
> > at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> > ionLocation(HConnectionManager.java:342)
> > at
> org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> > 91)
> > at
> org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> > putFormatBase.java:296)
> > at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> > at net.iridiant.crawler.mapred.DocumentParser.main(Unknown
> Source)
> > {code}
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
Re: [jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by Andrew Purtell <ap...@apache.org>.
If I roll one revision back from http://svn.apache.org/viewvc?view=rev&revision=782178 (via svn checkout -r782177) then I stop seeing the issues reported in HBASE-1500.
- Andy
________________________________
From: Ryan Rawson <ry...@gmail.com>
To: hbase-dev@hadoop.apache.org
Sent: Friday, June 12, 2009 12:30:50 PM
Subject: Re: [jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
More than 1 region suggests this is related to 1495....
On Jun 12, 2009 12:23 PM, "Andrew Purtell (JIRA)" <ji...@apache.org> wrote:
[
https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718949#action_12718949]
Andrew Purtell commented on HBASE-1500:
---------------------------------------
Some detail I added up on IRC:
<apurtell> i was wondering if it was just me...
<apurtell> yeah would be great if someone else can confirm the repro
works for them...
<apurtell> what's odd is the shell's count of all the rows comes back
before it even tries to advance the scanner to the next split. some kind of
META sort problem, parent region coming back before daughter? i'd like to
dig in but this code is all new to me
<apurtell> in the repro case, the split is triggered and confirmed, so
there is more than one region...
<apurtell> this is the same basic problem as with TableSplit, iterating
over META is going off the rails, giving bad results, triggering the out of
bounds exception
<apurtell> at least this is what it looks like to me
<apurtell> started immediately after commit of 1304
<apurtell> noticed it first while testing stargate scanners, scanners
hang/crash/fail on tables with more than one region
> KeyValue$KeyComparator array overrun >
------------------------------------ > > ...
> Assignee: ryan rawson
> Priority: Blocker > Fix For: 0.20.0 > > Attachments: hbase.log.gz, ...
Re: [jira] Commented: (HBASE-1500) KeyValue$KeyComparator array
overrun
Posted by Ryan Rawson <ry...@gmail.com>.
More than 1 region suggests this is related to 1495....
On Jun 12, 2009 12:23 PM, "Andrew Purtell (JIRA)" <ji...@apache.org> wrote:
[
https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718949#action_12718949]
Andrew Purtell commented on HBASE-1500:
---------------------------------------
Some detail I added up on IRC:
<apurtell> i was wondering if it was just me...
<apurtell> yeah would be great if someone else can confirm the repro
works for them...
<apurtell> what's odd is the shell's count of all the rows comes back
before it even tries to advance the scanner to the next split. some kind of
META sort problem, parent region coming back before daughter? i'd like to
dig in but this code is all new to me
<apurtell> in the repro case, the split is triggered and confirmed, so
there is more than one region...
<apurtell> this is the same basic problem as with TableSplit, iterating
over META is going off the rails, giving bad results, triggering the out of
bounds exception
<apurtell> at least this is what it looks like to me
<apurtell> started immediately after commit of 1304
<apurtell> noticed it first while testing stargate scanners, scanners
hang/crash/fail on tables with more than one region
> KeyValue$KeyComparator array overrun >
------------------------------------ > > ...
> Assignee: ryan rawson
> Priority: Blocker > Fix For: 0.20.0 > > Attachments: hbase.log.gz, ...
[jira] Commented: (HBASE-1500) ArrayIndexOutOfBoundsException out
of KeyValue$KeyComparator.compareRows from
HConnectionManager$TableServers.getCachedLocation
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717484#action_12717484 ]
Andrew Purtell commented on HBASE-1500:
---------------------------------------
I haven't converted this TableMap job to the new API yet.
> ArrayIndexOutOfBoundsException out of KeyValue$KeyComparator.compareRows from HConnectionManager$TableServers.getCachedLocation
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719240#action_12719240 ]
ryan rawson commented on HBASE-1500:
------------------------------------
i see this on my laptop. looks like the scanner is returning a null item in an array at some point which needless to say isnt handy.
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Assignee: ryan rawson
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-1500:
-------------------------------
Attachment: HBASE-1500.patch
here's a fix to the problem. I verified it fixes it.
Thanks for the great bug report apurtell, complete with single-machine repro test case!
Enjoy the fix.
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Assignee: ryan rawson
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1500.patch, hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1500:
----------------------------------
Comment: was deleted
(was: I haven't converted this TableMap job to the new API yet. )
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.20.0
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-1500:
-------------------------------
Status: Patch Available (was: Open)
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Assignee: ryan rawson
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1500.patch, hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718307#action_12718307 ]
Andrew Purtell commented on HBASE-1500:
---------------------------------------
This is pretty basic breakage. I inserted a few 1000 rows of data keyed with MD5 sums, then tried to run a count from the shell:
{code}
count 'content'
09/06/10 22:36:15 DEBUG client.HConnectionManager$TableServers: Cache hit for ro
w <> in tableName content: location server 192.168.56.1:39961, location region n
ame content,,1244694091453
09/06/10 22:36:15 DEBUG client.HConnectionManager$TableServers: Cache hit for ro
w <> in tableName .META.: location server 192.168.56.1:39961, location region na
me .META.,,1
09/06/10 22:36:15 DEBUG client.HTable$ClientScanner: Creating scanner over conte
nt starting at key ''
09/06/10 22:36:15 DEBUG client.HTable$ClientScanner: Advancing internal scanner
to startKey at ''
09/06/10 22:36:15 DEBUG client.HConnectionManager$TableServers: Cache hit for ro
w <> in tableName content: location server 192.168.56.1:39961, location region n
ame content,,1244694091453
09/06/10 22:36:17 DEBUG client.HConnectionManager$TableServers: address: 192.168
.56.1:39961, regioninfo: REGION => {NAME => 'content,,1244698495875', STARTKEY =
> '', ENDKEY => '7f47a51d4f7a8c882288d86a4c5cdc3d', ENCODED => 1221125875, TABLE
=> {{NAME => 'content', FAMILIES => [{NAME => 'content', VERSIONS => '1', COMPR
ESSION => 'GZ', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '1', COMPRESSION => 'NONE'
, TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}, {NAME => 'url', VERSIONS => '1', COMPRESSION => 'NONE', TTL => '21474
83647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
Current count: 1000, row: 0a917eadf27f4917f1a823f085d1e4bd
Current count: 2000, row: 156d46ad656a85f47e4e8d50c23167e9
Current count: 3000, row: 1fef03d4744e5252dee5f44c36f74a00
Current count: 4000, row: 2a8376e2718993eab5a7aa3cfc769e42
Current count: 5000, row: 34e566f2cd7eb35ea2d7ce6bc2312638
Current count: 6000, row: 3f68045f0aae18f432e0de665909f202
Current count: 7000, row: 49c514099448cfd9609445b37f5b161f
Current count: 8000, row: 5476db74043ccf5e40cb7974e7c07c83
Current count: 9000, row: 5f3998658a477b7cbf806f4082643f2a
Current count: 10000, row: 69fd32fcf76f64116602def18248938b
Current count: 11000, row: 74be3000056b58b8b487c5243aadf02b
Current count: 12000, row: 7ef67ca4c45a28d90847288300f05d4a
Current count: 13000, row: 89783606fa73bcbf10a7b850bfa57877
Current count: 14000, row: 945c830a5eecd63e948e736deada42f3
Current count: 15000, row: 9f41e1454905fd7416f89aa4380a65e1
Current count: 16000, row: aa54517d0450127c90e4dd4631eaa010
Current count: 17000, row: b57b55373bf1302135dcefb044a58528
Current count: 18000, row: c06d66ca04ab273cc96f050ded843160
Current count: 19000, row: cb36f29c8d12098e25f007d7f658f495
Current count: 20000, row: d59d9e495fd2f983b302e67d94aa5962
Current count: 21000, row: e10430e1fda5ad363c72bb5ab7e7f8dd
Current count: 22000, row: ec1d2d9cc89cd8ec67ffa1db1f3a7474
Current count: 23000, row: f73d7cb9759f556b94aad178e6a6e0d5
09/06/10 22:37:24 DEBUG client.HTable$ClientScanner: Advancing forward from regi
on REGION => {NAME => 'content,,1244698495875', STARTKEY => '', ENDKEY => '7f47a
51d4f7a8c882288d86a4c5cdc3d', ENCODED => 1221125875, TABLE => {{NAME => 'content
', FAMILIES => [{NAME => 'content', VERSIONS => '1', COMPRESSION => 'GZ', TTL =>
'2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'
}, {NAME => 'info', VERSIONS => '1', COMPRESSION => 'NONE', TTL => '2147483647',
BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'u
rl', VERSIONS => '1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '
65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
09/06/10 22:37:24 DEBUG client.HTable$ClientScanner: Advancing internal scanner
to startKey at '7f47a51d4f7a8c882288d86a4c5cdc3d'
NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.client.Retr
iesExhaustedException: Trying to contact region server null for region , row '7f
47a51d4f7a8c882288d86a4c5cdc3d', but failed after 5 attempts.
Exceptions:
java.lang.ArrayIndexOutOfBoundsException: 32
java.lang.ArrayIndexOutOfBoundsException: 32
java.lang.ArrayIndexOutOfBoundsException: 32
java.lang.ArrayIndexOutOfBoundsException: 32
java.lang.ArrayIndexOutOfBoundsException: 32
from org/apache/hadoop/hbase/client/HTable.java:2002:in `hasNext'
from sun.reflect.GeneratedMethodAccessor2:-1:in `invoke'
from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
from java/lang/reflect/Method.java:597:in `invoke'
from org/jruby/javasupport/JavaMethod.java:298:in `invokeWithExceptionHandling'
from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
from org/jruby/runtime/callsite/CachingCallSite.java:70:in `call'
from org/jruby/ast/CallNoArgNode.java:61:in `interpret'
from org/jruby/ast/WhileNode.java:127:in `interpret'
from org/jruby/ast/NewlineNode.java:104:in `interpret'
from org/jruby/ast/BlockNode.java:71:in `interpret'
from org/jruby/internal/runtime/methods/InterpretedMethod.java:163:in `call'
from org/jruby/internal/runtime/methods/DefaultMethod.java:144:in `call'
from org/jruby/runtime/callsite/CachingCallSite.java:110:in `call'
from C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb.rb:369:in `method_
_25$RUBY$count'
... 106 levels...
from C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb#start:-1:in `call'
from org/jruby/internal/runtime/methods/DynamicMethod.java:226:in `call'
from org/jruby/internal/runtime/methods/CompiledMethod.java:211:in `call'
from org/jruby/internal/runtime/methods/CompiledMethod.java:71:in `call'
from org/jruby/runtime/callsite/CachingCallSite.java:253:in `cacheAndCall'
from org/jruby/runtime/callsite/CachingCallSite.java:72:in `call'
from C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb.rb:450:in `__file__'
from C_3a_/cygwin/opt/HBASE_minus_7e_1_dot_3/bin/hirb.rb:-1:in `load'
from org/jruby/Ruby.java:577:in `runScript'
from org/jruby/Ruby.java:480:in `runNormally'
from org/jruby/Ruby.java:354:in `runFromMain'
from org/jruby/Main.java:229:in `run'
from org/jruby/Main.java:110:in `run'
from org/jruby/Main.java:94:in `main'
from C:\cygwin\opt\HBASE-~1.3\/bin/hirb.rb:369:in `count'
from (hbase):4
{code}
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.20.0
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718949#action_12718949 ]
Andrew Purtell commented on HBASE-1500:
---------------------------------------
Some detail I added up on IRC:
<apurtell> i was wondering if it was just me...
<apurtell> yeah would be great if someone else can confirm the repro works for them...
<apurtell> what's odd is the shell's count of all the rows comes back before it even tries to advance the scanner to the next split. some kind of META sort problem, parent region coming back before daughter? i'd like to dig in but this code is all new to me
<apurtell> in the repro case, the split is triggered and confirmed, so there is more than one region...
<apurtell> this is the same basic problem as with TableSplit, iterating over META is going off the rails, giving bad results, triggering the out of bounds exception
<apurtell> at least this is what it looks like to me
<apurtell> started immediately after commit of 1304
<apurtell> noticed it first while testing stargate scanners, scanners hang/crash/fail on tables with more than one region
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Assignee: ryan rawson
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1500:
----------------------------------
Attachment: hbase.log.gz
Complete log from initialization through first split through failed count attached. Test environment runs master and regionserver in single process, uses local filesystem, is fed by a Heritrix instance.
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase.log.gz
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719373#action_12719373 ]
stack commented on HBASE-1500:
------------------------------
+1
{code}
02:27 < St^Ack> why 1500 change getComparator?
02:27 < dj_ryan> heh
02:27 < dj_ryan> it fixes 1500
02:27 < dj_ryan> here's why
02:27 < dj_ryan> the old code
02:27 < dj_ryan> was treating the rowkey
02:27 < dj_ryan> as a _full_ KeyValue key
02:27 < dj_ryan> and was decoding ints
02:27 < dj_ryan> to figure out the length of thw row part
02:27 < dj_ryan> but we just wanted to compare the row portion
02:28 < St^Ack> oh
02:28 < St^Ack> bad
02:29 < dj_ryan> yes
02:29 < dj_ryan> very very bad
{code}
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Assignee: ryan rawson
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1500.patch, hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-1500:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Assignee: ryan rawson
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1500.patch, hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719296#action_12719296 ]
Andrew Purtell commented on HBASE-1500:
---------------------------------------
+1 confirmed fix.
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Assignee: ryan rawson
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1500.patch, hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson reassigned HBASE-1500:
----------------------------------
Assignee: ryan rawson
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Assignee: ryan rawson
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719229#action_12719229 ]
ryan rawson commented on HBASE-1500:
------------------------------------
the regionserver logs say something like so:
2009-06-14 00:44:43,751 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.NullPointerException
2009-06-14 00:44:43,752 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60021, call next(-1247002001233657007, 30) from 10.10.20.13:36528: error: java.io.IOExc
eption: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:832) at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:822)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:97)
at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:85)
at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:78)
at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:120)
at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:66)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:951)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:907)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1082) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:56)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:140)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1716)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1878)
... 5 more
2009-06-14 00:44:45,757 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.IllegalArgumentException
2009-06-14 00:44:45,758 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60021, call next(-1247002001233657007, 30) from 10.10.20.13:36528: error: java.io.IOExc
eption: java.lang.IllegalArgumentException
java.io.IOException: java.lang.IllegalArgumentException
at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:832)
at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:822)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218) at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1073)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:56)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:140)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1716)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1878)
... 5 more
2009-06-14 00:44:47,762 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.lang.NegativeArraySizeException
2009-06-14 00:44:47,763 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60021, call next(-1247002001233657007, 30) from 10.10.20.13:36528: error: java.io.IOException: java.lang.NegativeArraySizeException
java.io.IOException: java.lang.NegativeArraySizeException
at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:832)
at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:822)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
Caused by: java.lang.NegativeArraySizeException
at org.apache.hadoop.hbase.KeyValue.getRow(KeyValue.java:890)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1708)
at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1878)
... 5 more
then we get the negative array size exceptions from then on out. very strange.
this seems very related to 1503. I wonder how we can fix up hfile to be less errorful.
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Assignee: ryan rawson
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719173#action_12719173 ]
Andrew Purtell commented on HBASE-1500:
---------------------------------------
Testing with patch for HBASE-1495 changes the behavior of the test case some. The iteration of regions seems in order now, but the AIOBE remains.
{code}
Version: 0.20.0-dev, r784310, Sat Jun 13 18:51:23 UTC 2009
hbase(main):001:0> count 'test'
09/06/13 18:54:14 INFO zookeeper.ZooKeeperWrapper: Quorum servers: localhost:2181
NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.client.
RetriesExhaustedException: Trying to contact region server null for region , row
'86d6531b7fd76860445f8ffd1a0461fc', but failed after 5 attempts.
Exceptions:
java.lang.ArrayIndexOutOfBoundsException: 32
java.lang.ArrayIndexOutOfBoundsException: 32
java.lang.ArrayIndexOutOfBoundsException: 32
java.lang.ArrayIndexOutOfBoundsException: 32
java.lang.ArrayIndexOutOfBoundsException: 32
from org/apache/hadoop/hbase/client/HTable.java:2000:in `hasNext'
from sun.reflect.GeneratedMethodAccessor2:-1:in `invoke'
from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
{code}
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Assignee: ryan rawson
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1500:
----------------------------------
Priority: Blocker (was: Major)
Fix Version/s: 0.20.0
Summary: KeyValue$KeyComparator array overrun (was: ArrayIndexOutOfBoundsException out of KeyValue$KeyComparator.compareRows from HConnectionManager$TableServers.getCachedLocation)
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.20.0
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718183#action_12718183 ]
Lars George commented on HBASE-1500:
------------------------------------
It looks like it happens here from the above
{code}
public int compare(byte[] left, int loffset, int llength, byte[] right,
int roffset, int rlength) {
// Compare row
short lrowlength = Bytes.toShort(left, loffset);
short rrowlength = Bytes.toShort(right, roffset);
int compare = compareRows(left, loffset + Bytes.SIZEOF_SHORT,
lrowlength,
right, roffset + Bytes.SIZEOF_SHORT, rrowlength);
if (compare != 0) {
return compare;
}
{code}
It reads the lengths from the key byte array assuming they are correct and then applies those lengths to the compareRow() call, which then fails.
This indicates a borked length from a key array? Could you instrument the code above and log the lrowlength, rrowlength and the the real left.length and right.length?
If that mismatches, the question is why?
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.20.0
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1500:
----------------------------------
Attachment: test.log.gz
test.rb
To reproduce, run the attached script using the latest trunk.
{{./bin/hbase shell test.rb}}
Wait for the split to finish. Then, try a count:
{{./bin/hbase shell}}
{{hbase> count 'test'}}
{code}
count 'test'
09/06/11 14:23:32 DEBUG client.HConnectionManager$TableServers: address: 192.168
.56.1:22720, regioninfo: REGION => {NAME => 'test,,1244755386281', STARTKEY => '
', ENDKEY => '7fd99653bf71ea6c747d34db7846c708', ENCODED => 1840196805, TABLE =>
{{NAME => 'test', FAMILIES => [{NAME => 'content', VERSIONS => '3', COMPRESSION
=> 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLO
CKCACHE => 'false'}, {NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TT
L => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'fa
lse'}]}}
09/06/11 14:23:32 DEBUG client.HConnectionManager$TableServers: Cache hit for ro
w <> in tableName .META.: location server 192.168.56.1:22720, location region na
me .META.,,1
09/06/11 14:23:32 DEBUG client.HTable$ClientScanner: Creating scanner over test
starting at key ''
09/06/11 14:23:32 DEBUG client.HTable$ClientScanner: Advancing internal scanner
to startKey at ''
09/06/11 14:23:32 DEBUG client.HConnectionManager$TableServers: Cache hit for ro
w <> in tableName test: location server 192.168.56.1:22720, location region name
test,,1244755386281
Current count: 1000, row: ffe83c9f18ab2f057cd6435f04228377
09/06/11 14:23:33 DEBUG client.HTable$ClientScanner: Advancing forward from regi
on REGION => {NAME => 'test,,1244755386281', STARTKEY => '', ENDKEY => '7fd99653
bf71ea6c747d34db7846c708', ENCODED => 1840196805, TABLE => {{NAME => 'test', FAM
ILIES => [{NAME => 'content', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '21
47483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {
NAME => 'info', VERSIONS => '3', COMPRESSION => 'NONE', TTL => '2147483647', BLO
CKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
09/06/11 14:23:33 DEBUG client.HTable$ClientScanner: Advancing internal scanner
to startKey at '7fd99653bf71ea6c747d34db7846c708'
NativeException: java.lang.RuntimeException: org.apache.hadoop.hbase.client.Retr
iesExhaustedException: Trying to contact region server null for region , row '7f
d99653bf71ea6c747d34db7846c708', but failed after 5 attempts.
Exceptions:
java.lang.ArrayIndexOutOfBoundsException: 32
java.lang.ArrayIndexOutOfBoundsException: 32
java.lang.ArrayIndexOutOfBoundsException: 32
java.lang.ArrayIndexOutOfBoundsException: 32
java.lang.ArrayIndexOutOfBoundsException: 32
from org/apache/hadoop/hbase/client/HTable.java:2002:in `hasNext'
from sun.reflect.GeneratedMethodAccessor2:-1:in `invoke'
{code}
Probably the script does a bit more than is strictly necessary to reproduce.
Attached also is the master+regionserver log at DEBUG level from the test.
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718746#action_12718746 ]
Lars George commented on HBASE-1500:
------------------------------------
Just asking Andrew, since you lost me here a bit, the initial issue with the comparator and the first log dump, what has that do with the later client based issues? When I check you first set of logs there were hardly any issues (in terms of exceptions) in it. How does this all fit together? Sorry for being thick.
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1500) KeyValue$KeyComparator array overrun
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719228#action_12719228 ]
ryan rawson commented on HBASE-1500:
------------------------------------
this might be related to HBASE-1503 - the hfile might be going bad during a compaction and or split.
I saw this on a concurrent insert/count on my own cluster.
Either the data is bad (hmm) or the hfile is having a problem?
> KeyValue$KeyComparator array overrun
> ------------------------------------
>
> Key: HBASE-1500
> URL: https://issues.apache.org/jira/browse/HBASE-1500
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0
> Reporter: Andrew Purtell
> Assignee: ryan rawson
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: hbase.log.gz, test.log.gz, test.rb
>
>
> {code}
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@518bf072
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is
> false
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 04/172.20.3.231:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:40296 remote=BOA04/172.20.3.231
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
> e arguments. Applications should implement Tool for the same.
> 09/06/08 22:58:47 WARN mapred.JobClient: No job jar file set. User classes may
> not be found. See JobConf(Class) or JobConf#setJar(String).
> 09/06/08 22:58:47 INFO zookeeper.ZooKeeper: Initiating client connection, host=B
> OA03:2181,BOA02:2181,BOA01:2181,BOA04:2181 sessionTimeout=10000 watcher=org.apac
> he.hadoop.hbase.zookeeper.WatcherWrapper@362f0d54
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Attempting connection to server BOA
> 03/172.20.3.230:2181
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Priming connection to java.nio.chan
> nels.SocketChannel[connected local=/172.20.3.232:42792 remote=BOA03/172.20.3.230
> :2181]
> 09/06/08 22:58:47 INFO zookeeper.ClientCnxn: Server connection successful
> 09/06/08 22:58:48 INFO mapred.TableInputFormatBase: split: 0->BOA04.trendmicro.c
> om:,01e33c601a7a9dd0ddb5c8427438f2f1
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 32
> at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:798)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(KeyValue.j
> ava:1760)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1696)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1755)
> at org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:
> 1687)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getCac
> hedLocation(HConnectionManager.java:697)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> RegionInMeta(HConnectionManager.java:541)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:525)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locate
> Region(HConnectionManager.java:488)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getReg
> ionLocation(HConnectionManager.java:342)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:1
> 91)
> at org.apache.hadoop.hbase.mapred.TableInputFormatBase.getSplits(TableIn
> putFormatBase.java:296)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at net.iridiant.crawler.mapred.DocumentParser.main(Unknown Source)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.