You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Amit Sela <am...@infolinks.com> on 2014/05/25 11:02:44 UTC

Added a key not lexically larger than previous key

I'm running HBase 0.94.12 (Hadoop 1.0.4).
I have a daily bulk load job that runs for a long time now, suddenly I got
this exception:

java.io.IOException: Added a key not lexically larger than previous
key=\x00U20140523]org.cinetux\x09/2013/11/ver-pelicula-charlie-countryman-online-gratis-2013.html\x03gen\x03WMQ\x00\x00\x00\x05\x02MX\x0210\x010\
x11www.infolinks.com\x08tagcloud\x00\x00\x01F-\x83\xB0\x88\x04,
lastkey=\x00R20140523_org.cinetux\x09/2013/11/ver-pelicula-caza-al-asesino-online-gratis-2013.html\x03gen\x03WMQ\x00\x00\x00\x05\x02VE\x03901\x010\x0Cfacebook.com\x08tagcloud\x00\x00\x01F-\x83\xB0\x88\x04
at
org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
..........

I'm running with configureIncrementalLoad() and using
ImmutableBytesWritable as key and Put as value (leads to using
PutSortReducer).

The weird thing is after I re-executed the same job, it worked just fine,
no exceptions...

While I'm happy it works now, I really don't understand what could go wrong
there ?

Re: Added a key not lexically larger than previous key

Posted by Amit Sela <am...@infolinks.com>.
The files for bulk load are generated with MR job where output format is
sequence file, key is ImmutableBytesWritable and value is Put.
Bulk load MR map only writes out the key value as they are read by the
Mapper.
Don't have the files anymore... if it'll happen again I'll try to read them
with the hfile tool.
Thanks.


On Sun, May 25, 2014 at 1:41 PM, Stack <st...@duboce.net> wrote:

> On Sun, May 25, 2014 at 2:02 AM, Amit Sela <am...@infolinks.com> wrote:
>
> > I'm running HBase 0.94.12 (Hadoop 1.0.4).
> > I have a daily bulk load job that runs for a long time now, suddenly I
> got
> > this exception:
> >
> > java.io.IOException: Added a key not lexically larger than previous
> >
> >
> key=\x00U20140523]org.cinetux\x09/2013/11/ver-pelicula-charlie-countryman-online-gratis-2013.html\x03gen\x03WMQ\x00\x00\x00\x05\x02MX\x0210\x010\
> > x11www.infolinks.com\x08tagcloud\x00\x00\x01F-\x83\xB0\x88\x04,
> >
> >
> lastkey=\x00R20140523_org.cinetux\x09/2013/11/ver-pelicula-caza-al-asesino-online-gratis-2013.html\x03gen\x03WMQ\x00\x00\x00\x05\x02VE\x03901\x010\x0Cfacebook.com\x08tagcloud\x00\x00\x01F-\x83\xB0\x88\x04
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
> > ..........
> >
> > I'm running with configureIncrementalLoad() and using
> > ImmutableBytesWritable as key and Put as value (leads to using
> > PutSortReducer).
> >
> > The weird thing is after I re-executed the same job, it worked just fine,
> > no exceptions...
> >
> > While I'm happy it works now, I really don't understand what could go
> wrong
> > there ?
> >
>
> It does indeed look as though keys in the file are not properly ordered.
>  How are you generating the files you are bulk loading?  Do you have the
> file still?  IIRC, reading the file with the hfile tool will fail if it is
> not sorted properly: http://hbase.apache.org/book.html#hfile_tool2
>
> St.Ack
>

Re: Added a key not lexically larger than previous key

Posted by Stack <st...@duboce.net>.
On Sun, May 25, 2014 at 2:02 AM, Amit Sela <am...@infolinks.com> wrote:

> I'm running HBase 0.94.12 (Hadoop 1.0.4).
> I have a daily bulk load job that runs for a long time now, suddenly I got
> this exception:
>
> java.io.IOException: Added a key not lexically larger than previous
>
> key=\x00U20140523]org.cinetux\x09/2013/11/ver-pelicula-charlie-countryman-online-gratis-2013.html\x03gen\x03WMQ\x00\x00\x00\x05\x02MX\x0210\x010\
> x11www.infolinks.com\x08tagcloud\x00\x00\x01F-\x83\xB0\x88\x04,
>
> lastkey=\x00R20140523_org.cinetux\x09/2013/11/ver-pelicula-caza-al-asesino-online-gratis-2013.html\x03gen\x03WMQ\x00\x00\x00\x05\x02VE\x03901\x010\x0Cfacebook.com\x08tagcloud\x00\x00\x01F-\x83\xB0\x88\x04
> at
>
> org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
> ..........
>
> I'm running with configureIncrementalLoad() and using
> ImmutableBytesWritable as key and Put as value (leads to using
> PutSortReducer).
>
> The weird thing is after I re-executed the same job, it worked just fine,
> no exceptions...
>
> While I'm happy it works now, I really don't understand what could go wrong
> there ?
>

It does indeed look as though keys in the file are not properly ordered.
 How are you generating the files you are bulk loading?  Do you have the
file still?  IIRC, reading the file with the hfile tool will fail if it is
not sorted properly: http://hbase.apache.org/book.html#hfile_tool2

St.Ack