You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Guillermo Ortiz <ko...@gmail.com> on 2014/04/30 17:57:13 UTC
Error loading SHA-1 keys with load bulk
I'm using HBase with MapReduce to load a lot of data, so I have decide to
do it with bulk load.
I parse my keys with SHA1, but when I try to load them, I got this
exception.
java.io.IOException: Added a key not lexically larger than previous
key=\x00(6e9e59f36a7ec2ac54635b2d353e53e677839046\x01l\x00\x00\x01E\xB3>\xC9\xC7\x0E,
lastkey=\x00(b313a9f1f57c8a07c81dc3221c6151cf3637506a\x01l\x00\x00\x01E\xAE\x18k\x87\x0E
at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:324)
at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:289)
at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1206)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:168)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:124)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:551)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
I work with HBase 0.94.6. I have been loking for if I could define any
reducer, since, I have defined no one. I have read something about
KeyValueSortReducer but, I don'tknow if there's something that extends
TableReducer or I'm lookging for a wrong way.
Re: Error loading SHA-1 keys with load bulk
Posted by Guillermo Ortiz <ko...@gmail.com>.
The error was that when I was emitting the <K,V>, I was doing SHA about K,
not about the key in the Value.
The Value is a KeyValue and here's where I had to do the SHA1.
2014-05-02 0:42 GMT+02:00 Guillermo Ortiz <ko...@gmail.com>:
> Yes, I do,
>
>
> job.setMapperClass(EventMapper.class);
> job.setMapOutputKeyClass(ImmutableBytesWritable.class);
> job.setMapOutputValueClass(KeyValue.class);
>
> FileOutputFormat.setOutputPath(job, hbasePath);
> HTable table = new HTable(jConf, MEM_TABLE_HBASE);
> HFileOutputFormat.configureIncrementalLoad(job, table);
>
>
> The error is happeing in a MRUnit, I don't know if it changes something
> about the behavior, because I had some troubles in the past for the same
> reason about the serialization in Hbase 0.96 and MRUnit.
> . Besides, in the setup of the MRUnit test I load some data in hbase with
> keys in sha1 and it works.
>
> El jueves, 1 de mayo de 2014, Jean-Daniel Cryans <jd...@apache.org>
> escribió:
>
> Are you using HFileOutputFormat.configureIncrementalLoad() to set up the
>> partitioner and the reducers? That will take care of ordering your keys.
>>
>> J-D
>>
>>
>> On Thu, May 1, 2014 at 5:38 AM, Guillermo Ortiz <konstt2000@gmail.com
>> >wrote:
>>
>> > I have been looking at the code in HBase, but, I don't really understand
>> > what this error happens. Why can I put in HBase those keys?
>> >
>> >
>> > 2014-04-30 17:57 GMT+02:00 Guillermo Ortiz
>> > <konstt2000@gmail.com<javascript:_e(%7B%7D,'cvml','konstt2000@gmail.com
>> > ');>
>> > >:
>> >
>> > > I'm using HBase with MapReduce to load a lot of data, so I have
>> decide to
>> > > do it with bulk load.
>> > >
>> > >
>> > > I parse my keys with SHA1, but when I try to load them, I got this
>> > > exception.
>> > >
>> > > java.io.IOException: Added a key not lexically larger than previous
>> >
>> key=\x00(6e9e59f36a7ec2ac54635b2d353e53e677839046\x01l\x00\x00\x01E\xB3>\xC9\xC7\x0E,
>> >
>> lastkey=\x00(b313a9f1f57c8a07c81dc3221c6151cf3637506a\x01l\x00\x00\x01E\xAE\x18k\x87\x0E
>> > > at
>> >
>> org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
>> > > at
>> >
>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:324)
>> > > at
>> >
>> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:289)
>> > > at
>> >
>> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1206)
>> > > at
>> >
>> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:168)
>> > > at
>> >
>> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:124)
>> > > at
>> >
>> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:551)
>> > > at
>> >
>> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>> > >
>> > > I work with HBase 0.94.6. I have been loking for if I could define any
>> > reducer, since, I have defined no one. I have read something about
>> > KeyValueSortReducer but, I don'tknow if there's something that extends
>> > TableReducer or I'm lookging for a wrong way.
>> > >
>> > >
>> > >
>> >
>>
>
Re: Error loading SHA-1 keys with load bulk
Posted by Guillermo Ortiz <ko...@gmail.com>.
Yes, I do,
job.setMapperClass(EventMapper.class);
job.setMapOutputKeyClass(ImmutableBytesWritable.class);
job.setMapOutputValueClass(KeyValue.class);
FileOutputFormat.setOutputPath(job, hbasePath);
HTable table = new HTable(jConf, MEM_TABLE_HBASE);
HFileOutputFormat.configureIncrementalLoad(job, table);
The error is happeing in a MRUnit, I don't know if it changes something
about the behavior, because I had some troubles in the past for the same
reason about the serialization in Hbase 0.96 and MRUnit.
. Besides, in the setup of the MRUnit test I load some data in hbase with
keys in sha1 and it works.
El jueves, 1 de mayo de 2014, Jean-Daniel Cryans <jd...@apache.org>
escribió:
> Are you using HFileOutputFormat.configureIncrementalLoad() to set up the
> partitioner and the reducers? That will take care of ordering your keys.
>
> J-D
>
>
> On Thu, May 1, 2014 at 5:38 AM, Guillermo Ortiz <konstt2000@gmail.com<javascript:;>
> >wrote:
>
> > I have been looking at the code in HBase, but, I don't really understand
> > what this error happens. Why can I put in HBase those keys?
> >
> >
> > 2014-04-30 17:57 GMT+02:00 Guillermo Ortiz
> > <konstt2000@gmail.com <javascript:;><javascript:_e(%7B%7D,'cvml','
> konstt2000@gmail.com <javascript:;>
> > ');>
> > >:
> >
> > > I'm using HBase with MapReduce to load a lot of data, so I have decide
> to
> > > do it with bulk load.
> > >
> > >
> > > I parse my keys with SHA1, but when I try to load them, I got this
> > > exception.
> > >
> > > java.io.IOException: Added a key not lexically larger than previous
> >
> key=\x00(6e9e59f36a7ec2ac54635b2d353e53e677839046\x01l\x00\x00\x01E\xB3>\xC9\xC7\x0E,
> >
> lastkey=\x00(b313a9f1f57c8a07c81dc3221c6151cf3637506a\x01l\x00\x00\x01E\xAE\x18k\x87\x0E
> > > at
> >
> org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
> > > at
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:324)
> > > at
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:289)
> > > at
> >
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1206)
> > > at
> >
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:168)
> > > at
> >
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:124)
> > > at
> >
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:551)
> > > at
> >
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
> > >
> > > I work with HBase 0.94.6. I have been loking for if I could define any
> > reducer, since, I have defined no one. I have read something about
> > KeyValueSortReducer but, I don'tknow if there's something that extends
> > TableReducer or I'm lookging for a wrong way.
> > >
> > >
> > >
> >
>
Re: Error loading SHA-1 keys with load bulk
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Are you using HFileOutputFormat.configureIncrementalLoad() to set up the
partitioner and the reducers? That will take care of ordering your keys.
J-D
On Thu, May 1, 2014 at 5:38 AM, Guillermo Ortiz <ko...@gmail.com>wrote:
> I have been looking at the code in HBase, but, I don't really understand
> what this error happens. Why can I put in HBase those keys?
>
>
> 2014-04-30 17:57 GMT+02:00 Guillermo Ortiz
> <konstt2000@gmail.com<javascript:_e(%7B%7D,'cvml','konstt2000@gmail.com
> ');>
> >:
>
> > I'm using HBase with MapReduce to load a lot of data, so I have decide to
> > do it with bulk load.
> >
> >
> > I parse my keys with SHA1, but when I try to load them, I got this
> > exception.
> >
> > java.io.IOException: Added a key not lexically larger than previous
> key=\x00(6e9e59f36a7ec2ac54635b2d353e53e677839046\x01l\x00\x00\x01E\xB3>\xC9\xC7\x0E,
> lastkey=\x00(b313a9f1f57c8a07c81dc3221c6151cf3637506a\x01l\x00\x00\x01E\xAE\x18k\x87\x0E
> > at
> org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
> > at
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:324)
> > at
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:289)
> > at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1206)
> > at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:168)
> > at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:124)
> > at
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:551)
> > at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
> >
> > I work with HBase 0.94.6. I have been loking for if I could define any
> reducer, since, I have defined no one. I have read something about
> KeyValueSortReducer but, I don'tknow if there's something that extends
> TableReducer or I'm lookging for a wrong way.
> >
> >
> >
>
Error loading SHA-1 keys with load bulk
Posted by Guillermo Ortiz <ko...@gmail.com>.
I have been looking at the code in HBase, but, I don't really understand
what this error happens. Why can I put in HBase those keys?
2014-04-30 17:57 GMT+02:00 Guillermo Ortiz
<konstt2000@gmail.com<javascript:_e(%7B%7D,'cvml','konstt2000@gmail.com');>
>:
> I'm using HBase with MapReduce to load a lot of data, so I have decide to
> do it with bulk load.
>
>
> I parse my keys with SHA1, but when I try to load them, I got this
> exception.
>
> java.io.IOException: Added a key not lexically larger than previous key=\x00(6e9e59f36a7ec2ac54635b2d353e53e677839046\x01l\x00\x00\x01E\xB3>\xC9\xC7\x0E, lastkey=\x00(b313a9f1f57c8a07c81dc3221c6151cf3637506a\x01l\x00\x00\x01E\xAE\x18k\x87\x0E
> at org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:207)
> at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:324)
> at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:289)
> at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1206)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:168)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.write(HFileOutputFormat.java:124)
> at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:551)
> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
>
> I work with HBase 0.94.6. I have been loking for if I could define any reducer, since, I have defined no one. I have read something about KeyValueSortReducer but, I don'tknow if there's something that extends TableReducer or I'm lookging for a wrong way.
>
>
>