You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Serega Sheypak <se...@gmail.com> on 2014/11/02 22:02:41 UTC
No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
I had a unit test for mapper used to create HFile and bulk load later.
I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
Now I've started to get exception
java.lang.IllegalStateException: No applicable class implementing
Serialization in conf at io.serializations: class
org.apache.hadoop.hbase.client.Put
at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at
org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
at
org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
at
org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
at
org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
at
org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
at
org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
at
org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
at
org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
at
org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
at
org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(<generated>)
And here is mapper code:
public class ItemRecommendationHBaseMapper extends Mapper<LongWritable,
BytesWritable, ImmutableBytesWritable, Put> {
private final ImmutableBytesWritable hbaseKey = new
ImmutableBytesWritable();
private final DynamicObjectSerDe<ItemRecommendation> serde = new
DynamicObjectSerDe<ItemRecommendation>(ItemRecommendation.class);
@Override
protected void map(LongWritable key, BytesWritable value, Context
context) throws IOException, InterruptedException {
checkPreconditions(key, value);
hbaseKey.set(Bytes.toBytes(key.get()));
ItemRecommendation item = serde.deserialize(value.getBytes());
checkPreconditions(item);
Put put = PutFactory.createPut(serde, item, getColumnFamily());
context.write(hbaseKey, put); //Exception here
}
Whatcan i do in order to make unit-test pass?
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Ted Yu <yu...@gmail.com>.
Since your code uses mrunit which hbase 0.98 is not dependent on, have you
considered asking this question on:
mrunit-user@incubator.apache.org
Cheers
On Sun, Nov 2, 2014 at 1:22 PM, Serega Sheypak <se...@gmail.com>
wrote:
> public static Put createPut(DynamicObjectSerDe<ItemRecommendation> serde,
> ItemRecommendation item, String columnFamily){
> Put put = new Put(Bytes.toBytes(Long.valueOf(item.getId())));
> put.add(Bytes.toBytes(columnFamily), Bytes.toBytes(item.getRank()),
> serde.serialize(item));
> return put;
> }
>
> 2014-11-03 0:12 GMT+03:00 Ted Yu <yu...@gmail.com>:
>
> > bq. PutFactory.createPut(
> >
> > Can you reveal how PutFactory creates the Put ?
> >
> > Thanks
> >
> > On Sun, Nov 2, 2014 at 1:02 PM, Serega Sheypak <serega.sheypak@gmail.com
> >
> > wrote:
> >
> > > Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
> > > I had a unit test for mapper used to create HFile and bulk load later.
> > >
> > > I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
> > > Now I've started to get exception
> > >
> > > java.lang.IllegalStateException: No applicable class implementing
> > > Serialization in conf at io.serializations: class
> > > org.apache.hadoop.hbase.client.Put
> > > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
> > > at
> > >
> > >
> >
> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> > > at
> > >
> > >
> >
> org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> > > at
> > >
> > >
> >
> org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> > > at
> > >
> > >
> >
> org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> > > at
> > >
> > >
> >
> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(<generated>)
> > >
> > >
> > > And here is mapper code:
> > >
> > >
> > >
> > > public class ItemRecommendationHBaseMapper extends Mapper<LongWritable,
> > > BytesWritable, ImmutableBytesWritable, Put> {
> > >
> > > private final ImmutableBytesWritable hbaseKey = new
> > > ImmutableBytesWritable();
> > > private final DynamicObjectSerDe<ItemRecommendation> serde = new
> > > DynamicObjectSerDe<ItemRecommendation>(ItemRecommendation.class);
> > >
> > > @Override
> > > protected void map(LongWritable key, BytesWritable value, Context
> > > context) throws IOException, InterruptedException {
> > > checkPreconditions(key, value);
> > > hbaseKey.set(Bytes.toBytes(key.get()));
> > >
> > > ItemRecommendation item = serde.deserialize(value.getBytes());
> > > checkPreconditions(item);
> > > Put put = PutFactory.createPut(serde, item, getColumnFamily());
> > >
> > > context.write(hbaseKey, put); //Exception here
> > > }
> > >
> > > Whatcan i do in order to make unit-test pass?
> > >
> >
>
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Serega Sheypak <se...@gmail.com>.
public static Put createPut(DynamicObjectSerDe<ItemRecommendation> serde,
ItemRecommendation item, String columnFamily){
Put put = new Put(Bytes.toBytes(Long.valueOf(item.getId())));
put.add(Bytes.toBytes(columnFamily), Bytes.toBytes(item.getRank()),
serde.serialize(item));
return put;
}
2014-11-03 0:12 GMT+03:00 Ted Yu <yu...@gmail.com>:
> bq. PutFactory.createPut(
>
> Can you reveal how PutFactory creates the Put ?
>
> Thanks
>
> On Sun, Nov 2, 2014 at 1:02 PM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
> > I had a unit test for mapper used to create HFile and bulk load later.
> >
> > I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
> > Now I've started to get exception
> >
> > java.lang.IllegalStateException: No applicable class implementing
> > Serialization in conf at io.serializations: class
> > org.apache.hadoop.hbase.client.Put
> > at
> com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> > at
> >
> >
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
> > at
> >
> >
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
> > at
> >
> >
> org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
> > at
> >
> >
> org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
> > at
> >
> >
> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> > at
> >
> >
> org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> > at
> >
> >
> org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> > at
> >
> >
> org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> > at
> >
> >
> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> > at
> >
> >
> org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(<generated>)
> >
> >
> > And here is mapper code:
> >
> >
> >
> > public class ItemRecommendationHBaseMapper extends Mapper<LongWritable,
> > BytesWritable, ImmutableBytesWritable, Put> {
> >
> > private final ImmutableBytesWritable hbaseKey = new
> > ImmutableBytesWritable();
> > private final DynamicObjectSerDe<ItemRecommendation> serde = new
> > DynamicObjectSerDe<ItemRecommendation>(ItemRecommendation.class);
> >
> > @Override
> > protected void map(LongWritable key, BytesWritable value, Context
> > context) throws IOException, InterruptedException {
> > checkPreconditions(key, value);
> > hbaseKey.set(Bytes.toBytes(key.get()));
> >
> > ItemRecommendation item = serde.deserialize(value.getBytes());
> > checkPreconditions(item);
> > Put put = PutFactory.createPut(serde, item, getColumnFamily());
> >
> > context.write(hbaseKey, put); //Exception here
> > }
> >
> > Whatcan i do in order to make unit-test pass?
> >
>
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Ted Yu <yu...@gmail.com>.
bq. PutFactory.createPut(
Can you reveal how PutFactory creates the Put ?
Thanks
On Sun, Nov 2, 2014 at 1:02 PM, Serega Sheypak <se...@gmail.com>
wrote:
> Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
> I had a unit test for mapper used to create HFile and bulk load later.
>
> I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
> Now I've started to get exception
>
> java.lang.IllegalStateException: No applicable class implementing
> Serialization in conf at io.serializations: class
> org.apache.hadoop.hbase.client.Put
> at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> at
>
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
> at
>
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
> at
>
> org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
> at
>
> org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
> at
>
> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> at
>
> org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> at
>
> org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> at
>
> org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> at
>
> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> at
>
> org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(<generated>)
>
>
> And here is mapper code:
>
>
>
> public class ItemRecommendationHBaseMapper extends Mapper<LongWritable,
> BytesWritable, ImmutableBytesWritable, Put> {
>
> private final ImmutableBytesWritable hbaseKey = new
> ImmutableBytesWritable();
> private final DynamicObjectSerDe<ItemRecommendation> serde = new
> DynamicObjectSerDe<ItemRecommendation>(ItemRecommendation.class);
>
> @Override
> protected void map(LongWritable key, BytesWritable value, Context
> context) throws IOException, InterruptedException {
> checkPreconditions(key, value);
> hbaseKey.set(Bytes.toBytes(key.get()));
>
> ItemRecommendation item = serde.deserialize(value.getBytes());
> checkPreconditions(item);
> Put put = PutFactory.createPut(serde, item, getColumnFamily());
>
> context.write(hbaseKey, put); //Exception here
> }
>
> Whatcan i do in order to make unit-test pass?
>
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Sean Busbey <bu...@cloudera.com>.
On Sun, Nov 2, 2014 at 4:14 PM, Serega Sheypak <se...@gmail.com>
wrote:
> Cool, is it this stuff?
> http://hbase.apache.org/book/hfilev2.html
>
>
No, that's all details on the update to the backing HFile format that
started in HBase 0.92. The change in output format is detailed here:
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.html
It's basically just that the output format changed from using the old
private KeyValue class to the public one for Cell.
--
Sean
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Serega Sheypak <se...@gmail.com>.
Cool, is it this stuff?
http://hbase.apache.org/book/hfilev2.html
2014-11-03 1:10 GMT+03:00 Sean Busbey <bu...@cloudera.com>:
> On Sun, Nov 2, 2014 at 3:53 PM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > P.S.
> > HFileOutputFormat is also deprecated... so many changes... (((
> >
> >
>
> Incidentally, you should consider switching to HFileOutputFormat2, since
> you rely on the version that has a Mapper outputting Put values instead of
> KeyValue the impact on you should be negligible.
>
>
> --
> Sean
>
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Sean Busbey <bu...@cloudera.com>.
On Sun, Nov 2, 2014 at 3:53 PM, Serega Sheypak <se...@gmail.com>
wrote:
> P.S.
> HFileOutputFormat is also deprecated... so many changes... (((
>
>
Incidentally, you should consider switching to HFileOutputFormat2, since
you rely on the version that has a Mapper outputting Put values instead of
KeyValue the impact on you should be negligible.
--
Sean
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Serega Sheypak <se...@gmail.com>.
Ok, I got it. Thank you!
2014-11-03 2:20 GMT+03:00 Sean Busbey <bu...@cloudera.com>:
> On Sun, Nov 2, 2014 at 5:09 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. context.write(hbaseKey, put); //Exception here
> >
> > I am not mrunit expert. But as long as you call the following method
> prior
> > to the above method invocation, you should be able to proceed:
> >
> > conf.setStrings("io.serializations", conf.get("io.serializations"),
> >
> > MutationSerialization.class.getName(), ResultSerialization.class
> > .getName(),
> >
> > KeyValueSerialization.class.getName());
> >
> >
>
> Those classes are not a part of the public HBase API, so directly
> referencing them is a bad idea. Doing so just sets them up to break on some
> future HBase upgrade.
>
> The OP needs a place in MRUnit to call one of
> HFileOutputFormat.configureIncrementalLoad,
> HFileOutputFormat2.configureIncrementalLoad, or
> TableMapReduceUtil.initTableReducerJob. Those are the only public API ways
> to configure the needed Serialization.
>
> --
> Sean
>
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Sean Busbey <bu...@cloudera.com>.
On Sun, Nov 2, 2014 at 5:09 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. context.write(hbaseKey, put); //Exception here
>
> I am not mrunit expert. But as long as you call the following method prior
> to the above method invocation, you should be able to proceed:
>
> conf.setStrings("io.serializations", conf.get("io.serializations"),
>
> MutationSerialization.class.getName(), ResultSerialization.class
> .getName(),
>
> KeyValueSerialization.class.getName());
>
>
Those classes are not a part of the public HBase API, so directly
referencing them is a bad idea. Doing so just sets them up to break on some
future HBase upgrade.
The OP needs a place in MRUnit to call one of
HFileOutputFormat.configureIncrementalLoad,
HFileOutputFormat2.configureIncrementalLoad, or
TableMapReduceUtil.initTableReducerJob. Those are the only public API ways
to configure the needed Serialization.
--
Sean
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Ted Yu <yu...@gmail.com>.
bq. context.write(hbaseKey, put); //Exception here
I am not mrunit expert. But as long as you call the following method prior
to the above method invocation, you should be able to proceed:
conf.setStrings("io.serializations", conf.get("io.serializations"),
MutationSerialization.class.getName(), ResultSerialization.class
.getName(),
KeyValueSerialization.class.getName());
Cheers
On Sun, Nov 2, 2014 at 2:24 PM, Sean Busbey <bu...@cloudera.com> wrote:
> On Sun, Nov 2, 2014 at 4:11 PM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > Sean, I've started to catch that serialization problem on unit-test level
> > while using mrunit.
> > I don't see any possibility to call
> > HFileOutputFormat.configureIncrementalLoad
> > before mrunit mocking stuff.
> > I was workngi w/o any problem in 0.94 :)
> >
> >
> >
> Ah. Well, that sounds like a bug in MRUnit for dealing with HBase 0.96+.
>
> As Ted mentioned, you'll have more luck on the mrunit mailing list for
> figuring that bit out.
>
> --
> Sean
>
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Sean Busbey <bu...@cloudera.com>.
On Sun, Nov 2, 2014 at 4:11 PM, Serega Sheypak <se...@gmail.com>
wrote:
> Sean, I've started to catch that serialization problem on unit-test level
> while using mrunit.
> I don't see any possibility to call
> HFileOutputFormat.configureIncrementalLoad
> before mrunit mocking stuff.
> I was workngi w/o any problem in 0.94 :)
>
>
>
Ah. Well, that sounds like a bug in MRUnit for dealing with HBase 0.96+.
As Ted mentioned, you'll have more luck on the mrunit mailing list for
figuring that bit out.
--
Sean
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Serega Sheypak <se...@gmail.com>.
Sean, I've started to catch that serialization problem on unit-test level
while using mrunit.
I don't see any possibility to call HFileOutputFormat.configureIncrementalLoad
before mrunit mocking stuff.
I was workngi w/o any problem in 0.94 :)
2014-11-03 1:08 GMT+03:00 Sean Busbey <bu...@cloudera.com>:
> If you're calling HFileOutputFormat.configureIncrementalLoad, that should
> be setting up the Serialization for you.
>
> Can you look at the job configuration and see what's present for the key
> "io.serializations"?
>
> -Sean
>
> On Sun, Nov 2, 2014 at 3:53 PM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > I use it to prepare HFile using my custom mapper emitting Put and
> > HFileOutputFormat.configureIncrementalLoad(job, createHTable())
> > //connection to target table
> >
> > and then bulk load data to table using LoadIncrementalHFiles
> >
> > P.S.
> > HFileOutputFormat is also deprecated... so many changes... (((
> >
> >
> > 2014-11-03 0:41 GMT+03:00 Sean Busbey <bu...@cloudera.com>:
> >
> > > In the 0.94.x API, Put implemented Writable[1]. This meant that MR
> code,
> > > like yours, could use it as a Key or Value between Mapper and Reducer.
> > >
> > > In 0.96 and later APIs, Put no longer directly implements Writable[2].
> > > Instead, HBase now includes a Hadoop Seriazliation implementation.
> > > Normally, this would be configured via the TableMapReduceUtil class for
> > > either a TableMapper or TableReducer.
> > >
> > > Presuming that the intention of your MR job is to have all the Puts
> write
> > > to some HBase table, you should be able to follow the "write to HBase"
> > part
> > > of the examples for reading and writing HBase via mapreduce in the
> > > reference guide[3].
> > >
> > > Specifically, you should have your Driver call one of the
> > > initTableReducerJob methods on TableMapReduceUtil, where it currently
> > sets
> > > the Mapper class for your application[4].
> > >
> > > -Sean
> > >
> > > [1]:
> > >
> > >
> >
> http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/Put.html
> > > [2]:
> > >
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html
> > > [3]: http://hbase.apache.org/book/mapreduce.example.html
> > > [4]:
> > >
> > >
> >
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.html
> > >
> > >
> > > On Sun, Nov 2, 2014 at 3:02 PM, Serega Sheypak <
> serega.sheypak@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
> > > > I had a unit test for mapper used to create HFile and bulk load
> later.
> > > >
> > > > I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
> > > > Now I've started to get exception
> > > >
> > > > java.lang.IllegalStateException: No applicable class implementing
> > > > Serialization in conf at io.serializations: class
> > > > org.apache.hadoop.hbase.client.Put
> > > > at
> > > com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
> > > > at
> > > >
> > > >
> > >
> >
> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> > > > at
> > > >
> > > >
> > >
> >
> org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> > > > at
> > > >
> > > >
> > >
> >
> org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> > > > at
> > > >
> > > >
> > >
> >
> org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> > > > at
> > > >
> > > >
> > >
> >
> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(<generated>)
> > > >
> > > >
> > > > And here is mapper code:
> > > >
> > > >
> > > >
> > > > public class ItemRecommendationHBaseMapper extends
> Mapper<LongWritable,
> > > > BytesWritable, ImmutableBytesWritable, Put> {
> > > >
> > > > private final ImmutableBytesWritable hbaseKey = new
> > > > ImmutableBytesWritable();
> > > > private final DynamicObjectSerDe<ItemRecommendation> serde = new
> > > > DynamicObjectSerDe<ItemRecommendation>(ItemRecommendation.class);
> > > >
> > > > @Override
> > > > protected void map(LongWritable key, BytesWritable value, Context
> > > > context) throws IOException, InterruptedException {
> > > > checkPreconditions(key, value);
> > > > hbaseKey.set(Bytes.toBytes(key.get()));
> > > >
> > > > ItemRecommendation item =
> serde.deserialize(value.getBytes());
> > > > checkPreconditions(item);
> > > > Put put = PutFactory.createPut(serde, item,
> getColumnFamily());
> > > >
> > > > context.write(hbaseKey, put); //Exception here
> > > > }
> > > >
> > > > Whatcan i do in order to make unit-test pass?
> > > >
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
>
>
>
> --
> Sean
>
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Sean Busbey <bu...@cloudera.com>.
If you're calling HFileOutputFormat.configureIncrementalLoad, that should
be setting up the Serialization for you.
Can you look at the job configuration and see what's present for the key
"io.serializations"?
-Sean
On Sun, Nov 2, 2014 at 3:53 PM, Serega Sheypak <se...@gmail.com>
wrote:
> I use it to prepare HFile using my custom mapper emitting Put and
> HFileOutputFormat.configureIncrementalLoad(job, createHTable())
> //connection to target table
>
> and then bulk load data to table using LoadIncrementalHFiles
>
> P.S.
> HFileOutputFormat is also deprecated... so many changes... (((
>
>
> 2014-11-03 0:41 GMT+03:00 Sean Busbey <bu...@cloudera.com>:
>
> > In the 0.94.x API, Put implemented Writable[1]. This meant that MR code,
> > like yours, could use it as a Key or Value between Mapper and Reducer.
> >
> > In 0.96 and later APIs, Put no longer directly implements Writable[2].
> > Instead, HBase now includes a Hadoop Seriazliation implementation.
> > Normally, this would be configured via the TableMapReduceUtil class for
> > either a TableMapper or TableReducer.
> >
> > Presuming that the intention of your MR job is to have all the Puts write
> > to some HBase table, you should be able to follow the "write to HBase"
> part
> > of the examples for reading and writing HBase via mapreduce in the
> > reference guide[3].
> >
> > Specifically, you should have your Driver call one of the
> > initTableReducerJob methods on TableMapReduceUtil, where it currently
> sets
> > the Mapper class for your application[4].
> >
> > -Sean
> >
> > [1]:
> >
> >
> http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/Put.html
> > [2]:
> > http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html
> > [3]: http://hbase.apache.org/book/mapreduce.example.html
> > [4]:
> >
> >
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.html
> >
> >
> > On Sun, Nov 2, 2014 at 3:02 PM, Serega Sheypak <serega.sheypak@gmail.com
> >
> > wrote:
> >
> > > Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
> > > I had a unit test for mapper used to create HFile and bulk load later.
> > >
> > > I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
> > > Now I've started to get exception
> > >
> > > java.lang.IllegalStateException: No applicable class implementing
> > > Serialization in conf at io.serializations: class
> > > org.apache.hadoop.hbase.client.Put
> > > at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
> > > at
> > >
> > >
> >
> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> > > at
> > >
> > >
> >
> org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> > > at
> > >
> > >
> >
> org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> > > at
> > >
> > >
> >
> org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> > > at
> > >
> > >
> >
> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(<generated>)
> > >
> > >
> > > And here is mapper code:
> > >
> > >
> > >
> > > public class ItemRecommendationHBaseMapper extends Mapper<LongWritable,
> > > BytesWritable, ImmutableBytesWritable, Put> {
> > >
> > > private final ImmutableBytesWritable hbaseKey = new
> > > ImmutableBytesWritable();
> > > private final DynamicObjectSerDe<ItemRecommendation> serde = new
> > > DynamicObjectSerDe<ItemRecommendation>(ItemRecommendation.class);
> > >
> > > @Override
> > > protected void map(LongWritable key, BytesWritable value, Context
> > > context) throws IOException, InterruptedException {
> > > checkPreconditions(key, value);
> > > hbaseKey.set(Bytes.toBytes(key.get()));
> > >
> > > ItemRecommendation item = serde.deserialize(value.getBytes());
> > > checkPreconditions(item);
> > > Put put = PutFactory.createPut(serde, item, getColumnFamily());
> > >
> > > context.write(hbaseKey, put); //Exception here
> > > }
> > >
> > > Whatcan i do in order to make unit-test pass?
> > >
> >
> >
> >
> > --
> > Sean
> >
>
--
Sean
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Serega Sheypak <se...@gmail.com>.
I use it to prepare HFile using my custom mapper emitting Put and
HFileOutputFormat.configureIncrementalLoad(job, createHTable())
//connection to target table
and then bulk load data to table using LoadIncrementalHFiles
P.S.
HFileOutputFormat is also deprecated... so many changes... (((
2014-11-03 0:41 GMT+03:00 Sean Busbey <bu...@cloudera.com>:
> In the 0.94.x API, Put implemented Writable[1]. This meant that MR code,
> like yours, could use it as a Key or Value between Mapper and Reducer.
>
> In 0.96 and later APIs, Put no longer directly implements Writable[2].
> Instead, HBase now includes a Hadoop Seriazliation implementation.
> Normally, this would be configured via the TableMapReduceUtil class for
> either a TableMapper or TableReducer.
>
> Presuming that the intention of your MR job is to have all the Puts write
> to some HBase table, you should be able to follow the "write to HBase" part
> of the examples for reading and writing HBase via mapreduce in the
> reference guide[3].
>
> Specifically, you should have your Driver call one of the
> initTableReducerJob methods on TableMapReduceUtil, where it currently sets
> the Mapper class for your application[4].
>
> -Sean
>
> [1]:
>
> http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/Put.html
> [2]:
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html
> [3]: http://hbase.apache.org/book/mapreduce.example.html
> [4]:
>
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.html
>
>
> On Sun, Nov 2, 2014 at 3:02 PM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
> > I had a unit test for mapper used to create HFile and bulk load later.
> >
> > I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
> > Now I've started to get exception
> >
> > java.lang.IllegalStateException: No applicable class implementing
> > Serialization in conf at io.serializations: class
> > org.apache.hadoop.hbase.client.Put
> > at
> com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> > at
> >
> >
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
> > at
> >
> >
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
> > at
> >
> >
> org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
> > at
> >
> >
> org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
> > at
> >
> >
> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> > at
> >
> >
> org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> > at
> >
> >
> org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> > at
> >
> >
> org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> > at
> >
> >
> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> > at
> >
> >
> org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(<generated>)
> >
> >
> > And here is mapper code:
> >
> >
> >
> > public class ItemRecommendationHBaseMapper extends Mapper<LongWritable,
> > BytesWritable, ImmutableBytesWritable, Put> {
> >
> > private final ImmutableBytesWritable hbaseKey = new
> > ImmutableBytesWritable();
> > private final DynamicObjectSerDe<ItemRecommendation> serde = new
> > DynamicObjectSerDe<ItemRecommendation>(ItemRecommendation.class);
> >
> > @Override
> > protected void map(LongWritable key, BytesWritable value, Context
> > context) throws IOException, InterruptedException {
> > checkPreconditions(key, value);
> > hbaseKey.set(Bytes.toBytes(key.get()));
> >
> > ItemRecommendation item = serde.deserialize(value.getBytes());
> > checkPreconditions(item);
> > Put put = PutFactory.createPut(serde, item, getColumnFamily());
> >
> > context.write(hbaseKey, put); //Exception here
> > }
> >
> > Whatcan i do in order to make unit-test pass?
> >
>
>
>
> --
> Sean
>
Re: No applicable class implementing Serialization in conf at
io.serializations: class org.apache.hadoop.hbase.client.Put
Posted by Sean Busbey <bu...@cloudera.com>.
In the 0.94.x API, Put implemented Writable[1]. This meant that MR code,
like yours, could use it as a Key or Value between Mapper and Reducer.
In 0.96 and later APIs, Put no longer directly implements Writable[2].
Instead, HBase now includes a Hadoop Seriazliation implementation.
Normally, this would be configured via the TableMapReduceUtil class for
either a TableMapper or TableReducer.
Presuming that the intention of your MR job is to have all the Puts write
to some HBase table, you should be able to follow the "write to HBase" part
of the examples for reading and writing HBase via mapreduce in the
reference guide[3].
Specifically, you should have your Driver call one of the
initTableReducerJob methods on TableMapReduceUtil, where it currently sets
the Mapper class for your application[4].
-Sean
[1]:
http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/Put.html
[2]: http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Put.html
[3]: http://hbase.apache.org/book/mapreduce.example.html
[4]:
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.html
On Sun, Nov 2, 2014 at 3:02 PM, Serega Sheypak <se...@gmail.com>
wrote:
> Hi, I'm migrating from CDH4 to CDH5 (hbase 0.98.6-cdh5.2.0)
> I had a unit test for mapper used to create HFile and bulk load later.
>
> I've bumped maven deps from cdh4 to cdh5 0.98.6-cdh5.2.0
> Now I've started to get exception
>
> java.lang.IllegalStateException: No applicable class implementing
> Serialization in conf at io.serializations: class
> org.apache.hadoop.hbase.client.Put
> at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> at
>
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:75)
> at
>
> org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
> at
>
> org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
> at
>
> org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
> at
>
> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> at
>
> org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:91)
> at
>
> org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> at
>
> org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
> at
>
> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
> at
>
> org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$ba4633fb.write(<generated>)
>
>
> And here is mapper code:
>
>
>
> public class ItemRecommendationHBaseMapper extends Mapper<LongWritable,
> BytesWritable, ImmutableBytesWritable, Put> {
>
> private final ImmutableBytesWritable hbaseKey = new
> ImmutableBytesWritable();
> private final DynamicObjectSerDe<ItemRecommendation> serde = new
> DynamicObjectSerDe<ItemRecommendation>(ItemRecommendation.class);
>
> @Override
> protected void map(LongWritable key, BytesWritable value, Context
> context) throws IOException, InterruptedException {
> checkPreconditions(key, value);
> hbaseKey.set(Bytes.toBytes(key.get()));
>
> ItemRecommendation item = serde.deserialize(value.getBytes());
> checkPreconditions(item);
> Put put = PutFactory.createPut(serde, item, getColumnFamily());
>
> context.write(hbaseKey, put); //Exception here
> }
>
> Whatcan i do in order to make unit-test pass?
>
--
Sean