You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jeetendra Gangele <ga...@gmail.com> on 2015/04/06 20:37:07 UTC
Re: java.io.NotSerializableException: org.apache.hadoop.hbase.client.Result
I hit again same issue This time I tried to return the Object it failed
with task not serialized below is the code
here vendor record is serializable
private static JavaRDD<VendorRecord>
getVendorDataToProcess(JavaSparkContext sc) throws IOException {
return sc
.newAPIHadoopRDD(getVendorDataRowKeyScannerConfiguration(),
TableInputFormat.class,
ImmutableBytesWritable.class, Result.class)
.map(new Function<Tuple2<ImmutableBytesWritable, Result>,
VendorRecord>() {
@Override
public VendorRecord call(Tuple2<ImmutableBytesWritable, Result> v1)
throws Exception {
String rowKey = new String(v1._1.get());
VendorRecord vd=vendorDataDAO.getVendorDataForRowkey(rowKey);
return vd;
}
});
}
On 1 April 2015 at 02:07, Ted Yu <yu...@gmail.com> wrote:
> Jeetendra:
> Please extract the information you need from Result and return the
> extracted portion - instead of returning Result itself.
>
> Cheers
>
> On Tue, Mar 31, 2015 at 1:14 PM, Nan Zhu <zh...@gmail.com> wrote:
>
>> The example in
>> https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/HBaseTest.scala might
>> help
>>
>> Best,
>>
>> --
>> Nan Zhu
>> http://codingcat.me
>>
>> On Tuesday, March 31, 2015 at 3:56 PM, Sean Owen wrote:
>>
>> Yep, it's not serializable:
>>
>> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Result.html
>>
>> You can't return this from a distributed operation since that would
>> mean it has to travel over the network and you haven't supplied any
>> way to convert the thing into bytes.
>>
>> On Tue, Mar 31, 2015 at 8:51 PM, Jeetendra Gangele <ga...@gmail.com>
>> wrote:
>>
>> When I am trying to get the result from Hbase and running mapToPair
>> function
>> of RRD its giving the error
>> java.io.NotSerializableException: org.apache.hadoop.hbase.client.Result
>>
>> Here is the code
>>
>> // private static JavaPairRDD<Integer, Result>
>> getCompanyDataRDD(JavaSparkContext sc) throws IOException {
>> // return sc.newAPIHadoopRDD(companyDAO.getCompnayDataConfiguration(),
>> TableInputFormat.class, ImmutableBytesWritable.class,
>> // Result.class).mapToPair(new
>> PairFunction<Tuple2<ImmutableBytesWritable, Result>, Integer, Result>() {
>> //
>> // public Tuple2<Integer, Result> call(Tuple2<ImmutableBytesWritable,
>> Result> t) throws Exception {
>> // System.out.println("In getCompanyDataRDD"+t._2);
>> //
>> // String cknid = Bytes.toString(t._1.get());
>> // System.out.println("processing cknids is:"+cknid);
>> // Integer cknidInt = Integer.parseInt(cknid);
>> // Tuple2<Integer, Result> returnTuple = new Tuple2<Integer,
>> Result>(cknidInt, t._2);
>> // return returnTuple;
>> // }
>> // });
>> // }
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>>
>
Re: java.io.NotSerializableException: org.apache.hadoop.hbase.client.Result
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
One hack you can put in would be to bring Result class
<http://grepcode.com/file_/repository.cloudera.com/content/repositories/releases/com.cloudera.hbase/hbase/0.89.20100924-28/org/apache/hadoop/hbase/client/Result.java/?v=source>
locally and serialize it (implements serializable) and use it.
Thanks
Best Regards
On Tue, Apr 7, 2015 at 12:07 AM, Jeetendra Gangele <ga...@gmail.com>
wrote:
> I hit again same issue This time I tried to return the Object it failed
> with task not serialized below is the code
> here vendor record is serializable
>
> private static JavaRDD<VendorRecord>
> getVendorDataToProcess(JavaSparkContext sc) throws IOException {
> return sc
> .newAPIHadoopRDD(getVendorDataRowKeyScannerConfiguration(),
> TableInputFormat.class,
> ImmutableBytesWritable.class, Result.class)
> .map(new Function<Tuple2<ImmutableBytesWritable, Result>,
> VendorRecord>() {
> @Override
> public VendorRecord call(Tuple2<ImmutableBytesWritable, Result> v1)
> throws Exception {
> String rowKey = new String(v1._1.get());
> VendorRecord vd=vendorDataDAO.getVendorDataForRowkey(rowKey);
> return vd;
> }
> });
> }
>
>
> On 1 April 2015 at 02:07, Ted Yu <yu...@gmail.com> wrote:
>
>> Jeetendra:
>> Please extract the information you need from Result and return the
>> extracted portion - instead of returning Result itself.
>>
>> Cheers
>>
>> On Tue, Mar 31, 2015 at 1:14 PM, Nan Zhu <zh...@gmail.com> wrote:
>>
>>> The example in
>>> https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/HBaseTest.scala might
>>> help
>>>
>>> Best,
>>>
>>> --
>>> Nan Zhu
>>> http://codingcat.me
>>>
>>> On Tuesday, March 31, 2015 at 3:56 PM, Sean Owen wrote:
>>>
>>> Yep, it's not serializable:
>>>
>>> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Result.html
>>>
>>> You can't return this from a distributed operation since that would
>>> mean it has to travel over the network and you haven't supplied any
>>> way to convert the thing into bytes.
>>>
>>> On Tue, Mar 31, 2015 at 8:51 PM, Jeetendra Gangele <ga...@gmail.com>
>>> wrote:
>>>
>>> When I am trying to get the result from Hbase and running mapToPair
>>> function
>>> of RRD its giving the error
>>> java.io.NotSerializableException: org.apache.hadoop.hbase.client.Result
>>>
>>> Here is the code
>>>
>>> // private static JavaPairRDD<Integer, Result>
>>> getCompanyDataRDD(JavaSparkContext sc) throws IOException {
>>> // return sc.newAPIHadoopRDD(companyDAO.getCompnayDataConfiguration(),
>>> TableInputFormat.class, ImmutableBytesWritable.class,
>>> // Result.class).mapToPair(new
>>> PairFunction<Tuple2<ImmutableBytesWritable, Result>, Integer, Result>() {
>>> //
>>> // public Tuple2<Integer, Result> call(Tuple2<ImmutableBytesWritable,
>>> Result> t) throws Exception {
>>> // System.out.println("In getCompanyDataRDD"+t._2);
>>> //
>>> // String cknid = Bytes.toString(t._1.get());
>>> // System.out.println("processing cknids is:"+cknid);
>>> // Integer cknidInt = Integer.parseInt(cknid);
>>> // Tuple2<Integer, Result> returnTuple = new Tuple2<Integer,
>>> Result>(cknidInt, t._2);
>>> // return returnTuple;
>>> // }
>>> // });
>>> // }
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>>
>>
>
>
>
>