You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jeetendra Gangele <ga...@gmail.com> on 2015/04/06 22:30:39 UTC
task not serialize
In this code in foreach I am getting task not serialized exception
@SuppressWarnings("serial")
public static void matchAndMerge(JavaRDD<VendorRecord> matchRdd, final
JavaSparkContext jsc) throws IOException{
log.info("Company matcher started");
//final JavaSparkContext jsc = getSparkContext();
matchRdd.foreachAsync(new VoidFunction<VendorRecord>(){
@Override
public void call(VendorRecord t) throws Exception {
if(t !=null){
try{
CompanyMatcherHelper.UpdateMatchedRecord(jsc,t);
} catch (Exception e) {
log.error("ERROR while running Matcher for company " + t.getCompanyId(), e);
}
}
}
});
}
Re: task not serialize
Posted by Jeetendra Gangele <ga...@gmail.com>.
I thinking to follow the below approach(in my class hbase also return the
same object which i will get in RDD)
.1 First run the flatMapPairf
JavaPairRDD<VendorRecord, Iterable<VendorRecord>> pairvendorData
=matchRdd.flatMapToPair( new PairFlatMapFunction<VendorRecord,
VendorRecord, VendorRecord>(){
@Override
public Iterable<Tuple2<VendorRecord,VendorRecord>> call(
VendorRecord t) throws Exception {
List<Tuple2<VendorRecord, VendorRecord>> pairs = new
LinkedList<Tuple2<VendorRecord, VendorRecord>>();
MatcherKeys matchkeys=CompanyMatcherHelper.getBlockinkeys(t);
List<VendorRecord> Matchedrecords
=ckdao.getMatchingRecordsWithscan(matchkeys);
for(int i=0;i<Matchedrecords.size();i++){
pairs.add( new Tuple2<VendorRecord,VendorRecord>(t,Matchedrecords.get(i)));
}
return pairs;
}
}
).groupByKey(200);
Question will it store the returned RDD in one Node? or it only bring when
I run the second step? in groupBy if I increase the partiotionNumber will
it increase the prformance
2. Then apply mapPartition on this RDD and do logistic regression here.my
my issue is my logistic regression function take
On 7 April 2015 at 18:38, Dean Wampler <de...@gmail.com> wrote:
> Foreach() runs in parallel across the cluster, like map, flatMap, etc.
> You'll only run into problems if you call collect(), which brings the
> entire RDD into memory in the driver program.
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
> On Tue, Apr 7, 2015 at 3:50 AM, Jeetendra Gangele <ga...@gmail.com>
> wrote:
>
>> Lets say I follow below approach and I got RddPair with huge size ..
>> which can not fit into one machine ... what to run foreach on this RDD?
>>
>> On 7 April 2015 at 04:25, Jeetendra Gangele <ga...@gmail.com> wrote:
>>
>>>
>>>
>>> On 7 April 2015 at 04:03, Dean Wampler <de...@gmail.com> wrote:
>>>
>>>>
>>>> On Mon, Apr 6, 2015 at 6:20 PM, Jeetendra Gangele <gangele397@gmail.com
>>>> > wrote:
>>>>
>>>>> Thanks a lot.That means Spark does not support the nested RDD?
>>>>> if I pass the javaSparkContext that also wont work. I mean passing
>>>>> SparkContext not possible since its not serializable
>>>>>
>>>>> That's right. RDD don't nest and SparkContexts aren't serializable.
>>>>
>>>>
>>>>> i have a requirement where I will get JavaRDD<VendorRecord> matchRdd
>>>>> and I need to return the postential matches for this record from Hbase. so
>>>>> for each field of VendorRecord I have to do following
>>>>>
>>>>> 1. query Hbase to get the list of potential record in RDD
>>>>> 2. run logistic regression on RDD return from steps 1 and each element
>>>>> of the passed matchRdd.
>>>>>
>>>>> If I understand you correctly, each VectorRecord could correspond to
>>>> 0-to-N records in HBase, which you need to fetch, true?
>>>>
>>> yes thats correct each Vendorrecord corresponds to 0 to N matches
>>>
>>>
>>>> If so, you could use the RDD flatMap method, which takes a function a
>>>> that accepts each record, then returns a sequence of 0-to-N new records of
>>>> some other type, like your HBase records. However, running an HBase query
>>>> for each VendorRecord could be expensive. If you can turn this into a range
>>>> query or something like that, it would help. I haven't used HBase much, so
>>>> I don't have good advice on optimizing this, if necessary.
>>>>
>>>> Alternatively, can you do some sort of join on the VendorRecord RDD and
>>>> an RDD of query results from HBase?
>>>>
>>> Join will give too big result RDD of query result is returning around
>>> 10000 for each record and i have 2 millions to process so it will be huge
>>> to have this. 2 m*10000 big number
>>>
>>>>
>>>> For #2, it sounds like you need flatMap to return records that combine
>>>> the input VendorRecords and fields pulled from HBase.
>>>>
>>>> Whatever you can do to make this work like "table scans" and joins will
>>>> probably be most efficient.
>>>>
>>>> dean
>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 7 April 2015 at 03:33, Dean Wampler <de...@gmail.com> wrote:
>>>>>
>>>>>> The "log" instance won't be serializable, because it will have a file
>>>>>> handle to write to. Try defining another static method outside
>>>>>> matchAndMerge that encapsulates the call to log.error. CompanyMatcherHelper
>>>>>> might not be serializable either, but you didn't provide it. If it holds a
>>>>>> database connection, same problem.
>>>>>>
>>>>>> You can't suppress the warning because it's actually an error. The
>>>>>> VoidFunction can't be serialized to send it over the cluster's network.
>>>>>>
>>>>>> dean
>>>>>>
>>>>>> Dean Wampler, Ph.D.
>>>>>> Author: Programming Scala, 2nd Edition
>>>>>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>>>>>> Typesafe <http://typesafe.com>
>>>>>> @deanwampler <http://twitter.com/deanwampler>
>>>>>> http://polyglotprogramming.com
>>>>>>
>>>>>> On Mon, Apr 6, 2015 at 4:30 PM, Jeetendra Gangele <
>>>>>> gangele397@gmail.com> wrote:
>>>>>>
>>>>>>> In this code in foreach I am getting task not serialized exception
>>>>>>>
>>>>>>>
>>>>>>> @SuppressWarnings("serial")
>>>>>>> public static void matchAndMerge(JavaRDD<VendorRecord> matchRdd,
>>>>>>> final JavaSparkContext jsc) throws IOException{
>>>>>>> log.info("Company matcher started");
>>>>>>> //final JavaSparkContext jsc = getSparkContext();
>>>>>>> matchRdd.foreachAsync(new VoidFunction<VendorRecord>(){
>>>>>>> @Override
>>>>>>> public void call(VendorRecord t) throws Exception {
>>>>>>> if(t !=null){
>>>>>>> try{
>>>>>>> CompanyMatcherHelper.UpdateMatchedRecord(jsc,t);
>>>>>>> } catch (Exception e) {
>>>>>>> log.error("ERROR while running Matcher for company " +
>>>>>>> t.getCompanyId(), e);
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> });
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>> -
>>
>
>
Re: task not serialize
Posted by Dean Wampler <de...@gmail.com>.
Foreach() runs in parallel across the cluster, like map, flatMap, etc.
You'll only run into problems if you call collect(), which brings the
entire RDD into memory in the driver program.
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Tue, Apr 7, 2015 at 3:50 AM, Jeetendra Gangele <ga...@gmail.com>
wrote:
> Lets say I follow below approach and I got RddPair with huge size .. which
> can not fit into one machine ... what to run foreach on this RDD?
>
> On 7 April 2015 at 04:25, Jeetendra Gangele <ga...@gmail.com> wrote:
>
>>
>>
>> On 7 April 2015 at 04:03, Dean Wampler <de...@gmail.com> wrote:
>>
>>>
>>> On Mon, Apr 6, 2015 at 6:20 PM, Jeetendra Gangele <ga...@gmail.com>
>>> wrote:
>>>
>>>> Thanks a lot.That means Spark does not support the nested RDD?
>>>> if I pass the javaSparkContext that also wont work. I mean passing
>>>> SparkContext not possible since its not serializable
>>>>
>>>> That's right. RDD don't nest and SparkContexts aren't serializable.
>>>
>>>
>>>> i have a requirement where I will get JavaRDD<VendorRecord> matchRdd
>>>> and I need to return the postential matches for this record from Hbase. so
>>>> for each field of VendorRecord I have to do following
>>>>
>>>> 1. query Hbase to get the list of potential record in RDD
>>>> 2. run logistic regression on RDD return from steps 1 and each element
>>>> of the passed matchRdd.
>>>>
>>>> If I understand you correctly, each VectorRecord could correspond to
>>> 0-to-N records in HBase, which you need to fetch, true?
>>>
>> yes thats correct each Vendorrecord corresponds to 0 to N matches
>>
>>
>>> If so, you could use the RDD flatMap method, which takes a function a
>>> that accepts each record, then returns a sequence of 0-to-N new records of
>>> some other type, like your HBase records. However, running an HBase query
>>> for each VendorRecord could be expensive. If you can turn this into a range
>>> query or something like that, it would help. I haven't used HBase much, so
>>> I don't have good advice on optimizing this, if necessary.
>>>
>>> Alternatively, can you do some sort of join on the VendorRecord RDD and
>>> an RDD of query results from HBase?
>>>
>> Join will give too big result RDD of query result is returning around
>> 10000 for each record and i have 2 millions to process so it will be huge
>> to have this. 2 m*10000 big number
>>
>>>
>>> For #2, it sounds like you need flatMap to return records that combine
>>> the input VendorRecords and fields pulled from HBase.
>>>
>>> Whatever you can do to make this work like "table scans" and joins will
>>> probably be most efficient.
>>>
>>> dean
>>>
>>>>
>>>>
>>>>
>>>> On 7 April 2015 at 03:33, Dean Wampler <de...@gmail.com> wrote:
>>>>
>>>>> The "log" instance won't be serializable, because it will have a file
>>>>> handle to write to. Try defining another static method outside
>>>>> matchAndMerge that encapsulates the call to log.error. CompanyMatcherHelper
>>>>> might not be serializable either, but you didn't provide it. If it holds a
>>>>> database connection, same problem.
>>>>>
>>>>> You can't suppress the warning because it's actually an error. The
>>>>> VoidFunction can't be serialized to send it over the cluster's network.
>>>>>
>>>>> dean
>>>>>
>>>>> Dean Wampler, Ph.D.
>>>>> Author: Programming Scala, 2nd Edition
>>>>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>>>>> Typesafe <http://typesafe.com>
>>>>> @deanwampler <http://twitter.com/deanwampler>
>>>>> http://polyglotprogramming.com
>>>>>
>>>>> On Mon, Apr 6, 2015 at 4:30 PM, Jeetendra Gangele <
>>>>> gangele397@gmail.com> wrote:
>>>>>
>>>>>> In this code in foreach I am getting task not serialized exception
>>>>>>
>>>>>>
>>>>>> @SuppressWarnings("serial")
>>>>>> public static void matchAndMerge(JavaRDD<VendorRecord> matchRdd,
>>>>>> final JavaSparkContext jsc) throws IOException{
>>>>>> log.info("Company matcher started");
>>>>>> //final JavaSparkContext jsc = getSparkContext();
>>>>>> matchRdd.foreachAsync(new VoidFunction<VendorRecord>(){
>>>>>> @Override
>>>>>> public void call(VendorRecord t) throws Exception {
>>>>>> if(t !=null){
>>>>>> try{
>>>>>> CompanyMatcherHelper.UpdateMatchedRecord(jsc,t);
>>>>>> } catch (Exception e) {
>>>>>> log.error("ERROR while running Matcher for company " +
>>>>>> t.getCompanyId(), e);
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>> });
>>>>>>
>>>>>> }
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>>
>
>
> -
>
Re: task not serialize
Posted by Jeetendra Gangele <ga...@gmail.com>.
Lets say I follow below approach and I got RddPair with huge size .. which
can not fit into one machine ... what to run foreach on this RDD?
On 7 April 2015 at 04:25, Jeetendra Gangele <ga...@gmail.com> wrote:
>
>
> On 7 April 2015 at 04:03, Dean Wampler <de...@gmail.com> wrote:
>
>>
>> On Mon, Apr 6, 2015 at 6:20 PM, Jeetendra Gangele <ga...@gmail.com>
>> wrote:
>>
>>> Thanks a lot.That means Spark does not support the nested RDD?
>>> if I pass the javaSparkContext that also wont work. I mean passing
>>> SparkContext not possible since its not serializable
>>>
>>> That's right. RDD don't nest and SparkContexts aren't serializable.
>>
>>
>>> i have a requirement where I will get JavaRDD<VendorRecord> matchRdd
>>> and I need to return the postential matches for this record from Hbase. so
>>> for each field of VendorRecord I have to do following
>>>
>>> 1. query Hbase to get the list of potential record in RDD
>>> 2. run logistic regression on RDD return from steps 1 and each element
>>> of the passed matchRdd.
>>>
>>> If I understand you correctly, each VectorRecord could correspond to
>> 0-to-N records in HBase, which you need to fetch, true?
>>
> yes thats correct each Vendorrecord corresponds to 0 to N matches
>
>
>> If so, you could use the RDD flatMap method, which takes a function a
>> that accepts each record, then returns a sequence of 0-to-N new records of
>> some other type, like your HBase records. However, running an HBase query
>> for each VendorRecord could be expensive. If you can turn this into a range
>> query or something like that, it would help. I haven't used HBase much, so
>> I don't have good advice on optimizing this, if necessary.
>>
>> Alternatively, can you do some sort of join on the VendorRecord RDD and
>> an RDD of query results from HBase?
>>
> Join will give too big result RDD of query result is returning around
> 10000 for each record and i have 2 millions to process so it will be huge
> to have this. 2 m*10000 big number
>
>>
>> For #2, it sounds like you need flatMap to return records that combine
>> the input VendorRecords and fields pulled from HBase.
>>
>> Whatever you can do to make this work like "table scans" and joins will
>> probably be most efficient.
>>
>> dean
>>
>>>
>>>
>>>
>>> On 7 April 2015 at 03:33, Dean Wampler <de...@gmail.com> wrote:
>>>
>>>> The "log" instance won't be serializable, because it will have a file
>>>> handle to write to. Try defining another static method outside
>>>> matchAndMerge that encapsulates the call to log.error. CompanyMatcherHelper
>>>> might not be serializable either, but you didn't provide it. If it holds a
>>>> database connection, same problem.
>>>>
>>>> You can't suppress the warning because it's actually an error. The
>>>> VoidFunction can't be serialized to send it over the cluster's network.
>>>>
>>>> dean
>>>>
>>>> Dean Wampler, Ph.D.
>>>> Author: Programming Scala, 2nd Edition
>>>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>>>> Typesafe <http://typesafe.com>
>>>> @deanwampler <http://twitter.com/deanwampler>
>>>> http://polyglotprogramming.com
>>>>
>>>> On Mon, Apr 6, 2015 at 4:30 PM, Jeetendra Gangele <gangele397@gmail.com
>>>> > wrote:
>>>>
>>>>> In this code in foreach I am getting task not serialized exception
>>>>>
>>>>>
>>>>> @SuppressWarnings("serial")
>>>>> public static void matchAndMerge(JavaRDD<VendorRecord> matchRdd,
>>>>> final JavaSparkContext jsc) throws IOException{
>>>>> log.info("Company matcher started");
>>>>> //final JavaSparkContext jsc = getSparkContext();
>>>>> matchRdd.foreachAsync(new VoidFunction<VendorRecord>(){
>>>>> @Override
>>>>> public void call(VendorRecord t) throws Exception {
>>>>> if(t !=null){
>>>>> try{
>>>>> CompanyMatcherHelper.UpdateMatchedRecord(jsc,t);
>>>>> } catch (Exception e) {
>>>>> log.error("ERROR while running Matcher for company " +
>>>>> t.getCompanyId(), e);
>>>>> }
>>>>> }
>>>>> }
>>>>> });
>>>>>
>>>>> }
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>
>
>
>
-
Re: task not serialize
Posted by Jeetendra Gangele <ga...@gmail.com>.
On 7 April 2015 at 04:03, Dean Wampler <de...@gmail.com> wrote:
>
> On Mon, Apr 6, 2015 at 6:20 PM, Jeetendra Gangele <ga...@gmail.com>
> wrote:
>
>> Thanks a lot.That means Spark does not support the nested RDD?
>> if I pass the javaSparkContext that also wont work. I mean passing
>> SparkContext not possible since its not serializable
>>
>> That's right. RDD don't nest and SparkContexts aren't serializable.
>
>
>
>> i have a requirement where I will get JavaRDD<VendorRecord> matchRdd and
>> I need to return the postential matches for this record from Hbase. so for
>> each field of VendorRecord I have to do following
>>
>> 1. query Hbase to get the list of potential record in RDD
>> 2. run logistic regression on RDD return from steps 1 and each element of
>> the passed matchRdd.
>>
>> If I understand you correctly, each VectorRecord could correspond to
> 0-to-N records in HBase, which you need to fetch, true?
>
yes thats correct each Vendorrecord corresponds to 0 to N matches
> If so, you could use the RDD flatMap method, which takes a function a
> that accepts each record, then returns a sequence of 0-to-N new records of
> some other type, like your HBase records. However, running an HBase query
> for each VendorRecord could be expensive. If you can turn this into a range
> query or something like that, it would help. I haven't used HBase much, so
> I don't have good advice on optimizing this, if necessary.
>
> Alternatively, can you do some sort of join on the VendorRecord RDD and an
> RDD of query results from HBase?
>
Join will give too big result RDD of query result is returning around
10000 for each record and i have 2 millions to process so it will be huge
to have this. 2 m*10000 big number
>
> For #2, it sounds like you need flatMap to return records that combine the
> input VendorRecords and fields pulled from HBase.
>
> Whatever you can do to make this work like "table scans" and joins will
> probably be most efficient.
>
> dean
>
>>
>>
>>
>> On 7 April 2015 at 03:33, Dean Wampler <de...@gmail.com> wrote:
>>
>>> The "log" instance won't be serializable, because it will have a file
>>> handle to write to. Try defining another static method outside
>>> matchAndMerge that encapsulates the call to log.error. CompanyMatcherHelper
>>> might not be serializable either, but you didn't provide it. If it holds a
>>> database connection, same problem.
>>>
>>> You can't suppress the warning because it's actually an error. The
>>> VoidFunction can't be serialized to send it over the cluster's network.
>>>
>>> dean
>>>
>>> Dean Wampler, Ph.D.
>>> Author: Programming Scala, 2nd Edition
>>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>>> Typesafe <http://typesafe.com>
>>> @deanwampler <http://twitter.com/deanwampler>
>>> http://polyglotprogramming.com
>>>
>>> On Mon, Apr 6, 2015 at 4:30 PM, Jeetendra Gangele <ga...@gmail.com>
>>> wrote:
>>>
>>>> In this code in foreach I am getting task not serialized exception
>>>>
>>>>
>>>> @SuppressWarnings("serial")
>>>> public static void matchAndMerge(JavaRDD<VendorRecord> matchRdd,
>>>> final JavaSparkContext jsc) throws IOException{
>>>> log.info("Company matcher started");
>>>> //final JavaSparkContext jsc = getSparkContext();
>>>> matchRdd.foreachAsync(new VoidFunction<VendorRecord>(){
>>>> @Override
>>>> public void call(VendorRecord t) throws Exception {
>>>> if(t !=null){
>>>> try{
>>>> CompanyMatcherHelper.UpdateMatchedRecord(jsc,t);
>>>> } catch (Exception e) {
>>>> log.error("ERROR while running Matcher for company " +
>>>> t.getCompanyId(), e);
>>>> }
>>>> }
>>>> }
>>>> });
>>>>
>>>> }
>>>>
>>>
>>>
>>
>>
>>
>>
>
Re: task not serialize
Posted by Dean Wampler <de...@gmail.com>.
On Mon, Apr 6, 2015 at 6:20 PM, Jeetendra Gangele <ga...@gmail.com>
wrote:
> Thanks a lot.That means Spark does not support the nested RDD?
> if I pass the javaSparkContext that also wont work. I mean passing
> SparkContext not possible since its not serializable
>
> That's right. RDD don't nest and SparkContexts aren't serializable.
> i have a requirement where I will get JavaRDD<VendorRecord> matchRdd and
> I need to return the postential matches for this record from Hbase. so for
> each field of VendorRecord I have to do following
>
> 1. query Hbase to get the list of potential record in RDD
> 2. run logistic regression on RDD return from steps 1 and each element of
> the passed matchRdd.
>
> If I understand you correctly, each VectorRecord could correspond to
0-to-N records in HBase, which you need to fetch, true?
If so, you could use the RDD flatMap method, which takes a function a that
accepts each record, then returns a sequence of 0-to-N new records of some
other type, like your HBase records. However, running an HBase query for
each VendorRecord could be expensive. If you can turn this into a range
query or something like that, it would help. I haven't used HBase much, so
I don't have good advice on optimizing this, if necessary.
Alternatively, can you do some sort of join on the VendorRecord RDD and an
RDD of query results from HBase?
For #2, it sounds like you need flatMap to return records that combine the
input VendorRecords and fields pulled from HBase.
Whatever you can do to make this work like "table scans" and joins will
probably be most efficient.
dean
>
>
>
> On 7 April 2015 at 03:33, Dean Wampler <de...@gmail.com> wrote:
>
>> The "log" instance won't be serializable, because it will have a file
>> handle to write to. Try defining another static method outside
>> matchAndMerge that encapsulates the call to log.error. CompanyMatcherHelper
>> might not be serializable either, but you didn't provide it. If it holds a
>> database connection, same problem.
>>
>> You can't suppress the warning because it's actually an error. The
>> VoidFunction can't be serialized to send it over the cluster's network.
>>
>> dean
>>
>> Dean Wampler, Ph.D.
>> Author: Programming Scala, 2nd Edition
>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>> Typesafe <http://typesafe.com>
>> @deanwampler <http://twitter.com/deanwampler>
>> http://polyglotprogramming.com
>>
>> On Mon, Apr 6, 2015 at 4:30 PM, Jeetendra Gangele <ga...@gmail.com>
>> wrote:
>>
>>> In this code in foreach I am getting task not serialized exception
>>>
>>>
>>> @SuppressWarnings("serial")
>>> public static void matchAndMerge(JavaRDD<VendorRecord> matchRdd,
>>> final JavaSparkContext jsc) throws IOException{
>>> log.info("Company matcher started");
>>> //final JavaSparkContext jsc = getSparkContext();
>>> matchRdd.foreachAsync(new VoidFunction<VendorRecord>(){
>>> @Override
>>> public void call(VendorRecord t) throws Exception {
>>> if(t !=null){
>>> try{
>>> CompanyMatcherHelper.UpdateMatchedRecord(jsc,t);
>>> } catch (Exception e) {
>>> log.error("ERROR while running Matcher for company " + t.getCompanyId(),
>>> e);
>>> }
>>> }
>>> }
>>> });
>>>
>>> }
>>>
>>
>>
>
>
>
>
Re: task not serialize
Posted by Jeetendra Gangele <ga...@gmail.com>.
Thanks a lot.That means Spark does not support the nested RDD?
if I pass the javaSparkContext that also wont work. I mean passing
SparkContext not possible since its not serializable
i have a requirement where I will get JavaRDD<VendorRecord> matchRdd and I
need to return the postential matches for this record from Hbase. so for
each field of VendorRecord I have to do following
1. query Hbase to get the list of potential record in RDD
2. run logistic regression on RDD return from steps 1 and each element of
the passed matchRdd.
On 7 April 2015 at 03:33, Dean Wampler <de...@gmail.com> wrote:
> The "log" instance won't be serializable, because it will have a file
> handle to write to. Try defining another static method outside
> matchAndMerge that encapsulates the call to log.error. CompanyMatcherHelper
> might not be serializable either, but you didn't provide it. If it holds a
> database connection, same problem.
>
> You can't suppress the warning because it's actually an error. The
> VoidFunction can't be serialized to send it over the cluster's network.
>
> dean
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
> On Mon, Apr 6, 2015 at 4:30 PM, Jeetendra Gangele <ga...@gmail.com>
> wrote:
>
>> In this code in foreach I am getting task not serialized exception
>>
>>
>> @SuppressWarnings("serial")
>> public static void matchAndMerge(JavaRDD<VendorRecord> matchRdd, final
>> JavaSparkContext jsc) throws IOException{
>> log.info("Company matcher started");
>> //final JavaSparkContext jsc = getSparkContext();
>> matchRdd.foreachAsync(new VoidFunction<VendorRecord>(){
>> @Override
>> public void call(VendorRecord t) throws Exception {
>> if(t !=null){
>> try{
>> CompanyMatcherHelper.UpdateMatchedRecord(jsc,t);
>> } catch (Exception e) {
>> log.error("ERROR while running Matcher for company " + t.getCompanyId(),
>> e);
>> }
>> }
>> }
>> });
>>
>> }
>>
>
>
Re: task not serialize
Posted by Dean Wampler <de...@gmail.com>.
The "log" instance won't be serializable, because it will have a file
handle to write to. Try defining another static method outside
matchAndMerge that encapsulates the call to log.error. CompanyMatcherHelper
might not be serializable either, but you didn't provide it. If it holds a
database connection, same problem.
You can't suppress the warning because it's actually an error. The
VoidFunction can't be serialized to send it over the cluster's network.
dean
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Mon, Apr 6, 2015 at 4:30 PM, Jeetendra Gangele <ga...@gmail.com>
wrote:
> In this code in foreach I am getting task not serialized exception
>
>
> @SuppressWarnings("serial")
> public static void matchAndMerge(JavaRDD<VendorRecord> matchRdd, final
> JavaSparkContext jsc) throws IOException{
> log.info("Company matcher started");
> //final JavaSparkContext jsc = getSparkContext();
> matchRdd.foreachAsync(new VoidFunction<VendorRecord>(){
> @Override
> public void call(VendorRecord t) throws Exception {
> if(t !=null){
> try{
> CompanyMatcherHelper.UpdateMatchedRecord(jsc,t);
> } catch (Exception e) {
> log.error("ERROR while running Matcher for company " + t.getCompanyId(),
> e);
> }
> }
> }
> });
>
> }
>