You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Nirav Patel <np...@xactlycorp.com> on 2016/11/14 22:21:39 UTC
SparkException: Task not serializable for Closure variable
I have a following map:
final val idxMapArr = idxMap.collectAsMap
Which is being used in one of spark transformation here:
def labelStr(predictions: WrappedArray[Double]): Array[String] = {
predictions.array.map(*idxMapArr*.getOrElse(_, "Other"))
}
I get following error :
Caused by: java.io.NotSerializableException:
com.github.fommil.netlib.F2jBLAS
Serialization stack:
- object not serializable (class: com.github.fommil.netlib.F2jBLAS, value:
com.github.fommil.netlib.F2jBLAS@294770d3)
- field (class: org.apache.spark.ml.tuning.CrossValidator, name: f2jBLAS,
type: class com.github.fommil.netlib.F2jBLAS)
- object (class org.apache.spark.ml.tuning.CrossValidator, cv_891fd6b7d95f)
- field (class: $iw, name: crossValidator, type: class
org.apache.spark.ml.tuning.CrossValidator)
- object (class $iw, $iw@556a6aed)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@4a1212a0)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@34f1718d)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@1d4bb5f7)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@5a30765)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@39a33f4)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@335d6de3)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@2b4ccb90)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@1814c6c9)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@7cd91fb0)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@d6c803f)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@488b6cf6)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@4eee8af2)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@4301fc7)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@5dd61af2)
- field (class: $iw, name: $iw, type: class $iw)
- object (class $iw, $iw@4ce26a93)
- field (class: $line95653076098.$read, name: $iw, type: class $iw)
- object (class $line95653076098.$read, $line95653076098.$read@c7d550f)
- field (class: $iw, name: $line95653076098$read, type: class
$line95653076098.$read)
- object (class $iw, $iw@1c6701cd)
--
[image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
<https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
<https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
<https://twitter.com/Xactly> [image: Facebook]
<https://www.facebook.com/XactlyCorp> [image: YouTube]
<http://www.youtube.com/xactlycorporation>
Re: SparkException: Task not serializable for Closure variable
Posted by Nirav Patel <np...@xactlycorp.com>.
I have to try but I think it probably will happen with spark-shell as well.
I have found alternative to pass a Array to UDF as a parameter.
Thanks
On Thu, Nov 17, 2016 at 9:24 AM, moon soo Lee <mo...@apache.org> wrote:
> Are you able to run the same code in SPARK_HOME/bin/spark-shell ?
>
> Thanks,
> moon
>
> On Mon, Nov 14, 2016 at 2:21 PM Nirav Patel <np...@xactlycorp.com> wrote:
>
>> I have a following map:
>>
>> final val idxMapArr = idxMap.collectAsMap
>>
>> Which is being used in one of spark transformation here:
>>
>> def labelStr(predictions: WrappedArray[Double]): Array[String] = {
>>
>> predictions.array.map(*idxMapArr*.getOrElse(_, "Other"))
>> }
>>
>> I get following error :
>>
>>
>> Caused by: java.io.NotSerializableException: com.github.fommil.netlib.
>> F2jBLAS
>> Serialization stack:
>> - object not serializable (class: com.github.fommil.netlib.F2jBLAS,
>> value: com.github.fommil.netlib.F2jBLAS@294770d3)
>> - field (class: org.apache.spark.ml.tuning.CrossValidator, name:
>> f2jBLAS, type: class com.github.fommil.netlib.F2jBLAS)
>> - object (class org.apache.spark.ml.tuning.CrossValidator,
>> cv_891fd6b7d95f)
>> - field (class: $iw, name: crossValidator, type: class
>> org.apache.spark.ml.tuning.CrossValidator)
>> - object (class $iw, $iw@556a6aed)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@4a1212a0)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@34f1718d)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@1d4bb5f7)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@5a30765)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@39a33f4)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@335d6de3)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@2b4ccb90)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@1814c6c9)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@7cd91fb0)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@d6c803f)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@488b6cf6)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@4eee8af2)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@4301fc7)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@5dd61af2)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@4ce26a93)
>> - field (class: $line95653076098.$read, name: $iw, type: class $iw)
>> - object (class $line95653076098.$read, $line95653076098.$read@c7d550f)
>> - field (class: $iw, name: $line95653076098$read, type: class
>> $line95653076098.$read)
>> - object (class $iw, $iw@1c6701cd)
>>
>>
>>
>> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>>
>> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
>> <https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
>> <https://twitter.com/Xactly> [image: Facebook]
>> <https://www.facebook.com/XactlyCorp> [image: YouTube]
>> <http://www.youtube.com/xactlycorporation>
>
>
--
[image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
<https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
<https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
<https://twitter.com/Xactly> [image: Facebook]
<https://www.facebook.com/XactlyCorp> [image: YouTube]
<http://www.youtube.com/xactlycorporation>
Re: SparkException: Task not serializable for Closure variable
Posted by moon soo Lee <mo...@apache.org>.
Are you able to run the same code in SPARK_HOME/bin/spark-shell ?
Thanks,
moon
On Mon, Nov 14, 2016 at 2:21 PM Nirav Patel <np...@xactlycorp.com> wrote:
> I have a following map:
>
> final val idxMapArr = idxMap.collectAsMap
>
> Which is being used in one of spark transformation here:
>
> def labelStr(predictions: WrappedArray[Double]): Array[String] = {
>
> predictions.array.map(*idxMapArr*.getOrElse(_, "Other"))
> }
>
> I get following error :
>
>
> Caused by: java.io.NotSerializableException:
> com.github.fommil.netlib.F2jBLAS
> Serialization stack:
> - object not serializable (class: com.github.fommil.netlib.F2jBLAS, value:
> com.github.fommil.netlib.F2jBLAS@294770d3)
> - field (class: org.apache.spark.ml.tuning.CrossValidator, name: f2jBLAS,
> type: class com.github.fommil.netlib.F2jBLAS)
> - object (class org.apache.spark.ml.tuning.CrossValidator, cv_891fd6b7d95f)
> - field (class: $iw, name: crossValidator, type: class
> org.apache.spark.ml.tuning.CrossValidator)
> - object (class $iw, $iw@556a6aed)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@4a1212a0)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@34f1718d)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@1d4bb5f7)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@5a30765)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@39a33f4)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@335d6de3)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@2b4ccb90)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@1814c6c9)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@7cd91fb0)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@d6c803f)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@488b6cf6)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@4eee8af2)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@4301fc7)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@5dd61af2)
> - field (class: $iw, name: $iw, type: class $iw)
> - object (class $iw, $iw@4ce26a93)
> - field (class: $line95653076098.$read, name: $iw, type: class $iw)
> - object (class $line95653076098.$read, $line95653076098.$read@c7d550f)
> - field (class: $iw, name: $line95653076098$read, type: class
> $line95653076098.$read)
> - object (class $iw, $iw@1c6701cd)
>
>
>
> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>
> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
> <https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
> <https://twitter.com/Xactly> [image: Facebook]
> <https://www.facebook.com/XactlyCorp> [image: YouTube]
> <http://www.youtube.com/xactlycorporation>