You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jianshi Huang <ji...@gmail.com> on 2014/11/21 11:11:27 UTC
How to deal with BigInt in my case class for RDD => SchemaRDD convertion
Hi,
I got an error during rdd.registerTempTable(...) saying scala.MatchError:
scala.BigInt
Looks like BigInt cannot be used in SchemaRDD, is that correct?
So what would you recommend to deal with it?
Thanks,
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: How to deal with BigInt in my case class for RDD => SchemaRDD convertion
Posted by Jianshi Huang <ji...@gmail.com>.
Ah yes. I found it too in the manual. Thanks for the help anyway!
Since BigDecimal is just a wrapper around BigInt, let's also convert to
BigInt to Decimal.
I created a ticket. https://issues.apache.org/jira/browse/SPARK-4549
Jianshi
On Fri, Nov 21, 2014 at 11:30 PM, Yin Huai <hu...@gmail.com> wrote:
> Hello Jianshi,
>
> The reason of that error is that we do not have a Spark SQL data type for
> Scala BigInt. You can use Decimal for your case.
>
> Thanks,
>
> Yin
>
> On Fri, Nov 21, 2014 at 5:11 AM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I got an error during rdd.registerTempTable(...) saying scala.MatchError:
>> scala.BigInt
>>
>> Looks like BigInt cannot be used in SchemaRDD, is that correct?
>>
>> So what would you recommend to deal with it?
>>
>> Thanks,
>> --
>> Jianshi Huang
>>
>> LinkedIn: jianshi
>> Twitter: @jshuang
>> Github & Blog: http://huangjs.github.com/
>>
>
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: How to deal with BigInt in my case class for RDD => SchemaRDD convertion
Posted by Yin Huai <hu...@gmail.com>.
Hello Jianshi,
The reason of that error is that we do not have a Spark SQL data type for
Scala BigInt. You can use Decimal for your case.
Thanks,
Yin
On Fri, Nov 21, 2014 at 5:11 AM, Jianshi Huang <ji...@gmail.com>
wrote:
> Hi,
>
> I got an error during rdd.registerTempTable(...) saying scala.MatchError:
> scala.BigInt
>
> Looks like BigInt cannot be used in SchemaRDD, is that correct?
>
> So what would you recommend to deal with it?
>
> Thanks,
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>