Spark SQL Dataset and BigDecimal

[Ivan Petrov <ca...@gmail.com>]

Hi, I'm using Spark Scala Dataset API to write spark sql jobs.
I've noticed that Spark dataset accepts scala BigDecimal as the value but
it always returns java.math.BigDecimal when you read it back.

Is it by design?
Should I use java.math.BigDecimal everywhere instead?
Is there any performance penalty for  using scala BigDecimal? it's more
convenient from an API point of view than java.math.BigDecimal.

Re: Spark SQL Dataset and BigDecimal

[Khalid Mammadov <kh...@gmail.com>]

As Scala book says Value types are mapped/transformed to java primitive types. So when you use Integer for example it will compile to int. So Integer is a syntactic sugar and makes it more readable in Scala code than plain int and plus Scala adds extra perks through implicits etc. I think the same goes for this BigDecimal case. So I personally would go with Scala types and compiler should do the rest for you.

Cheers,
Khalid Mammadov

> On 18 Feb 2021, at 09:39, Ivan Petrov <ca...@gmail.com> wrote:
> 
> 
> I'm fine with both. So does it make sense to use java.math.BigDecimal everywhere to avoid perf penalty for value conversion? 
> scala BigMath looks like a wrapper around java.math.BigDecimal though...
> 
> чт, 18 февр. 2021 г. в 00:33, Takeshi Yamamuro <li...@gmail.com>:
>> Yea, I think that's because it's needed for interoperability between scala/java.
>> If it returns a scala decimal, java code cannot handle it.
>> 
>> If you want a scala decimal, you need to convert it by yourself.
>> 
>> Bests,
>> Takeshi
>> 
>>> On Wed, Feb 17, 2021 at 9:48 PM Ivan Petrov <ca...@gmail.com> wrote:
>>> Hi, I'm using Spark Scala Dataset API to write spark sql jobs.
>>> I've noticed that Spark dataset accepts scala BigDecimal as the value but it always returns java.math.BigDecimal when you read it back.
>>> 
>>> Is it by design?
>>> Should I use java.math.BigDecimal everywhere instead?
>>> Is there any performance penalty for  using scala BigDecimal? it's more convenient from an API point of view than java.math.BigDecimal.
>> 
>> 
>> -- 
>> ---
>> Takeshi Yamamuro

Re: Spark SQL Dataset and BigDecimal

[Ivan Petrov <ca...@gmail.com>]

I'm fine with both. So does it make sense to use java.math.BigDecimal
everywhere to avoid perf penalty for value conversion?
scala BigMath looks like a wrapper around java.math.BigDecimal though...

чт, 18 февр. 2021 г. в 00:33, Takeshi Yamamuro <li...@gmail.com>:

> Yea, I think that's because it's needed for interoperability between
> scala/java.
> If it returns a scala decimal, java code cannot handle it.
>
> If you want a scala decimal, you need to convert it by yourself.
>
> Bests,
> Takeshi
>
> On Wed, Feb 17, 2021 at 9:48 PM Ivan Petrov <ca...@gmail.com> wrote:
>
>> Hi, I'm using Spark Scala Dataset API to write spark sql jobs.
>> I've noticed that Spark dataset accepts scala BigDecimal as the value but
>> it always returns java.math.BigDecimal when you read it back.
>>
>> Is it by design?
>> Should I use java.math.BigDecimal everywhere instead?
>> Is there any performance penalty for  using scala BigDecimal? it's more
>> convenient from an API point of view than java.math.BigDecimal.
>>
>
>
> --
> ---
> Takeshi Yamamuro
>

Re: Spark SQL Dataset and BigDecimal

[Takeshi Yamamuro <li...@gmail.com>]

Yea, I think that's because it's needed for interoperability between
scala/java.
If it returns a scala decimal, java code cannot handle it.

If you want a scala decimal, you need to convert it by yourself.

Bests,
Takeshi

On Wed, Feb 17, 2021 at 9:48 PM Ivan Petrov <ca...@gmail.com> wrote:

> Hi, I'm using Spark Scala Dataset API to write spark sql jobs.
> I've noticed that Spark dataset accepts scala BigDecimal as the value but
> it always returns java.math.BigDecimal when you read it back.
>
> Is it by design?
> Should I use java.math.BigDecimal everywhere instead?
> Is there any performance penalty for  using scala BigDecimal? it's more
> convenient from an API point of view than java.math.BigDecimal.
>


-- 
---
Takeshi Yamamuro