Encoder for JValue

[Arko Provo Mukherjee <ar...@gmail.com>]

Hello Spark Gurus,

I am running into an issue with Encoding and wanted your help.

I have a case class with a JObject in it. Ex:
*case class SomeClass(a: String, b: JObject)*

I also have an encoder for this case class:
*val encoder = Encoders.product[**SomeClass**]*

Now I am creating a DataFrame with the tuple (a, b) from my transformations
and converting into a DataSet:
*df.as <http://df.as>[SomeClass](encoder)*

When I do this, I get the following error:
*java.lang.UnsupportedOperationException: No Encoder found for
org.json4s.JsonAST.JValue*

Appreciate any help regarding this issue.

Many thanks in advance!
Warm regards
Arko

Re: Encoder for JValue

[Arko Provo Mukherjee <ar...@gmail.com>]

Hello Muthu,

Many thanks for your reply. That is what we are currently doing.

However, we finally load the data somewhere and we need to have JSON
objects rather than serialized strings.

Hence I was wondering if there are encoders our there for JObject and if I
can somehow pass that information to Spark.

Thanks & regards
Arko


On Tue, Sep 18, 2018 at 11:39 PM Muthu Jayakumar <ba...@gmail.com> wrote:

> A naive workaround may be to transform the json4s JValue to String (using
> something like compact()) and process it as String? Once you are done with
> the last action, you could write it back as JValue (using something like
> parse())
>
> Thanks,
> Muthu
>
> On Wed, Sep 19, 2018 at 6:35 AM Arko Provo Mukherjee <
> arkoprovomukherjee@gmail.com> wrote:
>
>> Hello Spark Gurus,
>>
>> I am running into an issue with Encoding and wanted your help.
>>
>> I have a case class with a JObject in it. Ex:
>> *case class SomeClass(a: String, b: JObject)*
>>
>> I also have an encoder for this case class:
>> *val encoder = Encoders.product[**SomeClass**]*
>>
>> Now I am creating a DataFrame with the tuple (a, b) from my
>> transformations and converting into a DataSet:
>> *df.as <http://df.as>[SomeClass](encoder)*
>>
>> When I do this, I get the following error:
>> *java.lang.UnsupportedOperationException: No Encoder found for
>> org.json4s.JsonAST.JValue*
>>
>> Appreciate any help regarding this issue.
>>
>> Many thanks in advance!
>> Warm regards
>> Arko
>>
>>
>>

Re: Encoder for JValue

[Muthu Jayakumar <ba...@gmail.com>]

A naive workaround may be to transform the json4s JValue to String (using
something like compact()) and process it as String? Once you are done with
the last action, you could write it back as JValue (using something like
parse())

Thanks,
Muthu

On Wed, Sep 19, 2018 at 6:35 AM Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:

> Hello Spark Gurus,
>
> I am running into an issue with Encoding and wanted your help.
>
> I have a case class with a JObject in it. Ex:
> *case class SomeClass(a: String, b: JObject)*
>
> I also have an encoder for this case class:
> *val encoder = Encoders.product[**SomeClass**]*
>
> Now I am creating a DataFrame with the tuple (a, b) from my
> transformations and converting into a DataSet:
> *df.as <http://df.as>[SomeClass](encoder)*
>
> When I do this, I get the following error:
> *java.lang.UnsupportedOperationException: No Encoder found for
> org.json4s.JsonAST.JValue*
>
> Appreciate any help regarding this issue.
>
> Many thanks in advance!
> Warm regards
> Arko
>
>
>