You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mungeol Heo <mu...@gmail.com> on 2017/04/05 09:52:36 UTC

JSON lib works differently in spark-shell and IDE like intellij

Hello,

I am using "minidev" which is a JSON lib to remove duplicated keys in
JSON object.

--------------------
minidev
--------------------

<dependency>
  <groupId>net.minidev</groupId>
  <artifactId>json-smart</artifactId>
  <version>2.3</version>
</dependency>

--------------------
Test Code
--------------------

import net.minidev.json.parser.JSONParser
val badJson = "{\"keyA\":\"valueA\",\"keyB\":\"valueB\",\"keyA\":\"valueA\"}"
val json = new JSONParser(JSONParser.MODE_PERMISSIVE).parse(badJson.toLowerCase())
println(json)

--------------------

The source code placed above works at IDE like intellij.
But, it gives error at spark-shell

--------------------
Error
--------------------

net.minidev.json.parser.ParseException: Unexpected duplicate key:keya
at position 33.

--------------------

BTW, both IDE and spark-shell using same version of scala which is 2.11.8.
And, of course, same version of "minidev"

Any help will be great.
Thank you.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: JSON lib works differently in spark-shell and IDE like intellij

Posted by Mungeol Heo <mu...@gmail.com>.
It will work with spark-submit, if putting the configuration, which is
addressed below, under the maven-shade-plugin.

<configuration>
  <relocations>
    <relocation>
      <pattern>net.minidev</pattern>
      <shadedPattern>shaded.net.minidev</shadedPattern>
    </relocation>
  </relocations>
</configuration>

Still, need a way to make it work with spark-shell for testing purpose.
Any idea will be grate.

Thank you.

On Wed, Apr 5, 2017 at 6:52 PM, Mungeol Heo <mu...@gmail.com> wrote:
> Hello,
>
> I am using "minidev" which is a JSON lib to remove duplicated keys in
> JSON object.
>
> --------------------
> minidev
> --------------------
>
> <dependency>
>   <groupId>net.minidev</groupId>
>   <artifactId>json-smart</artifactId>
>   <version>2.3</version>
> </dependency>
>
> --------------------
> Test Code
> --------------------
>
> import net.minidev.json.parser.JSONParser
> val badJson = "{\"keyA\":\"valueA\",\"keyB\":\"valueB\",\"keyA\":\"valueA\"}"
> val json = new JSONParser(JSONParser.MODE_PERMISSIVE).parse(badJson.toLowerCase())
> println(json)
>
> --------------------
>
> The source code placed above works at IDE like intellij.
> But, it gives error at spark-shell
>
> --------------------
> Error
> --------------------
>
> net.minidev.json.parser.ParseException: Unexpected duplicate key:keya
> at position 33.
>
> --------------------
>
> BTW, both IDE and spark-shell using same version of scala which is 2.11.8.
> And, of course, same version of "minidev"
>
> Any help will be great.
> Thank you.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org