[jira] [Created] (SPARK-15156) String fields in Dataframe behaves weirdly when executor-memory >= 32GB

["Varadharajan (JIRA)" <ji...@apache.org>]

Varadharajan created SPARK-15156:
------------------------------------

             Summary: String fields in Dataframe behaves weirdly when executor-memory >= 32GB
                 Key: SPARK-15156
                 URL: https://issues.apache.org/jira/browse/SPARK-15156
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 1.5.0
            Reporter: Varadharajan
            Priority: Critical


Consider the below snippet. This issue was faced on an infrastructure with CDH 5.5.2 (which ships with Spark 1.5.0).

{code:title=test.scala|borderStyle=solid}
case class A(id: Int, name: String)
sc.parallelize(List(A(1, "varadharajan"), A(2, "thiyagu"))).toDF.show
{code}

When i execute the above snippet with the below command line arguments, it works as expected.

{code}
spark-shell --master yarn  --num-executors 4 --executor-cores 10 --executor-memory 31g
{code}

{quote}
| id|        name|
|  1|varadharajan|
|  2|     thiyagu|
{quote}

But things become weird, when i increase executor memory beyond 32GB. 

{code}
spark-shell --master yarn  --num-executors 4 --executor-cores 10 --executor-memory 32g
{code}

{quote}
| id|        name|
|  1|ajan|
|  2|     |
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org