Spark sql with large sql syntax job failed with outofmemory error and grows beyond 64k warn

[FangFang Chen <lu...@163.com>]

Hi all,
With large sql command, job failed with following error. Please give your suggestion on how to resolve it. Thanks


Sql file size: 676k
Log:
16/04/25 10:55:00 WARN TaskSetManager: Lost task 84.0 in stage 0.0 (TID 6, BJHC-HADOOP-HERA-17493.jd.local): java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Code of method "(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 KB


public Object generate(org.apache.spark.sql.catalyst.expressions.Expression[] exprs) {
  return new SpecificUnsafeProjection(exprs);
}


class SpecificUnsafeProjection extends org.apache.spark.sql.catalyst.expressions.UnsafeProjection {




......


java.lang.OutOfMemoryError: Java heap space
  at com.google.protobuf.ByteString.copyFrom(ByteString.java:192)
  at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:324)
  at akka.remote.WireFormats$AkkaProtocolMessage.<init>(WireFormats.java:6657)
  at akka.remote.WireFormats$AkkaProtocolMessage.<init>(WireFormats.java:6607)
  at akka.remote.WireFormats$AkkaProtocolMessage$1.parsePartialFrom(WireFormats.java:6703)
  at akka.remote.WireFormats$AkkaProtocolMessage$1.parsePartialFrom(WireFormats.java:6698)
  at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:141)
  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176)
  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188)
  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193)
  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
  at akka.remote.WireFormats$AkkaProtocolMessage.parseFrom(WireFormats.java:6821)
  at akka.remote.transport.AkkaPduProtobufCodec$.decodePdu(AkkaPduCodec.scala:168)


  


发自 网易邮箱大师