You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Peter Liu <pe...@gmail.com> on 2018/07/01 14:01:04 UTC
Re: spark 2.3.1 with kafka spark-streaming-kafka-0-10 (java.lang.AbstractMethodError)
Hello there,
I didn't get any response/help from the user list for the following
question and thought people on the dev list might be able to help?:
I upgraded to spark 2.3.1 from spark 2.2.1, ran my streaming workload and
got the error (java.lang.AbstractMethodError) never seen before; see the
error exception stack attached in (a) bellow.
anyone knows if spark 2.3.1 works well with kafka
spark-streaming-kafka-0-10?
this link of spark kafka integration page doesn't say anything about any
limitation:
https://spark.apache.org/docs/2.3.1/streaming-kafka-integration.html
<https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_2.3.1_streaming-2Dkafka-2Dintegration.html&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=LORvnW8MyMYoQuiRJpv4qo1519IpQ57V-MxUgro0vP0&e=>
but this discussion seems to say there is indeed an issue when upgrading to
spark 2.3.1:
https://stackoverflow.com/questions/49180931/abstractmethoderror-creating-kafka-stream
<https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_49180931_abstractmethoderror-2Dcreating-2Dkafka-2Dstream&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=xIl7y-rgXq2AUuNg8cKeBIRApmRMLnrJbPsf4kc5Zzg&e=>
i also rebuilt the workload with some spark 2.3.1 jars (see (b) below). it
doesn't seem to help.
Would be great if anyone could kindly share any insights here.
Thanks!
Peter
(a) the exception
Exception in thread "stream execution thread for [id =
5adae836-268a-4ebf-adc4-e3cc9fbe5acf, runId =
70e78d5c-665e-4c6f-a0cc-41a56e488e30]" java.lang.AbstractMethodError
at
org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)
at
org.apache.spark.sql.kafka010.KafkaSourceProvider$.initializeLogIfNecessary(KafkaSourceProvider.scala:369)
at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
at
org.apache.spark.sql.kafka010.KafkaSourceProvider$.log(KafkaSourceProvider.scala:369)
at
org.apache.spark.internal.Logging$class.logDebug(Logging.scala:58)
at
org.apache.spark.sql.kafka010.KafkaSourceProvider$.logDebug(KafkaSourceProvider.scala:369)
at
org.apache.spark.sql.kafka010.KafkaSourceProvider$ConfigUpdater.set(KafkaSourceProvider.scala:439)
at
org.apache.spark.sql.kafka010.KafkaSourceProvider$.kafkaParamsForDriver(KafkaSourceProvider.scala:394)
at
org.apache.spark.sql.kafka010.KafkaSourceProvider.createSource(KafkaSourceProvider.scala:90)
at
org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:277)
at
org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:80)
at
org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:77)
at
scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)
at
scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80)
at
org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:77)
at
org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:75)
at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256)
at
org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan$lzycompute(MicroBatchExecution.scala:75)
at
org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan(MicroBatchExecution.scala:61)
at org.apache.spark.sql.execution.streaming.StreamExecution.org
<https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.sql.execution.streaming.StreamExecution.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=lU8N2IsjW321Bu4T4BHLHvj1OSsKdPI_R_ahmOgq9rA&e=>
$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:265)
at
org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189)
(b)* the build script update:*
[pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$ diff
build.sbt spk211-build.sbt.original
10,11c10,11
< libraryDependencies += "org.apache.spark" % "spark-sql_2.11" %* "2.3.1"*
< libraryDependencies += "org.apache.spark" % "spark-core_2.11" %* "2.3.1"*
---
> libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.1"
> libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.1"
[pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$
Re: spark 2.3.1 with kafka spark-streaming-kafka-0-10 (java.lang.AbstractMethodError)
Posted by Peter Liu <pe...@gmail.com>.
Thank you, Ryan!
I'll keep you guys updated.
Cheers,
Peter
On Tue, Jul 3, 2018 at 4:22 PM, Shixiong(Ryan) Zhu <sh...@databricks.com>
wrote:
> -dev +user
>
> Hey Peter,
>
> You should upgrade "org.apache.spark_spark-sql-kafka-0-10_2.11-2.2.0.jar"
> to 2.3.1. The doc link you posted is for DStream. You should take a look at
> https://spark.apache.org/docs/2.3.1/structured-
> streaming-kafka-integration.html. You don't need
> "spark-streaming-kafka-0-10" at all if you are using Structured Streaming.
> This jar is for DStream.
>
>
>
> Best Regards,
> Ryan
>
> On Tue, Jul 3, 2018 at 8:55 AM, Peter Liu <pe...@gmail.com> wrote:
>
>> Hi Sean,
>>
>> thanks for your reply (sorry for the late response - it was a long
>> weekend here).
>>
>> do you happen to have any suggestion for any specific area or jars I
>> should take a close look? (i have the spark environment page attached
>> below).
>>
>> This is very much appreciated!
>>
>> Peter
>>
>> Environment
>> Runtime Information
>> NameValue
>> Java Home /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre
>> Java Version 1.8.0-internal (Oracle Corporation)
>> Scala Version version 2.11.8 Spark Properties
>> NameValue
>> spark.serializer org.apache.spark.serializer.KryoSerializer
>> spark.executor.extraJavaOptions -XX:ParallelGCThreads=4
>> spark.driver.host isvhdp11lab.spark.ibm.com
>> spark.history.fs.logDirectory hdfs://isvhdp7lab/sparkLogs
>> spark.eventLog.enabled true
>> spark.executor.heartbeatInterval 60s
>> spark.ui.port 0
>> spark.driver.port 39641
>> spark.yarn.app.id application_1530116533665_0008
>> spark.app.name Consumer Cfg: Stream_Dur300000_RecPerSec6000
>> 0_camp100_recGenNumP10
>> spark.network.timeout 1200
>> spark.scheduler.mode FIFO
>> spark.driver.memory 4g
>> spark.executor.instances 60
>> spark.yarn.secondary.jars org.apache.kafka_kafka-clients
>> -0.10.0.1.jar,org.apache.spark_spark-sql-kafka-0-10_2.
>> 11-2.2.0.jar,org.spark-project.spark_unused-1.0.0.jar,net.
>> jpountz.lz4_lz4-1.3.0.jar,org.xerial.snappy_snappy-java-1.1.
>> 2.6.jar,org.slf4j_slf4j-api-1.7.16.jar
>> spark.executor.id driver
>> spark.streaming.stopGracefullyOnShutdown true
>> spark.yarn.app.container.log.dir *********(redacted)
>> spark.submit.deployMode cluster
>> spark.master yarn
>> spark.ui.filters org.apache.hadoop.yarn.server.
>> webproxy.amfilter.AmIpFilter
>> spark.executor.memory 3g
>> spark.eventLog.dir hdfs://isvhdp7lab/sparkLogs
>> spark.executor.cores 1
>> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFi
>> lter.param.PROXY_HOSTS isvhdp7.aus.stglabs.ibm.com
>> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFi
>> lter.param.PROXY_URI_BASES http://isvhdp7.aus.stglabs.ibm
>> .com:8088/proxy/application_1530116533665_0008
>> spark.yarn.dist.jars file:///home/pgl/SparkStreamin
>> gBenchmark-RemoteConsumer/org.apache.kafka_kafka-clients-0.
>> 10.0.1.jar,file:///home/pgl/.ivy2/jars/org.apache.spark_
>> spark-sql-kafka-0-10_2.11-2.2.0.jar,file:///home/pgl/.ivy2/
>> jars/org.apache.kafka_kafka-clients-0.10.0.1.jar,file:///hom
>> e/pgl/.ivy2/jars/org.spark-project.spark_unused-1.0.0.jar,
>> file:///home/pgl/.ivy2/jars/net.jpountz.lz4_lz4-1.3.0.jar,
>> file:///home/pgl/.ivy2/jars/org.xerial.snappy_snappy-java-
>> 1.1.2.6.jar,file:///home/pgl/.ivy2/jars/org.slf4j_slf4j-api-1.7.16.jar
>> spark.app.id application_1530116533665_0008
>> spark.sql.shuffle.partitions 40 System Properties
>> NameValue
>> java.io.tmpdir /spark2/pgl/yarn/usercache/pgl
>> /appcache/application_1530116533665_0008/container_153011653
>> 3665_0008_02_000001/tmp
>> line.separator
>> path.separator :
>> sun.management.compiler HotSpot 64-Bit Tiered Compilers
>> sun.cpu.endian little
>> java.specification.version 1.8
>> java.vm.specification.name Java Virtual Machine Specification
>> java.vendor Oracle Corporation
>> java.vm.specification.version 1.8
>> user.home /home/pgl
>> file.encoding.pkg sun.io
>> sun.nio.ch.bugLevel
>> sun.arch.data.model 64
>> sun.boot.library.path /home/pgl/root-dir/jdk8_trlopt
>> _55cf01d/jre/lib/ppc64le
>> user.dir /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001
>> java.library.path /home/maha/hadoop-2.9.0/lib/na
>> tive/:/home/maha/hadoop-2.9.0/lib/native/:/home/pgl/hadoop-
>> 2.9.0/lib/native::/usr/java/packages/lib/ppc64le:/usr/
>> lib64:/lib64:/lib:/usr/lib
>> sun.cpu.isalist
>> os.arch
>> ppc64le
>> java.vm.version 25.71-b00
>> java.endorsed.dirs /home/pgl/root-dir/jdk8_trlopt
>> _55cf01d/jre/lib/endorsed
>> java.runtime.version 1.8.0-internal-root_2018_04_19_13_32-b00
>> java.vm.info mixed mode
>> java.ext.dirs /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/ext:/usr/
>> java/packages/lib/ext
>> java.runtime.name OpenJDK Runtime Environment
>> file.separator /
>> java.class.version 52.0
>> java.specification.name Java Platform API Specification
>> sun.boot.class.path /home/pgl/root-dir/jdk8_trlopt
>> _55cf01d/jre/lib/resources.jar:/home/pgl/root-dir/jdk8_
>> trlopt_55cf01d/jre/lib/rt.jar:/home/pgl/root-dir/jdk8_
>> trlopt_55cf01d/jre/lib/sunrsasign.jar:/home/pgl/root-dir/
>> jdk8_trlopt_55cf01d/jre/lib/jsse.jar:/home/pgl/root-dir/
>> jdk8_trlopt_55cf01d/jre/lib/jce.jar:/home/pgl/root-dir/
>> jdk8_trlopt_55cf01d/jre/lib/charsets.jar:/home/pgl/root-
>> dir/jdk8_trlopt_55cf01d/jre/lib/jfr.jar:/home/pgl/root-
>> dir/jdk8_trlopt_55cf01d/jre/classes
>> file.encoding UTF-8
>> user.timezone America/Chicago
>> java.specification.vendor Oracle Corporation
>> sun.java.launcher SUN_STANDARD
>> java.vm.compressedOopsMode Zero based
>> os.version 4.14.0-81.el7a.ppc64le
>> sun.os.patch.level unknown
>> java.vm.specification.vendor Oracle Corporation
>> user.country US
>> sun.jnu.encoding UTF-8
>> user.language en
>> java.vendor.url http://java.oracle.com/
>> java.awt.printerjob sun.print.PSPrinterJob
>> java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
>> awt.toolkit sun.awt.X11.XToolkit
>> os.name Linux
>> java.vm.vendor Oracle Corporation
>> java.vendor.url.bug http://bugreport.sun.com/bugreport/
>> user.name pgl
>> java.vm.name OpenJDK 64-Bit Server VM
>> sun.java.command org.apache.spark.deploy.yarn.ApplicationMaster --class
>> SparkStreamingTest --jar file:/home/pgl/SparkStreamingB
>> enchmark-RemoteConsumer/target/scala-2.11/sparkstreamingbenchmark-remoteconsumer_2.11-0.1-SNAPSHOT.jar
>> --arg 300000 --arg 60000 --arg 20 --arg 10 --arg 100 --arg 80 --arg 80
>> --arg 80 --arg Stream_Dur300000_RecPerSec60000_camp100_recGenNumP10
>> --properties-file /spark2/pgl/yarn/usercache/pgl
>> /appcache/application_1530116533665_0008/container_153011653
>> 3665_0008_02_000001/__spark_conf__/__spark_conf__.properties
>> java.home /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre
>> java.version 1.8.0-internal
>> sun.io.unicode.encoding UnicodeLittle Classpath Entries
>> ResourceSource
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/stax-api-1.0-2.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-beanutils-core-1.8.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guava-11.0.2.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/datanucleus-api-jdo-3.2.6.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/guava-11.0.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/java-util-1.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-hive_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/javax.inject-1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-recipes-2.7.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/libthrift-0.9.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/okio-1.13.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-cli-1.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-shuffle-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guice-3.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/leveldbjni-all-1.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-network-common_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> nodemanager-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/okhttp-3.8.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/junit-4.11.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/json4s-ast_2.11-3.2.11.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-configuration-1.6.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/avro-1.7.7.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/generex-1.0.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/arpack_combined_all-0.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-common-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-jaxrs-1.9.13.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-cli-1.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/kubernetes-client-3.0.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/eigenbase-properties-1.1.5.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/pyrolite-4.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-client-2.9.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang-2.6.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-jobclient-2.9.0-tests.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-hs-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-sslengine-6.1.26.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-kubernetes_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/log4j-1.2.17.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spire_2.11-0.13.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-encoding-1.8.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-common-2.22.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-6.1.26.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-test-2.7.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-core-
>> asl-1.9.13.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-net-2.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/avro-1.7.7.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/logging-interceptor-3.8.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-compress-1.4.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-auth-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-net-3.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/metrics-jvm-3.1.5.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-registry-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/machinist_2.11-0.6.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/xz-1.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/httpclient-4.5.2.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jetty-util-6.1.26.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/stax-api-1.0.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jsr305-1.3.9.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/netty-all-4.1.17.Final.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/arrow-memory-0.8.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/antlr-runtime-3.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javax.inject-1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/orc-core-1.4.4-nohive.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/json-io-2.5.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/metrics-core-3.1.5.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-graphx_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-beanu
>> tils-1.7.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/chill_2.11-0.8.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-launcher_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/paranamer-2.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-compress-1.4.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-client-2.7.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/httpclient-4.5.2.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-hadoop-bundle-1.6.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/avro-1.7.7.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> sharedcachemanager-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/zjsonpatch-0.3.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-framework-2.7.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-guava-2.22.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/gson-2.2.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-catalyst_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/geronimo-jcache
>> _1.0_spec-1.0-alpha-1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/snappy-0.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/log4j-1.2.17.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/compress-lzf-1.0.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hive-beeline-1.2.1.spark2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-digester-1.8.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/metrics-core-3.0.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/super-csv-2.2.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-dataformat-yaml-2.6.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-core-2.6.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/JavaEWAH-0.3.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javax.ws.rs-api-2.0.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-mapreduce-client-core-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/apache-log4j-extras-1.2.17.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/asm-3.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/ehcache-3.3.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-json-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/asm-3.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsp-api-2.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javolution-5.5.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-databind-2.6.7.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/xmlenc-0.52.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-mesos_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-mapper-
>> asl-1.9.13.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/janino-3.0.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/htrace-core-3.1.0-incubating.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-common-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/breeze_2.11-0.13.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-lang-2.6.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-format-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/snappy-java-1.0.5.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/mesos-1.4.0-shaded-protobuf.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/woodstox-core-5.0.3.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-nfs-2.9.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/servlet-api-2.5.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-dbcp-1.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-core-asl-1.9.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-logging-1.1.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/asm-3.2.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jcl-over-slf4j-1.7.16.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math-2.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-i18n-2.0.0-M15.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/javax.inject-1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jets3t-0.9.4.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-app-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/libfb303-0.9.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/core-1.1.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-mllib_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-mapreduce-client-shuffle-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jpam-1.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-util-6.1.26.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-client-2.7.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-core_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/api-asn1-api-1.0.0-M20.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/apacheds-i18n
>> -2.0.0-M15.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jtransforms-2.4.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-network-shuffle_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-math3-3.4.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/snappy-java-1.0.5.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-column-1.8.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/mockito-all-1.8.5.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jaxb-api-2.2.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/breeze-macros_2.11-0.13.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-kvstore_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-mapreduce-client-jobclient-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/base64-2.3.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/scala-xml_2.11-1.0.5.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/oro-2.0.8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-beanutils-1.7.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jetty-6.1.26.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/aopalliance-repackaged-2.4.0-b34.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-logging-1.1.3.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-hdfs-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jets3t-0.9.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/guava-11.0.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/apacheds-kerb
>> eros-codec-2.0.0-M15.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-mapreduce-client-common-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/metrics-graphite-3.1.5.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/scala-compiler-2.11.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-server-2.22.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/aopalliance-1.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/joda-time-2.9.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> resourcemanager-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/netty-3.9.9.Final.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/activation-1.1.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/stax2-api-3.1.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/gson-2.2.4.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsch-0.1.54.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-module-jaxb-annotations-2.6.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/scalap-2.11.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/py4j-0.10.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/xmlenc-0.52.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> tests-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/curator-client-2.7.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javassist-3.18.1-GA.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-core-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-beanu
>> tils-core-1.8.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/kryo-shaded-3.0.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/httpcore-4.4.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/calcite-core-1.2.0-incubating.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-compiler-3.0.8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-mapper-
>> asl-1.9.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-yarn-server-common-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/zookeeper-3.4.6.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-confi
>> guration-1.6.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/stringtemplate-3.2.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/curator-framework-2.7.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/antlr-2.7.7.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsr305-3.0.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> timeline-pluginstorage-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-math3-3.1.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-jackson-1.8.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-mllib-local_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/scala-parser-combinators_2.11-1.0.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javax.inject-2.4.0-b34.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-client-2.22.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsch-0.1.54.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/httpclient-4.5.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-httpclient-3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jsr305-3.0.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/lz4-java-1.4.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jta-1.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/protobuf-java-2.5.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/snakeyaml-1.15.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hadoop-auth-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/datanucleus-core-3.2.10.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/avro-mapred-1.7.7-hadoop2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/aopalliance-1.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javax.servlet-api-3.1.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/etc/hadoop System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-media-jaxb-2.22.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-unsafe_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-xc-1.9.13.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/antlr4-runtime-4.7.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-native-
>> client-2.9.0-tests.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-yarn-server-web-proxy-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-sketch_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/api-asn1-api-1.0.0-M20.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/objenesis-2.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/guava-14.0.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/apacheds-kerberos-codec-2.0.0-M15.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-co
>> mpress-1.4.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/slf4j-api-1.7.16.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/avro-1.7.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/ivy-2.4.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-yarn-client-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> conf__/__hadoop_conf__ System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/slf4j-api-1.7.25.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> web-proxy-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-pool-1.5.4.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-frame
>> work-2.7.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/xz-1.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hk2-api-2.4.0-b34.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jettison-1.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/calcite-linq4j-1.2.0-incubating.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hppc-0.7.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hadoop-annota
>> tions-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-collect
>> ions-3.2.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-module-scala_2.11-2.6.7.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/guice-3.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/kubernetes-model-2.0.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hive-jdbc-1.2.1.spark2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-io-2.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-repl_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-hadoop-1.8.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/osgi-resource-locator-1.0.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-annotations-2.6.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/opencsv-2.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/log4j-1.2.17.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/apacheds-i18n-2.0.0-M15.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-colle
>> ctions-3.2.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-common-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-3.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/activation-1.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-io-2.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-codec-1.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/aircompressor-0.8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-annotat
>> ions-2.7.8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/nimbus-jose-jwt-3.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> router-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/zstd-jni-1.3.2-2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-yarn_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-kerber
>> os-codec-2.0.0-M15.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> applicationhistoryservice-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/json4s-jackson_2.11-3.2.11.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/validation-api-1.1.0.Final.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/parquet-common-1.8.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-lang3-3.5.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/okio-1.4.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/metrics-json-3.1.5.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jline-2.12.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/paranamer-2.3.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-codec-1.10.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/fst-2.50.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-configu
>> ration-1.6.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/json-smart-1.1.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-client-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-core-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/xmlenc-0.52.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/derby-10.12.1.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-hs-plugins-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_conf__ System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/okhttp-2.4.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0-tests.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jcip-annotations-1.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/minlog-1.3.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-io-2.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/flatbuffers-1.2.0-3f79e055.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/stax2-api-3.1.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/json4s-core_2.11-3.2.11.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hive-cli-1.2.1.spark2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/httpcore-4.4.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/netty-3.6.2.Final.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hk2-utils-2.4.0-b34.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-yarn-api-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jettison-1.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/curator-recipes-2.7.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-container-servlet-core-2.22.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/api-util-1.0.0-M20.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hamcrest-core-1.3.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math3-3.1.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hk2-locator-2.4.0-b34.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-crypto-1.0.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-applica
>> tions-distributedshell-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/ST4-4.0.4.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-core-1.9.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/httpcore-4.4.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/guice-servlet-3.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-lang-2.6.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0-tests.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-sql_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/json-smart-1.1.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/snappy-java-1.0.5.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-native-
>> client-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/xercesImpl-2.9.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/scala-library-2.11.8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-6.1.26.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
>> common-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-digester-1.8.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-beanutils-1.7.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/htrace-core4-
>> 4.1.0-incubating.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsp-api-2.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-yarn-common-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/api-util-1.0.0-M20.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/automaton-1.11-8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-net-3.1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/htrace-core4-4.
>> 1.0-incubating.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/woodstox-core-5.0.3.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-api-2.9.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jets3t-0.9.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/univocity-parsers-2.5.9.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-codec-1.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/stream-2.7.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spire-macros_2.11-0.13.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/datanucleus-rdbms-3.2.9.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-lang3-3.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/hadoop-ann
>> otations-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-co
>> re-asl-1.9.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-streaming_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang3-3.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-ma
>> pper-asl-1.9.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-server-1.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/zookeeper-3.4.6.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/snappy-java-1.1.2.6.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001 System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-io-2.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-tags_2.11-2.3.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jodd-core-3.5.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/RoaringBitmap-0.5.11.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jersey-container-servlet-2.22.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-client-jobclient-2.9.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapredu
>> ce-examples-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/slf4j-log4j12-1.7.16.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/java-xmlbuilder-0.4.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/activation-1.1.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/arrow-vector-0.8.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/paranamer-2.3.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-annotations-2.7.3.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/asm-3.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/arrow-format-0.8.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/servlet-api-2.5.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/junit-4.11.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/shapeless_2.11-2.3.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-client-2.9.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-module-paranamer-2.7.9.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jsp-api-2.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-beanuti
>> ls-core-1.8.0.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/macro-compat_2.11-1.1.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-client-1.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/spark-hive-thriftserver_2.11-2.3.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-applica
>> tions-unmanaged-am-launcher-2.9.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/stax-api-1.0-2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-cli-1.2.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/xz-1.0.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-mappe
>> r-asl-1.9.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-json-1.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/bonecp-0.8.0.RELEASE.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-client-
>> 2.9.0-tests.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/bcprov-jdk15on-1.58.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-server-1.9.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/chill-java-0.8.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-collections-3.2.2.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jackson-mapper-asl-1.9.13.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/javax.annotation-api-1.2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/gson-2.2.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hadoop-mapreduce-client-app-2.7.3.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jdo-api-3.0.1.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/avro-ipc-1.7.7.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/xz-1.0.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/jul-to-slf4j-1.7.16.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsr305-3.0.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/calcite-avatica-1.2.0-incubating.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/htrace-core4-4.
>> 1.0-incubating.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/java-xmlbuilder-1.1.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hive-metastore-1.2.1.spark2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar System
>> Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/scala-reflect-2.11.8.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/orc-mapreduce-1.4.4-nohive.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/hive-exec-1.2.1.spark2.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar System
>> Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/xbean-asm5-shaded-4.4.jar System Classpath
>> /spark2/pgl/yarn/usercache/pgl/appcache/application_15301165
>> 33665_0008/container_1530116533665_0008_02_000001/__spark_
>> libs__/commons-digester-1.8.jar System Classpath
>> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jcip-annotations-1.0.jar
>>
>>
>> On Sun, Jul 1, 2018 at 10:09 AM, Sean Owen <sr...@gmail.com> wrote:
>>
>>> Somewhere, you have mismatched versions of Spark on your classpath.
>>>
>>>
>>> On Sun, Jul 1, 2018, 9:01 AM Peter Liu <pe...@gmail.com> wrote:
>>>
>>>> Hello there,
>>>>
>>>> I didn't get any response/help from the user list for the following
>>>> question and thought people on the dev list might be able to help?:
>>>>
>>>> I upgraded to spark 2.3.1 from spark 2.2.1, ran my streaming workload
>>>> and got the error (java.lang.AbstractMethodError) never seen before;
>>>> see the error exception stack attached in (a) bellow.
>>>>
>>>> anyone knows if spark 2.3.1 works well with kafka
>>>> spark-streaming-kafka-0-10?
>>>>
>>>> this link of spark kafka integration page doesn't say anything about
>>>> any limitation:
>>>> https://spark.apache.org/docs/2.3.1/streaming-kafka-integration.html
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_2.3.1_streaming-2Dkafka-2Dintegration.html&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=LORvnW8MyMYoQuiRJpv4qo1519IpQ57V-MxUgro0vP0&e=>
>>>>
>>>> but this discussion seems to say there is indeed an issue when
>>>> upgrading to spark 2.3.1:
>>>> https://stackoverflow.com/questions/49180931/abstractmethode
>>>> rror-creating-kafka-stream
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_49180931_abstractmethoderror-2Dcreating-2Dkafka-2Dstream&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=xIl7y-rgXq2AUuNg8cKeBIRApmRMLnrJbPsf4kc5Zzg&e=>
>>>>
>>>> i also rebuilt the workload with some spark 2.3.1 jars (see (b) below).
>>>> it doesn't seem to help.
>>>>
>>>> Would be great if anyone could kindly share any insights here.
>>>>
>>>> Thanks!
>>>>
>>>> Peter
>>>>
>>>> (a) the exception
>>>> Exception in thread "stream execution thread for [id =
>>>> 5adae836-268a-4ebf-adc4-e3cc9fbe5acf, runId =
>>>> 70e78d5c-665e-4c6f-a0cc-41a56e488e30]" java.lang.AbstractMethodError
>>>> at org.apache.spark.internal.Logging$class.initializeLogIfNeces
>>>> sary(Logging.scala:99)
>>>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$.initializ
>>>> eLogIfNecessary(KafkaSourceProvider.scala:369)
>>>> at org.apache.spark.internal.Logging$class.log(Logging.scala:46
>>>> )
>>>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$.log(Kafka
>>>> SourceProvider.scala:369)
>>>> at org.apache.spark.internal.Logging$class.logDebug(Logging.sca
>>>> la:58)
>>>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$.logDebug(
>>>> KafkaSourceProvider.scala:369)
>>>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$ConfigUpda
>>>> ter.set(KafkaSourceProvider.scala:439)
>>>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$.kafkaPara
>>>> msForDriver(KafkaSourceProvider.scala:394)
>>>> at org.apache.spark.sql.kafka010.KafkaSourceProvider.createSour
>>>> ce(KafkaSourceProvider.scala:90)
>>>> at org.apache.spark.sql.execution.datasources.DataSource.create
>>>> Source(DataSource.scala:277)
>>>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>>> $$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:80)
>>>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>>> $$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:77)
>>>> at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLi
>>>> ke.scala:194)
>>>> at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.sca
>>>> la:80)
>>>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>>> $$anonfun$1.applyOrElse(MicroBatchExecution.scala:77)
>>>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>>> $$anonfun$1.applyOrElse(MicroBatchExecution.scala:75)
>>>> at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.appl
>>>> y(TreeNode.scala:267)
>>>> at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.appl
>>>> y(TreeNode.scala:267)
>>>> at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigi
>>>> n(TreeNode.scala:70)
>>>> at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(T
>>>> reeNode.scala:266)
>>>> at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeN
>>>> ode.scala:256)
>>>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>>> .logicalPlan$lzycompute(MicroBatchExecution.scala:75)
>>>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>>> .logicalPlan(MicroBatchExecution.scala:61)
>>>> at org.apache.spark.sql.execution.streaming.StreamExecution.org
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.sql.execution.streaming.StreamExecution.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=lU8N2IsjW321Bu4T4BHLHvj1OSsKdPI_R_ahmOgq9rA&e=>
>>>> $apache$spark$sql$execution$streaming$StreamExecution$$runSt
>>>> ream(StreamExecution.scala:265)
>>>> at org.apache.spark.sql.execution.streaming.StreamExecution$$an
>>>> on$1.run(StreamExecution.scala:189)
>>>>
>>>> (b)* the build script update:*
>>>>
>>>> [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$ diff
>>>> build.sbt spk211-build.sbt.original
>>>> 10,11c10,11
>>>> < libraryDependencies += "org.apache.spark" % "spark-sql_2.11" %*
>>>> "2.3.1"*
>>>> < libraryDependencies += "org.apache.spark" % "spark-core_2.11" %*
>>>> "2.3.1"*
>>>> ---
>>>> > libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.1"
>>>> > libraryDependencies += "org.apache.spark" % "spark-core_2.11" %
>>>> "2.2.1"
>>>> [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$
>>>>
>>>>
>>>>
>>>>
>>
>
Re: spark 2.3.1 with kafka spark-streaming-kafka-0-10 (java.lang.AbstractMethodError)
Posted by "Shixiong(Ryan) Zhu" <sh...@databricks.com>.
-dev +user
Hey Peter,
You should upgrade "org.apache.spark_spark-sql-kafka-0-10_2.11-2.2.0.jar"
to 2.3.1. The doc link you posted is for DStream. You should take a look at
https://spark.apache.org/docs/2.3.1/structured-streaming-kafka-integration.html.
You don't need "spark-streaming-kafka-0-10" at all if you are using
Structured Streaming. This jar is for DStream.
Best Regards,
Ryan
On Tue, Jul 3, 2018 at 8:55 AM, Peter Liu <pe...@gmail.com> wrote:
> Hi Sean,
>
> thanks for your reply (sorry for the late response - it was a long weekend
> here).
>
> do you happen to have any suggestion for any specific area or jars I
> should take a close look? (i have the spark environment page attached
> below).
>
> This is very much appreciated!
>
> Peter
>
> Environment
> Runtime Information
> NameValue
> Java Home /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre
> Java Version 1.8.0-internal (Oracle Corporation)
> Scala Version version 2.11.8 Spark Properties
> NameValue
> spark.serializer org.apache.spark.serializer.KryoSerializer
> spark.executor.extraJavaOptions -XX:ParallelGCThreads=4
> spark.driver.host isvhdp11lab.spark.ibm.com
> spark.history.fs.logDirectory hdfs://isvhdp7lab/sparkLogs
> spark.eventLog.enabled true
> spark.executor.heartbeatInterval 60s
> spark.ui.port 0
> spark.driver.port 39641
> spark.yarn.app.id application_1530116533665_0008
> spark.app.name Consumer Cfg: Stream_Dur300000_RecPerSec60000_camp100_
> recGenNumP10
> spark.network.timeout 1200
> spark.scheduler.mode FIFO
> spark.driver.memory 4g
> spark.executor.instances 60
> spark.yarn.secondary.jars org.apache.kafka_kafka-clients-0.10.0.1.jar,org.
> apache.spark_spark-sql-kafka-0-10_2.11-2.2.0.jar,org.spark-
> project.spark_unused-1.0.0.jar,net.jpountz.lz4_lz4-1.3.0.
> jar,org.xerial.snappy_snappy-java-1.1.2.6.jar,org.slf4j_
> slf4j-api-1.7.16.jar
> spark.executor.id driver
> spark.streaming.stopGracefullyOnShutdown true
> spark.yarn.app.container.log.dir *********(redacted)
> spark.submit.deployMode cluster
> spark.master yarn
> spark.ui.filters org.apache.hadoop.yarn.server.
> webproxy.amfilter.AmIpFilter
> spark.executor.memory 3g
> spark.eventLog.dir hdfs://isvhdp7lab/sparkLogs
> spark.executor.cores 1
> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.
> AmIpFilter.param.PROXY_HOSTS isvhdp7.aus.stglabs.ibm.com
> spark.org.apache.hadoop.yarn.server.webproxy.amfilter.
> AmIpFilter.param.PROXY_URI_BASES http://isvhdp7.aus.stglabs.
> ibm.com:8088/proxy/application_1530116533665_0008
> spark.yarn.dist.jars file:///home/pgl/SparkStreamingBenchmark-
> RemoteConsumer/org.apache.kafka_kafka-clients-0.10.0.1.
> jar,file:///home/pgl/.ivy2/jars/org.apache.spark_spark-
> sql-kafka-0-10_2.11-2.2.0.jar,file:///home/pgl/.ivy2/jars/
> org.apache.kafka_kafka-clients-0.10.0.1.jar,file:///
> home/pgl/.ivy2/jars/org.spark-project.spark_unused-1.0.0.
> jar,file:///home/pgl/.ivy2/jars/net.jpountz.lz4_lz4-1.3.
> 0.jar,file:///home/pgl/.ivy2/jars/org.xerial.snappy_snappy-
> java-1.1.2.6.jar,file:///home/pgl/.ivy2/jars/org.slf4j_
> slf4j-api-1.7.16.jar
> spark.app.id application_1530116533665_0008
> spark.sql.shuffle.partitions 40 System Properties
> NameValue
> java.io.tmpdir /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/tmp
> line.separator
> path.separator :
> sun.management.compiler HotSpot 64-Bit Tiered Compilers
> sun.cpu.endian little
> java.specification.version 1.8
> java.vm.specification.name Java Virtual Machine Specification
> java.vendor Oracle Corporation
> java.vm.specification.version 1.8
> user.home /home/pgl
> file.encoding.pkg sun.io
> sun.nio.ch.bugLevel
> sun.arch.data.model 64
> sun.boot.library.path /home/pgl/root-dir/jdk8_
> trlopt_55cf01d/jre/lib/ppc64le
> user.dir /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001
> java.library.path /home/maha/hadoop-2.9.0/lib/
> native/:/home/maha/hadoop-2.9.0/lib/native/:/home/pgl/
> hadoop-2.9.0/lib/native::/usr/java/packages/lib/ppc64le:/
> usr/lib64:/lib64:/lib:/usr/lib
> sun.cpu.isalist
> os.arch
> ppc64le
> java.vm.version 25.71-b00
> java.endorsed.dirs /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/endorsed
> java.runtime.version 1.8.0-internal-root_2018_04_19_13_32-b00
> java.vm.info mixed mode
> java.ext.dirs /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/ext:/
> usr/java/packages/lib/ext
> java.runtime.name OpenJDK Runtime Environment
> file.separator /
> java.class.version 52.0
> java.specification.name Java Platform API Specification
> sun.boot.class.path /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/
> resources.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/
> lib/rt.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/
> sunrsasign.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/
> lib/jsse.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/
> lib/jce.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/
> lib/charsets.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/
> jre/lib/jfr.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/classes
> file.encoding UTF-8
> user.timezone America/Chicago
> java.specification.vendor Oracle Corporation
> sun.java.launcher SUN_STANDARD
> java.vm.compressedOopsMode Zero based
> os.version 4.14.0-81.el7a.ppc64le
> sun.os.patch.level unknown
> java.vm.specification.vendor Oracle Corporation
> user.country US
> sun.jnu.encoding UTF-8
> user.language en
> java.vendor.url http://java.oracle.com/
> java.awt.printerjob sun.print.PSPrinterJob
> java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
> awt.toolkit sun.awt.X11.XToolkit
> os.name Linux
> java.vm.vendor Oracle Corporation
> java.vendor.url.bug http://bugreport.sun.com/bugreport/
> user.name pgl
> java.vm.name OpenJDK 64-Bit Server VM
> sun.java.command org.apache.spark.deploy.yarn.ApplicationMaster --class
> SparkStreamingTest --jar file:/home/pgl/SparkStreamingBenchmark-
> RemoteConsumer/target/scala-2.11/sparkstreamingbenchmark-
> remoteconsumer_2.11-0.1-SNAPSHOT.jar --arg 300000 --arg 60000 --arg 20
> --arg 10 --arg 100 --arg 80 --arg 80 --arg 80 --arg Stream_Dur300000_
> RecPerSec60000_camp100_recGenNumP10 --properties-file
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_conf__/__spark_conf__.properties
> java.home /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre
> java.version 1.8.0-internal
> sun.io.unicode.encoding UnicodeLittle Classpath Entries
> ResourceSource
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/stax-api-1.0-2.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-beanutils-core-1.8.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guava-11.0.2.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/datanucleus-api-jdo-3.2.6.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/guava-11.0.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/java-util-1.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-hive_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/javax.inject-1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-recipes-2.7.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/libthrift-0.9.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/okio-1.13.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-cli-1.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-shuffle-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guice-3.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/leveldbjni-all-1.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-network-common_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-nodemanager-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/okhttp-3.8.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/junit-4.11.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/json4s-ast_2.11-3.2.11.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-configuration-1.6.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/avro-1.7.7.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/generex-1.0.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/arpack_combined_all-0.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-common-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-jaxrs-1.9.13.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-cli-1.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/kubernetes-client-3.0.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/eigenbase-properties-1.1.5.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/pyrolite-4.13.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-client-2.9.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang-2.6.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-jobclient-2.9.0-tests.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-hs-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-sslengine-6.1.26.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-kubernetes_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/log4j-1.2.17.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spire_2.11-0.13.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-encoding-1.8.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-common-2.22.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-6.1.26.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-test-2.7.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-net-2.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/avro-1.7.7.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/logging-interceptor-3.8.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-compress-1.4.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-auth-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-net-3.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/metrics-jvm-3.1.5.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-registry-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/machinist_2.11-0.6.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/xz-1.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/httpclient-4.5.2.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jetty-util-6.1.26.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/stax-api-1.0.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jsr305-1.3.9.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/netty-all-4.1.17.Final.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/arrow-memory-0.8.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/antlr-runtime-3.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javax.inject-1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/orc-core-1.4.4-nohive.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/json-io-2.5.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/metrics-core-3.1.5.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-graphx_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/chill_2.11-0.8.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-launcher_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/paranamer-2.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-compress-1.4.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-client-2.7.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/httpclient-4.5.2.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-hadoop-bundle-1.6.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/avro-1.7.7.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-sharedcachemanager-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/zjsonpatch-0.3.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-framework-2.7.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-guava-2.22.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/gson-2.2.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-catalyst_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/geronimo-
> jcache_1.0_spec-1.0-alpha-1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/snappy-0.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/log4j-1.2.17.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/compress-lzf-1.0.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hive-beeline-1.2.1.spark2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-digester-1.8.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/metrics-core-3.0.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/super-csv-2.2.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-dataformat-yaml-2.6.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-core-2.6.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/JavaEWAH-0.3.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javax.ws.rs-api-2.0.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-mapreduce-client-core-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/apache-log4j-extras-1.2.17.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/asm-3.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/ehcache-3.3.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-json-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/asm-3.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsp-api-2.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javolution-5.5.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-databind-2.6.7.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/xmlenc-0.52.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-mesos_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/janino-3.0.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/htrace-core-3.1.0-incubating.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-common-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/breeze_2.11-0.13.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-lang-2.6.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-format-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/snappy-java-1.0.5.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/mesos-1.4.0-shaded-protobuf.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/woodstox-core-5.0.3.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-nfs-2.9.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/servlet-api-2.5.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-dbcp-1.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-core-asl-1.9.13.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-logging-1.1.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/asm-3.2.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jcl-over-slf4j-1.7.16.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math-2.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-i18n-2.0.0-M15.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/javax.inject-1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jets3t-0.9.4.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-app-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/libfb303-0.9.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/core-1.1.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-mllib_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-mapreduce-client-shuffle-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jpam-1.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-util-6.1.26.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-client-2.7.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-core_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/api-asn1-api-1.0.0-M20.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jtransforms-2.4.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-network-shuffle_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-math3-3.4.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/snappy-java-1.0.5.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-column-1.8.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/mockito-all-1.8.5.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jaxb-api-2.2.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/breeze-macros_2.11-0.13.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-kvstore_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-mapreduce-client-jobclient-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/base64-2.3.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/scala-xml_2.11-1.0.5.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/oro-2.0.8.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-beanutils-1.7.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jetty-6.1.26.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/aopalliance-repackaged-2.4.0-b34.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-logging-1.1.3.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-hdfs-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jets3t-0.9.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/guava-11.0.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/apacheds-
> kerberos-codec-2.0.0-M15.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-mapreduce-client-common-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/metrics-graphite-3.1.5.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/scala-compiler-2.11.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-server-2.22.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/aopalliance-1.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/joda-time-2.9.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-resourcemanager-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/netty-3.9.9.Final.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/activation-1.1.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/stax2-api-3.1.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/gson-2.2.4.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsch-0.1.54.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-module-jaxb-annotations-2.6.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/scalap-2.11.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/py4j-0.10.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/xmlenc-0.52.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-tests-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/curator-client-2.7.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javassist-3.18.1-GA.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-core-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-
> beanutils-core-1.8.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/kryo-shaded-3.0.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/httpcore-4.4.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/calcite-core-1.2.0-incubating.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-compiler-3.0.8.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-yarn-server-common-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/zookeeper-3.4.6.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-
> configuration-1.6.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/stringtemplate-3.2.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/curator-framework-2.7.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/antlr-2.7.7.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsr305-3.0.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-timeline-pluginstorage-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-math3-3.1.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-jackson-1.8.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-mllib-local_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/scala-parser-combinators_2.11-1.0.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javax.inject-2.4.0-b34.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-client-2.22.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsch-0.1.54.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/httpclient-4.5.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-httpclient-3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jsr305-3.0.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/lz4-java-1.4.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jta-1.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/protobuf-java-2.5.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/snakeyaml-1.15.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hadoop-auth-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/datanucleus-core-3.2.10.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/avro-mapred-1.7.7-hadoop2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/aopalliance-1.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javax.servlet-api-3.1.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/etc/hadoop System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-media-jaxb-2.22.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-unsafe_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-xc-1.9.13.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/antlr4-runtime-4.7.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-
> native-client-2.9.0-tests.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-yarn-server-web-proxy-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-sketch_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/api-asn1-api-1.0.0-M20.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/objenesis-2.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/guava-14.0.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/apacheds-kerberos-codec-2.0.0-M15.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-
> compress-1.4.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/slf4j-api-1.7.16.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/avro-1.7.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/ivy-2.4.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-yarn-client-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_conf__/__hadoop_conf__ System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/slf4j-api-1.7.25.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-web-proxy-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-pool-1.5.4.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-framework-2.7.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/xz-1.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hk2-api-2.4.0-b34.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jettison-1.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/calcite-linq4j-1.2.0-incubating.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hppc-0.7.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hadoop-
> annotations-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-module-scala_2.11-2.6.7.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/guice-3.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/kubernetes-model-2.0.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hive-jdbc-1.2.1.spark2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-io-2.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-repl_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-hadoop-1.8.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/osgi-resource-locator-1.0.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-annotations-2.6.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/opencsv-2.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/log4j-1.2.17.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/apacheds-i18n-2.0.0-M15.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-
> collections-3.2.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-common-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-3.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/activation-1.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-io-2.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-codec-1.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/aircompressor-0.8.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/nimbus-jose-jwt-3.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-router-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/zstd-jni-1.3.2-2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-yarn_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-
> kerberos-codec-2.0.0-M15.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-
> applicationhistoryservice-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/json4s-jackson_2.11-3.2.11.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/validation-api-1.1.0.Final.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/parquet-common-1.8.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-lang3-3.5.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/okio-1.4.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/metrics-json-3.1.5.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jline-2.12.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/paranamer-2.3.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-codec-1.10.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/fst-2.50.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-configuration-1.6.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/json-smart-1.1.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-client-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-core-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/xmlenc-0.52.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/derby-10.12.1.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-hs-plugins-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/__spark_conf__ System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/okhttp-2.4.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0-tests.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jcip-annotations-1.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/minlog-1.3.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-io-2.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/flatbuffers-1.2.0-3f79e055.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/stax2-api-3.1.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/json4s-core_2.11-3.2.11.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hive-cli-1.2.1.spark2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/httpcore-4.4.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/netty-3.6.2.Final.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hk2-utils-2.4.0-b34.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-yarn-api-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jettison-1.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/curator-recipes-2.7.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-container-servlet-core-2.22.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/api-util-1.0.0-M20.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hamcrest-core-1.3.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math3-3.1.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hk2-locator-2.4.0-b34.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-crypto-1.0.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> applications-distributedshell-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/ST4-4.0.4.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-core-1.9.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/httpcore-4.4.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/guice-servlet-3.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-lang-2.6.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0-tests.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-sql_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/json-smart-1.1.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/snappy-java-1.0.5.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-
> native-client-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/xercesImpl-2.9.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/scala-library-2.11.8.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-6.1.26.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> server-common-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-digester-1.8.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-beanutils-1.7.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/htrace-
> core4-4.1.0-incubating.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsp-api-2.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-yarn-common-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/api-util-1.0.0-M20.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/automaton-1.11-8.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-net-3.1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/htrace-core4-
> 4.1.0-incubating.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/woodstox-core-5.0.3.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-api-2.9.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jets3t-0.9.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/univocity-parsers-2.5.9.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-codec-1.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/stream-2.7.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spire-macros_2.11-0.13.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/datanucleus-rdbms-3.2.9.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-lang3-3.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/hadoop-
> annotations-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-
> core-asl-1.9.13.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-streaming_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang3-3.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-
> mapper-asl-1.9.13.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-server-1.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/zookeeper-3.4.6.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/snappy-java-1.1.2.6.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001 System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-io-2.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-tags_2.11-2.3.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jodd-core-3.5.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/RoaringBitmap-0.5.11.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jersey-container-servlet-2.22.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-client-jobclient-2.9.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-
> mapreduce-examples-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/slf4j-log4j12-1.7.16.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/java-xmlbuilder-0.4.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/activation-1.1.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/arrow-vector-0.8.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/paranamer-2.3.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-annotations-2.7.3.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/asm-3.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/arrow-format-0.8.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/servlet-api-2.5.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/junit-4.11.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/shapeless_2.11-2.3.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-client-2.9.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-module-paranamer-2.7.9.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jsp-api-2.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-
> beanutils-core-1.8.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/macro-compat_2.11-1.1.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-client-1.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/spark-hive-thriftserver_2.11-2.3.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-
> applications-unmanaged-am-launcher-2.9.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/stax-api-1.0-2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-cli-1.2.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/xz-1.0.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-
> mapper-asl-1.9.13.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-json-1.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/bonecp-0.8.0.RELEASE.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-
> client-2.9.0-tests.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/bcprov-jdk15on-1.58.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-server-1.9.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/chill-java-0.8.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-collections-3.2.2.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jackson-mapper-asl-1.9.13.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/javax.annotation-api-1.2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/gson-2.2.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hadoop-mapreduce-client-app-2.7.3.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jdo-api-3.0.1.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/avro-ipc-1.7.7.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/xz-1.0.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/jul-to-slf4j-1.7.16.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsr305-3.0.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/calcite-avatica-1.2.0-incubating.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/htrace-core4-
> 4.1.0-incubating.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/java-xmlbuilder-1.1.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hive-metastore-1.2.1.spark2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar System
> Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/scala-reflect-2.11.8.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/orc-mapreduce-1.4.4-nohive.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/hive-exec-1.2.1.spark2.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar System
> Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/xbean-asm5-shaded-4.4.jar System Classpath
> /spark2/pgl/yarn/usercache/pgl/appcache/application_
> 1530116533665_0008/container_1530116533665_0008_02_000001/_
> _spark_libs__/commons-digester-1.8.jar System Classpath
> /home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jcip-annotations-1.0.jar
>
>
> On Sun, Jul 1, 2018 at 10:09 AM, Sean Owen <sr...@gmail.com> wrote:
>
>> Somewhere, you have mismatched versions of Spark on your classpath.
>>
>>
>> On Sun, Jul 1, 2018, 9:01 AM Peter Liu <pe...@gmail.com> wrote:
>>
>>> Hello there,
>>>
>>> I didn't get any response/help from the user list for the following
>>> question and thought people on the dev list might be able to help?:
>>>
>>> I upgraded to spark 2.3.1 from spark 2.2.1, ran my streaming workload
>>> and got the error (java.lang.AbstractMethodError) never seen before;
>>> see the error exception stack attached in (a) bellow.
>>>
>>> anyone knows if spark 2.3.1 works well with kafka
>>> spark-streaming-kafka-0-10?
>>>
>>> this link of spark kafka integration page doesn't say anything about any
>>> limitation:
>>> https://spark.apache.org/docs/2.3.1/streaming-kafka-integration.html
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_2.3.1_streaming-2Dkafka-2Dintegration.html&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=LORvnW8MyMYoQuiRJpv4qo1519IpQ57V-MxUgro0vP0&e=>
>>>
>>> but this discussion seems to say there is indeed an issue when upgrading
>>> to spark 2.3.1:
>>> https://stackoverflow.com/questions/49180931/abstractmethode
>>> rror-creating-kafka-stream
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_49180931_abstractmethoderror-2Dcreating-2Dkafka-2Dstream&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=xIl7y-rgXq2AUuNg8cKeBIRApmRMLnrJbPsf4kc5Zzg&e=>
>>>
>>> i also rebuilt the workload with some spark 2.3.1 jars (see (b) below).
>>> it doesn't seem to help.
>>>
>>> Would be great if anyone could kindly share any insights here.
>>>
>>> Thanks!
>>>
>>> Peter
>>>
>>> (a) the exception
>>> Exception in thread "stream execution thread for [id =
>>> 5adae836-268a-4ebf-adc4-e3cc9fbe5acf, runId =
>>> 70e78d5c-665e-4c6f-a0cc-41a56e488e30]" java.lang.AbstractMethodError
>>> at org.apache.spark.internal.Logging$class.initializeLogIfNeces
>>> sary(Logging.scala:99)
>>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$.initializ
>>> eLogIfNecessary(KafkaSourceProvider.scala:369)
>>> at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
>>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$.log(Kafka
>>> SourceProvider.scala:369)
>>> at org.apache.spark.internal.Logging$class.logDebug(Logging.
>>> scala:58)
>>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$.logDebug(
>>> KafkaSourceProvider.scala:369)
>>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$ConfigUpda
>>> ter.set(KafkaSourceProvider.scala:439)
>>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$.kafkaPara
>>> msForDriver(KafkaSourceProvider.scala:394)
>>> at org.apache.spark.sql.kafka010.KafkaSourceProvider.createSour
>>> ce(KafkaSourceProvider.scala:90)
>>> at org.apache.spark.sql.execution.datasources.DataSource.
>>> createSource(DataSource.scala:277)
>>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>> $$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:80)
>>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>> $$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:77)
>>> at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLi
>>> ke.scala:194)
>>> at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.
>>> scala:80)
>>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>> $$anonfun$1.applyOrElse(MicroBatchExecution.scala:77)
>>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>> $$anonfun$1.applyOrElse(MicroBatchExecution.scala:75)
>>> at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.appl
>>> y(TreeNode.scala:267)
>>> at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.appl
>>> y(TreeNode.scala:267)
>>> at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigi
>>> n(TreeNode.scala:70)
>>> at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(T
>>> reeNode.scala:266)
>>> at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeN
>>> ode.scala:256)
>>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>> .logicalPlan$lzycompute(MicroBatchExecution.scala:75)
>>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution
>>> .logicalPlan(MicroBatchExecution.scala:61)
>>> at org.apache.spark.sql.execution.streaming.StreamExecution.org
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.sql.execution.streaming.StreamExecution.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=lU8N2IsjW321Bu4T4BHLHvj1OSsKdPI_R_ahmOgq9rA&e=>
>>> $apache$spark$sql$execution$streaming$StreamExecution$$
>>> runStream(StreamExecution.scala:265)
>>> at org.apache.spark.sql.execution.streaming.StreamExecution$$
>>> anon$1.run(StreamExecution.scala:189)
>>>
>>> (b)* the build script update:*
>>>
>>> [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$ diff
>>> build.sbt spk211-build.sbt.original
>>> 10,11c10,11
>>> < libraryDependencies += "org.apache.spark" % "spark-sql_2.11" %*
>>> "2.3.1"*
>>> < libraryDependencies += "org.apache.spark" % "spark-core_2.11" %*
>>> "2.3.1"*
>>> ---
>>> > libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.1"
>>> > libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.1"
>>> [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$
>>>
>>>
>>>
>>>
>
Re: spark 2.3.1 with kafka spark-streaming-kafka-0-10 (java.lang.AbstractMethodError)
Posted by Peter Liu <pe...@gmail.com>.
Hi Sean,
thanks for your reply (sorry for the late response - it was a long weekend
here).
do you happen to have any suggestion for any specific area or jars I should
take a close look? (i have the spark environment page attached below).
This is very much appreciated!
Peter
Environment
Runtime Information
NameValue
Java Home /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre
Java Version 1.8.0-internal (Oracle Corporation)
Scala Version version 2.11.8 Spark Properties
NameValue
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.executor.extraJavaOptions -XX:ParallelGCThreads=4
spark.driver.host isvhdp11lab.spark.ibm.com
spark.history.fs.logDirectory hdfs://isvhdp7lab/sparkLogs
spark.eventLog.enabled true
spark.executor.heartbeatInterval 60s
spark.ui.port 0
spark.driver.port 39641
spark.yarn.app.id application_1530116533665_0008
spark.app.name Consumer Cfg:
Stream_Dur300000_RecPerSec60000_camp100_recGenNumP10
spark.network.timeout 1200
spark.scheduler.mode FIFO
spark.driver.memory 4g
spark.executor.instances 60
spark.yarn.secondary.jars
org.apache.kafka_kafka-clients-0.10.0.1.jar,org.apache.spark_spark-sql-kafka-0-10_2.11-2.2.0.jar,org.spark-project.spark_unused-1.0.0.jar,net.jpountz.lz4_lz4-1.3.0.jar,org.xerial.snappy_snappy-java-1.1.2.6.jar,org.slf4j_slf4j-api-1.7.16.jar
spark.executor.id driver
spark.streaming.stopGracefullyOnShutdown true
spark.yarn.app.container.log.dir *********(redacted)
spark.submit.deployMode cluster
spark.master yarn
spark.ui.filters org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
spark.executor.memory 3g
spark.eventLog.dir hdfs://isvhdp7lab/sparkLogs
spark.executor.cores 1
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTS
isvhdp7.aus.stglabs.ibm.com
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES
http://isvhdp7.aus.stglabs.ibm.com:8088/proxy/application_1530116533665_0008
spark.yarn.dist.jars
file:///home/pgl/SparkStreamingBenchmark-RemoteConsumer/org.apache.kafka_kafka-clients-0.10.0.1.jar,file:///home/pgl/.ivy2/jars/org.apache.spark_spark-sql-kafka-0-10_2.11-2.2.0.jar,file:///home/pgl/.ivy2/jars/org.apache.kafka_kafka-clients-0.10.0.1.jar,file:///home/pgl/.ivy2/jars/org.spark-project.spark_unused-1.0.0.jar,file:///home/pgl/.ivy2/jars/net.jpountz.lz4_lz4-1.3.0.jar,file:///home/pgl/.ivy2/jars/org.xerial.snappy_snappy-java-1.1.2.6.jar,file:///home/pgl/.ivy2/jars/org.slf4j_slf4j-api-1.7.16.jar
spark.app.id application_1530116533665_0008
spark.sql.shuffle.partitions 40 System Properties
NameValue
java.io.tmpdir
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/tmp
line.separator
path.separator :
sun.management.compiler HotSpot 64-Bit Tiered Compilers
sun.cpu.endian little
java.specification.version 1.8
java.vm.specification.name Java Virtual Machine Specification
java.vendor Oracle Corporation
java.vm.specification.version 1.8
user.home /home/pgl
file.encoding.pkg sun.io
sun.nio.ch.bugLevel
sun.arch.data.model 64
sun.boot.library.path /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/ppc64le
user.dir
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001
java.library.path
/home/maha/hadoop-2.9.0/lib/native/:/home/maha/hadoop-2.9.0/lib/native/:/home/pgl/hadoop-2.9.0/lib/native::/usr/java/packages/lib/ppc64le:/usr/lib64:/lib64:/lib:/usr/lib
sun.cpu.isalist
os.arch
ppc64le
java.vm.version 25.71-b00
java.endorsed.dirs /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/endorsed
java.runtime.version 1.8.0-internal-root_2018_04_19_13_32-b00
java.vm.info mixed mode
java.ext.dirs
/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/ext:/usr/java/packages/lib/ext
java.runtime.name OpenJDK Runtime Environment
file.separator /
java.class.version 52.0
java.specification.name Java Platform API Specification
sun.boot.class.path
/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/resources.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/rt.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/sunrsasign.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/jsse.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/jce.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/charsets.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/jfr.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/classes
file.encoding UTF-8
user.timezone America/Chicago
java.specification.vendor Oracle Corporation
sun.java.launcher SUN_STANDARD
java.vm.compressedOopsMode Zero based
os.version 4.14.0-81.el7a.ppc64le
sun.os.patch.level unknown
java.vm.specification.vendor Oracle Corporation
user.country US
sun.jnu.encoding UTF-8
user.language en
java.vendor.url http://java.oracle.com/
java.awt.printerjob sun.print.PSPrinterJob
java.awt.graphicsenv sun.awt.X11GraphicsEnvironment
awt.toolkit sun.awt.X11.XToolkit
os.name Linux
java.vm.vendor Oracle Corporation
java.vendor.url.bug http://bugreport.sun.com/bugreport/
user.name pgl
java.vm.name OpenJDK 64-Bit Server VM
sun.java.command org.apache.spark.deploy.yarn.ApplicationMaster --class
SparkStreamingTest --jar
file:/home/pgl/SparkStreamingBenchmark-RemoteConsumer/target/scala-2.11/sparkstreamingbenchmark-remoteconsumer_2.11-0.1-SNAPSHOT.jar
--arg 300000 --arg 60000 --arg 20 --arg 10 --arg 100 --arg 80 --arg 80
--arg 80 --arg Stream_Dur300000_RecPerSec60000_camp100_recGenNumP10
--properties-file
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_conf__/__spark_conf__.properties
java.home /home/pgl/root-dir/jdk8_trlopt_55cf01d/jre
java.version 1.8.0-internal
sun.io.unicode.encoding UnicodeLittle Classpath Entries
ResourceSource
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/stax-api-1.0-2.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-beanutils-core-1.8.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guava-11.0.2.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/datanucleus-api-jdo-3.2.6.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/guava-11.0.2.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/java-util-1.9.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-hive_2.11-2.3.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/javax.inject-1.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-recipes-2.7.1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/libthrift-0.9.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/okio-1.13.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-cli-1.2.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.9.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guice-3.0.jar System Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/leveldbjni-all-1.8.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-network-common_2.11-2.3.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.9.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/okhttp-3.8.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/junit-4.11.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/json4s-ast_2.11-3.2.11.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-configuration-1.6.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/avro-1.7.7.jar System Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/generex-1.0.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/arpack_combined_all-0.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-common-2.7.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-jaxrs-1.9.13.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-cli-1.2.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/kubernetes-client-3.0.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/eigenbase-properties-1.1.5.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/pyrolite-4.13.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-client-2.9.0.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang-2.6.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.0-tests.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.9.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-sslengine-6.1.26.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-kubernetes_2.11-2.3.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/log4j-1.2.17.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spire_2.11-0.13.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-encoding-1.8.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-common-2.22.2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-6.1.26.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-test-2.7.1.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-net-2.2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/avro-1.7.7.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/logging-interceptor-3.8.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-compress-1.4.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-auth-2.7.3.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-net-3.1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/metrics-jvm-3.1.5.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-registry-2.9.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/machinist_2.11-0.6.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/xz-1.0.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/httpclient-4.5.2.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jetty-util-6.1.26.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/stax-api-1.0.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jsr305-1.3.9.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/netty-all-4.1.17.Final.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/arrow-memory-0.8.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/antlr-runtime-3.4.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javax.inject-1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/orc-core-1.4.4-nohive.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/json-io-2.5.1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/metrics-core-3.1.5.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-graphx_2.11-2.3.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/chill_2.11-0.8.4.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-launcher_2.11-2.3.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/paranamer-2.8.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-compress-1.4.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-client-2.7.1.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/httpclient-4.5.2.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-hadoop-bundle-1.6.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/avro-1.7.7.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.9.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/zjsonpatch-0.3.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-framework-2.7.1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-guava-2.22.2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/gson-2.2.4.jar System Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-catalyst_2.11-2.3.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/snappy-0.2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/log4j-1.2.17.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/compress-lzf-1.0.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hive-beeline-1.2.1.spark2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-digester-1.8.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/metrics-core-3.0.1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/super-csv-2.2.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-dataformat-yaml-2.6.7.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-core-2.6.7.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/JavaEWAH-0.3.2.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javax.ws.rs-api-2.0.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-mapreduce-client-core-2.7.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/apache-log4j-extras-1.2.17.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/asm-3.2.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/ehcache-3.3.1.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-json-1.9.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/asm-3.2.jar System Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsp-api-2.1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javolution-5.5.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-databind-2.6.7.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/xmlenc-0.52.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-mesos_2.11-2.3.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/janino-3.0.8.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/htrace-core-3.1.0-incubating.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-common-2.9.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/breeze_2.11-0.13.2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-lang-2.6.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-format-2.3.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/snappy-java-1.0.5.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/mesos-1.4.0-shaded-protobuf.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/woodstox-core-5.0.3.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-nfs-2.9.0.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/servlet-api-2.5.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-dbcp-1.4.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-core-asl-1.9.13.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-logging-1.1.3.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/asm-3.2.jar System Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jcl-over-slf4j-1.7.16.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math-2.2.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-i18n-2.0.0-M15.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/javax.inject-1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jets3t-0.9.4.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.9.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/libfb303-0.9.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/core-1.1.2.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-mllib_2.11-2.3.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-mapreduce-client-shuffle-2.7.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jpam-1.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-util-6.1.26.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-client-2.7.1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-core_2.11-2.3.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/api-asn1-api-1.0.0-M20.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jtransforms-2.4.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-network-shuffle_2.11-2.3.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-math3-3.4.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/snappy-java-1.0.5.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-column-1.8.3.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/mockito-all-1.8.5.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jaxb-api-2.2.2.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/breeze-macros_2.11-0.13.2.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-kvstore_2.11-2.3.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-mapreduce-client-jobclient-2.7.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/base64-2.3.8.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/scala-xml_2.11-1.0.5.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/oro-2.0.8.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-beanutils-1.7.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jetty-6.1.26.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/aopalliance-repackaged-2.4.0-b34.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-logging-1.1.3.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-hdfs-2.7.3.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jets3t-0.9.0.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/guava-11.0.2.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-mapreduce-client-common-2.7.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/metrics-graphite-3.1.5.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/scala-compiler-2.11.8.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-server-2.22.2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/aopalliance-1.0.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/joda-time-2.9.3.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.9.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/netty-3.9.9.Final.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/activation-1.1.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/stax2-api-3.1.4.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/gson-2.2.4.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsch-0.1.54.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-module-jaxb-annotations-2.6.7.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/scalap-2.11.8.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/py4j-0.10.7.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/xmlenc-0.52.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.9.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/curator-client-2.7.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javassist-3.18.1-GA.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-core-1.9.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/kryo-shaded-3.0.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/httpcore-4.4.8.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/calcite-core-1.2.0-incubating.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-compiler-3.0.8.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-yarn-server-common-2.7.3.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/zookeeper-3.4.6.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-configuration-1.6.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/stringtemplate-3.2.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/curator-framework-2.7.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-io-2.4.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/antlr-2.7.7.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsr305-3.0.0.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.9.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-math3-3.1.1.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-jackson-1.8.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-mllib-local_2.11-2.3.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/scala-parser-combinators_2.11-1.0.4.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javax.inject-2.4.0-b34.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-client-2.22.2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsch-0.1.54.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/httpclient-4.5.4.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-httpclient-3.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jsr305-3.0.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/lz4-java-1.4.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jta-1.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/protobuf-java-2.5.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/snakeyaml-1.15.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hadoop-auth-2.9.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/datanucleus-core-3.2.10.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/avro-mapred-1.7.7-hadoop2.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/aopalliance-1.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javax.servlet-api-3.1.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/etc/hadoop System Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-media-jaxb-2.22.2.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-unsafe_2.11-2.3.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-xc-1.9.13.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/antlr4-runtime-4.7.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-native-client-2.9.0-tests.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-yarn-server-web-proxy-2.7.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-sketch_2.11-2.3.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/api-asn1-api-1.0.0-M20.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/objenesis-2.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/guava-14.0.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/apacheds-kerberos-codec-2.0.0-M15.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/slf4j-api-1.7.16.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/avro-1.7.7.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/ivy-2.4.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-yarn-client-2.7.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_conf__/__hadoop_conf__
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/slf4j-api-1.7.25.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.9.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-pool-1.5.4.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-framework-2.7.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/xz-1.0.jar System Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hk2-api-2.4.0-b34.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jettison-1.1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/calcite-linq4j-1.2.0-incubating.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hppc-0.7.2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hadoop-annotations-2.9.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-module-scala_2.11-2.6.7.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/guice-3.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/kubernetes-model-2.0.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hive-jdbc-1.2.1.spark2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-io-2.4.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-repl_2.11-2.3.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-hadoop-1.8.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/osgi-resource-locator-1.0.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.9.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-annotations-2.6.7.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/opencsv-2.3.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/log4j-1.2.17.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/apacheds-i18n-2.0.0-M15.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-collections-3.2.2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.9.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-3.0.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/activation-1.1.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-io-2.4.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-codec-1.4.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/aircompressor-0.8.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/nimbus-jose-jwt-3.9.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-router-2.9.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.9.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/zstd-jni-1.3.2-2.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-yarn_2.11-2.3.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.9.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/json4s-jackson_2.11-3.2.11.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/validation-api-1.1.0.Final.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-common-1.8.3.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-lang3-3.5.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/okio-1.4.0.jar System Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/metrics-json-3.1.5.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jline-2.12.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/paranamer-2.3.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-codec-1.10.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/fst-2.50.jar System Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-configuration-1.6.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/json-smart-1.1.1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-client-2.7.3.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.9.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/xmlenc-0.52.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/derby-10.12.1.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.9.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_conf__
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/paranamer-2.3.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/okhttp-2.4.0.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0-tests.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jcip-annotations-1.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/minlog-1.3.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-io-2.4.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/flatbuffers-1.2.0-3f79e055.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/stax2-api-3.1.4.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/json4s-core_2.11-3.2.11.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hive-cli-1.2.1.spark2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/httpcore-4.4.4.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/netty-3.6.2.Final.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hk2-utils-2.4.0-b34.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-yarn-api-2.7.3.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jettison-1.1.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/curator-recipes-2.7.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-container-servlet-core-2.22.2.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/api-util-1.0.0-M20.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hamcrest-core-1.3.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math3-3.1.1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hk2-locator-2.4.0-b34.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-crypto-1.0.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.9.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/ST4-4.0.4.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-core-1.9.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/httpcore-4.4.4.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/guice-servlet-3.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-lang-2.6.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0-tests.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-sql_2.11-2.3.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/json-smart-1.1.1.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/snappy-java-1.0.5.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-native-client-2.9.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/xercesImpl-2.9.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/scala-library-2.11.8.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-6.1.26.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-common-2.9.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-digester-1.8.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-beanutils-1.7.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsp-api-2.1.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-yarn-common-2.7.3.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/api-util-1.0.0-M20.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/automaton-1.11-8.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-net-3.1.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/htrace-core4-4.1.0-incubating.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/woodstox-core-5.0.3.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-api-2.9.0.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jets3t-0.9.0.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/univocity-parsers-2.5.9.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-codec-1.4.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/stream-2.7.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spire-macros_2.11-0.13.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/datanucleus-rdbms-3.2.9.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-lang3-3.4.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.9.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-streaming_2.11-2.3.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang3-3.4.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-server-1.9.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/zookeeper-3.4.6.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/snappy-java-1.1.2.6.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-io-2.4.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-tags_2.11-2.3.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jodd-core-3.5.2.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/RoaringBitmap-0.5.11.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-container-servlet-2.22.2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/slf4j-log4j12-1.7.16.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/java-xmlbuilder-0.4.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/activation-1.1.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/arrow-vector-0.8.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/paranamer-2.3.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-annotations-2.7.3.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/asm-3.2.jar System Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/arrow-format-0.8.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/servlet-api-2.5.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/junit-4.11.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/shapeless_2.11-2.3.2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-client-2.9.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-module-paranamer-2.7.9.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jsp-api-2.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-beanutils-core-1.8.0.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/macro-compat_2.11-1.1.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-client-1.9.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-hive-thriftserver_2.11-2.3.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.9.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/stax-api-1.0-2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-cli-1.2.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/xz-1.0.jar System Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-json-1.9.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/bonecp-0.8.0.RELEASE.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-client-2.9.0-tests.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/bcprov-jdk15on-1.58.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-server-1.9.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/chill-java-0.8.4.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-collections-3.2.2.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-mapper-asl-1.9.13.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javax.annotation-api-1.2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/gson-2.2.4.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-mapreduce-client-app-2.7.3.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jdo-api-3.0.1.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/avro-ipc-1.7.7.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/xz-1.0.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jul-to-slf4j-1.7.16.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsr305-3.0.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/calcite-avatica-1.2.0-incubating.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/java-xmlbuilder-1.1.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hive-metastore-1.2.1.spark2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/scala-reflect-2.11.8.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/orc-mapreduce-1.4.4-nohive.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hive-exec-1.2.1.spark2.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/xbean-asm5-shaded-4.4.jar
System
Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-digester-1.8.jar
System
Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jcip-annotations-1.0.jar
On Sun, Jul 1, 2018 at 10:09 AM, Sean Owen <sr...@gmail.com> wrote:
> Somewhere, you have mismatched versions of Spark on your classpath.
>
>
> On Sun, Jul 1, 2018, 9:01 AM Peter Liu <pe...@gmail.com> wrote:
>
>> Hello there,
>>
>> I didn't get any response/help from the user list for the following
>> question and thought people on the dev list might be able to help?:
>>
>> I upgraded to spark 2.3.1 from spark 2.2.1, ran my streaming workload and
>> got the error (java.lang.AbstractMethodError) never seen before; see the
>> error exception stack attached in (a) bellow.
>>
>> anyone knows if spark 2.3.1 works well with kafka
>> spark-streaming-kafka-0-10?
>>
>> this link of spark kafka integration page doesn't say anything about any
>> limitation:
>> https://spark.apache.org/docs/2.3.1/streaming-kafka-integration.html
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_2.3.1_streaming-2Dkafka-2Dintegration.html&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=LORvnW8MyMYoQuiRJpv4qo1519IpQ57V-MxUgro0vP0&e=>
>>
>> but this discussion seems to say there is indeed an issue when upgrading
>> to spark 2.3.1:
>> https://stackoverflow.com/questions/49180931/
>> abstractmethoderror-creating-kafka-stream
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_49180931_abstractmethoderror-2Dcreating-2Dkafka-2Dstream&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=xIl7y-rgXq2AUuNg8cKeBIRApmRMLnrJbPsf4kc5Zzg&e=>
>>
>> i also rebuilt the workload with some spark 2.3.1 jars (see (b) below).
>> it doesn't seem to help.
>>
>> Would be great if anyone could kindly share any insights here.
>>
>> Thanks!
>>
>> Peter
>>
>> (a) the exception
>> Exception in thread "stream execution thread for [id =
>> 5adae836-268a-4ebf-adc4-e3cc9fbe5acf, runId = 70e78d5c-665e-4c6f-a0cc-41a56e488e30]"
>> java.lang.AbstractMethodError
>> at org.apache.spark.internal.Logging$class.
>> initializeLogIfNecessary(Logging.scala:99)
>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$.
>> initializeLogIfNecessary(KafkaSourceProvider.scala:369)
>> at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$.log(
>> KafkaSourceProvider.scala:369)
>> at org.apache.spark.internal.Logging$class.logDebug(
>> Logging.scala:58)
>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$.logDebug(
>> KafkaSourceProvider.scala:369)
>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$
>> ConfigUpdater.set(KafkaSourceProvider.scala:439)
>> at org.apache.spark.sql.kafka010.KafkaSourceProvider$.
>> kafkaParamsForDriver(KafkaSourceProvider.scala:394)
>> at org.apache.spark.sql.kafka010.KafkaSourceProvider.
>> createSource(KafkaSourceProvider.scala:90)
>> at org.apache.spark.sql.execution.datasources.
>> DataSource.createSource(DataSource.scala:277)
>> at org.apache.spark.sql.execution.streaming.
>> MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.
>> apply(MicroBatchExecution.scala:80)
>> at org.apache.spark.sql.execution.streaming.
>> MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.
>> apply(MicroBatchExecution.scala:77)
>> at scala.collection.mutable.MapLike$class.getOrElseUpdate(
>> MapLike.scala:194)
>> at scala.collection.mutable.AbstractMap.getOrElseUpdate(
>> Map.scala:80)
>> at org.apache.spark.sql.execution.streaming.
>> MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:77)
>> at org.apache.spark.sql.execution.streaming.
>> MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:75)
>> at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.
>> apply(TreeNode.scala:267)
>> at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.
>> apply(TreeNode.scala:267)
>> at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.
>> withOrigin(TreeNode.scala:70)
>> at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(
>> TreeNode.scala:266)
>> at org.apache.spark.sql.catalyst.trees.TreeNode.transform(
>> TreeNode.scala:256)
>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution.
>> logicalPlan$lzycompute(MicroBatchExecution.scala:75)
>> at org.apache.spark.sql.execution.streaming.MicroBatchExecution.
>> logicalPlan(MicroBatchExecution.scala:61)
>> at org.apache.spark.sql.execution.streaming.StreamExecution.org
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.sql.execution.streaming.StreamExecution.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=lU8N2IsjW321Bu4T4BHLHvj1OSsKdPI_R_ahmOgq9rA&e=>
>> $apache$spark$sql$execution$streaming$StreamExecution$$runStream(
>> StreamExecution.scala:265)
>> at org.apache.spark.sql.execution.streaming.
>> StreamExecution$$anon$1.run(StreamExecution.scala:189)
>>
>> (b)* the build script update:*
>>
>> [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$ diff
>> build.sbt spk211-build.sbt.original
>> 10,11c10,11
>> < libraryDependencies += "org.apache.spark" % "spark-sql_2.11" %*
>> "2.3.1"*
>> < libraryDependencies += "org.apache.spark" % "spark-core_2.11" %*
>> "2.3.1"*
>> ---
>> > libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.1"
>> > libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.1"
>> [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$
>>
>>
>>
>>
Re: spark 2.3.1 with kafka spark-streaming-kafka-0-10 (java.lang.AbstractMethodError)
Posted by Sean Owen <sr...@gmail.com>.
Somewhere, you have mismatched versions of Spark on your classpath.
On Sun, Jul 1, 2018, 9:01 AM Peter Liu <pe...@gmail.com> wrote:
> Hello there,
>
> I didn't get any response/help from the user list for the following
> question and thought people on the dev list might be able to help?:
>
> I upgraded to spark 2.3.1 from spark 2.2.1, ran my streaming workload and
> got the error (java.lang.AbstractMethodError) never seen before; see the
> error exception stack attached in (a) bellow.
>
> anyone knows if spark 2.3.1 works well with kafka
> spark-streaming-kafka-0-10?
>
> this link of spark kafka integration page doesn't say anything about any
> limitation:
> https://spark.apache.org/docs/2.3.1/streaming-kafka-integration.html
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_2.3.1_streaming-2Dkafka-2Dintegration.html&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=LORvnW8MyMYoQuiRJpv4qo1519IpQ57V-MxUgro0vP0&e=>
>
> but this discussion seems to say there is indeed an issue when upgrading
> to spark 2.3.1:
>
> https://stackoverflow.com/questions/49180931/abstractmethoderror-creating-kafka-stream
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_49180931_abstractmethoderror-2Dcreating-2Dkafka-2Dstream&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=xIl7y-rgXq2AUuNg8cKeBIRApmRMLnrJbPsf4kc5Zzg&e=>
>
> i also rebuilt the workload with some spark 2.3.1 jars (see (b) below). it
> doesn't seem to help.
>
> Would be great if anyone could kindly share any insights here.
>
> Thanks!
>
> Peter
>
> (a) the exception
> Exception in thread "stream execution thread for [id =
> 5adae836-268a-4ebf-adc4-e3cc9fbe5acf, runId =
> 70e78d5c-665e-4c6f-a0cc-41a56e488e30]" java.lang.AbstractMethodError
> at
> org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)
> at
> org.apache.spark.sql.kafka010.KafkaSourceProvider$.initializeLogIfNecessary(KafkaSourceProvider.scala:369)
> at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
> at
> org.apache.spark.sql.kafka010.KafkaSourceProvider$.log(KafkaSourceProvider.scala:369)
> at
> org.apache.spark.internal.Logging$class.logDebug(Logging.scala:58)
> at
> org.apache.spark.sql.kafka010.KafkaSourceProvider$.logDebug(KafkaSourceProvider.scala:369)
> at
> org.apache.spark.sql.kafka010.KafkaSourceProvider$ConfigUpdater.set(KafkaSourceProvider.scala:439)
> at
> org.apache.spark.sql.kafka010.KafkaSourceProvider$.kafkaParamsForDriver(KafkaSourceProvider.scala:394)
> at
> org.apache.spark.sql.kafka010.KafkaSourceProvider.createSource(KafkaSourceProvider.scala:90)
> at
> org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:277)
> at
> org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:80)
> at
> org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:77)
> at
> scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)
> at
> scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80)
> at
> org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:77)
> at
> org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:75)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256)
> at
> org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan$lzycompute(MicroBatchExecution.scala:75)
> at
> org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan(MicroBatchExecution.scala:61)
> at org.apache.spark.sql.execution.streaming.StreamExecution.org
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.sql.execution.streaming.StreamExecution.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9YF85k6Q86ELSbXl40mkGw&m=BhWNMxHTJ5GKalIFY0_1N-6KmUXTHe6wwnFCkVkJZJU&s=lU8N2IsjW321Bu4T4BHLHvj1OSsKdPI_R_ahmOgq9rA&e=>
> $apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:265)
> at
> org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189)
>
> (b)* the build script update:*
>
> [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$ diff
> build.sbt spk211-build.sbt.original
> 10,11c10,11
> < libraryDependencies += "org.apache.spark" % "spark-sql_2.11" %* "2.3.1"*
> < libraryDependencies += "org.apache.spark" % "spark-core_2.11" %*
> "2.3.1"*
> ---
> > libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.1"
> > libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.1"
> [pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$
>
>
>
>