You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "xinzhang (JIRA)" <ji...@apache.org> on 2018/01/10 02:15:00 UTC
[jira] [Created] (SPARK-23022) Spark Thrift Server always cache
resource issues
xinzhang created SPARK-23022:
--------------------------------
Summary: Spark Thrift Server always cache resource issues
Key: SPARK-23022
URL: https://issues.apache.org/jira/browse/SPARK-23022
Project: Spark
Issue Type: Bug
Components: Deploy
Affects Versions: 2.2.1
Environment: CentOS6.x
Spark2.x
JDK1.8
Reporter: xinzhang
Hi. I use the Thrift Server for SparkSQL . I query muiltle query.I deply the Spark on Yarn.
When I finish my query.Thrift Server always cache the Yarn Resources. Any suggests will be helpful.
------------------------------------------------------------------------------------
Here is the img .
------------------------------------------------------------------------------------
!https://user-images.githubusercontent.com/8244097/34752652-8d224416-f5ee-11e7-89d3-5868c128378d.png!
!https://user-images.githubusercontent.com/8244097/34752397-215bea08-f5ed-11e7-89f4-13ef9ab78904.png!
!https://user-images.githubusercontent.com/8244097/34752403-2756d224-f5ed-11e7-97d3-9c7d21c48f3a.png!
!https://user-images.githubusercontent.com/8244097/34752409-2ba1d3ce-f5ed-11e7-85d6-9e46ae8a3e2b.png!
------------------------------------------------------------------------------------
Here is the Spark Conf .
------------------------------------------------------------------------------------
{code:java}
spark.dynamicAllocation.minExecutors 1
spark.dynamicAllocation.maxExecutors 500
spark.dynamicAllocation.enabled true
spark.shuffle.service.enabled true
spark.executor.instances 2
spark.executor.memory 6g
#serializer
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryoserializer.buffer.max 512m
#spark.kryo.unsafe true
spark.kryo.referenceTracking false
spark.rdd.compress true
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 1g
spark.yarn.archive hdfs://ns/data1/hadooptmp/spark2.2.1/jars
spark.yarn.am.memory 2g
spark.driver.memory 4g
spark.driver.maxResultSize 2g
#SPARK SQL
spark.sql.shuffle.partitions 500
spark.sql.statistics.fallBackToHdfs true
spark.sql.orc.filterPushdown true
spark.sql.autoBroadcastJoinThreshold 104857600
spark.sql.adaptive.enabled true
spark.history.fs.logDirectory hdfs://ns/data4/hadooptmp/spark-history
spark.eventLog.enabled true
spark.eventLog.compress true
spark.eventLog.dir hdfs://ns/data4/hadooptmp/spark-history
spark.yarn.historyServer.address 172.31.10.119:18080
spark.io.compression.codec snappy
spark.executor.logs.rolling.enableCompression true
spark.dynamicAllocation.executorIdleTimeout 10s
spark.network.timeout 600s
spark.sql.parquet.writeLegacyFormat true
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org