You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2018/01/10 03:36:00 UTC
[jira] [Resolved] (SPARK-23022) Spark Thrift Server always cache
resource issues
[ https://issues.apache.org/jira/browse/SPARK-23022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-23022.
-------------------------------
Resolution: Invalid
It's not at all clear what specific problem with Spark you're reporting. For general questions please use StackOverflow or the mailing list.
> Spark Thrift Server always cache resource issues
> ------------------------------------------------
>
> Key: SPARK-23022
> URL: https://issues.apache.org/jira/browse/SPARK-23022
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 2.2.1
> Environment: CentOS6.x
> Spark2.x
> JDK1.8
> Reporter: xinzhang
>
> Hi. I use the Thrift Server for SparkSQL . I queried muiltle queries.I deploy the Spark on Yarn.
> When I finish my query.Thrift Server always cache the Yarn Resources. Any suggests will be helpful.
> ------------------------------------------------------------------------------------
> Here is the img .
> ------------------------------------------------------------------------------------
> !https://user-images.githubusercontent.com/8244097/34752652-8d224416-f5ee-11e7-89d3-5868c128378d.png!
> !https://user-images.githubusercontent.com/8244097/34752397-215bea08-f5ed-11e7-89f4-13ef9ab78904.png!
> !https://user-images.githubusercontent.com/8244097/34752403-2756d224-f5ed-11e7-97d3-9c7d21c48f3a.png!
> !https://user-images.githubusercontent.com/8244097/34752409-2ba1d3ce-f5ed-11e7-85d6-9e46ae8a3e2b.png!
> ------------------------------------------------------------------------------------
> Here is the Spark Conf .
> ------------------------------------------------------------------------------------
> {code:java}
> spark.dynamicAllocation.minExecutors 1
> spark.dynamicAllocation.maxExecutors 500
> spark.dynamicAllocation.enabled true
> spark.shuffle.service.enabled true
> spark.executor.instances 2
> spark.executor.memory 6g
> #serializer
> spark.serializer org.apache.spark.serializer.KryoSerializer
> spark.kryoserializer.buffer.max 512m
> #spark.kryo.unsafe true
> spark.kryo.referenceTracking false
> spark.rdd.compress true
> spark.memory.offHeap.enabled true
> spark.memory.offHeap.size 1g
> spark.yarn.archive hdfs://ns/data1/hadooptmp/spark2.2.1/jars
> spark.yarn.am.memory 2g
> spark.driver.memory 4g
> spark.driver.maxResultSize 2g
> #SPARK SQL
> spark.sql.shuffle.partitions 500
> spark.sql.statistics.fallBackToHdfs true
> spark.sql.orc.filterPushdown true
> spark.sql.autoBroadcastJoinThreshold 104857600
> spark.sql.adaptive.enabled true
> spark.history.fs.logDirectory hdfs://ns/data4/hadooptmp/spark-history
> spark.eventLog.enabled true
> spark.eventLog.compress true
> spark.eventLog.dir hdfs://ns/data4/hadooptmp/spark-history
> spark.yarn.historyServer.address 172.31.10.119:18080
> spark.io.compression.codec snappy
> spark.executor.logs.rolling.enableCompression true
> spark.dynamicAllocation.executorIdleTimeout 10s
> spark.network.timeout 600s
> spark.sql.parquet.writeLegacyFormat true
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org