You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/08/31 09:13:22 UTC

[jira] [Resolved] (SPARK-17327) Throughput limitaion in spark standalone of simple task without calculation.

     [ https://issues.apache.org/jira/browse/SPARK-17327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-17327.
-------------------------------
          Resolution: Invalid
       Fix Version/s:     (was: 1.6.2)
    Target Version/s:   (was: 1.6.2)

Questions go to user@spark.apache.org, but you'd need to narrow this down further than "why does this take a certain amount of time?"

Read https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark first.

> Throughput limitaion in spark standalone of simple task without calculation.
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-17327
>                 URL: https://issues.apache.org/jira/browse/SPARK-17327
>             Project: Spark
>          Issue Type: Question
>          Components: Java API, Windows
>    Affects Versions: 1.6.2
>         Environment: windows server 2008 R2 standard
>            Reporter: xiefeng
>              Labels: performance
>
> I install a spark standalone and run the spark cluster(one master and one worker) in a windows 2008 server with 16cores and 24GB memory.
> I have done a simple test: Just create  a string RDD and simply return it. I use JMeter to test throughput but the highest is around 35/sec. I think spark is powerful at distribute calculation, but why the throughput is so limit in such simple test scenario only contains simple task dispatch and no calculation?
> 1. In JMeter I test both 10 threads or 100 threads, there is little difference around 2-3/sec.
> 2. I test both cache/not cache the RDD, there is little difference around 1-2/sec. 
> 3. During the test, the cpu and memory is in low level.
> Below is my test code:
> @RestController
> public class SimpleTest {	
> 	@RequestMapping(value = "/SimpleTest", method = RequestMethod.GET)
> 	@ResponseBody
> 	public String testProcessTransaction() {
> 		return SparkShardTest.simpleRDDTest();
> 	}
> }
> final static Map<String, JavaRDD<String>> simpleRDDs = initSimpleRDDs();
> public static Map<String, JavaRDD<String>> initSimpleRDDs()
> 	{
> 		Map<String, JavaRDD<String>> result = new ConcurrentHashMap<String, JavaRDD<String>>();
> 		JavaRDD<String> rddData = JavaSC.parallelize(data;
> 		rddData.cache().count();    //this cache will improve 1-2/sec
> 		result.put("MyRDD", rddData);
> 		return result;
> 	}
> 	
> 	public static String simpleRDDTest()
> 	{		
> 		JavaRDD<String> rddData = simpleRDDs.get("MyRDD");
> 		return rddData.first();
> 	}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org