You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by sachin Singh <sa...@gmail.com> on 2015/05/14 07:02:04 UTC
Spark performance in cluster mode using yarn
Hi Friends,
please someone can give the idea, Ideally what should be time(complete job
execution) for spark job,
I have data in a hive table, amount of data would be 1GB , 2 lacs rows for
whole month,
I want to do monthly aggregation, using SQL queries,groupby
I have only one node,1 cluster,below configuration for running job,
--num-executors 2 --driver-memory 3g --driver-java-options
"-XX:MaxPermSize=1G" --executor-memory 2g --executor-cores 2
how much approximate time require to finish the job,
or can someone suggest the best way to get quickly results,
Thanks in advance,
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-performance-in-cluster-mode-using-yarn-tp22877.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark performance in cluster mode using yarn
Posted by Sachin Singh <sa...@gmail.com>.
Hi Ayan,
I am asking general scenarios as per given info/configuration, from
experts, not specific,
java code is nothing get hive context and select query,
there is no serialization or any other complex things I kept,straight
forward, 10 lines of code,
Group Please suggest if any Idea,
Regards
Sachin
On Fri, May 15, 2015 at 6:57 AM, ayan guha <gu...@gmail.com> wrote:
> With this information it is hard to predict. What's the performance you
> are getting? What's your desired performance? Maybe you can post your code
> and experts can suggests improvement?
> On 14 May 2015 15:02, "sachin Singh" <sa...@gmail.com> wrote:
>
>> Hi Friends,
>> please someone can give the idea, Ideally what should be time(complete job
>> execution) for spark job,
>>
>> I have data in a hive table, amount of data would be 1GB , 2 lacs rows for
>> whole month,
>> I want to do monthly aggregation, using SQL queries,groupby
>>
>> I have only one node,1 cluster,below configuration for running job,
>> --num-executors 2 --driver-memory 3g --driver-java-options
>> "-XX:MaxPermSize=1G" --executor-memory 2g --executor-cores 2
>>
>> how much approximate time require to finish the job,
>>
>> or can someone suggest the best way to get quickly results,
>>
>> Thanks in advance,
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-performance-in-cluster-mode-using-yarn-tp22877.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
Re: Spark performance in cluster mode using yarn
Posted by ayan guha <gu...@gmail.com>.
With this information it is hard to predict. What's the performance you are
getting? What's your desired performance? Maybe you can post your code and
experts can suggests improvement?
On 14 May 2015 15:02, "sachin Singh" <sa...@gmail.com> wrote:
> Hi Friends,
> please someone can give the idea, Ideally what should be time(complete job
> execution) for spark job,
>
> I have data in a hive table, amount of data would be 1GB , 2 lacs rows for
> whole month,
> I want to do monthly aggregation, using SQL queries,groupby
>
> I have only one node,1 cluster,below configuration for running job,
> --num-executors 2 --driver-memory 3g --driver-java-options
> "-XX:MaxPermSize=1G" --executor-memory 2g --executor-cores 2
>
> how much approximate time require to finish the job,
>
> or can someone suggest the best way to get quickly results,
>
> Thanks in advance,
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-performance-in-cluster-mode-using-yarn-tp22877.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>