You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by sherine ahmed <sh...@hotmail.com> on 2015/05/03 09:19:59 UTC
Hardware requirements
I need to use spark to upload a 500 GB data from hadoop on standalone mode
cluster what are the minimum hardware requirements if it's known that it
will be used for advanced analysis (social network analysis)?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Hardware-requirements-tp22744.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Hardware requirements
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Assume your block size is 128MB.
Thanks
Best Regards
On Mon, May 4, 2015 at 2:38 PM, ayan guha <gu...@gmail.com> wrote:
> Hi
>
> How do you figure out 500gig~3900 partitions? I am trying to do the math.
> If I assume 64mb block size then 1G~16 blocks and 500g~8000 blocks. If we
> assume split and block sizes are same, shouldn't we end up with 8k
> partitions?
> On 4 May 2015 17:49, "Akhil Das" <ak...@sigmoidanalytics.com> wrote:
>
>> 500GB of data will have nearly 3900 partitions and if you can have nearly
>> that many number of cores and around 500GB of memory then things will be
>> lightening fast. :)
>>
>> Thanks
>> Best Regards
>>
>> On Sun, May 3, 2015 at 12:49 PM, sherine ahmed <
>> sherine.shafei@hotmail.com> wrote:
>>
>>> I need to use spark to upload a 500 GB data from hadoop on standalone
>>> mode
>>> cluster what are the minimum hardware requirements if it's known that it
>>> will be used for advanced analysis (social network analysis)?
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Hardware-requirements-tp22744.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
Re: Hardware requirements
Posted by ayan guha <gu...@gmail.com>.
Hi
How do you figure out 500gig~3900 partitions? I am trying to do the math.
If I assume 64mb block size then 1G~16 blocks and 500g~8000 blocks. If we
assume split and block sizes are same, shouldn't we end up with 8k
partitions?
On 4 May 2015 17:49, "Akhil Das" <ak...@sigmoidanalytics.com> wrote:
> 500GB of data will have nearly 3900 partitions and if you can have nearly
> that many number of cores and around 500GB of memory then things will be
> lightening fast. :)
>
> Thanks
> Best Regards
>
> On Sun, May 3, 2015 at 12:49 PM, sherine ahmed <sherine.shafei@hotmail.com
> > wrote:
>
>> I need to use spark to upload a 500 GB data from hadoop on standalone mode
>> cluster what are the minimum hardware requirements if it's known that it
>> will be used for advanced analysis (social network analysis)?
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Hardware-requirements-tp22744.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Hardware requirements
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
500GB of data will have nearly 3900 partitions and if you can have nearly
that many number of cores and around 500GB of memory then things will be
lightening fast. :)
Thanks
Best Regards
On Sun, May 3, 2015 at 12:49 PM, sherine ahmed <sh...@hotmail.com>
wrote:
> I need to use spark to upload a 500 GB data from hadoop on standalone mode
> cluster what are the minimum hardware requirements if it's known that it
> will be used for advanced analysis (social network analysis)?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Hardware-requirements-tp22744.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>