You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2016/05/12 10:08:14 UTC
My notes on Spark Performance & Tuning Guide
Hi Al,,
Following the threads in spark forum, I decided to write up on
configuration of Spark including allocation of resources and configuration
of driver, executors, threads, execution of Spark apps and general
troubleshooting taking into account the allocation of resources for Spark
applications and OS tools at the disposal.
Since the most widespread configuration as I notice is with "Spark
Standalone Mode", I have decided to write these notes starting with
Standalone and later on moving to Yarn
-
*Standalone *– a simple cluster manager included with Spark that makes
it easy to set up a cluster.
-
*YARN* – the resource manager in Hadoop 2.
I would appreciate if anyone interested in reading and commenting to get in
touch with me directly on mich.talebzadeh@gmail.com so I can send the
write-up for their review and comments.
Just to be clear this is not meant to be any commercial proposition or
anything like that. As I seem to get involved with members troubleshooting
issues and threads on this topic, I thought it is worthwhile writing a note
about it to summarise the findings for the benefit of the community.
Regards.
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
Re: My notes on Spark Performance & Tuning Guide
Posted by Abi <an...@gmail.com>.
Please include me too
On May 12, 2016 6:08:14 AM EDT, Mich Talebzadeh <mi...@gmail.com> wrote:
>Hi Al,,
>
>
>Following the threads in spark forum, I decided to write up on
>configuration of Spark including allocation of resources and
>configuration
>of driver, executors, threads, execution of Spark apps and general
>troubleshooting taking into account the allocation of resources for
>Spark
>applications and OS tools at the disposal.
>
>Since the most widespread configuration as I notice is with "Spark
>Standalone Mode", I have decided to write these notes starting with
>Standalone and later on moving to Yarn
>
>
> -
>
> *Standalone *\u2013 a simple cluster manager included with Spark that makes
> it easy to set up a cluster.
> -
>
> *YARN* \u2013 the resource manager in Hadoop 2.
>
>
>I would appreciate if anyone interested in reading and commenting to
>get in
>touch with me directly on mich.talebzadeh@gmail.com so I can send the
>write-up for their review and comments.
>
>
>Just to be clear this is not meant to be any commercial proposition or
>anything like that. As I seem to get involved with members
>troubleshooting
>issues and threads on this topic, I thought it is worthwhile writing a
>note
>about it to summarise the findings for the benefit of the community.
>
>
>Regards.
>
>
>Dr Mich Talebzadeh
>
>
>
>LinkedIn *
>https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
><https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
>http://talebzadehmich.wordpress.com
Re: My notes on Spark Performance & Tuning Guide
Posted by Cesar Flores <ce...@gmail.com>.
Please sent me to me too !
Thanks ! ! !
Cesar Flores
On Tue, May 17, 2016 at 4:55 PM, Femi Anthony <fe...@gmail.com> wrote:
> Please send it to me as well.
>
> Thanks
>
> Sent from my iPhone
>
> On May 17, 2016, at 12:09 PM, Raghavendra Pandey <
> raghavendra.pandey@gmail.com> wrote:
>
> Can you please send me as well.
>
> Thanks
> Raghav
> On 12 May 2016 20:02, "Tom Ellis" <te...@gmail.com> wrote:
>
>> I would like to also Mich, please send it through, thanks!
>>
>> On Thu, 12 May 2016 at 15:14 Alonso Isidoro <al...@gmail.com> wrote:
>>
>>> Me too, send me the guide.
>>>
>>> Enviado desde mi iPhone
>>>
>>> El 12 may 2016, a las 12:11, Ashok Kumar <ashok34668@yahoo.com.INVALID
>>> <as...@yahoo.com.invalid>> escribió:
>>>
>>> Hi Dr Mich,
>>>
>>> I will be very keen to have a look at it and review if possible.
>>>
>>> Please forward me a copy
>>>
>>> Thanking you warmly
>>>
>>>
>>> On Thursday, 12 May 2016, 11:08, Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>
>>> Hi Al,,
>>>
>>>
>>> Following the threads in spark forum, I decided to write up on
>>> configuration of Spark including allocation of resources and configuration
>>> of driver, executors, threads, execution of Spark apps and general
>>> troubleshooting taking into account the allocation of resources for Spark
>>> applications and OS tools at the disposal.
>>>
>>> Since the most widespread configuration as I notice is with "Spark
>>> Standalone Mode", I have decided to write these notes starting with
>>> Standalone and later on moving to Yarn
>>>
>>>
>>> - *Standalone *– a simple cluster manager included with Spark that
>>> makes it easy to set up a cluster.
>>> - *YARN* – the resource manager in Hadoop 2.
>>>
>>>
>>> I would appreciate if anyone interested in reading and commenting to get
>>> in touch with me directly on mich.talebzadeh@gmail.com so I can send
>>> the write-up for their review and comments.
>>>
>>> Just to be clear this is not meant to be any commercial proposition or
>>> anything like that. As I seem to get involved with members troubleshooting
>>> issues and threads on this topic, I thought it is worthwhile writing a note
>>> about it to summarise the findings for the benefit of the community.
>>>
>>> Regards.
>>>
>>> Dr Mich Talebzadeh
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>>
--
Cesar Flores
Re: My notes on Spark Performance & Tuning Guide
Posted by Femi Anthony <fe...@gmail.com>.
Please send it to me as well.
Thanks
Sent from my iPhone
> On May 17, 2016, at 12:09 PM, Raghavendra Pandey <ra...@gmail.com> wrote:
>
> Can you please send me as well.
>
> Thanks
> Raghav
>
>> On 12 May 2016 20:02, "Tom Ellis" <te...@gmail.com> wrote:
>> I would like to also Mich, please send it through, thanks!
>>
>>> On Thu, 12 May 2016 at 15:14 Alonso Isidoro <al...@gmail.com> wrote:
>>> Me too, send me the guide.
>>>
>>> Enviado desde mi iPhone
>>>
>>>> El 12 may 2016, a las 12:11, Ashok Kumar <as...@yahoo.com.INVALID> escribió:
>>>>
>>>> Hi Dr Mich,
>>>>
>>>> I will be very keen to have a look at it and review if possible.
>>>>
>>>> Please forward me a copy
>>>>
>>>> Thanking you warmly
>>>>
>>>>
>>>> On Thursday, 12 May 2016, 11:08, Mich Talebzadeh <mi...@gmail.com> wrote:
>>>>
>>>>
>>>> Hi Al,,
>>>>
>>>>
>>>> Following the threads in spark forum, I decided to write up on configuration of Spark including allocation of resources and configuration of driver, executors, threads, execution of Spark apps and general troubleshooting taking into account the allocation of resources for Spark applications and OS tools at the disposal.
>>>>
>>>> Since the most widespread configuration as I notice is with "Spark Standalone Mode", I have decided to write these notes starting with Standalone and later on moving to Yarn
>>>>
>>>> Standalone – a simple cluster manager included with Spark that makes it easy to set up a cluster.
>>>> YARN – the resource manager in Hadoop 2.
>>>>
>>>> I would appreciate if anyone interested in reading and commenting to get in touch with me directly on mich.talebzadeh@gmail.com so I can send the write-up for their review and comments.
>>>>
>>>> Just to be clear this is not meant to be any commercial proposition or anything like that. As I seem to get involved with members troubleshooting issues and threads on this topic, I thought it is worthwhile writing a note about it to summarise the findings for the benefit of the community.
>>>>
>>>> Regards.
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>> LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>
>>>> http://talebzadehmich.wordpress.com
Re: My notes on Spark Performance & Tuning Guide
Posted by Raghavendra Pandey <ra...@gmail.com>.
Can you please send me as well.
Thanks
Raghav
On 12 May 2016 20:02, "Tom Ellis" <te...@gmail.com> wrote:
> I would like to also Mich, please send it through, thanks!
>
> On Thu, 12 May 2016 at 15:14 Alonso Isidoro <al...@gmail.com> wrote:
>
>> Me too, send me the guide.
>>
>> Enviado desde mi iPhone
>>
>> El 12 may 2016, a las 12:11, Ashok Kumar <ashok34668@yahoo.com.INVALID
>> <as...@yahoo.com.invalid>> escribió:
>>
>> Hi Dr Mich,
>>
>> I will be very keen to have a look at it and review if possible.
>>
>> Please forward me a copy
>>
>> Thanking you warmly
>>
>>
>> On Thursday, 12 May 2016, 11:08, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>
>> Hi Al,,
>>
>>
>> Following the threads in spark forum, I decided to write up on
>> configuration of Spark including allocation of resources and configuration
>> of driver, executors, threads, execution of Spark apps and general
>> troubleshooting taking into account the allocation of resources for Spark
>> applications and OS tools at the disposal.
>>
>> Since the most widespread configuration as I notice is with "Spark
>> Standalone Mode", I have decided to write these notes starting with
>> Standalone and later on moving to Yarn
>>
>>
>> - *Standalone *– a simple cluster manager included with Spark that
>> makes it easy to set up a cluster.
>> - *YARN* – the resource manager in Hadoop 2.
>>
>>
>> I would appreciate if anyone interested in reading and commenting to get
>> in touch with me directly on mich.talebzadeh@gmail.com so I can send the
>> write-up for their review and comments.
>>
>> Just to be clear this is not meant to be any commercial proposition or
>> anything like that. As I seem to get involved with members troubleshooting
>> issues and threads on this topic, I thought it is worthwhile writing a note
>> about it to summarise the findings for the benefit of the community.
>>
>> Regards.
>>
>> Dr Mich Talebzadeh
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>>
Re: My notes on Spark Performance & Tuning Guide
Posted by Tom Ellis <te...@gmail.com>.
I would like to also Mich, please send it through, thanks!
On Thu, 12 May 2016 at 15:14 Alonso Isidoro <al...@gmail.com> wrote:
> Me too, send me the guide.
>
> Enviado desde mi iPhone
>
> El 12 may 2016, a las 12:11, Ashok Kumar <ashok34668@yahoo.com.INVALID
> <as...@yahoo.com.invalid>> escribió:
>
> Hi Dr Mich,
>
> I will be very keen to have a look at it and review if possible.
>
> Please forward me a copy
>
> Thanking you warmly
>
>
> On Thursday, 12 May 2016, 11:08, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>
> Hi Al,,
>
>
> Following the threads in spark forum, I decided to write up on
> configuration of Spark including allocation of resources and configuration
> of driver, executors, threads, execution of Spark apps and general
> troubleshooting taking into account the allocation of resources for Spark
> applications and OS tools at the disposal.
>
> Since the most widespread configuration as I notice is with "Spark
> Standalone Mode", I have decided to write these notes starting with
> Standalone and later on moving to Yarn
>
>
> - *Standalone *– a simple cluster manager included with Spark that
> makes it easy to set up a cluster.
> - *YARN* – the resource manager in Hadoop 2.
>
>
> I would appreciate if anyone interested in reading and commenting to get
> in touch with me directly on mich.talebzadeh@gmail.com so I can send the
> write-up for their review and comments.
>
> Just to be clear this is not meant to be any commercial proposition or
> anything like that. As I seem to get involved with members troubleshooting
> issues and threads on this topic, I thought it is worthwhile writing a note
> about it to summarise the findings for the benefit of the community.
>
> Regards.
>
> Dr Mich Talebzadeh
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
> http://talebzadehmich.wordpress.com
>
>
>
>
Re: My notes on Spark Performance & Tuning Guide
Posted by Alonso Isidoro <al...@gmail.com>.
Me too, send me the guide.
Enviado desde mi iPhone
> El 12 may 2016, a las 12:11, Ashok Kumar <as...@yahoo.com.INVALID> escribió:
>
> Hi Dr Mich,
>
> I will be very keen to have a look at it and review if possible.
>
> Please forward me a copy
>
> Thanking you warmly
>
>
> On Thursday, 12 May 2016, 11:08, Mich Talebzadeh <mi...@gmail.com> wrote:
>
>
> Hi Al,,
>
>
> Following the threads in spark forum, I decided to write up on configuration of Spark including allocation of resources and configuration of driver, executors, threads, execution of Spark apps and general troubleshooting taking into account the allocation of resources for Spark applications and OS tools at the disposal.
>
> Since the most widespread configuration as I notice is with "Spark Standalone Mode", I have decided to write these notes starting with Standalone and later on moving to Yarn
>
> Standalone – a simple cluster manager included with Spark that makes it easy to set up a cluster.
> YARN – the resource manager in Hadoop 2.
>
> I would appreciate if anyone interested in reading and commenting to get in touch with me directly on mich.talebzadeh@gmail.com so I can send the write-up for their review and comments.
>
> Just to be clear this is not meant to be any commercial proposition or anything like that. As I seem to get involved with members troubleshooting issues and threads on this topic, I thought it is worthwhile writing a note about it to summarise the findings for the benefit of the community.
>
> Regards.
>
> Dr Mich Talebzadeh
>
> LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
> http://talebzadehmich.wordpress.com
>
>
>
Re: My notes on Spark Performance & Tuning Guide
Posted by Ashok Kumar <as...@yahoo.com.INVALID>.
Hi Dr Mich,
I will be very keen to have a look at it and review if possible.
Please forward me a copy
Thanking you warmly
On Thursday, 12 May 2016, 11:08, Mich Talebzadeh <mi...@gmail.com> wrote:
Hi Al,,
Following the threads in spark forum, I decided to write up on configuration of Spark including allocation of resources and configuration of driver, executors, threads, execution of Spark apps and general troubleshooting taking into account the allocation of resources for Spark applications and OS tools at the disposal.
Since the most widespread configuration as I notice is with "Spark Standalone Mode", I have decided to write these notes starting with Standalone and later on moving to Yarn
- Standalone – a simple cluster managerincluded with Spark that makes it easy to set up a cluster.
- YARN – the resource manager inHadoop 2.
I would appreciate if anyone interested in reading and commenting to get in touch with me directly on mich.talebzadeh@gmail.com so I can send the write-up for their review and comments.
Just to be clear this is not meant to be any commercial proposition or anything like that. As I seem to get involved with members troubleshooting issues and threads on this topic, I thought it is worthwhile writing a note about it to summarise the findings for the benefit of the community.
Regards.
Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com
Re: My notes on Spark Performance & Tuning Guide
Posted by rakesh sharma <ra...@hotmail.com>.
It would be a rare doc. Please share
Get Outlook for Android<https://aka.ms/ghei36>
On Tue, May 17, 2016 at 9:14 AM -0700, "Natu Lauchande" <nl...@gmail.com>> wrote:
Hi Mich,
I am also interested in the write up.
Regards,
Natu
On Thu, May 12, 2016 at 12:08 PM, Mich Talebzadeh <mi...@gmail.com>> wrote:
Hi Al,,
Following the threads in spark forum, I decided to write up on configuration of Spark including allocation of resources and configuration of driver, executors, threads, execution of Spark apps and general troubleshooting taking into account the allocation of resources for Spark applications and OS tools at the disposal.
Since the most widespread configuration as I notice is with "Spark Standalone Mode", I have decided to write these notes starting with Standalone and later on moving to Yarn
* Standalone - a simple cluster manager included with Spark that makes it easy to set up a cluster.
* YARN - the resource manager in Hadoop 2.
I would appreciate if anyone interested in reading and commenting to get in touch with me directly on mich.talebzadeh@gmail.com<ma...@gmail.com> so I can send the write-up for their review and comments.
Just to be clear this is not meant to be any commercial proposition or anything like that. As I seem to get involved with members troubleshooting issues and threads on this topic, I thought it is worthwhile writing a note about it to summarise the findings for the benefit of the community.
Regards.
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/>
Re: My notes on Spark Performance & Tuning Guide
Posted by Natu Lauchande <nl...@gmail.com>.
Hi Mich,
I am also interested in the write up.
Regards,
Natu
On Thu, May 12, 2016 at 12:08 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> wrote:
> Hi Al,,
>
>
> Following the threads in spark forum, I decided to write up on
> configuration of Spark including allocation of resources and configuration
> of driver, executors, threads, execution of Spark apps and general
> troubleshooting taking into account the allocation of resources for Spark
> applications and OS tools at the disposal.
>
> Since the most widespread configuration as I notice is with "Spark
> Standalone Mode", I have decided to write these notes starting with
> Standalone and later on moving to Yarn
>
>
> -
>
> *Standalone *– a simple cluster manager included with Spark that makes
> it easy to set up a cluster.
> -
>
> *YARN* – the resource manager in Hadoop 2.
>
>
> I would appreciate if anyone interested in reading and commenting to get
> in touch with me directly on mich.talebzadeh@gmail.com so I can send the
> write-up for their review and comments.
>
>
> Just to be clear this is not meant to be any commercial proposition or
> anything like that. As I seem to get involved with members troubleshooting
> issues and threads on this topic, I thought it is worthwhile writing a note
> about it to summarise the findings for the benefit of the community.
>
>
> Regards.
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
Re: My notes on Spark Performance & Tuning Guide
Posted by Jeff Zhang <zj...@gmail.com>.
I think you can write it in gitbook and share it in user mail list then
everyone can comment on that.
On Wed, May 18, 2016 at 10:12 AM, Vinayak Agrawal <
vinayakagrawal88@gmail.com> wrote:
> Please include me too.
>
> Vinayak Agrawal
> Big Data Analytics
> IBM
>
> "To Strive, To Seek, To Find and Not to Yield!"
> ~Lord Alfred Tennyson
>
> On May 17, 2016, at 2:15 PM, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
> Hi all,
>
> Many thanks for your tremendous interest in the forthcoming notes. I have
> had nearly thirty requests and many supporting kind words from the
> colleagues in this forum.
>
> I will strive to get the first draft ready as soon as possible. Apologies
> for not being more specific. However, hopefully not too long for your
> perusal.
>
>
> Regards,
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 12 May 2016 at 11:08, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Hi Al,,
>>
>>
>> Following the threads in spark forum, I decided to write up on
>> configuration of Spark including allocation of resources and configuration
>> of driver, executors, threads, execution of Spark apps and general
>> troubleshooting taking into account the allocation of resources for Spark
>> applications and OS tools at the disposal.
>>
>> Since the most widespread configuration as I notice is with "Spark
>> Standalone Mode", I have decided to write these notes starting with
>> Standalone and later on moving to Yarn
>>
>>
>> -
>>
>> *Standalone *– a simple cluster manager included with Spark that
>> makes it easy to set up a cluster.
>> -
>>
>> *YARN* – the resource manager in Hadoop 2.
>>
>>
>> I would appreciate if anyone interested in reading and commenting to get
>> in touch with me directly on mich.talebzadeh@gmail.com so I can send the
>> write-up for their review and comments.
>>
>>
>> Just to be clear this is not meant to be any commercial proposition or
>> anything like that. As I seem to get involved with members troubleshooting
>> issues and threads on this topic, I thought it is worthwhile writing a note
>> about it to summarise the findings for the benefit of the community.
>>
>>
>> Regards.
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>
>
--
Best Regards
Jeff Zhang
Re: My notes on Spark Performance & Tuning Guide
Posted by Vinayak Agrawal <vi...@gmail.com>.
Please include me too.
Vinayak Agrawal
Big Data Analytics
IBM
"To Strive, To Seek, To Find and Not to Yield!"
~Lord Alfred Tennyson
> On May 17, 2016, at 2:15 PM, Mich Talebzadeh <mi...@gmail.com> wrote:
>
> Hi all,
>
> Many thanks for your tremendous interest in the forthcoming notes. I have had nearly thirty requests and many supporting kind words from the colleagues in this forum.
>
> I will strive to get the first draft ready as soon as possible. Apologies for not being more specific. However, hopefully not too long for your perusal.
>
>
> Regards,
>
>
> Dr Mich Talebzadeh
>
> LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>
> http://talebzadehmich.wordpress.com
>
>
>> On 12 May 2016 at 11:08, Mich Talebzadeh <mi...@gmail.com> wrote:
>> Hi Al,,
>>
>>
>> Following the threads in spark forum, I decided to write up on configuration of Spark including allocation of resources and configuration of driver, executors, threads, execution of Spark apps and general troubleshooting taking into account the allocation of resources for Spark applications and OS tools at the disposal.
>>
>> Since the most widespread configuration as I notice is with "Spark Standalone Mode", I have decided to write these notes starting with Standalone and later on moving to Yarn
>>
>> Standalone – a simple cluster manager included with Spark that makes it easy to set up a cluster.
>> YARN – the resource manager in Hadoop 2.
>>
>> I would appreciate if anyone interested in reading and commenting to get in touch with me directly on mich.talebzadeh@gmail.com so I can send the write-up for their review and comments.
>>
>> Just to be clear this is not meant to be any commercial proposition or anything like that. As I seem to get involved with members troubleshooting issues and threads on this topic, I thought it is worthwhile writing a note about it to summarise the findings for the benefit of the community.
>>
>> Regards.
>>
>> Dr Mich Talebzadeh
>>
>> LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>
>> http://talebzadehmich.wordpress.com
>
Re: My notes on Spark Performance & Tuning Guide
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi all,
Many thanks for your tremendous interest in the forthcoming notes. I have
had nearly thirty requests and many supporting kind words from the
colleagues in this forum.
I will strive to get the first draft ready as soon as possible. Apologies
for not being more specific. However, hopefully not too long for your
perusal.
Regards,
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 12 May 2016 at 11:08, Mich Talebzadeh <mi...@gmail.com> wrote:
> Hi Al,,
>
>
> Following the threads in spark forum, I decided to write up on
> configuration of Spark including allocation of resources and configuration
> of driver, executors, threads, execution of Spark apps and general
> troubleshooting taking into account the allocation of resources for Spark
> applications and OS tools at the disposal.
>
> Since the most widespread configuration as I notice is with "Spark
> Standalone Mode", I have decided to write these notes starting with
> Standalone and later on moving to Yarn
>
>
> -
>
> *Standalone *– a simple cluster manager included with Spark that makes
> it easy to set up a cluster.
> -
>
> *YARN* – the resource manager in Hadoop 2.
>
>
> I would appreciate if anyone interested in reading and commenting to get
> in touch with me directly on mich.talebzadeh@gmail.com so I can send the
> write-up for their review and comments.
>
>
> Just to be clear this is not meant to be any commercial proposition or
> anything like that. As I seem to get involved with members troubleshooting
> issues and threads on this topic, I thought it is worthwhile writing a note
> about it to summarise the findings for the benefit of the community.
>
>
> Regards.
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>