set up spark 1.4.1 as default spark engine in HDP 2.2/2.3

[Divya Gehlot <di...@gmail.com>]

Hi,
As per requirement I need to use Spark 1.4.1.But HDP doesnt comes with
Spark 1.4.1 version.
As instructed in  this hortonworks page
<http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/>
I am able to set up Spark 1.4 in HDP ,but when I run the spark shell It
shows Spark 1.3.1 REPL instead of spark 1.4.1 .
Do I need to make any configuration changes apart from instructions given
in above mentioned page.
How do I set spark 1.4.1 as the default spark engine.
Options :
1.Do I need to remove the current spark 1.3.1 dir.
2. I named my spark installation dir names spark 1.4.1,Do I need to rename
as Spark.
3.Is there configuration needs to change  in HDP to set spark 1.4.1 as
default spark engine .

Would really appreciate your help.

Thanks,
Regards,

Re: set up spark 1.4.1 as default spark engine in HDP 2.2/2.3

[Saisai Shao <sa...@gmail.com>]

Please make sure the spark shell script you're running is pointed to
<dir-to-spark-1.4.1>/bin/spark-shell

Just follow the instructions to correctly configure your spark 1.4.1 and
execute correct script are enough.

On Wed, Dec 9, 2015 at 11:28 AM, Divya Gehlot <di...@gmail.com>
wrote:

> Hi,
> As per requirement I need to use Spark 1.4.1.But HDP doesnt comes with
> Spark 1.4.1 version.
> As instructed in  this hortonworks page
> <http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/>
> I am able to set up Spark 1.4 in HDP ,but when I run the spark shell It
> shows Spark 1.3.1 REPL instead of spark 1.4.1 .
> Do I need to make any configuration changes apart from instructions given
> in above mentioned page.
> How do I set spark 1.4.1 as the default spark engine.
> Options :
> 1.Do I need to remove the current spark 1.3.1 dir.
> 2. I named my spark installation dir names spark 1.4.1,Do I need to rename
> as Spark.
> 3.Is there configuration needs to change  in HDP to set spark 1.4.1 as
> default spark engine .
>
> Would really appreciate your help.
>
> Thanks,
> Regards,
>