Config for Yarn

[Schmirr Wurst <sc...@gmail.com>]

Hi,

I've seen some post there, so I tried to follow the advices, I
compiled everything with :
 mvn clean package -Pspark-1.3 -Dhadoop.version=2.6.0 -Phadoop-2.6
-Pyarn -DskipTests

The binary is working, I've also the the tutorial that is working.

- If the tutorial is working, that mean -Pspark mean a local instance
of spark is build...
Do this is always needed ? or If the connection to the remote spark
environment is working, it is nomore needed ?
- second point, is where and how do I configure my Spark Context...
I'm using cloudera 5.4 (that's the reason why I've compiled with hdp
2.6 and spark 1.3)...

Regards,
S

Re: Config for Yarn

[moon soo Lee <mo...@apache.org>]

Hi,

-Pspark-1.x flag sets proper dependency library version. So even if you're
not connecting to remote spark cluster, it's recommended to use it.

In GUI, you can go 'Interpreter' menu, and you can create new configuration
of edit existing configurations for the SparkContext.

Best,
moon

On Thu, Jul 2, 2015 at 8:16 AM Schmirr Wurst <sc...@gmail.com> wrote:

> Hi,
>
> I've seen some post there, so I tried to follow the advices, I
> compiled everything with :
>  mvn clean package -Pspark-1.3 -Dhadoop.version=2.6.0 -Phadoop-2.6
> -Pyarn -DskipTests
>
> The binary is working, I've also the the tutorial that is working.
>
> - If the tutorial is working, that mean -Pspark mean a local instance
> of spark is build...
> Do this is always needed ? or If the connection to the remote spark
> environment is working, it is nomore needed ?
> - second point, is where and how do I configure my Spark Context...
> I'm using cloudera 5.4 (that's the reason why I've compiled with hdp
> 2.6 and spark 1.3)...
>
> Regards,
> S
>