You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Hao REN <ju...@gmail.com> on 2013/10/07 16:40:20 UTC
Deploying spark on EC2 failed
Hi,
I am trying to deploy spark 0.8.0 on EC2 with the CDH4 prebuild patch.
Here is the cmd:
./spark-ec2 -k sparkkp -i sparkkp.pem -s 3 -t m1.medium -v 0.8.0
--ebs-vol-size=100
launch spark-test
I just added -v 0.8.0. However, during the deployment, there is an error
thrown as below :
*~/spark-ec2*
*Initializing spark-standalone*
*Creating local config files...*
* File "./deploy_templates.py", line 29*
* if system_ram_mb > 100*1024;*
* ^*
*SyntaxError: invalid syntax*
I have also checked repo on github:
https://github.com/mesos/spark-ec2/blob/master/deploy_templates.py
I notice that there are somehow some modification done there. It's not the
exacte one that the launching script uses.
In the end, the cluster is launched, but without hdfs running on it.
I've tried to start hdfs manually. And the it says: JAVA_HOME is not set.
I thought there mb other config not done yet, so I terminated my deployment.
I believe that the launching script does not use the latest deploying
script.
Could someone help me out ? or some fix here ?
Thank you in advance.
Hao.
--
REN Hao
Data Engineer @ ClaraVista
Paris, France
Tel: +33 06 14 54 57 24
Re: Deploying spark on EC2 failed
Posted by Hao REN <ju...@gmail.com>.
Thank you for the lightning quick fix !
Hao
On Mon, Oct 7, 2013 at 6:39 PM, Shivaram Venkataraman <
shivaram.venkataraman@gmail.com> wrote:
> Ah - This was a recently introduced bug in deploy_templates.py. Should
> be fixed when we merge https://github.com/mesos/spark-ec2/pull/24
>
> Thanks
> Shivaram
>
> On Mon, Oct 7, 2013 at 7:40 AM, Hao REN <ju...@gmail.com> wrote:
> > Hi,
> >
> > I am trying to deploy spark 0.8.0 on EC2 with the CDH4 prebuild patch.
> >
> > Here is the cmd:
> >
> > ./spark-ec2 -k sparkkp -i sparkkp.pem -s 3 -t m1.medium -v 0.8.0
> > --ebs-vol-size=100 launch spark-test
> >
> > I just added -v 0.8.0. However, during the deployment, there is an error
> > thrown as below :
> >
> > ~/spark-ec2
> > Initializing spark-standalone
> > Creating local config files...
> > File "./deploy_templates.py", line 29
> > if system_ram_mb > 100*1024;
> > ^
> > SyntaxError: invalid syntax
> >
> > I have also checked repo on github:
> > https://github.com/mesos/spark-ec2/blob/master/deploy_templates.py
> >
> > I notice that there are somehow some modification done there. It's not
> the
> > exacte one that the launching script uses.
> >
> > In the end, the cluster is launched, but without hdfs running on it.
> >
> > I've tried to start hdfs manually. And the it says: JAVA_HOME is not set.
> >
> > I thought there mb other config not done yet, so I terminated my
> deployment.
> >
> > I believe that the launching script does not use the latest deploying
> > script.
> >
> > Could someone help me out ? or some fix here ?
> >
> > Thank you in advance.
> >
> > Hao.
> > --
> > REN Hao
> >
> > Data Engineer @ ClaraVista
> >
> > Paris, France
> >
> > Tel: +33 06 14 54 57 24
>
--
REN Hao
Data Engineer @ ClaraVista
Paris, France
Tel: +33 06 14 54 57 24
Re: Deploying spark on EC2 failed
Posted by Shivaram Venkataraman <sh...@gmail.com>.
Ah - This was a recently introduced bug in deploy_templates.py. Should
be fixed when we merge https://github.com/mesos/spark-ec2/pull/24
Thanks
Shivaram
On Mon, Oct 7, 2013 at 7:40 AM, Hao REN <ju...@gmail.com> wrote:
> Hi,
>
> I am trying to deploy spark 0.8.0 on EC2 with the CDH4 prebuild patch.
>
> Here is the cmd:
>
> ./spark-ec2 -k sparkkp -i sparkkp.pem -s 3 -t m1.medium -v 0.8.0
> --ebs-vol-size=100 launch spark-test
>
> I just added -v 0.8.0. However, during the deployment, there is an error
> thrown as below :
>
> ~/spark-ec2
> Initializing spark-standalone
> Creating local config files...
> File "./deploy_templates.py", line 29
> if system_ram_mb > 100*1024;
> ^
> SyntaxError: invalid syntax
>
> I have also checked repo on github:
> https://github.com/mesos/spark-ec2/blob/master/deploy_templates.py
>
> I notice that there are somehow some modification done there. It's not the
> exacte one that the launching script uses.
>
> In the end, the cluster is launched, but without hdfs running on it.
>
> I've tried to start hdfs manually. And the it says: JAVA_HOME is not set.
>
> I thought there mb other config not done yet, so I terminated my deployment.
>
> I believe that the launching script does not use the latest deploying
> script.
>
> Could someone help me out ? or some fix here ?
>
> Thank you in advance.
>
> Hao.
> --
> REN Hao
>
> Data Engineer @ ClaraVista
>
> Paris, France
>
> Tel: +33 06 14 54 57 24