You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by lu...@sina.com on 2016/07/20 10:00:37 UTC
run spark apps in linux crontab
hi guys: I add a spark-submit job into my Linux crontab list by the means below ,however none of them works. If I change it to a normal shell script, it is ok. I don't quite understand why. I checked the 8080 web ui of my spark cluster, no job submitted, and there is not messages in /home/hadoop/log. Any idea is welcome.
[hadoop@master ~]$ crontab -e1.22 21 * * * sh /home/hadoop/shellscripts/run4.sh > /home/hadoop/log
and in run4.sh,it wrote:$SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar > /home/hadoop/log
2.
22 21 * * * $SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar > /home/hadoop/log
3.22 21 * * * /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar > /home/hadoop/log
4.22 21 * * * hadoop /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar > /home/hadoop/log
--------------------------------
Thanks&Best regards!
San.Luo
RE: run spark apps in linux crontab
Posted by Joaquin Alzola <Jo...@lebara.com>.
Remember that the you need to souce your .bashrc
For your PATH to be set up.
From: luohui20001@sina.com [mailto:luohui20001@sina.com]
Sent: 20 July 2016 11:01
To: user <us...@spark.apache.org>
Subject: run spark apps in linux crontab
hi guys:
I add a spark-submit job into my Linux crontab list by the means below ,however none of them works. If I change it to a normal shell script, it is ok. I don't quite understand why. I checked the 8080 web ui of my spark cluster, no job submitted, and there is not messages in /home/hadoop/log.
Any idea is welcome.
[hadoop@master ~]$ crontab -e
1.
22 21 * * * sh /home/hadoop/shellscripts/run4.sh > /home/hadoop/log
and in run4.sh,it wrote:
$SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar > /home/hadoop/log
2.
22 21 * * * $SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar > /home/hadoop/log
3.
22 21 * * * /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar > /home/hadoop/log
4.
22 21 * * * hadoop /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar > /home/hadoop/log
--------------------------------
Thanks&Best regards!
San.Luo
This email is confidential and may be subject to privilege. If you are not the intended recipient, please do not copy or disclose its content but contact the sender immediately upon receipt.
Re: run spark apps in linux crontab
Posted by Rabin Banerjee <de...@gmail.com>.
HI ,
Please check your deploy mode and master , For example if you want to
deploy in yarn cluster you should use --master yarn-cluster , if you want
to do it on yarn client mode you should use --master yarn-client .
Please note for your case deploying yarn-cluster will be better as cluster
mode is async , it will not die if the launcher dies, in yarn-client mode
your driver die as soon as launcher die so as your executors .
Regards,
Rabin
On Wed, Jul 20, 2016 at 3:41 PM, focus <fo...@qq.com> wrote:
> Hi, I just meet this problem, too! The reason is crontab runtime doesn't
> have the variables you defined, such as $SPARK_HOME.
> I defined the $SPARK_HOME and other variables in /etc/profile like this:
>
> export $MYSCRIPTS=/opt/myscripts
> export $SPARK_HOME=/opt/spark
>
> then, in my crontab job script daily_job.sh
>
> #!/bin/sh
>
> . /etc/profile
>
> $SPARK_HOME/bin/spark-submit $MYSCRIPTS/fix_fh_yesterday.py
>
> then, in crontab -e
>
> 0 8 * * * /home/user/daily_job.sh
>
> hope this helps~
>
>
>
>
> ------------------ Original ------------------
> *From:* "luohui20001"<lu...@sina.com>;
> *Date:* 2016年7月20日(星期三) 晚上6:00
> *To:* "user@spark.apache.org"<us...@spark.apache.org>;
> *Subject:* run spark apps in linux crontab
>
> hi guys:
> I add a spark-submit job into my Linux crontab list by the means
> below ,however none of them works. If I change it to a normal shell script,
> it is ok. I don't quite understand why. I checked the 8080 web ui of my
> spark cluster, no job submitted, and there is not messages in
> /home/hadoop/log.
> Any idea is welcome.
>
> [hadoop@master ~]$ crontab -e
> 1.
> 22 21 * * * sh /home/hadoop/shellscripts/run4.sh > /home/hadoop/log
>
> and in run4.sh,it wrote:
> $SPARK_HOME/bin/spark-submit --class com.abc.myclass
> --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar
> $SPARK_HOME/MyJar.jar > /home/hadoop/log
>
> 2.
> 22 21 * * * $SPARK_HOME/bin/spark-submit --class com.abc.myclass
> --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar
> $SPARK_HOME/MyJar.jar > /home/hadoop/log
>
> 3.
> 22 21 * * * /usr/lib/spark/bin/spark-submit --class com.abc.myclass
> --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar
> /usr/lib/spark/MyJar.jar > /home/hadoop/log
>
> 4.
> 22 21 * * * hadoop /usr/lib/spark/bin/spark-submit --class com.abc.myclass
> --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar
> /usr/lib/spark/MyJar.jar > /home/hadoop/log
>
> --------------------------------
>
> Thanks&Best regards!
> San.Luo
>
Re:run spark apps in linux crontab
Posted by focus <fo...@qq.com>.
Hi, I just meet this problem, too! The reason is crontab runtime doesn't have the variables you defined, such as $SPARK_HOME.
I defined the $SPARK_HOME and other variables in /etc/profile like this:
export $MYSCRIPTS=/opt/myscripts
export $SPARK_HOME=/opt/spark
then, in my crontab job script daily_job.sh
#!/bin/sh
. /etc/profile
$SPARK_HOME/bin/spark-submit $MYSCRIPTS/fix_fh_yesterday.py
then, in crontab -e
0 8 * * * /home/user/daily_job.sh
hope this helps~
------------------ Original ------------------
From: "luohui20001"<lu...@sina.com>;
Date: 2016年7月20日(星期三) 晚上6:00
To: "user@spark.apache.org"<us...@spark.apache.org>;
Subject: run spark apps in linux crontab
hi guys:
I add a spark-submit job into my Linux crontab list by the means below ,however none of them works. If I change it to a normal shell script, it is ok. I don't quite understand why. I checked the 8080 web ui of my spark cluster, no job submitted, and there is not messages in /home/hadoop/log.
Any idea is welcome.
[hadoop@master ~]$ crontab -e
1.
22 21 * * * sh /home/hadoop/shellscripts/run4.sh > /home/hadoop/log
and in run4.sh,it wrote:
$SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar > /home/hadoop/log
2.
22 21 * * * $SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar > /home/hadoop/log
3.
22 21 * * * /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar > /home/hadoop/log
4.
22 21 * * * hadoop /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores 10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar > /home/hadoop/log
--------------------------------
Thanks&Best regards!
San.Luo