[jira] [Created] (BIGTOP-3462) Deploying Livy fails if the SPARK_HOME directory doesn't exist

["Kengo Seki (Jira)" <ji...@apache.org>]

Kengo Seki created BIGTOP-3462:
----------------------------------

             Summary: Deploying Livy fails if the SPARK_HOME directory doesn't exist
                 Key: BIGTOP-3462
                 URL: https://issues.apache.org/jira/browse/BIGTOP-3462
             Project: Bigtop
          Issue Type: Bug
          Components: deployment
            Reporter: Kengo Seki
            Assignee: Kengo Seki
             Fix For: 1.5.0


On some distros/platforms, deploying Livy fails with the following messages.
{code:java}
Error: Systemd start for livy-server failed!
journalctl log for livy-server:
-- Logs begin at Thu 2020-12-03 09:48:16 UTC, end at Thu 2020-12-03 10:01:07 UTC. --
Dec 03 10:00:59 59fa1cab214f systemd[1]: Starting LSB: Livy Server...
Dec 03 10:00:59 59fa1cab214f runuser[4287]: pam_unix(runuser:session): session opened for user livy by (uid=0)
Dec 03 10:00:59 59fa1cab214f livy-server[4270]: starting /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-1.el8_2.aarch64/bin/java  -cp /usr/lib/livy/jars/*:/usr/lib/livy/conf:/etc/hadoop/conf: org.apache.livy.server.LivyServer, logging to /var/log/livy/livy-livy-server.out
Dec 03 10:01:01 59fa1cab214f runuser[4287]: pam_unix(runuser:session): session closed for user livy
Dec 03 10:01:06 59fa1cab214f livy-server[4270]: Failed to start Livy Server. Return value: 1[FAILED]
Dec 03 10:01:06 59fa1cab214f systemd[1]: livy-server.service: Control process exited, code=exited status=1
Dec 03 10:01:06 59fa1cab214f systemd[1]: livy-server.service: Failed with result 'exit-code'.
Dec 03 10:01:06 59fa1cab214f systemd[1]: Failed to start LSB: Livy Server.
{code}
[https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=centos-8-aarch64-deploy/2/console]
 [https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=centos-8-x86_64-deploy/2/console]
 [https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=fedora-31-aarch64-deploy/2/console]
 [https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=fedora-31-x86_64-deploy/2/console]

So I reproduced it locally and found the following error in /var/log/livy/livy-livy-server.out, referred to above:
{code:java}
$ docker exec -it 20201204_103251_r9202_bigtop_1 cat /var/log/livy/livy-livy-server.out
20/12/04 10:40:51 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: ;other allowed users: *
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: SPARK_HOME path does not exist
	at scala.Predef$.require(Predef.scala:224)
	at org.apache.livy.utils.LivySparkUtils$.testSparkHome(LivySparkUtils.scala:56)
	at org.apache.livy.server.LivyServer.start(LivyServer.scala:77)
	at org.apache.livy.server.LivyServer$.main(LivyServer.scala:423)
	at org.apache.livy.server.LivyServer.main(LivyServer.scala)
{code}
To avoid this situation, Livy server should be started after the spark-core package is installed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)