You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Jonathan Hurley <jh...@hortonworks.com> on 2017/12/15 03:03:42 UTC
Review Request 64637: Livy/Livy2 Unable To Start Due to Address
Already In Use
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64637/
-----------------------------------------------------------
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
Bugs: AMBARI-22655
https://issues.apache.org/jira/browse/AMBARI-22655
Repository: ambari
Description
-------
While restarting Livy and Livy2 on a non-root cluster, the following is seen:
```
17/12/14 14:36:23 WARN LivyConf: The configuration key livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.repl.enable-hive-context instead.
17/12/14 14:36:23 WARN LivyConf: The configuration key livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.server.csrf-protection.enabled instead.
17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: cstm-zeppelin;other allowed users: *
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
17/12/14 14:36:28 INFO LineBufferedStream: stdout: ____ __
17/12/14 14:36:28 INFO LineBufferedStream: stdout: / __/__ ___ _____/ /__
17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/ '_/
17/12/14 14:36:28 INFO LineBufferedStream: stdout: /___/ .__/_,_/_/ /_/_\ version 2.2.0.2.6.4.0-73
17/12/14 14:36:28 INFO LineBufferedStream: stdout: /_/
17/12/14 14:36:28 INFO LineBufferedStream: stdout:
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins on 2017-12-13T19:08:32Z
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision a24017869f5450397136ee8b11be818e7cd3facb
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url git@github.com:hortonworks/spark2.git
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more information.
17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal = HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM)
17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM
17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server: nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal: [HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM], added = true
17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
```
This occurs because the PID file cannot be accessed by the non-root agent and returns an exit code of 1:
{code}
call returned (1, 'cat: /var/run/livy/livy-cstm-livy-server.pid: Permission denied')
{code}
This tricks out PID detection into thinking that there is no process running (or a missing PID file) and therefore, we do not need to stop Livy/Livy2.
Diffs
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148ab6
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84613
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_livy2.py c4284889a1
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b91bd
Diff: https://reviews.apache.org/r/64637/diff/1/
Testing
-------
Manually verified PID detection on non-root cluster.
Thanks,
Jonathan Hurley
Re: Review Request 64637: Livy/Livy2 Unable To Start Due to Address
Already In Use
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On Dec. 14, 2017, 10:08 p.m., Dmytro Grinenko wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
> > Line 48 (original), 52 (patched)
> > <https://reviews.apache.org/r/64637/diff/1/?file=1917958#file1917958line56>
> >
> > i don't understand why we removing the pid file, but let's fix this later
Yeah, neither do I. However, this process is working right now, so we can fix it at a later time. Thx for the +1
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64637/#review193874
-----------------------------------------------------------
On Dec. 14, 2017, 10:03 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64637/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2017, 10:03 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22655
> https://issues.apache.org/jira/browse/AMBARI-22655
>
>
> Repository: ambari
>
>
> Description
> -------
>
> While restarting Livy and Livy2 on a non-root cluster, the following is seen:
>
> ```
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.repl.enable-hive-context instead.
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.server.csrf-protection.enabled instead.
> 17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: cstm-zeppelin;other allowed users: *
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: ____ __
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: / __/__ ___ _____/ /__
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/ '_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /___/ .__/_,_/_/ /_/_\ version 2.2.0.2.6.4.0-73
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins on 2017-12-13T19:08:32Z
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision a24017869f5450397136ee8b11be818e7cd3facb
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url git@github.com:hortonworks/spark2.git
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more information.
> 17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
> 17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
> 17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
> 17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
> 17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal = HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM)
> 17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server: nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal: [HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM], added = true
> 17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException: Address already in use
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> ```
>
> This occurs because the PID file cannot be accessed by the non-root agent and returns an exit code of 1:
> {code}
> call returned (1, 'cat: /var/run/livy/livy-cstm-livy-server.pid: Permission denied')
> {code}
>
> This tricks out PID detection into thinking that there is no process running (or a missing PID file) and therefore, we do not need to stop Livy/Livy2.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148ab6
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84613
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_livy2.py c4284889a1
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b91bd
>
>
> Diff: https://reviews.apache.org/r/64637/diff/1/
>
>
> Testing
> -------
>
> Manually verified PID detection on non-root cluster.
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 64637: Livy/Livy2 Unable To Start Due to Address
Already In Use
Posted by Dmytro Grinenko <ha...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64637/#review193874
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py
Line 48 (original), 52 (patched)
<https://reviews.apache.org/r/64637/#comment272532>
i don't understand why we removing the pid file, but let's fix this later
- Dmytro Grinenko
On Dec. 15, 2017, 3:03 a.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64637/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2017, 3:03 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22655
> https://issues.apache.org/jira/browse/AMBARI-22655
>
>
> Repository: ambari
>
>
> Description
> -------
>
> While restarting Livy and Livy2 on a non-root cluster, the following is seen:
>
> ```
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.repl.enable-hive-context instead.
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.server.csrf-protection.enabled instead.
> 17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: cstm-zeppelin;other allowed users: *
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: ____ __
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: / __/__ ___ _____/ /__
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/ '_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /___/ .__/_,_/_/ /_/_\ version 2.2.0.2.6.4.0-73
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins on 2017-12-13T19:08:32Z
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision a24017869f5450397136ee8b11be818e7cd3facb
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url git@github.com:hortonworks/spark2.git
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more information.
> 17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
> 17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
> 17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
> 17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
> 17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal = HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM)
> 17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server: nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal: [HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM], added = true
> 17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException: Address already in use
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> ```
>
> This occurs because the PID file cannot be accessed by the non-root agent and returns an exit code of 1:
> {code}
> call returned (1, 'cat: /var/run/livy/livy-cstm-livy-server.pid: Permission denied')
> {code}
>
> This tricks out PID detection into thinking that there is no process running (or a missing PID file) and therefore, we do not need to stop Livy/Livy2.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148ab6
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84613
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_livy2.py c4284889a1
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b91bd
>
>
> Diff: https://reviews.apache.org/r/64637/diff/1/
>
>
> Testing
> -------
>
> Manually verified PID detection on non-root cluster.
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 64637: Livy/Livy2 Unable To Start Due to Address
Already In Use
Posted by Dmytro Grinenko <ha...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64637/#review193873
-----------------------------------------------------------
Ship it!
Ship It!
- Dmytro Grinenko
On Dec. 15, 2017, 3:03 a.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64637/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2017, 3:03 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22655
> https://issues.apache.org/jira/browse/AMBARI-22655
>
>
> Repository: ambari
>
>
> Description
> -------
>
> While restarting Livy and Livy2 on a non-root cluster, the following is seen:
>
> ```
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.repl.enable-hive-context instead.
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.server.csrf-protection.enabled instead.
> 17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: cstm-zeppelin;other allowed users: *
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: ____ __
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: / __/__ ___ _____/ /__
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/ '_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /___/ .__/_,_/_/ /_/_\ version 2.2.0.2.6.4.0-73
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins on 2017-12-13T19:08:32Z
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision a24017869f5450397136ee8b11be818e7cd3facb
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url git@github.com:hortonworks/spark2.git
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more information.
> 17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
> 17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
> 17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
> 17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
> 17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal = HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM)
> 17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server: nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal: [HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM], added = true
> 17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException: Address already in use
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> ```
>
> This occurs because the PID file cannot be accessed by the non-root agent and returns an exit code of 1:
> {code}
> call returned (1, 'cat: /var/run/livy/livy-cstm-livy-server.pid: Permission denied')
> {code}
>
> This tricks out PID detection into thinking that there is no process running (or a missing PID file) and therefore, we do not need to stop Livy/Livy2.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148ab6
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84613
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_livy2.py c4284889a1
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b91bd
>
>
> Diff: https://reviews.apache.org/r/64637/diff/1/
>
>
> Testing
> -------
>
> Manually verified PID detection on non-root cluster.
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 64637: Livy/Livy2 Unable To Start Due to Address
Already In Use
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64637/#review193872
-----------------------------------------------------------
Ship it!
Ship It!
- Jayush Luniya
On Dec. 15, 2017, 3:03 a.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64637/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2017, 3:03 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22655
> https://issues.apache.org/jira/browse/AMBARI-22655
>
>
> Repository: ambari
>
>
> Description
> -------
>
> While restarting Livy and Livy2 on a non-root cluster, the following is seen:
>
> ```
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.repl.enable-hive-context instead.
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.server.csrf-protection.enabled instead.
> 17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: cstm-zeppelin;other allowed users: *
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: ____ __
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: / __/__ ___ _____/ /__
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/ '_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /___/ .__/_,_/_/ /_/_\ version 2.2.0.2.6.4.0-73
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins on 2017-12-13T19:08:32Z
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision a24017869f5450397136ee8b11be818e7cd3facb
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url git@github.com:hortonworks/spark2.git
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more information.
> 17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
> 17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
> 17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
> 17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
> 17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal = HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM)
> 17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server: nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal: [HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM], added = true
> 17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException: Address already in use
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> ```
>
> This occurs because the PID file cannot be accessed by the non-root agent and returns an exit code of 1:
> {code}
> call returned (1, 'cat: /var/run/livy/livy-cstm-livy-server.pid: Permission denied')
> {code}
>
> This tricks out PID detection into thinking that there is no process running (or a missing PID file) and therefore, we do not need to stop Livy/Livy2.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148ab6
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84613
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_livy2.py c4284889a1
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b91bd
>
>
> Diff: https://reviews.apache.org/r/64637/diff/1/
>
>
> Testing
> -------
>
> Manually verified PID detection on non-root cluster.
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 64637: Livy/Livy2 Unable To Start Due to Address
Already In Use
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64637/#review194016
-----------------------------------------------------------
Ship it!
Ship It!
- Dmitro Lisnichenko
On Dec. 15, 2017, 5:38 a.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64637/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2017, 5:38 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22655
> https://issues.apache.org/jira/browse/AMBARI-22655
>
>
> Repository: ambari
>
>
> Description
> -------
>
> While restarting Livy and Livy2 on a non-root cluster, the following is seen:
>
> ```
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.repl.enable-hive-context instead.
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.server.csrf-protection.enabled instead.
> 17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: cstm-zeppelin;other allowed users: *
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: ____ __
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: / __/__ ___ _____/ /__
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/ '_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /___/ .__/_,_/_/ /_/_\ version 2.2.0.2.6.4.0-73
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins on 2017-12-13T19:08:32Z
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision a24017869f5450397136ee8b11be818e7cd3facb
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url git@github.com:hortonworks/spark2.git
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more information.
> 17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
> 17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
> 17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
> 17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
> 17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal = HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM)
> 17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server: nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal: [HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM], added = true
> 17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException: Address already in use
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> ```
>
> This occurs because the PID file cannot be accessed by the non-root agent and returns an exit code of 1:
> {code}
> call returned (1, 'cat: /var/run/livy/livy-cstm-livy-server.pid: Permission denied')
> {code}
>
> This tricks out PID detection into thinking that there is no process running (or a missing PID file) and therefore, we do not need to stop Livy/Livy2.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148ab6
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84613
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_livy2.py c4284889a1
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b91bd
> ambari-server/src/test/python/stacks/2.5/SPARK/test_spark_livy.py 4f1fb17b64
> ambari-server/src/test/python/stacks/2.6/SPARK2/test_spark_livy2.py 60e7fd2a42
>
>
> Diff: https://reviews.apache.org/r/64637/diff/2/
>
>
> Testing
> -------
>
> ----------------------------------------------------------------------
> Total run:1200
> Total errors:0
> Total failures:0
> OK
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 02:37 min
> [INFO] Finished at: 2017-12-14T22:37:14-05:00
> [INFO] Final Memory: 21M/619M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 64637: Livy/Livy2 Unable To Start Due to Address
Already In Use
Posted by Dmytro Grinenko <ha...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64637/#review193904
-----------------------------------------------------------
Ship it!
Ship It!
- Dmytro Grinenko
On Dec. 15, 2017, 3:38 a.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64637/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2017, 3:38 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22655
> https://issues.apache.org/jira/browse/AMBARI-22655
>
>
> Repository: ambari
>
>
> Description
> -------
>
> While restarting Livy and Livy2 on a non-root cluster, the following is seen:
>
> ```
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.repl.enable-hive-context instead.
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.server.csrf-protection.enabled instead.
> 17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: cstm-zeppelin;other allowed users: *
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: ____ __
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: / __/__ ___ _____/ /__
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/ '_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /___/ .__/_,_/_/ /_/_\ version 2.2.0.2.6.4.0-73
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins on 2017-12-13T19:08:32Z
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision a24017869f5450397136ee8b11be818e7cd3facb
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url git@github.com:hortonworks/spark2.git
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more information.
> 17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
> 17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
> 17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
> 17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
> 17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal = HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM)
> 17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server: nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal: [HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM], added = true
> 17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException: Address already in use
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> ```
>
> This occurs because the PID file cannot be accessed by the non-root agent and returns an exit code of 1:
> {code}
> call returned (1, 'cat: /var/run/livy/livy-cstm-livy-server.pid: Permission denied')
> {code}
>
> This tricks out PID detection into thinking that there is no process running (or a missing PID file) and therefore, we do not need to stop Livy/Livy2.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148ab6
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84613
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_livy2.py c4284889a1
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b91bd
> ambari-server/src/test/python/stacks/2.5/SPARK/test_spark_livy.py 4f1fb17b64
> ambari-server/src/test/python/stacks/2.6/SPARK2/test_spark_livy2.py 60e7fd2a42
>
>
> Diff: https://reviews.apache.org/r/64637/diff/2/
>
>
> Testing
> -------
>
> ----------------------------------------------------------------------
> Total run:1200
> Total errors:0
> Total failures:0
> OK
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 02:37 min
> [INFO] Finished at: 2017-12-14T22:37:14-05:00
> [INFO] Final Memory: 21M/619M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 64637: Livy/Livy2 Unable To Start Due to Address
Already In Use
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64637/#review193918
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Dec. 14, 2017, 10:38 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64637/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2017, 10:38 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-22655
> https://issues.apache.org/jira/browse/AMBARI-22655
>
>
> Repository: ambari
>
>
> Description
> -------
>
> While restarting Livy and Livy2 on a non-root cluster, the following is seen:
>
> ```
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.repl.enable-hive-context instead.
> 17/12/14 14:36:23 WARN LivyConf: The configuration key livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.server.csrf-protection.enabled instead.
> 17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: cstm-zeppelin;other allowed users: *
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: ____ __
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: / __/__ ___ _____/ /__
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/ '_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /___/ .__/_,_/_/ /_/_\ version 2.2.0.2.6.4.0-73
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: /_/
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout:
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins on 2017-12-13T19:08:32Z
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision a24017869f5450397136ee8b11be818e7cd3facb
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url git@github.com:hortonworks/spark2.git
> 17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more information.
> 17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
> 17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
> 17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
> 17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
> 17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
> 17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal = HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM)
> 17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM
> 17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server: nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal: [HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM], added = true
> 17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException: Address already in use
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> ```
>
> This occurs because the PID file cannot be accessed by the non-root agent and returns an exit code of 1:
> {code}
> call returned (1, 'cat: /var/run/livy/livy-cstm-livy-server.pid: Permission denied')
> {code}
>
> This tricks out PID detection into thinking that there is no process running (or a missing PID file) and therefore, we do not need to stop Livy/Livy2.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148ab6
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84613
> ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_livy2.py c4284889a1
> ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b91bd
> ambari-server/src/test/python/stacks/2.5/SPARK/test_spark_livy.py 4f1fb17b64
> ambari-server/src/test/python/stacks/2.6/SPARK2/test_spark_livy2.py 60e7fd2a42
>
>
> Diff: https://reviews.apache.org/r/64637/diff/2/
>
>
> Testing
> -------
>
> ----------------------------------------------------------------------
> Total run:1200
> Total errors:0
> Total failures:0
> OK
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 02:37 min
> [INFO] Finished at: 2017-12-14T22:37:14-05:00
> [INFO] Final Memory: 21M/619M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 64637: Livy/Livy2 Unable To Start Due to Address
Already In Use
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64637/
-----------------------------------------------------------
(Updated Dec. 14, 2017, 10:38 p.m.)
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
Bugs: AMBARI-22655
https://issues.apache.org/jira/browse/AMBARI-22655
Repository: ambari
Description
-------
While restarting Livy and Livy2 on a non-root cluster, the following is seen:
```
17/12/14 14:36:23 WARN LivyConf: The configuration key livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.repl.enable-hive-context instead.
17/12/14 14:36:23 WARN LivyConf: The configuration key livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.server.csrf-protection.enabled instead.
17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: cstm-zeppelin;other allowed users: *
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
17/12/14 14:36:28 INFO LineBufferedStream: stdout: ____ __
17/12/14 14:36:28 INFO LineBufferedStream: stdout: / __/__ ___ _____/ /__
17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/ '_/
17/12/14 14:36:28 INFO LineBufferedStream: stdout: /___/ .__/_,_/_/ /_/_\ version 2.2.0.2.6.4.0-73
17/12/14 14:36:28 INFO LineBufferedStream: stdout: /_/
17/12/14 14:36:28 INFO LineBufferedStream: stdout:
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins on 2017-12-13T19:08:32Z
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision a24017869f5450397136ee8b11be818e7cd3facb
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url git@github.com:hortonworks/spark2.git
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more information.
17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0
17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive sessions. Next session id: 0
17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread started.
17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal = HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM)
17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM
17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server: nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal: [HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal@EXAMPLE.COM], added = true
17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
```
This occurs because the PID file cannot be accessed by the non-root agent and returns an exit code of 1:
{code}
call returned (1, 'cat: /var/run/livy/livy-cstm-livy-server.pid: Permission denied')
{code}
This tricks out PID detection into thinking that there is no process running (or a missing PID file) and therefore, we do not need to stop Livy/Livy2.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_service.py cce2148ab6
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_service.py dfadd84613
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_livy2.py c4284889a1
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_packages.json 62a46b91bd
ambari-server/src/test/python/stacks/2.5/SPARK/test_spark_livy.py 4f1fb17b64
ambari-server/src/test/python/stacks/2.6/SPARK2/test_spark_livy2.py 60e7fd2a42
Diff: https://reviews.apache.org/r/64637/diff/2/
Changes: https://reviews.apache.org/r/64637/diff/1-2/
Testing (updated)
-------
----------------------------------------------------------------------
Total run:1200
Total errors:0
Total failures:0
OK
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:37 min
[INFO] Finished at: 2017-12-14T22:37:14-05:00
[INFO] Final Memory: 21M/619M
[INFO] ------------------------------------------------------------------------
Thanks,
Jonathan Hurley