You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Emil Anca <ea...@hortonworks.com> on 2015/04/23 14:05:25 UTC
Review Request 33477: Issues when log/pid dirs are customized - Hive
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33477/
-----------------------------------------------------------
Review request for Ambari, Andrew Onischuk and Robert Levas.
Bugs: AMBARI-10691
https://issues.apache.org/jira/browse/AMBARI-10691
Repository: ambari
Description
-------
Problem:
I ran Ambari 1.7 to install HDP 2.2 on a VM after changing all the /var/log, /var/run, /hadoop references form defaults to instead be under some other dir. I am encountering a number of Ambari failures which I needed help on.
After the install wizard error out I also noticed:
HS2 does not stay up. From the logs I see below error
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hive, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
Solution:
When hive uses a custom hive.exec.scratchdir path on the Hadoop Filesystem, it automatically tries to create said path at start and it will fail to do so as it does not have the required permissions.
Ambari will create said custom folder if needed and set the appropiate permissions.
Diffs
-----
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py f7266f1
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py b93d056
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ecba234
ambari-server/src/test/python/stacks/2.0.6/configs/default.json 90cae08
ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 6cce47c
Diff: https://reviews.apache.org/r/33477/diff/
Testing
-------
mvn clean test
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [6.912s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.116s]
[INFO] Ambari Web ........................................ SUCCESS [29.655s]
[INFO] Ambari Views ...................................... SUCCESS [1.196s]
[INFO] Ambari Admin View ................................. SUCCESS [19.873s]
[INFO] Ambari Metrics Common ............................. SUCCESS [1.116s]
[INFO] Ambari Server ..................................... SUCCESS [2:39.489s]
[INFO] Ambari Agent ...................................... SUCCESS [9.179s]
[INFO] Ambari Client ..................................... SUCCESS [0.202s]
[INFO] Ambari Python Client .............................. SUCCESS [1.121s]
[INFO] Ambari Groovy Client .............................. SUCCESS [2.917s]
[INFO] Ambari Shell ...................................... SUCCESS [0.108s]
[INFO] Ambari Python Shell ............................... SUCCESS [0.637s]
[INFO] Ambari Groovy Shell ............................... SUCCESS [1.760s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
Thanks,
Emil Anca
Re: Review Request 33477: Issues when log/pid dirs are customized -
Hive
Posted by Emil Anca <ea...@hortonworks.com>.
> On April 23, 2015, 1:15 p.m., Robert Levas wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py, line 270
> > <https://reviews.apache.org/r/33477/diff/1/?file=940346#file940346line270>
> >
> > Where is `/configurations/hive-site/hive.exec.scratchdir` comming from? Shouldn't there be a property set up for it in common-services/HIVE/0.12.0.2.0/configuration/hive-site.xml?
>
> Andrew Onischuk wrote:
> Seems like it's in hive starting from Hive 2.2. For Hive on previous versions it will cause exception. We should fix this as well.
>
> Emil, please add change like this:
>
> if not params.hive_exec_scratchdir.startswith("/tmp"):
> ->
> is not is_empty(hive_exec_scratchdir) and not params.hive_exec_scratchdir.startswith("/tmp"):
Good point Rob; Applied Andrew's suggestion and also changed the hive.exec.scratchdir asignation to a safer approach just to be more consitent as it might not always be set.
- Emil
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33477/#review81309
-----------------------------------------------------------
On April 24, 2015, 11:15 a.m., Emil Anca wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33477/
> -----------------------------------------------------------
>
> (Updated April 24, 2015, 11:15 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk and Robert Levas.
>
>
> Bugs: AMBARI-10691
> https://issues.apache.org/jira/browse/AMBARI-10691
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Problem:
>
> I ran Ambari 1.7 to install HDP 2.2 on a VM after changing all the /var/log, /var/run, /hadoop references form defaults to instead be under some other dir. I am encountering a number of Ambari failures which I needed help on.
> After the install wizard error out I also noticed:
> HS2 does not stay up. From the logs I see below error
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hive, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
>
> Solution:
>
> When hive uses a custom hive.exec.scratchdir path on the Hadoop Filesystem, it automatically tries to create said path at start and it will fail to do so as it does not have the required permissions.
> Ambari will create said custom folder if needed and set the appropiate permissions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py f7266f1
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py b93d056
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ecba234
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json 5e4a2d1
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 6796061
>
> Diff: https://reviews.apache.org/r/33477/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [6.912s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.116s]
> [INFO] Ambari Web ........................................ SUCCESS [29.655s]
> [INFO] Ambari Views ...................................... SUCCESS [1.196s]
> [INFO] Ambari Admin View ................................. SUCCESS [19.873s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [1.116s]
> [INFO] Ambari Server ..................................... SUCCESS [2:39.489s]
> [INFO] Ambari Agent ...................................... SUCCESS [9.179s]
> [INFO] Ambari Client ..................................... SUCCESS [0.202s]
> [INFO] Ambari Python Client .............................. SUCCESS [1.121s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [2.917s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.108s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.637s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [1.760s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
>
>
> File Attachments
> ----------------
>
> AMBARI-10691_02.patch
> https://reviews.apache.org/media/uploaded/files/2015/04/24/9277fae5-ef8b-4154-8a24-cb9a42e73e3f__AMBARI-10691_02.patch
>
>
> Thanks,
>
> Emil Anca
>
>
Re: Review Request 33477: Issues when log/pid dirs are customized -
Hive
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On April 23, 2015, 1:15 p.m., Robert Levas wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py, line 270
> > <https://reviews.apache.org/r/33477/diff/1/?file=940346#file940346line270>
> >
> > Where is `/configurations/hive-site/hive.exec.scratchdir` comming from? Shouldn't there be a property set up for it in common-services/HIVE/0.12.0.2.0/configuration/hive-site.xml?
Seems like it's in hive starting from Hive 2.2. For Hive on previous versions it will cause exception. We should fix this as well.
Emil, please add change like this:
if not params.hive_exec_scratchdir.startswith("/tmp"):
->
is not is_empty(hive_exec_scratchdir) and not params.hive_exec_scratchdir.startswith("/tmp"):
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33477/#review81309
-----------------------------------------------------------
On April 23, 2015, 12:05 p.m., Emil Anca wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33477/
> -----------------------------------------------------------
>
> (Updated April 23, 2015, 12:05 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk and Robert Levas.
>
>
> Bugs: AMBARI-10691
> https://issues.apache.org/jira/browse/AMBARI-10691
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Problem:
>
> I ran Ambari 1.7 to install HDP 2.2 on a VM after changing all the /var/log, /var/run, /hadoop references form defaults to instead be under some other dir. I am encountering a number of Ambari failures which I needed help on.
> After the install wizard error out I also noticed:
> HS2 does not stay up. From the logs I see below error
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hive, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
>
> Solution:
>
> When hive uses a custom hive.exec.scratchdir path on the Hadoop Filesystem, it automatically tries to create said path at start and it will fail to do so as it does not have the required permissions.
> Ambari will create said custom folder if needed and set the appropiate permissions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py f7266f1
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py b93d056
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ecba234
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json 90cae08
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 6cce47c
>
> Diff: https://reviews.apache.org/r/33477/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [6.912s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.116s]
> [INFO] Ambari Web ........................................ SUCCESS [29.655s]
> [INFO] Ambari Views ...................................... SUCCESS [1.196s]
> [INFO] Ambari Admin View ................................. SUCCESS [19.873s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [1.116s]
> [INFO] Ambari Server ..................................... SUCCESS [2:39.489s]
> [INFO] Ambari Agent ...................................... SUCCESS [9.179s]
> [INFO] Ambari Client ..................................... SUCCESS [0.202s]
> [INFO] Ambari Python Client .............................. SUCCESS [1.121s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [2.917s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.108s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.637s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [1.760s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Emil Anca
>
>
Re: Review Request 33477: Issues when log/pid dirs are customized -
Hive
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33477/#review81309
-----------------------------------------------------------
Ship it!
Other than Andrew's concerns and my one potential issue, looks good.
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
<https://reviews.apache.org/r/33477/#comment131636>
Where is `/configurations/hive-site/hive.exec.scratchdir` comming from? Shouldn't there be a property set up for it in common-services/HIVE/0.12.0.2.0/configuration/hive-site.xml?
- Robert Levas
On April 23, 2015, 8:05 a.m., Emil Anca wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33477/
> -----------------------------------------------------------
>
> (Updated April 23, 2015, 8:05 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk and Robert Levas.
>
>
> Bugs: AMBARI-10691
> https://issues.apache.org/jira/browse/AMBARI-10691
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Problem:
>
> I ran Ambari 1.7 to install HDP 2.2 on a VM after changing all the /var/log, /var/run, /hadoop references form defaults to instead be under some other dir. I am encountering a number of Ambari failures which I needed help on.
> After the install wizard error out I also noticed:
> HS2 does not stay up. From the logs I see below error
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hive, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
>
> Solution:
>
> When hive uses a custom hive.exec.scratchdir path on the Hadoop Filesystem, it automatically tries to create said path at start and it will fail to do so as it does not have the required permissions.
> Ambari will create said custom folder if needed and set the appropiate permissions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py f7266f1
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py b93d056
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ecba234
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json 90cae08
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 6cce47c
>
> Diff: https://reviews.apache.org/r/33477/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [6.912s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.116s]
> [INFO] Ambari Web ........................................ SUCCESS [29.655s]
> [INFO] Ambari Views ...................................... SUCCESS [1.196s]
> [INFO] Ambari Admin View ................................. SUCCESS [19.873s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [1.116s]
> [INFO] Ambari Server ..................................... SUCCESS [2:39.489s]
> [INFO] Ambari Agent ...................................... SUCCESS [9.179s]
> [INFO] Ambari Client ..................................... SUCCESS [0.202s]
> [INFO] Ambari Python Client .............................. SUCCESS [1.121s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [2.917s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.108s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.637s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [1.760s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Emil Anca
>
>
Re: Review Request 33477: Issues when log/pid dirs are customized -
Hive
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33477/#review81302
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py
<https://reviews.apache.org/r/33477/#comment131625>
Hey Emil it should be better to use HdfsDirectory Resource which contains all that functionality.
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py
<https://reviews.apache.org/r/33477/#comment131626>
Why do we mock this function. Let's include it in testcase.
- Andrew Onischuk
On April 23, 2015, 12:05 p.m., Emil Anca wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33477/
> -----------------------------------------------------------
>
> (Updated April 23, 2015, 12:05 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk and Robert Levas.
>
>
> Bugs: AMBARI-10691
> https://issues.apache.org/jira/browse/AMBARI-10691
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Problem:
>
> I ran Ambari 1.7 to install HDP 2.2 on a VM after changing all the /var/log, /var/run, /hadoop references form defaults to instead be under some other dir. I am encountering a number of Ambari failures which I needed help on.
> After the install wizard error out I also noticed:
> HS2 does not stay up. From the logs I see below error
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hive, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
>
> Solution:
>
> When hive uses a custom hive.exec.scratchdir path on the Hadoop Filesystem, it automatically tries to create said path at start and it will fail to do so as it does not have the required permissions.
> Ambari will create said custom folder if needed and set the appropiate permissions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py f7266f1
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py b93d056
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ecba234
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json 90cae08
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 6cce47c
>
> Diff: https://reviews.apache.org/r/33477/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [6.912s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.116s]
> [INFO] Ambari Web ........................................ SUCCESS [29.655s]
> [INFO] Ambari Views ...................................... SUCCESS [1.196s]
> [INFO] Ambari Admin View ................................. SUCCESS [19.873s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [1.116s]
> [INFO] Ambari Server ..................................... SUCCESS [2:39.489s]
> [INFO] Ambari Agent ...................................... SUCCESS [9.179s]
> [INFO] Ambari Client ..................................... SUCCESS [0.202s]
> [INFO] Ambari Python Client .............................. SUCCESS [1.121s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [2.917s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.108s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.637s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [1.760s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Emil Anca
>
>
Re: Review Request 33477: Issues when log/pid dirs are customized -
Hive
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33477/#review81303
-----------------------------------------------------------
ambari-server/src/test/python/stacks/2.0.6/configs/default.json
<https://reviews.apache.org/r/33477/#comment131627>
Also for testcase let's make it not to start with /tmp
- Andrew Onischuk
On April 23, 2015, 12:05 p.m., Emil Anca wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33477/
> -----------------------------------------------------------
>
> (Updated April 23, 2015, 12:05 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk and Robert Levas.
>
>
> Bugs: AMBARI-10691
> https://issues.apache.org/jira/browse/AMBARI-10691
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Problem:
>
> I ran Ambari 1.7 to install HDP 2.2 on a VM after changing all the /var/log, /var/run, /hadoop references form defaults to instead be under some other dir. I am encountering a number of Ambari failures which I needed help on.
> After the install wizard error out I also noticed:
> HS2 does not stay up. From the logs I see below error
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hive, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
>
> Solution:
>
> When hive uses a custom hive.exec.scratchdir path on the Hadoop Filesystem, it automatically tries to create said path at start and it will fail to do so as it does not have the required permissions.
> Ambari will create said custom folder if needed and set the appropiate permissions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py f7266f1
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py b93d056
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ecba234
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json 90cae08
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 6cce47c
>
> Diff: https://reviews.apache.org/r/33477/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [6.912s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.116s]
> [INFO] Ambari Web ........................................ SUCCESS [29.655s]
> [INFO] Ambari Views ...................................... SUCCESS [1.196s]
> [INFO] Ambari Admin View ................................. SUCCESS [19.873s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [1.116s]
> [INFO] Ambari Server ..................................... SUCCESS [2:39.489s]
> [INFO] Ambari Agent ...................................... SUCCESS [9.179s]
> [INFO] Ambari Client ..................................... SUCCESS [0.202s]
> [INFO] Ambari Python Client .............................. SUCCESS [1.121s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [2.917s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.108s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.637s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [1.760s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Emil Anca
>
>
Re: Review Request 33477: Issues when log/pid dirs are customized -
Hive
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On April 24, 2015, 2:56 p.m., Andrew Onischuk wrote:
> > Ship It!
seems you removed that mock, nice :)
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33477/#review81482
-----------------------------------------------------------
On April 24, 2015, 2:54 p.m., Emil Anca wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33477/
> -----------------------------------------------------------
>
> (Updated April 24, 2015, 2:54 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk and Robert Levas.
>
>
> Bugs: AMBARI-10691
> https://issues.apache.org/jira/browse/AMBARI-10691
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Problem:
>
> I ran Ambari 1.7 to install HDP 2.2 on a VM after changing all the /var/log, /var/run, /hadoop references form defaults to instead be under some other dir. I am encountering a number of Ambari failures which I needed help on.
> After the install wizard error out I also noticed:
> HS2 does not stay up. From the logs I see below error
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hive, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
>
> Solution:
>
> When hive uses a custom hive.exec.scratchdir path on the Hadoop Filesystem, it automatically tries to create said path at start and it will fail to do so as it does not have the required permissions.
> Ambari will create said custom folder if needed and set the appropiate permissions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py a352797
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py f7266f1
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py b93d056
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ecba234
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json 5e4a2d1
> ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json b1be0ff
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 6796061
>
> Diff: https://reviews.apache.org/r/33477/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [6.912s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.116s]
> [INFO] Ambari Web ........................................ SUCCESS [29.655s]
> [INFO] Ambari Views ...................................... SUCCESS [1.196s]
> [INFO] Ambari Admin View ................................. SUCCESS [19.873s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [1.116s]
> [INFO] Ambari Server ..................................... SUCCESS [2:39.489s]
> [INFO] Ambari Agent ...................................... SUCCESS [9.179s]
> [INFO] Ambari Client ..................................... SUCCESS [0.202s]
> [INFO] Ambari Python Client .............................. SUCCESS [1.121s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [2.917s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.108s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.637s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [1.760s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
>
>
> File Attachments
> ----------------
>
> AMBARI-10691_02.patch
> https://reviews.apache.org/media/uploaded/files/2015/04/24/9277fae5-ef8b-4154-8a24-cb9a42e73e3f__AMBARI-10691_02.patch
> AMBARI-10691_03.patch
> https://reviews.apache.org/media/uploaded/files/2015/04/24/41b65b78-913c-426e-bb94-934c4902e5a6__AMBARI-10691_03.patch
>
>
> Thanks,
>
> Emil Anca
>
>
Re: Review Request 33477: Issues when log/pid dirs are customized -
Hive
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33477/#review81482
-----------------------------------------------------------
Ship it!
Ship It!
- Andrew Onischuk
On April 24, 2015, 2:54 p.m., Emil Anca wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33477/
> -----------------------------------------------------------
>
> (Updated April 24, 2015, 2:54 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk and Robert Levas.
>
>
> Bugs: AMBARI-10691
> https://issues.apache.org/jira/browse/AMBARI-10691
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Problem:
>
> I ran Ambari 1.7 to install HDP 2.2 on a VM after changing all the /var/log, /var/run, /hadoop references form defaults to instead be under some other dir. I am encountering a number of Ambari failures which I needed help on.
> After the install wizard error out I also noticed:
> HS2 does not stay up. From the logs I see below error
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hive, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
>
> Solution:
>
> When hive uses a custom hive.exec.scratchdir path on the Hadoop Filesystem, it automatically tries to create said path at start and it will fail to do so as it does not have the required permissions.
> Ambari will create said custom folder if needed and set the appropiate permissions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py a352797
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py f7266f1
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py b93d056
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ecba234
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json 5e4a2d1
> ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json b1be0ff
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 6796061
>
> Diff: https://reviews.apache.org/r/33477/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [6.912s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.116s]
> [INFO] Ambari Web ........................................ SUCCESS [29.655s]
> [INFO] Ambari Views ...................................... SUCCESS [1.196s]
> [INFO] Ambari Admin View ................................. SUCCESS [19.873s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [1.116s]
> [INFO] Ambari Server ..................................... SUCCESS [2:39.489s]
> [INFO] Ambari Agent ...................................... SUCCESS [9.179s]
> [INFO] Ambari Client ..................................... SUCCESS [0.202s]
> [INFO] Ambari Python Client .............................. SUCCESS [1.121s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [2.917s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.108s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.637s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [1.760s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
>
>
> File Attachments
> ----------------
>
> AMBARI-10691_02.patch
> https://reviews.apache.org/media/uploaded/files/2015/04/24/9277fae5-ef8b-4154-8a24-cb9a42e73e3f__AMBARI-10691_02.patch
> AMBARI-10691_03.patch
> https://reviews.apache.org/media/uploaded/files/2015/04/24/41b65b78-913c-426e-bb94-934c4902e5a6__AMBARI-10691_03.patch
>
>
> Thanks,
>
> Emil Anca
>
>
Re: Review Request 33477: Issues when log/pid dirs are customized -
Hive
Posted by Emil Anca <ea...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33477/
-----------------------------------------------------------
(Updated April 24, 2015, 2:54 p.m.)
Review request for Ambari, Andrew Onischuk and Robert Levas.
Bugs: AMBARI-10691
https://issues.apache.org/jira/browse/AMBARI-10691
Repository: ambari
Description
-------
Problem:
I ran Ambari 1.7 to install HDP 2.2 on a VM after changing all the /var/log, /var/run, /hadoop references form defaults to instead be under some other dir. I am encountering a number of Ambari failures which I needed help on.
After the install wizard error out I also noticed:
HS2 does not stay up. From the logs I see below error
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hive, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
Solution:
When hive uses a custom hive.exec.scratchdir path on the Hadoop Filesystem, it automatically tries to create said path at start and it will fail to do so as it does not have the required permissions.
Ambari will create said custom folder if needed and set the appropiate permissions.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py a352797
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py f7266f1
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py b93d056
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ecba234
ambari-server/src/test/python/stacks/2.0.6/configs/default.json 5e4a2d1
ambari-server/src/test/python/stacks/2.0.6/configs/default_no_install.json b1be0ff
ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 6796061
Diff: https://reviews.apache.org/r/33477/diff/
Testing
-------
mvn clean test
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [6.912s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.116s]
[INFO] Ambari Web ........................................ SUCCESS [29.655s]
[INFO] Ambari Views ...................................... SUCCESS [1.196s]
[INFO] Ambari Admin View ................................. SUCCESS [19.873s]
[INFO] Ambari Metrics Common ............................. SUCCESS [1.116s]
[INFO] Ambari Server ..................................... SUCCESS [2:39.489s]
[INFO] Ambari Agent ...................................... SUCCESS [9.179s]
[INFO] Ambari Client ..................................... SUCCESS [0.202s]
[INFO] Ambari Python Client .............................. SUCCESS [1.121s]
[INFO] Ambari Groovy Client .............................. SUCCESS [2.917s]
[INFO] Ambari Shell ...................................... SUCCESS [0.108s]
[INFO] Ambari Python Shell ............................... SUCCESS [0.637s]
[INFO] Ambari Groovy Shell ............................... SUCCESS [1.760s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
File Attachments
----------------
AMBARI-10691_02.patch
https://reviews.apache.org/media/uploaded/files/2015/04/24/9277fae5-ef8b-4154-8a24-cb9a42e73e3f__AMBARI-10691_02.patch
AMBARI-10691_03.patch
https://reviews.apache.org/media/uploaded/files/2015/04/24/41b65b78-913c-426e-bb94-934c4902e5a6__AMBARI-10691_03.patch
Thanks,
Emil Anca
Re: Review Request 33477: Issues when log/pid dirs are customized -
Hive
Posted by Emil Anca <ea...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33477/
-----------------------------------------------------------
(Updated April 24, 2015, 12:02 p.m.)
Review request for Ambari, Andrew Onischuk and Robert Levas.
Bugs: AMBARI-10691
https://issues.apache.org/jira/browse/AMBARI-10691
Repository: ambari
Description
-------
Problem:
I ran Ambari 1.7 to install HDP 2.2 on a VM after changing all the /var/log, /var/run, /hadoop references form defaults to instead be under some other dir. I am encountering a number of Ambari failures which I needed help on.
After the install wizard error out I also noticed:
HS2 does not stay up. From the logs I see below error
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hive, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
Solution:
When hive uses a custom hive.exec.scratchdir path on the Hadoop Filesystem, it automatically tries to create said path at start and it will fail to do so as it does not have the required permissions.
Ambari will create said custom folder if needed and set the appropiate permissions.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py f7266f1
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py b93d056
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ecba234
ambari-server/src/test/python/stacks/2.0.6/configs/default.json 5e4a2d1
ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 6796061
Diff: https://reviews.apache.org/r/33477/diff/
Testing
-------
mvn clean test
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [6.912s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.116s]
[INFO] Ambari Web ........................................ SUCCESS [29.655s]
[INFO] Ambari Views ...................................... SUCCESS [1.196s]
[INFO] Ambari Admin View ................................. SUCCESS [19.873s]
[INFO] Ambari Metrics Common ............................. SUCCESS [1.116s]
[INFO] Ambari Server ..................................... SUCCESS [2:39.489s]
[INFO] Ambari Agent ...................................... SUCCESS [9.179s]
[INFO] Ambari Client ..................................... SUCCESS [0.202s]
[INFO] Ambari Python Client .............................. SUCCESS [1.121s]
[INFO] Ambari Groovy Client .............................. SUCCESS [2.917s]
[INFO] Ambari Shell ...................................... SUCCESS [0.108s]
[INFO] Ambari Python Shell ............................... SUCCESS [0.637s]
[INFO] Ambari Groovy Shell ............................... SUCCESS [1.760s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
File Attachments (updated)
----------------
AMBARI-10691_02.patch
https://reviews.apache.org/media/uploaded/files/2015/04/24/9277fae5-ef8b-4154-8a24-cb9a42e73e3f__AMBARI-10691_02.patch
AMBARI-10691_03.patch
https://reviews.apache.org/media/uploaded/files/2015/04/24/41b65b78-913c-426e-bb94-934c4902e5a6__AMBARI-10691_03.patch
Thanks,
Emil Anca
Re: Review Request 33477: Issues when log/pid dirs are customized -
Hive
Posted by Emil Anca <ea...@hortonworks.com>.
> On April 24, 2015, 11:50 a.m., Andrew Onischuk wrote:
> > ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py, line 97
> > <https://reviews.apache.org/r/33477/diff/2/?file=940935#file940935line97>
> >
> > Seems like you forgot to remove this one mock
There are already 2 tests in place that go through the scratchdir functionality, for this test I left the mock there as it does not need another test which would only add time on the test execution.
- Emil
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33477/#review81467
-----------------------------------------------------------
On April 24, 2015, 11:15 a.m., Emil Anca wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33477/
> -----------------------------------------------------------
>
> (Updated April 24, 2015, 11:15 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk and Robert Levas.
>
>
> Bugs: AMBARI-10691
> https://issues.apache.org/jira/browse/AMBARI-10691
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Problem:
>
> I ran Ambari 1.7 to install HDP 2.2 on a VM after changing all the /var/log, /var/run, /hadoop references form defaults to instead be under some other dir. I am encountering a number of Ambari failures which I needed help on.
> After the install wizard error out I also noticed:
> HS2 does not stay up. From the logs I see below error
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hive, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
>
> Solution:
>
> When hive uses a custom hive.exec.scratchdir path on the Hadoop Filesystem, it automatically tries to create said path at start and it will fail to do so as it does not have the required permissions.
> Ambari will create said custom folder if needed and set the appropiate permissions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py f7266f1
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py b93d056
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ecba234
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json 5e4a2d1
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 6796061
>
> Diff: https://reviews.apache.org/r/33477/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [6.912s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.116s]
> [INFO] Ambari Web ........................................ SUCCESS [29.655s]
> [INFO] Ambari Views ...................................... SUCCESS [1.196s]
> [INFO] Ambari Admin View ................................. SUCCESS [19.873s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [1.116s]
> [INFO] Ambari Server ..................................... SUCCESS [2:39.489s]
> [INFO] Ambari Agent ...................................... SUCCESS [9.179s]
> [INFO] Ambari Client ..................................... SUCCESS [0.202s]
> [INFO] Ambari Python Client .............................. SUCCESS [1.121s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [2.917s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.108s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.637s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [1.760s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
>
>
> File Attachments
> ----------------
>
> AMBARI-10691_02.patch
> https://reviews.apache.org/media/uploaded/files/2015/04/24/9277fae5-ef8b-4154-8a24-cb9a42e73e3f__AMBARI-10691_02.patch
>
>
> Thanks,
>
> Emil Anca
>
>
Re: Review Request 33477: Issues when log/pid dirs are customized -
Hive
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33477/#review81467
-----------------------------------------------------------
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py
<https://reviews.apache.org/r/33477/#comment131852>
Seems like you forgot to remove this one mock
- Andrew Onischuk
On April 24, 2015, 11:15 a.m., Emil Anca wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33477/
> -----------------------------------------------------------
>
> (Updated April 24, 2015, 11:15 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk and Robert Levas.
>
>
> Bugs: AMBARI-10691
> https://issues.apache.org/jira/browse/AMBARI-10691
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Problem:
>
> I ran Ambari 1.7 to install HDP 2.2 on a VM after changing all the /var/log, /var/run, /hadoop references form defaults to instead be under some other dir. I am encountering a number of Ambari failures which I needed help on.
> After the install wizard error out I also noticed:
> HS2 does not stay up. From the logs I see below error
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hive, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
>
> Solution:
>
> When hive uses a custom hive.exec.scratchdir path on the Hadoop Filesystem, it automatically tries to create said path at start and it will fail to do so as it does not have the required permissions.
> Ambari will create said custom folder if needed and set the appropiate permissions.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py f7266f1
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py b93d056
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ecba234
> ambari-server/src/test/python/stacks/2.0.6/configs/default.json 5e4a2d1
> ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 6796061
>
> Diff: https://reviews.apache.org/r/33477/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ....................................... SUCCESS [6.912s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.116s]
> [INFO] Ambari Web ........................................ SUCCESS [29.655s]
> [INFO] Ambari Views ...................................... SUCCESS [1.196s]
> [INFO] Ambari Admin View ................................. SUCCESS [19.873s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [1.116s]
> [INFO] Ambari Server ..................................... SUCCESS [2:39.489s]
> [INFO] Ambari Agent ...................................... SUCCESS [9.179s]
> [INFO] Ambari Client ..................................... SUCCESS [0.202s]
> [INFO] Ambari Python Client .............................. SUCCESS [1.121s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [2.917s]
> [INFO] Ambari Shell ...................................... SUCCESS [0.108s]
> [INFO] Ambari Python Shell ............................... SUCCESS [0.637s]
> [INFO] Ambari Groovy Shell ............................... SUCCESS [1.760s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
>
>
> File Attachments
> ----------------
>
> AMBARI-10691_02.patch
> https://reviews.apache.org/media/uploaded/files/2015/04/24/9277fae5-ef8b-4154-8a24-cb9a42e73e3f__AMBARI-10691_02.patch
>
>
> Thanks,
>
> Emil Anca
>
>
Re: Review Request 33477: Issues when log/pid dirs are customized -
Hive
Posted by Emil Anca <ea...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33477/
-----------------------------------------------------------
(Updated April 24, 2015, 11:15 a.m.)
Review request for Ambari, Andrew Onischuk and Robert Levas.
Bugs: AMBARI-10691
https://issues.apache.org/jira/browse/AMBARI-10691
Repository: ambari
Description
-------
Problem:
I ran Ambari 1.7 to install HDP 2.2 on a VM after changing all the /var/log, /var/run, /hadoop references form defaults to instead be under some other dir. I am encountering a number of Ambari failures which I needed help on.
After the install wizard error out I also noticed:
HS2 does not stay up. From the logs I see below error
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hive, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
Solution:
When hive uses a custom hive.exec.scratchdir path on the Hadoop Filesystem, it automatically tries to create said path at start and it will fail to do so as it does not have the required permissions.
Ambari will create said custom folder if needed and set the appropiate permissions.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py f7266f1
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py b93d056
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ecba234
ambari-server/src/test/python/stacks/2.0.6/configs/default.json 5e4a2d1
ambari-server/src/test/python/stacks/2.0.6/configs/secured.json 6796061
Diff: https://reviews.apache.org/r/33477/diff/
Testing
-------
mvn clean test
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [6.912s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.116s]
[INFO] Ambari Web ........................................ SUCCESS [29.655s]
[INFO] Ambari Views ...................................... SUCCESS [1.196s]
[INFO] Ambari Admin View ................................. SUCCESS [19.873s]
[INFO] Ambari Metrics Common ............................. SUCCESS [1.116s]
[INFO] Ambari Server ..................................... SUCCESS [2:39.489s]
[INFO] Ambari Agent ...................................... SUCCESS [9.179s]
[INFO] Ambari Client ..................................... SUCCESS [0.202s]
[INFO] Ambari Python Client .............................. SUCCESS [1.121s]
[INFO] Ambari Groovy Client .............................. SUCCESS [2.917s]
[INFO] Ambari Shell ...................................... SUCCESS [0.108s]
[INFO] Ambari Python Shell ............................... SUCCESS [0.637s]
[INFO] Ambari Groovy Shell ............................... SUCCESS [1.760s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
File Attachments (updated)
----------------
AMBARI-10691_02.patch
https://reviews.apache.org/media/uploaded/files/2015/04/24/9277fae5-ef8b-4154-8a24-cb9a42e73e3f__AMBARI-10691_02.patch
Thanks,
Emil Anca