You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Jonathan Maron <jm...@hortonworks.com> on 2015/06/01 16:59:58 UTC
Re: Review Request 34864: Add installation of atlas hive hook
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34864/
-----------------------------------------------------------
(Updated June 1, 2015, 2:59 p.m.)
Review request for Ambari, Arpit Gupta and Sumit Mohanty.
Changes
-------
Some code clean up and the Ambari unit tests now pass fully.
Bugs: AMBARI-11579
https://issues.apache.org/jira/browse/AMBARI-11579
Repository: ambari
Description
-------
This patch include the modifications to hive definition enabling the installation and configuration of the atlas hive plugin. It:
- adds 3 properties to hive-site
- adds an optional classpath modification to hive-env.sh (only necessary if the hook is installed).
- enables the installation of the atlas hook if atlas is added as a service to a cluster and hive is restarted. Had to write code to modify the command config to add the package list containing the hook to the hostLevelParams for a RESTART.
I'd like the code for the latter reviewed carefully to ensure no issues.
I also need help with the following:
- can the property modification remaing in params_linux.py, or should that code be moved to params.py to enable the same property changes across both windows and linux? Atlas isn't currently in the windows MSI (I believe)
- I can't seem to come up with the appropriate modification to mock the new code (atlas_plugin_utils.configure_for_plugin) in test_hive_metastore. I am able to do so for other hive component tests, by the hive metastore tests fail with the following:
Traceback (most recent call last):
File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1191, in patched
arg = patching.__enter__()
File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1250, in __enter__
self.target = self.getter()
File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1412, in <lambda>
getter = lambda: _importer(target)
File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1096, in _importer
thing = __import__(import_path)
ImportError: No module named atlas_plugin_utils
I have yet to come up with a way to make the test code aware of the new module I created in these tests.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hcat-env.xml 91b402b
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/atlas_plugin_utils.py PRE-CREATION
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 79096e4
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py cf08e1c
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py 7ae738a
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py f817c68
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py 0d13bea
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py ce62309
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_server.py 012d1e2
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/configuration/hive-env.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml 58627ea
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 2f6bd40
Diff: https://reviews.apache.org/r/34864/diff/
Testing
-------
Functional testing with a local VM spawned cluster
Ambari unit tests.
Thanks,
Jonathan Maron
Re: Review Request 34864: Add installation of atlas hive hook
Posted by Jonathan Maron <jm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34864/#review86105
-----------------------------------------------------------
I performed the following tests on a local VM cluster:
1) Installed both Hive and Atlas
2) Installed Hive then added Atlas, restarted Hive
3) Installed Atlas then added Hive
All passed successfully.
- Jonathan Maron
On June 1, 2015, 9:27 p.m., Jonathan Maron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34864/
> -----------------------------------------------------------
>
> (Updated June 1, 2015, 9:27 p.m.)
>
>
> Review request for Ambari, Arpit Gupta and Sumit Mohanty.
>
>
> Bugs: AMBARI-11579
> https://issues.apache.org/jira/browse/AMBARI-11579
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch include the modifications to hive definition enabling the installation and configuration of the atlas hive plugin. It:
>
> - adds 3 properties to hive-site
> - adds an optional classpath modification to hive-env.sh (only necessary if the hook is installed).
> - enables the installation of the atlas hook if atlas is added as a service to a cluster and hive is restarted. Had to write code to modify the command config to add the package list containing the hook to the hostLevelParams for a RESTART.
>
> I'd like the code for the latter reviewed carefully to ensure no issues.
>
> I also need help with the following:
> - can the property modification remaing in params_linux.py, or should that code be moved to params.py to enable the same property changes across both windows and linux? Atlas isn't currently in the windows MSI (I believe)
> - I can't seem to come up with the appropriate modification to mock the new code (atlas_plugin_utils.configure_for_plugin) in test_hive_metastore. I am able to do so for other hive component tests, by the hive metastore tests fail with the following:
>
> Traceback (most recent call last):
> File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1191, in patched
> arg = patching.__enter__()
> File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1250, in __enter__
> self.target = self.getter()
> File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1412, in <lambda>
> getter = lambda: _importer(target)
> File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1096, in _importer
> thing = __import__(import_path)
> ImportError: No module named atlas_plugin_utils
>
> I have yet to come up with a way to make the test code aware of the new module I created in these tests.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/atlas_plugin_utils.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 79096e4
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py cf08e1c
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py 7ae738a
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py f817c68
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py 0d13bea
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py ce62309
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_server.py 012d1e2
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml 58627ea
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 2f6bd40
>
> Diff: https://reviews.apache.org/r/34864/diff/
>
>
> Testing
> -------
>
> Functional testing with a local VM spawned cluster
> Ambari unit tests.
>
>
> Thanks,
>
> Jonathan Maron
>
>
Re: Review Request 34864: Add installation of atlas hive hook
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34864/#review86123
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On June 1, 2015, 9:27 p.m., Jonathan Maron wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34864/
> -----------------------------------------------------------
>
> (Updated June 1, 2015, 9:27 p.m.)
>
>
> Review request for Ambari, Arpit Gupta and Sumit Mohanty.
>
>
> Bugs: AMBARI-11579
> https://issues.apache.org/jira/browse/AMBARI-11579
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch include the modifications to hive definition enabling the installation and configuration of the atlas hive plugin. It:
>
> - adds 3 properties to hive-site
> - adds an optional classpath modification to hive-env.sh (only necessary if the hook is installed).
> - enables the installation of the atlas hook if atlas is added as a service to a cluster and hive is restarted. Had to write code to modify the command config to add the package list containing the hook to the hostLevelParams for a RESTART.
>
> I'd like the code for the latter reviewed carefully to ensure no issues.
>
> I also need help with the following:
> - can the property modification remaing in params_linux.py, or should that code be moved to params.py to enable the same property changes across both windows and linux? Atlas isn't currently in the windows MSI (I believe)
> - I can't seem to come up with the appropriate modification to mock the new code (atlas_plugin_utils.configure_for_plugin) in test_hive_metastore. I am able to do so for other hive component tests, by the hive metastore tests fail with the following:
>
> Traceback (most recent call last):
> File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1191, in patched
> arg = patching.__enter__()
> File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1250, in __enter__
> self.target = self.getter()
> File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1412, in <lambda>
> getter = lambda: _importer(target)
> File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1096, in _importer
> thing = __import__(import_path)
> ImportError: No module named atlas_plugin_utils
>
> I have yet to come up with a way to make the test code aware of the new module I created in these tests.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/atlas_plugin_utils.py PRE-CREATION
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 79096e4
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py cf08e1c
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py 7ae738a
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py f817c68
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py 0d13bea
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py ce62309
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_server.py 012d1e2
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml 58627ea
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 2f6bd40
>
> Diff: https://reviews.apache.org/r/34864/diff/
>
>
> Testing
> -------
>
> Functional testing with a local VM spawned cluster
> Ambari unit tests.
>
>
> Thanks,
>
> Jonathan Maron
>
>
Re: Review Request 34864: Add installation of atlas hive hook
Posted by Jonathan Maron <jm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34864/
-----------------------------------------------------------
(Updated June 1, 2015, 9:27 p.m.)
Review request for Ambari, Arpit Gupta and Sumit Mohanty.
Changes
-------
Modified to accomodate recent changes to hive/hcat shell scripts.
Still looking ofr direction on whether the property additions belong in params_linux.py or elsewhere (param.py?)
Bugs: AMBARI-11579
https://issues.apache.org/jira/browse/AMBARI-11579
Repository: ambari
Description
-------
This patch include the modifications to hive definition enabling the installation and configuration of the atlas hive plugin. It:
- adds 3 properties to hive-site
- adds an optional classpath modification to hive-env.sh (only necessary if the hook is installed).
- enables the installation of the atlas hook if atlas is added as a service to a cluster and hive is restarted. Had to write code to modify the command config to add the package list containing the hook to the hostLevelParams for a RESTART.
I'd like the code for the latter reviewed carefully to ensure no issues.
I also need help with the following:
- can the property modification remaing in params_linux.py, or should that code be moved to params.py to enable the same property changes across both windows and linux? Atlas isn't currently in the windows MSI (I believe)
- I can't seem to come up with the appropriate modification to mock the new code (atlas_plugin_utils.configure_for_plugin) in test_hive_metastore. I am able to do so for other hive component tests, by the hive metastore tests fail with the following:
Traceback (most recent call last):
File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1191, in patched
arg = patching.__enter__()
File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1250, in __enter__
self.target = self.getter()
File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1412, in <lambda>
getter = lambda: _importer(target)
File "/Users/jmaron/dev/workspaces/ambari/ambari-server/../ambari-common/src/test/python/mock/mock.py", line 1096, in _importer
thing = __import__(import_path)
ImportError: No module named atlas_plugin_utils
I have yet to come up with a way to make the test code aware of the new module I created in these tests.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/atlas_plugin_utils.py PRE-CREATION
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py 79096e4
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py cf08e1c
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py 7ae738a
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py f817c68
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py 0d13bea
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py ce62309
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_server.py 012d1e2
ambari-server/src/main/resources/stacks/HDP/2.3/services/HIVE/metainfo.xml 58627ea
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 2f6bd40
Diff: https://reviews.apache.org/r/34864/diff/
Testing
-------
Functional testing with a local VM spawned cluster
Ambari unit tests.
Thanks,
Jonathan Maron