You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Andrew Onischuk <ao...@hortonworks.com> on 2015/02/10 14:22:15 UTC
Review Request 30826: Updating FSRoot for Hive Metastore doesn't seem
to
work (breaks Hive functionality after move NameNode and NameNode HA)
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30826/
-----------------------------------------------------------
Review request for Ambari and Vitalyi Brodetskyi.
Bugs: AMBARI-9548
https://issues.apache.org/jira/browse/AMBARI-9548
Repository: ambari
Description
-------
In hive_service.py, we have some logic to check and update FS root for Hive.
I believe this was added to handle the situation when the NameNode is moved.
I also think that this code would also automate updating the pointer to
nameservice ID rather than a specific namenode after enabling NameNode HA.
However, this code does not seem correct:
def check_fs_root():
import params
fs_root_url = format("{fs_root}{hive_apps_whs_dir}")
cmd = format("metatool -listFSRoot 2>/dev/null | grep hdfs:// | grep -v '.db$'")
code, out = shell.call(cmd, user=params.hive_user)
if code == 0 and fs_root_url.strip() != out.strip():
cmd = format("metatool -updateLocation {fs_root}{hive_apps_whs_dir} {out}")
Execute(cmd,
environment= {'PATH' : params.execute_path },
user=params.hive_user)
Problem 1: "metatool" is not in the PATH for Hive user, so this does not run.
Problem 2: "metatool" is a wrapper around "hive --service metatool". However,
it does not take any arguments for the hive config directory. Simply running
"metatool -listRootFS", even if the correct PATH was set for metatool, uses
/etc/hive/conf. However, we do not set "hive.metastore.warehouse.dir"
parameter in that file. We need to explicitly use /etc/hive/conf.server.
/usr/hdp/current/hive-client/bin/metatool -listFSRoot
incorrectly yields
hdfs://yusaku-1.c.pramod-thangali.internal:8020/user/hive/warehouse
The above command is equivalent to:
hive --service metatool -listFSRoot
and shows the same incorrect output.
The following correctly shows the fs root:
hive --config /etc/hive/conf.server--service metatool -listFSRoot
hdfs://yusaku-1.c.pramod-thangali.internal:8020/apps/hive/warehouse
Net net, we should be using "hive" command with config correctly set, rather
than the "metatool" wrapper.
Diffs
-----
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py 5f8029f
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 19f5168
Diff: https://reviews.apache.org/r/30826/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
Re: Review Request 30826: Updating FSRoot for Hive Metastore doesn't
seem to
work (breaks Hive functionality after move NameNode and NameNode HA)
Posted by Vitalyi Brodetskyi <vb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30826/#review71781
-----------------------------------------------------------
Ship it!
Ship It!
- Vitalyi Brodetskyi
On Лют. 10, 2015, 1:22 після полудня, Andrew Onischuk wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30826/
> -----------------------------------------------------------
>
> (Updated Лют. 10, 2015, 1:22 після полудня)
>
>
> Review request for Ambari and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-9548
> https://issues.apache.org/jira/browse/AMBARI-9548
>
>
> Repository: ambari
>
>
> Description
> -------
>
> In hive_service.py, we have some logic to check and update FS root for Hive.
> I believe this was added to handle the situation when the NameNode is moved.
> I also think that this code would also automate updating the pointer to
> nameservice ID rather than a specific namenode after enabling NameNode HA.
> However, this code does not seem correct:
>
>
>
>
> def check_fs_root():
> import params
> fs_root_url = format("{fs_root}{hive_apps_whs_dir}")
> cmd = format("metatool -listFSRoot 2>/dev/null | grep hdfs:// | grep -v '.db$'")
> code, out = shell.call(cmd, user=params.hive_user)
> if code == 0 and fs_root_url.strip() != out.strip():
> cmd = format("metatool -updateLocation {fs_root}{hive_apps_whs_dir} {out}")
> Execute(cmd,
> environment= {'PATH' : params.execute_path },
> user=params.hive_user)
>
>
> Problem 1: "metatool" is not in the PATH for Hive user, so this does not run.
> Problem 2: "metatool" is a wrapper around "hive --service metatool". However,
> it does not take any arguments for the hive config directory. Simply running
> "metatool -listRootFS", even if the correct PATH was set for metatool, uses
> /etc/hive/conf. However, we do not set "hive.metastore.warehouse.dir"
> parameter in that file. We need to explicitly use /etc/hive/conf.server.
>
>
>
>
> /usr/hdp/current/hive-client/bin/metatool -listFSRoot
>
>
> incorrectly yields
>
>
>
>
> hdfs://yusaku-1.c.pramod-thangali.internal:8020/user/hive/warehouse
>
>
> The above command is equivalent to:
>
>
>
>
> hive --service metatool -listFSRoot
>
>
> and shows the same incorrect output.
>
> The following correctly shows the fs root:
>
>
>
>
> hive --config /etc/hive/conf.server--service metatool -listFSRoot
>
>
> hdfs://yusaku-1.c.pramod-thangali.internal:8020/apps/hive/warehouse
>
> Net net, we should be using "hive" command with config correctly set, rather
> than the "metatool" wrapper.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py 5f8029f
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 19f5168
>
> Diff: https://reviews.apache.org/r/30826/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Andrew Onischuk
>
>