You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ambari.apache.org by "Jason Golieb (JIRA)" <ji...@apache.org> on 2018/03/19 17:29:00 UTC
[jira] [Created] (AMBARI-23286) check_fs_root() in hive_service.py
may replace Hive HDFS locations with ":8020:8020"
Jason Golieb created AMBARI-23286:
-------------------------------------
Summary: check_fs_root() in hive_service.py may replace Hive HDFS locations with ":8020:8020"
Key: AMBARI-23286
URL: https://issues.apache.org/jira/browse/AMBARI-23286
Project: Ambari
Issue Type: Bug
Components: ambari-server
Affects Versions: 2.6.1
Reporter: Jason Golieb
Assignee: Mahadev konar
Fix For: 2.6.1
Modify check_fs_root() in hive_services.py to check if "fs_root" and "out" are not endswith('/'), append '/'
+*Background*+
When hive's metatstore database table 'DBS' contains a HDFS location which doesn't contain any port number, Ambari may incorrectly replace the HDFS locations.
For example, "hdfs://sandbox-hdp.hortonworks.com/apps/hive/warehouse/dummies.db" should be a valid location:
{code:java}
[hive@sandbox-hdp ~]$ hive --config /usr/hdp/current/hive-server2/conf/conf.server --service metatool -listFSRoot 2>/dev/null
Initializing HiveMetaTool..
Listing FS Roots..
hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db
hdfs://sandbox-hdp.hortonworks.com/apps/hive/warehouse/dummies.db
hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/xademo.db
hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse
{code}
Ambari uses same command as below:
{code:java}
[hive@sandbox-hdp ~]$ hive --config /usr/hdp/current/hive-server2/conf/conf.server --service metatool -listFSRoot 2>/dev/null | grep hdfs:// | cut -f1,2,3 -d '/' | grep -v 'hdfs://sandbox-hdp.hortonworks.com:8020' | head -1
hdfs://sandbox-hdp.hortonworks.com
{code}
Then tries to update location with same command as below (except dryRun), which breaks Hive completely.
{code:java}
[hive@sandbox-hdp ~]$ hive --config /usr/hdp/current/hive-server2/conf/conf.server --service metatool -updateLocation hdfs://sandbox-hdp.hortonworks.com:8020 hdfs://sandbox-hdp.hortonworks.com -dryRun
Initializing HiveMetaTool..
... (snip)...
Looking for LOCATION_URI field in DBS table to update..
Dry Run of updateLocation on table DBS..
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/foodmart.db
old location: hdfs://sandbox-hdp.hortonworks.com/apps/hive/warehouse/dummies.db new location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/xademo.db new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/xademo.db
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse
Found 4 records in DBS table to update
Looking for LOCATION field in SDS table to update..
Dry Run of updateLocation on table SDS..
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=A new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=A
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=B new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=B
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/hdfs_audit new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/hdfs_audit
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/test2 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/test2
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/xademo.db/recharge_details new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/xademo.db/recharge_details
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/sample_07_orc new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/sample_07_orc
old location: hdfs://sandbox-hdp.hortonworks.com/tmp/emp_stage_data new location: hdfs://sandbox-hdp.hortonworks.com:8020/tmp/emp_stage_data
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=C new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=C
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/emp_part_bckt new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/emp_part_bckt
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=D new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/emp_part_bckt/department=D
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/xademo.db/call_detail_records new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/xademo.db/call_detail_records
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/xademo.db/customer_details new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/xademo.db/customer_details
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db/customer new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/foodmart.db/customer
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/sample_08 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/sample_08
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db/product new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/foodmart.db/product
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/sample_07 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/sample_07
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db/sales_fact_1998 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/foodmart.db/sales_fact_1998
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db/inventory_fact_1998 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/foodmart.db/inventory_fact_1998
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/census new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/census
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/foodmart.db/store new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/foodmart.db/store
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/census_clus new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/census_clus
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/sample_08 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/sample_08
old location: hdfs://sandbox-hdp.hortonworks.com:8020/apps/hive/warehouse/dummies.db/sample_07 new location: hdfs://sandbox-hdp.hortonworks.com:8020:8020/apps/hive/warehouse/dummies.db/sample_07
Found 23 records in SDS table to update
{code}
Lines from output-9505.txt
{code:java}
2018-02-27 20:16:42,747 - call['ambari-sudo.sh su hive -l -s /bin/bash -c 'hive --config /usr/hdp/current/hive-server2/conf/conf.server --service metatool -listFSRoot' 2>/dev/null | grep hdfs:// | cut -f1,2,3 -d '/' | grep -v 'hdfs://farmerm.hw262.com:8020' | head -1'] {}
2018-02-27 20:16:48,897 - call returned (0, 'hdfs://farmerm.hw262.com')
2018-02-27 20:16:48,898 - Execute['hive --config /usr/hdp/current/hive-server2/conf/conf.server --service metatool -updateLocation hdfs://farmerm.hw262.com:8020 hdfs://farmerm.hw262.com'] {'environment': {'PATH': u'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hive-server2/bin:/usr/hdp/current/hadoop-client/bin'}, 'user': 'hive'}{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)