Re: Review Request 48159: HIVE-13901: Hivemetastore add partitions can be slow depending on filesystems

[Rajesh Balamohan <ra...@gmail.com>]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48159/
-----------------------------------------------------------

(Updated July 8, 2016, 3:52 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
-------

The earlier patch caused test failures in TestHiveMetastore where it was trying to have duplicate partitions.  In earlier patch, any entry in addedPartitions was getting nuked in the finally block, so even valid partitions were getting deleted.

Changes in latest patch:
1. Making use of Map<PartValEqWrapperLite, Boolean> for addedPartitions (in synchronized hashmap) to take care of the condition mentioned earlier.
2. Still adding entry to addPartitions isn't threadsafe, when something goes wrong between time of creating the directory and adding it to addedPartitions (e.g killing server). But that is something which was already existing and patch isn't trying to change the behaviour.

Ran the following for verification with latest patch.

mvn clean install -DskipTests && cd itests && mvn clean install -DskipTests && cd hive-unit && mvn test -Dtest=TestEmbeddedHiveMetaStore,TestRemoteHiveMetaStore,TestSetUGIOnBothClientServer,TestSetUGIOnOnlyClient,TestSetUGIOnOnlyServer


Bugs: HIVE-13901
    https://issues.apache.org/jira/browse/HIVE-13901


Repository: hive-git


Description
-------

Depending on FS, creating external tables & adding partitions can be expensive (e.g msck which adds all partitions).


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b13fc65 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java f45b90d 

Diff: https://reviews.apache.org/r/48159/diff/


Testing
-------


Thanks,

Rajesh Balamohan