Review Request 34636: DATAFU-58 Update to Hadoop 2.7.0 and Pig 0.14.0 and fix all tests

[Matthew Hayes <ma...@gmail.com>]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34636/
-----------------------------------------------------------

Review request for DataFu.


Repository: datafu


Description
-------

This updates DataFu to use Hadoop 2.7.0 and Pig 0.14.0.  I've fixed all issues that I could find.  All the unit tests pass.  Some of the issues were already addressd by Daniel Dai's earlier patch, but I found some more problems.

Summary of issues:

* Hourglass tests failed because we were pulling in avro-tools, which contains Hadoop classes and therefore conflicts.  I removed the dependency.  I'm not sure why we had this.  I also added a handy script to help with this analysis, which uses jarfish.
* Hourglass used .toString() for many Path instances.  This caused a problem because the resulting string is prefixed with "file:" and therefore isn't a valid path you can use in File.  The general fix is to use getName() instead, which gets the simple name and is usually suitable.
* Some Pig unit tests had assumptions on tuple ordering in bags.  I applied a sort to the actual and expected bags to ensure a consistent comparison.


Diffs
-----

  README.md 8e1b67d 
  build-plugin/src/main/java/org/adrianwalker/multilinestring/MultilineProcessor.java 9abdba5 
  datafu-hourglass/.gitignore 942515e 
  datafu-hourglass/build.gradle 75a2876 
  datafu-hourglass/find_dupes.rb PRE-CREATION 
  datafu-hourglass/src/main/java/datafu/hourglass/fs/PathUtils.java c270c7b 
  datafu-hourglass/src/main/java/datafu/hourglass/mapreduce/DistributedCacheHelper.java 62975d1 
  datafu-hourglass/src/test/java/datafu/hourglass/demo/Examples.java 039822c 
  datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionCollapsingExecutionPlannerTests.java d68ea83 
  datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionCollapsingJoinTest.java 02aa342 
  datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionCollapsingTests.java fff1cfd 
  datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionPreservingCollapsingIntegrationTests.java a8f020b 
  datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionPreservingJoinTests.java c41fd39 
  datafu-hourglass/src/test/java/datafu/hourglass/test/PartitionPreservingTests.java acae96c 
  datafu-hourglass/src/test/java/datafu/hourglass/test/TestAvroJob.java b428003 
  datafu-hourglass/src/test/java/datafu/hourglass/test/TestBase.java bc52977 
  datafu-pig/build.gradle ea385d2 
  datafu-pig/src/test/java/datafu/test/pig/bags/BagTests.java 9bcc384 
  gradle/dependency-versions.gradle 3b0835f 

Diff: https://reviews.apache.org/r/34636/diff/


Testing
-------

./gradlew test


Thanks,

Matthew Hayes