You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by Jason Dere <jd...@hortonworks.com> on 2017/11/03 19:36:17 UTC

Review Request 63544: HIVE-17963: Fix for HIVE-17113 can be improved for non-blobstore filesystems

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63544/
-----------------------------------------------------------

Review request for hive, Ashutosh Chauhan and Owen O'Malley.


Bugs: HIVE-17963
    https://issues.apache.org/jira/browse/HIVE-17963


Repository: hive-git


Description
-------

For non-blobstore FS:
1) Move the temp directory to a new directory name, to prevent additional files from being added by any runaway processes.
2) Run removeTempOrDuplicateFiles() on this renamed temp directory
3) Run renameOrMoveFiles() to move the renamed temp directory to the final location.

For blobstore FS:
1) Remove duplicates from tmpPath
2) Use moveSpecifiedFiles() to perform a file-by-file move of the de-duped files to specPath

Also remove hive.exec.move.files.from.source.dir setting - the above changes will always be enabled.


Diffs
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbe4de5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1be7eab 
  ql/src/test/queries/clientpositive/insert_with_move_files_from_source_dir.q 0117755 
  ql/src/test/queries/clientpositive/skewjoin.q 9ad4b77 


Diff: https://reviews.apache.org/r/63544/diff/1/


Testing
-------


Thanks,

Jason Dere