You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2023/01/04 10:20:51 UTC
[GitHub] [hive] warriersruthi opened a new pull request, #3909: HIVE-26894: After using scratchdir for staging final job, CTAS and IOW on ACID tables are failing with wrongFS exception
warriersruthi opened a new pull request, #3909:
URL: https://github.com/apache/hive/pull/3909
###What changes were proposed in this pull request?
moveAcidFiles() method in Hive.java is modified to cater to the case where the src FS and dest FS are different. Instead of rename() operation, in the above case, we need to do copy().
###Why are the changes needed?
With the Jira: HIVE-26815, we can change the hive staging directory to be the same as that of scratchdir. In that case, staging files will be in HDFS while the target location would be blob or some other FS. In such scenarios, while creating and updating ACID tables, the final rename operation was not working fine (throws Wrong FS error) since the src and dest FS are different. This is fixed in this patch by changing the rename() operation to copy() if FS are in different encryption zones.
###Does this PR introduce any user-facing change?
Nope.
###How was this patch tested?
Tested with patch on hive 3.1.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh commented on a diff in pull request #3909: HIVE-26894: After using scratchdir for staging final job, CTAS and IOW on ACID tables are failing with wrongFS exception
Posted by GitBox <gi...@apache.org>.
sankarh commented on code in PR #3909:
URL: https://github.com/apache/hive/pull/3909#discussion_r1062211047
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##########
@@ -5263,23 +5263,38 @@ public Void call() throws HiveException {
private static void moveAcidFilesForDelta(String deltaFileType, FileSystem fs,
Path dst, Set<Path> createdDeltaDirs,
- List<Path> newFiles, FileStatus deltaStat) throws HiveException {
-
+ List<Path> newFiles, FileStatus deltaStat, HiveConf conf) throws HiveException {
+ String configuredOwner = HiveConf.getVar(conf, ConfVars.HIVE_LOAD_DATA_OWNER);
Path deltaPath = deltaStat.getPath();
// Create the delta directory. Don't worry if it already exists,
// as that likely means another task got to it first. Then move each of the buckets.
// it would be more efficient to try to move the delta with it's buckets but that is
// harder to make race condition proof.
Path deltaDest = new Path(dst, deltaPath.getName());
try {
+ FileSystem destFs = deltaDest.getFileSystem(conf);
if (!createdDeltaDirs.contains(deltaDest)) {
try {
- if(fs.mkdirs(deltaDest)) {
+ // Check if the src and dest filesystems are same or not
+ // if the src and dest filesystems are different, then we need to do copy, instead of rename
+ if (needToCopy(conf, deltaStat.getPath(), deltaDest, fs, destFs, configuredOwner, true)) {
Review Comment:
Can you add tests that cover these 2 newly added flows?
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##########
@@ -5263,23 +5263,38 @@ public Void call() throws HiveException {
private static void moveAcidFilesForDelta(String deltaFileType, FileSystem fs,
Path dst, Set<Path> createdDeltaDirs,
- List<Path> newFiles, FileStatus deltaStat) throws HiveException {
-
+ List<Path> newFiles, FileStatus deltaStat, HiveConf conf) throws HiveException {
+ String configuredOwner = HiveConf.getVar(conf, ConfVars.HIVE_LOAD_DATA_OWNER);
Path deltaPath = deltaStat.getPath();
// Create the delta directory. Don't worry if it already exists,
// as that likely means another task got to it first. Then move each of the buckets.
// it would be more efficient to try to move the delta with it's buckets but that is
// harder to make race condition proof.
Path deltaDest = new Path(dst, deltaPath.getName());
try {
+ FileSystem destFs = deltaDest.getFileSystem(conf);
if (!createdDeltaDirs.contains(deltaDest)) {
try {
- if(fs.mkdirs(deltaDest)) {
+ // Check if the src and dest filesystems are same or not
+ // if the src and dest filesystems are different, then we need to do copy, instead of rename
+ if (needToCopy(conf, deltaStat.getPath(), deltaDest, fs, destFs, configuredOwner, true)) {
+ //copy if across file system or encryption zones.
+ LOG.debug("Copying source " + deltaStat.getPath() + " to " + deltaDest + " because HDFS encryption zones are different.");
Review Comment:
The log msg can also highlight, the copy is across different file systems or encryption zones.
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##########
@@ -5294,11 +5309,24 @@ private static void moveAcidFilesForDelta(String deltaFileType, FileSystem fs,
for (FileStatus bucketStat : bucketStats) {
Path bucketSrc = bucketStat.getPath();
Path bucketDest = new Path(deltaDest, bucketSrc.getName());
+ FileSystem bucketDestFs = bucketDest.getFileSystem(conf);
LOG.info("Moving bucket " + bucketSrc.toUri().toString() + " to " +
bucketDest.toUri().toString());
try {
- fs.rename(bucketSrc, bucketDest);
- if (newFiles != null) {
+ // Check if the src and dest filesystems are same or not
+ // if the src and dest filesystems are different, then we need to do copy, instead of rename
+ if (needToCopy(conf, bucketSrc, bucketDest, fs, bucketDestFs, configuredOwner, true)) {
+ //copy if across file system or encryption zones.
+ LOG.debug("Copying source " + bucketSrc + " to " + bucketDest + " because HDFS encryption zones are different.");
+ FileUtils.copy(fs, bucketSrc, bucketDestFs, bucketDest,
+ true, // delete source
+ true, // overwrite destination
+ conf);
+ } else {
+ // do the rename
+ bucketDestFs.rename(bucketSrc, bucketDest);
+ }
+ if (newFiles != null) {
newFiles.add(bucketDest);
Review Comment:
Need to add extra tab space for the body of "if" path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] closed pull request #3909: HIVE-26894: After using scratchdir for staging final job, CTAS and IOW on ACID tables are failing with wrongFS exception
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3909: HIVE-26894: After using scratchdir for staging final job, CTAS and IOW on ACID tables are failing with wrongFS exception
URL: https://github.com/apache/hive/pull/3909
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #3909: HIVE-26894: After using scratchdir for staging final job, CTAS and IOW on ACID tables are failing with wrongFS exception
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3909:
URL: https://github.com/apache/hive/pull/3909#issuecomment-1457263005
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3909: HIVE-26894: After using scratchdir for staging final job, CTAS and IOW on ACID tables are failing with wrongFS exception
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3909:
URL: https://github.com/apache/hive/pull/3909#issuecomment-1371361113
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3909)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3909&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3909&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3909&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=CODE_SMELL) [8 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3909&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3909&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3909&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org