You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/03/01 09:23:50 UTC
[GitHub] [hadoop] sumangala-patki opened a new pull request #2729: Hadoop 17548
sumangala-patki opened a new pull request #2729:
URL: https://github.com/apache/hadoop/pull/2729
The call to mkdirs with overwrite set to true results in an overhead set property call (LMT update, etc), which is unnecessary. Moreover, mkdirs on an existing file path returns success. This PR provides an option to set the overwrite parameter to false, and ensures that mkdirs on a file throws an exception.
New config: `fs.azure.enable.mkdir.overwrite` [true by default]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #2729: HADOOP-17548. ABFS: Config for Mkdir overwrite
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #2729:
URL: https://github.com/apache/hadoop/pull/2729#discussion_r585454114
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -302,7 +302,16 @@ public AbfsRestOperation createPath(final String path, final boolean isFile, fin
HTTP_METHOD_PUT,
url,
requestHeaders);
- op.execute();
+ try {
+ op.execute();
+ } catch (AzureBlobFileSystemException ex) {
+ String existingResource =
Review comment:
first check if the httpstatus code is 409 and (!isFile), then retrieve the existingResource header.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on pull request #2729: HADOOP-17548. ABFS: Toggle Store Mkdirs request overwrite parameter
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on pull request #2729:
URL: https://github.com/apache/hadoop/pull/2729#issuecomment-791305493
TEST RESULTS
HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US
Overwrite=true
```
HNS OAuth
[INFO] Tests run: 93, Failures: 0, Errors: 0, Skipped: 0
[INFO] Tests run: 504, Failures: 0, Errors: 0, Skipped: 70
[WARNING] Tests run: 257, Failures: 0, Errors: 0, Skipped: 48
HNS SharedKey
[INFO] Tests run: 93, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 513, Failures: 0, Errors: 0, Skipped: 26
[WARNING] Tests run: 257, Failures: 0, Errors: 0, Skipped: 40
Non-HNS SharedKey
[INFO] Tests run: 93, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 504, Failures: 0, Errors: 0, Skipped: 250
[WARNING] Tests run: 257, Failures: 0, Errors: 0, Skipped: 40
```
Dev Fabric (Xns account)
Overwrite=false
```
Tests run: 868, passed: 762, failed: 19, ignored: 87
Errors:
ITestAbfsNetworkStatistics, ITestAzureBlobFileSystemCheckAccess, ITestAzureBlobFileSystemFileStatus, ITestClientUrlScheme
ITestFileSystemInitialization, ITestFileSystemRegistration, TestAbfsConfigurationFieldsValidation, ITestAbfsDelegationTokens
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] surendralilhore merged pull request #2729: HADOOP-17548. ABFS: Toggle Store Mkdirs request overwrite parameter
Posted by GitBox <gi...@apache.org>.
surendralilhore merged pull request #2729:
URL: https://github.com/apache/hadoop/pull/2729
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki commented on a change in pull request #2729: HADOOP-17548. ABFS: Toggle Store Mkdirs request overwrite parameter
Posted by GitBox <gi...@apache.org>.
sumangala-patki commented on a change in pull request #2729:
URL: https://github.com/apache/hadoop/pull/2729#discussion_r585498405
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -302,7 +302,16 @@ public AbfsRestOperation createPath(final String path, final boolean isFile, fin
HTTP_METHOD_PUT,
url,
requestHeaders);
- op.execute();
+ try {
+ op.execute();
+ } catch (AzureBlobFileSystemException ex) {
+ String existingResource =
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sumangala-patki edited a comment on pull request #2729: HADOOP-17548. ABFS: Toggle Store Mkdirs request overwrite parameter
Posted by GitBox <gi...@apache.org>.
sumangala-patki edited a comment on pull request #2729:
URL: https://github.com/apache/hadoop/pull/2729#issuecomment-791305493
TEST RESULTS
HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US
Overwrite=true
```
HNS OAuth
[INFO] Tests run: 93, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 513, Failures: 0, Errors: 0, Skipped: 70
[WARNING] Tests run: 257, Failures: 0, Errors: 0, Skipped: 48
HNS SharedKey
[INFO] Tests run: 93, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 513, Failures: 0, Errors: 0, Skipped: 26
[WARNING] Tests run: 257, Failures: 0, Errors: 0, Skipped: 40
Non-HNS SharedKey
[INFO] Tests run: 93, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 504, Failures: 0, Errors: 0, Skipped: 250
[WARNING] Tests run: 257, Failures: 0, Errors: 0, Skipped: 40
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org