You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/02/12 10:05:14 UTC
[GitHub] [hadoop] snvijaya opened a new pull request #1842: Hadoop 16730:
Add Authorizer Interface
snvijaya opened a new pull request #1842: Hadoop 16730: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842
Enabling Authorizer Interface as a plugin
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379629072
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/authentication/AuthorizationStatus.java
##########
@@ -0,0 +1,183 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.authentication;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResource;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult;
+
+import java.net.URI;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.HashMap;
+
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.DEFAULT_SAS_REFRESH_INTERVAL_BEFORE_EXPIRY;
+
+/**
+ * AuthorizationStatus maintains the status of Authorization and also SAS
+ * token if authorizer is providing it.
+ */
+public class AuthorizationStatus {
+ private boolean isAuthorized;
+ private HashMap<URI, SasTokenData> sasTokenMap;
+
+ public AuthorizationStatus() {
+ sasTokenMap = new HashMap<>();
+ }
+
+ /**
+ * Fetch the SAS token
+ *
+ * @return
+ */
+ public String getSasTokenQuery(URI storePathUri) {
+ if (sasTokenMap.containsKey(storePathUri)) {
+ SasTokenData sasTokenData = sasTokenMap.get(storePathUri);
+ if (isValidSas(sasTokenData)) {
+ return sasTokenData.sasToken;
+ }
+ }
+
+ return null;
+ }
+
+ /**
+ * Update authTokenMap
+ * Also update the refresh interval for each SAS token.
+ *
+ * @param authorizationResource
+ * @param authResult - Authorizer AuthorizationResult
+ */
+ public void setSasToken(AuthorizationResource[] authorizationResource,
+ AuthorizationResult authResult) throws AbfsAuthorizationException {
+
+ AuthorizationResourceResult[] resourceResult = authResult
+ .getAuthResourceResult();
+
+ int i = 0;
+ for (AuthorizationResourceResult singleResourceAuth : resourceResult) {
+ // First check if the requested resource matches the resource
+ // for which authToken is returned
+ AuthorizationResource authorizationRequestedForResource =
+ authorizationResource[i];
+ if ((singleResourceAuth == null) || (singleResourceAuth.storePathUri
+ == null) || (singleResourceAuth.authorizerAction == null)) {
+ throw new AbfsAuthorizationException("Invalid authorization response");
Review comment:
Have put in detailed checks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya edited a comment on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya edited a comment on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-586471534
@steveloughran - Thanks for taking time to review this change. I have addressed almost all the comments and few I have retained with explanation in the comments.
Re-ran the tests again with the current new changes and all the tests passed.
Can you please have a look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378935772
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/authentication/AuthorizationStatus.java
##########
@@ -0,0 +1,183 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.authentication;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResource;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult;
+
+import java.net.URI;
Review comment:
general: the import java. block should be a the top of each file, followed by the non org.apache, then the org.apache.* imports
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378949937
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -604,9 +622,10 @@ public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
| NoSuchMethodException
| SecurityException
| AbfsAuthorizationException e) {
- throw new IOException(e);
+ throw new AbfsAuthorizationException("Unable to initialize "
Review comment:
no need to catch and wrap AbfsAuthException
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ThomasMarquardt commented on a change in pull request
#1842: HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
ThomasMarquardt commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384665889
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -65,6 +66,21 @@ public void setup() throws Exception {
super.setup();
}
+ @Test
+ public void testSASTokenProviderInitializeException() throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+
+ final AzureBlobFileSystem testFs = new AzureBlobFileSystem();
+
+ MockSASTokenProvider.setThrowExceptionAtInit(true);
Review comment:
I think this will fail intermittently (since it is stored in a static) when the tests are run in parallel, which is how we usually run the tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r383185199
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -578,35 +585,37 @@ public AccessTokenProvider getTokenProvider() throws TokenAccessProviderExceptio
}
}
- public String getAbfsExternalAuthorizationClass() {
- return this.abfsExternalAuthorizationClass;
- }
-
- public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
- String authClassName = getAbfsExternalAuthorizationClass();
- AbfsAuthorizer authorizer = null;
+ public SASTokenProvider getSASTokenProvider() throws AzureBlobFileSystemException {
+ AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME, AuthType.SharedKey);
+ if (authType != AuthType.SAS) {
+ throw new SASTokenProviderException(String.format(
+ "Invalid auth type: %s is being used, expecting SAS", authType));
+ }
try {
- if (authClassName != null && !authClassName.isEmpty()) {
- @SuppressWarnings("unchecked")
- Class<AbfsAuthorizer> authClass = (Class<AbfsAuthorizer>) rawConfig.getClassByName(authClassName);
- authorizer = authClass.getConstructor(new Class[] {Configuration.class}).newInstance(rawConfig);
- LOG.trace("Initializing {}", authClassName);
- authorizer.init();
- LOG.trace("{} init complete", authClassName);
+ String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+ Class<? extends SASTokenProvider> sasTokenProviderClass =
+ getClass(configKey, null, SASTokenProvider.class);
+ if (sasTokenProviderClass == null) {
+ throw new IllegalArgumentException(
+ String.format("The configuration value for \"%s\" is invalid.", configKey));
}
- } catch (
- IllegalAccessException
- | InstantiationException
- | ClassNotFoundException
- | IllegalArgumentException
- | InvocationTargetException
- | NoSuchMethodException
- | SecurityException
- | AbfsAuthorizationException e) {
- throw new IOException(e);
+
+ SASTokenProvider sasTokenProvider = ReflectionUtils
+ .newInstance(sasTokenProviderClass, rawConfig);
+ if (sasTokenProvider == null) {
+ throw new IllegalArgumentException("Failed to initialize " + sasTokenProviderClass);
+ }
+
+ LOG.trace("Initializing {}", sasTokenProviderClass.getName());
+ sasTokenProvider.initialize(rawConfig, accountName);
+ LOG.trace("{} init complete", sasTokenProviderClass.getName());
+ return sasTokenProvider;
+ } catch(IllegalArgumentException e) {
+ throw e;
+ } catch (Exception e) {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on issue #1842: HADOOP-16730 : ABFS:
Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-589607680
Have fixed the open comments. @steveloughran - Can you please check. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382511733
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -29,6 +30,8 @@
import java.util.Locale;
import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException;
+import org.apache.hadoop.fs.azurebfs.extensions.SASTokenProvider;
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378959131
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -36,331 +40,485 @@
import static org.apache.hadoop.fs.permission.AclEntryScope.ACCESS;
import static org.apache.hadoop.fs.permission.AclEntryType.GROUP;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
-import static org.junit.Assume.assumeTrue;
/**
* Test Perform Authorization Check operation
*/
-public class ITestAzureBlobFileSystemAuthorization extends AbstractAbfsIntegrationTest {
-
- private static final Path TEST_READ_ONLY_FILE_PATH_0 = new Path(TEST_READ_ONLY_FILE_0);
- private static final Path TEST_READ_ONLY_FOLDER_PATH = new Path(TEST_READ_ONLY_FOLDER);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_0 = new Path(TEST_WRITE_ONLY_FILE_0);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_1 = new Path(TEST_WRITE_ONLY_FILE_1);
- private static final Path TEST_READ_WRITE_FILE_PATH_0 = new Path(TEST_READ_WRITE_FILE_0);
- private static final Path TEST_READ_WRITE_FILE_PATH_1 = new Path(TEST_READ_WRITE_FILE_1);
- private static final Path TEST_WRITE_ONLY_FOLDER_PATH = new Path(TEST_WRITE_ONLY_FOLDER);
- private static final Path TEST_WRITE_THEN_READ_ONLY_PATH = new Path(TEST_WRITE_THEN_READ_ONLY);
- private static final String TEST_AUTHZ_CLASS = "org.apache.hadoop.fs.azurebfs.extensions.MockAbfsAuthorizer";
+public class ITestAzureBlobFileSystemAuthorization
Review comment:
There's a lot of repetition in these tests which could be factored out into a pair of methods and so make for much shorter test cases. The test suites form the largest body of code in the project and we need to think about maintenance here too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379622525
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/AuthorizationResource.java
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.extensions;
+
+import java.net.URI;
+
+/**
+ * AuthorizationResource with action and store path URI
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382526421
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -578,35 +585,37 @@ public AccessTokenProvider getTokenProvider() throws TokenAccessProviderExceptio
}
}
- public String getAbfsExternalAuthorizationClass() {
- return this.abfsExternalAuthorizationClass;
- }
-
- public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
- String authClassName = getAbfsExternalAuthorizationClass();
- AbfsAuthorizer authorizer = null;
+ public SASTokenProvider getSASTokenProvider() throws AzureBlobFileSystemException {
+ AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME, AuthType.SharedKey);
+ if (authType != AuthType.SAS) {
+ throw new SASTokenProviderException(String.format(
+ "Invalid auth type: %s is being used, expecting SAS", authType));
+ }
try {
- if (authClassName != null && !authClassName.isEmpty()) {
- @SuppressWarnings("unchecked")
- Class<AbfsAuthorizer> authClass = (Class<AbfsAuthorizer>) rawConfig.getClassByName(authClassName);
- authorizer = authClass.getConstructor(new Class[] {Configuration.class}).newInstance(rawConfig);
- LOG.trace("Initializing {}", authClassName);
- authorizer.init();
- LOG.trace("{} init complete", authClassName);
+ String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+ Class<? extends SASTokenProvider> sasTokenProviderClass =
+ getClass(configKey, null, SASTokenProvider.class);
+ if (sasTokenProviderClass == null) {
Review comment:
do you really need to do that? because that just increases the LOC and overall maintenance costs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ThomasMarquardt commented on a change in pull request
#1842: HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
ThomasMarquardt commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384049325
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/abfs.md
##########
@@ -626,7 +626,7 @@ points for third-parties to integrate their authentication and authorization
services into the ABFS client.
* `CustomDelegationTokenManager` : adds ability to issue Hadoop Delegation Tokens.
-* `AbfsAuthorizer` permits client-side authorization of file operations.
+* `SASTokenProvider` permits client-side authorization of file operations.
Review comment:
Client-side makes it sound like the client authorizes itself, which is not the case. I suggest " `SASTokenProvider` allows for custom provision of Azure Storage Shared Access Signature (SAS) tokens." to keep with the pattern on line 630.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r383183047
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -550,6 +631,17 @@ public AbfsRestOperation checkAccess(String path, String rwx)
return op;
}
+ private void appendSASTokenToQuery(String path, String operation, AbfsUriQueryBuilder queryBuilder) throws SASTokenProviderException {
+ try {
+ LOG.trace("Fetch SAS token for {}", path);
Review comment:
Added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ThomasMarquardt commented on a change in pull request
#1842: HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
ThomasMarquardt commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384072576
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -550,6 +595,30 @@ public AbfsRestOperation checkAccess(String path, String rwx)
return op;
}
+ /**
+ * If configured for SAS AuthType, appends SAS token to queryBuilder
+ * @param path
+ * @param operation
+ * @param queryBuilder
+ * @throws SASTokenProviderException
+ */
+ private void appendSASTokenToQuery(String path, String operation, AbfsUriQueryBuilder queryBuilder) throws SASTokenProviderException {
+ if (this.authType == AuthType.SAS) {
+ try {
+ LOG.trace("Fetch SAS token for {} on {}", operation, path);
Review comment:
Good to know, then we can resolve this comment. (I don't see an option to resolve comments.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382640519
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -57,25 +59,30 @@
private final SharedKeyCredentials sharedKeyCredentials;
private final String xMsVersion = "2018-11-09";
private final ExponentialRetryPolicy retryPolicy;
+ private final String accountName;
private final String filesystem;
+ private final AuthType authType;
Review comment:
can you move down to line 68 along with the accountName, in the hope it will make merge easier
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382510664
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/authentication/AuthorizationStatus.java
##########
@@ -0,0 +1,183 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.authentication;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResource;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult;
+
+import java.net.URI;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.HashMap;
+
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.DEFAULT_SAS_REFRESH_INTERVAL_BEFORE_EXPIRY;
+
+/**
+ * AuthorizationStatus maintains the status of Authorization and also SAS
+ * token if authorizer is providing it.
+ */
+public class AuthorizationStatus {
+ private boolean isAuthorized;
+ private HashMap<URI, SasTokenData> sasTokenMap;
Review comment:
Code is modified. Resolving this comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378962892
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsAuthorizerConstants.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidAbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperationType;
+
+/**
+ * Defines the string literals that need to be set to authorizer as action
+ */
+public class AbfsAuthorizerConstants {
+
+ public static final String RENAME_SOURCE_ACTION = "rename-source";
+ public static final String RENAME_DESTINATION_ACTION = "rename-destination";
+
+ public static final String CONCAT_SOURCE_ACTION = "concat-source";
+ public static final String CONCAT_TARGET_ACTION = "concat-target";
+
+ public static final String CHECKACCESS_ACTION_PREFIX_ACTION = "access-";
+
+ public static final String LISTSTATUS_ACTION = "liststatus";
+ public static final String DELETE_ACTION = "delete";
+ public static final String CREATEFILE_ACTION = "create";
+ public static final String MKDIR_ACTION = "mkdir";
+ public static final String GETACL_ACTION = "getaclstatus";
+ public static final String GETFILESTATUS_ACTION = "getfilestatus";
+ public static final String SETACL_ACTION = "setacl"; // Modify, removeacl,
+ // setacl
+ public static final String SETOWNER_ACTION = "setowner";
+ public static final String SETPERMISSION_ACTION = "setpermission";
+ public static final String APPEND_ACTION = "write";
+ public static final String READ_ACTION = "read";
+ public static final String EXECUTE_ACTION = "execute";
+
+ /**
+ * Converts AbfsRestOperation to Authorizer action
Review comment:
nit: .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378936931
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsAuthorizerConstants.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidAbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperationType;
+
+/**
+ * Defines the string literals that need to be set to authorizer as action
Review comment:
nit: this and a lot of other javadocs need a "." at the end. Some javadoc compiler versions complain a lot when there's not one at the end of the sentence
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-593460073
ok,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ThomasMarquardt commented on a change in pull request
#1842: HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
ThomasMarquardt commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384665889
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -65,6 +66,21 @@ public void setup() throws Exception {
super.setup();
}
+ @Test
+ public void testSASTokenProviderInitializeException() throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+
+ final AzureBlobFileSystem testFs = new AzureBlobFileSystem();
+
+ MockSASTokenProvider.setThrowExceptionAtInit(true);
Review comment:
I think this will fail intermittently when the tests are run in parallel, which is how we usually run the tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r381992480
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -578,35 +585,37 @@ public AccessTokenProvider getTokenProvider() throws TokenAccessProviderExceptio
}
}
- public String getAbfsExternalAuthorizationClass() {
- return this.abfsExternalAuthorizationClass;
- }
-
- public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
- String authClassName = getAbfsExternalAuthorizationClass();
- AbfsAuthorizer authorizer = null;
+ public SASTokenProvider getSASTokenProvider() throws AzureBlobFileSystemException {
+ AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME, AuthType.SharedKey);
+ if (authType != AuthType.SAS) {
+ throw new SASTokenProviderException(String.format(
+ "Invalid auth type: %s is being used, expecting SAS", authType));
+ }
try {
- if (authClassName != null && !authClassName.isEmpty()) {
- @SuppressWarnings("unchecked")
- Class<AbfsAuthorizer> authClass = (Class<AbfsAuthorizer>) rawConfig.getClassByName(authClassName);
- authorizer = authClass.getConstructor(new Class[] {Configuration.class}).newInstance(rawConfig);
- LOG.trace("Initializing {}", authClassName);
- authorizer.init();
- LOG.trace("{} init complete", authClassName);
+ String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+ Class<? extends SASTokenProvider> sasTokenProviderClass =
+ getClass(configKey, null, SASTokenProvider.class);
+ if (sasTokenProviderClass == null) {
Review comment:
Preconditions.checkArgument does this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-591775214
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 40s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 16s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 30s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
| -0 :warning: | patch | 1m 15s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed |
| +1 :green_heart: | javac | 0m 26s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 4 new + 8 unchanged - 1 fixed = 12 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 30s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 40s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed |
| -1 :x: | findbugs | 0m 57s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 18s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 61m 35s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Nullcheck of sasToken at line 612 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsClient.appendSASTokenToQuery(String, String, AbfsUriQueryBuilder) At AbfsClient.java:612 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsClient.appendSASTokenToQuery(String, String, AbfsUriQueryBuilder) At AbfsClient.java:[line 612] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux ec90e2b8bf69 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 7dfa37e |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/15/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/15/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/15/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/15/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379621403
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsAuthorizerConstants.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidAbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperationType;
+
+/**
+ * Defines the string literals that need to be set to authorizer as action
Review comment:
Updated as suggested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378944874
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -294,18 +298,20 @@ private synchronized void writeCurrentBufferToService() throws IOException {
waitForTaskToComplete();
}
- final Future<Void> job = completionService.submit(new Callable<Void>() {
+ final Future<AuthorizationStatus> job = completionService.submit(new Callable<AuthorizationStatus>() {
Review comment:
just go to a java 8 lambda here rather than callable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382649934
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java
##########
@@ -189,6 +195,32 @@ public void teardown() throws Exception {
}
}
+
+ public void loadConfiguredFileSystem() throws Exception {
+ // disable auto-creation of filesystem
+ abfsConfig.setBoolean(AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION,
+ false);
+
+ // AbstractAbfsIntegrationTest always uses a new instance of FileSystem,
+ // need to disable that and force filesystem provided in test configs.
+ String[] authorityParts =
+ (new java.net.URI(rawConfig.get(FS_AZURE_CONTRACT_TEST_URI))).getRawAuthority().split(
Review comment:
just use `URI` as below
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382649460
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/SASGenerator.java
##########
@@ -0,0 +1,127 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.io.UnsupportedEncodingException;
+import java.time.format.DateTimeFormatter;
+import java.time.Instant;
+import java.time.ZoneId;
+import java.util.Locale;
+import javax.crypto.Mac;
+import javax.crypto.spec.SecretKeySpec;
+
+import org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants;
+import org.apache.hadoop.fs.azurebfs.services.AbfsUriQueryBuilder;
+
+/**
+ * Created by tmarq on 2/17/20.
Review comment:
replace with a summary of what it does
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378947082
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/UriUtils.java
##########
@@ -18,8 +18,17 @@
package org.apache.hadoop.fs.azurebfs.utils;
+import org.apache.http.client.utils.URIBuilder;
Review comment:
nit: move down
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-589618887
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 19s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 9s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 9s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 1m 1s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 59s | trunk passed |
| -0 :warning: | patch | 1m 18s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 33 new + 8 unchanged - 1 fixed = 41 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 24s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 26s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 57s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 64m 57s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 97361ab0a33e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 7f35676 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/9/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/9/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/9/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378956690
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -36,331 +40,485 @@
import static org.apache.hadoop.fs.permission.AclEntryScope.ACCESS;
import static org.apache.hadoop.fs.permission.AclEntryType.GROUP;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
-import static org.junit.Assume.assumeTrue;
/**
* Test Perform Authorization Check operation
*/
-public class ITestAzureBlobFileSystemAuthorization extends AbstractAbfsIntegrationTest {
-
- private static final Path TEST_READ_ONLY_FILE_PATH_0 = new Path(TEST_READ_ONLY_FILE_0);
- private static final Path TEST_READ_ONLY_FOLDER_PATH = new Path(TEST_READ_ONLY_FOLDER);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_0 = new Path(TEST_WRITE_ONLY_FILE_0);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_1 = new Path(TEST_WRITE_ONLY_FILE_1);
- private static final Path TEST_READ_WRITE_FILE_PATH_0 = new Path(TEST_READ_WRITE_FILE_0);
- private static final Path TEST_READ_WRITE_FILE_PATH_1 = new Path(TEST_READ_WRITE_FILE_1);
- private static final Path TEST_WRITE_ONLY_FOLDER_PATH = new Path(TEST_WRITE_ONLY_FOLDER);
- private static final Path TEST_WRITE_THEN_READ_ONLY_PATH = new Path(TEST_WRITE_THEN_READ_ONLY);
- private static final String TEST_AUTHZ_CLASS = "org.apache.hadoop.fs.azurebfs.extensions.MockAbfsAuthorizer";
+public class ITestAzureBlobFileSystemAuthorization
+ extends AbstractAbfsIntegrationTest {
+
+ private static final String TEST_READ_ONLY_FILE_PATH_PREFIX_0 =
+ TEST_READ_ONLY_FILE_0;
+ private static final String TEST_READ_ONLY_FILE_PATH_PREFIX_1 =
+ TEST_READ_ONLY_FILE_1;
+ private static final String TEST_READ_ONLY_FOLDER_PATH_PREFIX =
+ TEST_READ_ONLY_FOLDER;
+ private static final String TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 =
+ TEST_WRITE_ONLY_FILE_0;
+ private static final String TEST_WRITE_ONLY_FILE_PATH_PREFIX_1 =
+ TEST_WRITE_ONLY_FILE_1;
+ private static final String TEST_READ_WRITE_FILE_PATH_PREFIX_0 =
+ TEST_READ_WRITE_FILE_0;
+ private static final String TEST_READ_WRITE_FILE_PATH_PREFIX_1 =
+ TEST_READ_WRITE_FILE_1;
+ private static final String TEST_WRITE_ONLY_FOLDER_PATH_PREFIX =
+ TEST_WRITE_ONLY_FOLDER;
+ private static final String TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX =
+ TEST_WRITE_THEN_READ_ONLY;
+ private static final String TEST_AUTHZ_CLASS =
+ "org.apache.hadoop.fs" + ".azurebfs.extensions.MockAbfsAuthorizer";
private static final String TEST_USER = UUID.randomUUID().toString();
private static final String TEST_GROUP = UUID.randomUUID().toString();
private static final String BAR = UUID.randomUUID().toString();
+ @Rule
+ public TestName name = new TestName();
public ITestAzureBlobFileSystemAuthorization() throws Exception {
}
@Override
public void setup() throws Exception {
- this.getConfiguration().set(ConfigurationKeys.ABFS_EXTERNAL_AUTHORIZATION_CLASS, TEST_AUTHZ_CLASS);
+ boolean isHNSEnabled = this.getConfiguration().getBoolean(
+ TestConfigurationKeys.FS_AZURE_TEST_NAMESPACE_ENABLED_ACCOUNT, false);
+ Assume.assumeTrue(isHNSEnabled == true);
+ this.getConfiguration().setAbfsAuthorizerClass(TEST_AUTHZ_CLASS);
+ loadAuthorizer();
super.setup();
}
@Test
public void testOpenFileWithInvalidPath() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- intercept(IllegalArgumentException.class,
- ()-> {
- fs.open(new Path("")).close();
+ intercept(IllegalArgumentException.class, () -> {
+ fs.open(new Path("")).close();
});
}
@Test
public void testOpenFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_THEN_READ_ONLY_PATH).close();
- fs.open(TEST_WRITE_THEN_READ_ONLY_PATH).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX + name.getMethodName());
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.WRITE_MODE);
+ fs.create(testFilePath).close();
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.READ_MODE);
+ fs.open(testFilePath).close();
}
@Test
public void testOpenFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.open(TEST_WRITE_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
Review comment:
this is recurrent enough its worth having some method to take a prefix and return a path with the getMethodName attached
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379627813
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -36,331 +40,485 @@
import static org.apache.hadoop.fs.permission.AclEntryScope.ACCESS;
import static org.apache.hadoop.fs.permission.AclEntryType.GROUP;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
-import static org.junit.Assume.assumeTrue;
/**
* Test Perform Authorization Check operation
*/
-public class ITestAzureBlobFileSystemAuthorization extends AbstractAbfsIntegrationTest {
-
- private static final Path TEST_READ_ONLY_FILE_PATH_0 = new Path(TEST_READ_ONLY_FILE_0);
- private static final Path TEST_READ_ONLY_FOLDER_PATH = new Path(TEST_READ_ONLY_FOLDER);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_0 = new Path(TEST_WRITE_ONLY_FILE_0);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_1 = new Path(TEST_WRITE_ONLY_FILE_1);
- private static final Path TEST_READ_WRITE_FILE_PATH_0 = new Path(TEST_READ_WRITE_FILE_0);
- private static final Path TEST_READ_WRITE_FILE_PATH_1 = new Path(TEST_READ_WRITE_FILE_1);
- private static final Path TEST_WRITE_ONLY_FOLDER_PATH = new Path(TEST_WRITE_ONLY_FOLDER);
- private static final Path TEST_WRITE_THEN_READ_ONLY_PATH = new Path(TEST_WRITE_THEN_READ_ONLY);
- private static final String TEST_AUTHZ_CLASS = "org.apache.hadoop.fs.azurebfs.extensions.MockAbfsAuthorizer";
+public class ITestAzureBlobFileSystemAuthorization
+ extends AbstractAbfsIntegrationTest {
+
+ private static final String TEST_READ_ONLY_FILE_PATH_PREFIX_0 =
+ TEST_READ_ONLY_FILE_0;
+ private static final String TEST_READ_ONLY_FILE_PATH_PREFIX_1 =
+ TEST_READ_ONLY_FILE_1;
+ private static final String TEST_READ_ONLY_FOLDER_PATH_PREFIX =
+ TEST_READ_ONLY_FOLDER;
+ private static final String TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 =
+ TEST_WRITE_ONLY_FILE_0;
+ private static final String TEST_WRITE_ONLY_FILE_PATH_PREFIX_1 =
+ TEST_WRITE_ONLY_FILE_1;
+ private static final String TEST_READ_WRITE_FILE_PATH_PREFIX_0 =
+ TEST_READ_WRITE_FILE_0;
+ private static final String TEST_READ_WRITE_FILE_PATH_PREFIX_1 =
+ TEST_READ_WRITE_FILE_1;
+ private static final String TEST_WRITE_ONLY_FOLDER_PATH_PREFIX =
+ TEST_WRITE_ONLY_FOLDER;
+ private static final String TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX =
+ TEST_WRITE_THEN_READ_ONLY;
+ private static final String TEST_AUTHZ_CLASS =
+ "org.apache.hadoop.fs" + ".azurebfs.extensions.MockAbfsAuthorizer";
private static final String TEST_USER = UUID.randomUUID().toString();
private static final String TEST_GROUP = UUID.randomUUID().toString();
private static final String BAR = UUID.randomUUID().toString();
+ @Rule
+ public TestName name = new TestName();
public ITestAzureBlobFileSystemAuthorization() throws Exception {
}
@Override
public void setup() throws Exception {
- this.getConfiguration().set(ConfigurationKeys.ABFS_EXTERNAL_AUTHORIZATION_CLASS, TEST_AUTHZ_CLASS);
+ boolean isHNSEnabled = this.getConfiguration().getBoolean(
+ TestConfigurationKeys.FS_AZURE_TEST_NAMESPACE_ENABLED_ACCOUNT, false);
+ Assume.assumeTrue(isHNSEnabled == true);
+ this.getConfiguration().setAbfsAuthorizerClass(TEST_AUTHZ_CLASS);
+ loadAuthorizer();
super.setup();
}
@Test
public void testOpenFileWithInvalidPath() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- intercept(IllegalArgumentException.class,
- ()-> {
- fs.open(new Path("")).close();
+ intercept(IllegalArgumentException.class, () -> {
+ fs.open(new Path("")).close();
});
}
@Test
public void testOpenFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_THEN_READ_ONLY_PATH).close();
- fs.open(TEST_WRITE_THEN_READ_ONLY_PATH).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX + name.getMethodName());
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.WRITE_MODE);
+ fs.create(testFilePath).close();
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.READ_MODE);
+ fs.open(testFilePath).close();
}
@Test
public void testOpenFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.open(TEST_WRITE_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
Review comment:
Refactored
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379622749
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/AuthorizationResult.java
##########
@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.extensions;
+
+/**
+ * AuthorizationResult will be returned by Authorizer with store path URI and
+ * supported Auth token if the AuthorizationResource is found authorized for
+ * specified action
Review comment:
The intention was to make a copy of the values too. Have made the changes accordingly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384357453
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java
##########
@@ -189,6 +195,32 @@ public void teardown() throws Exception {
}
}
+
+ public void loadConfiguredFileSystem() throws Exception {
+ // disable auto-creation of filesystem
+ abfsConfig.setBoolean(AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION,
+ false);
+
+ // AbstractAbfsIntegrationTest always uses a new instance of FileSystem,
+ // need to disable that and force filesystem provided in test configs.
+ String[] authorityParts =
+ (new java.net.URI(rawConfig.get(FS_AZURE_CONTRACT_TEST_URI))).getRawAuthority().split(
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ThomasMarquardt commented on a change in pull request
#1842: HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
ThomasMarquardt commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384036257
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -550,6 +595,30 @@ public AbfsRestOperation checkAccess(String path, String rwx)
return op;
}
+ /**
+ * If configured for SAS AuthType, appends SAS token to queryBuilder
+ * @param path
+ * @param operation
+ * @param queryBuilder
+ * @throws SASTokenProviderException
+ */
+ private void appendSASTokenToQuery(String path, String operation, AbfsUriQueryBuilder queryBuilder) throws SASTokenProviderException {
+ if (this.authType == AuthType.SAS) {
+ try {
+ LOG.trace("Fetch SAS token for {} on {}", operation, path);
Review comment:
This is a hot path and we are calling LOG.trace twice (L608 and L612), is this expensive? Could you step thru in debugger to confirm this is a light weight Boolean check only, when the level is not TRACE?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379375362
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/AbfsAuthorizationException.java
##########
@@ -16,16 +16,12 @@
* limitations under the License.
*/
-package org.apache.hadoop.fs.azurebfs.extensions;
-
-import java.io.IOException;
+package org.apache.hadoop.fs.azurebfs.contracts.exceptions;
Review comment:
All ABFS related exceptions are in contracts, hence moved it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382643267
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -266,17 +299,29 @@ public AbfsRestOperation createPath(final String path, final boolean isFile, fin
return op;
}
- public AbfsRestOperation renamePath(final String source, final String destination, final String continuation)
+ public AbfsRestOperation renamePath(String source, final String destination, final String continuation)
throws AzureBlobFileSystemException {
final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
+ String sasToken = "";
+
+ if (authType == AuthType.SAS) {
+ final AbfsUriQueryBuilder queryBuilder = new AbfsUriQueryBuilder();
+ appendSASTokenToQuery(source, SASTokenProvider.RENAME_SOURCE_OPERATION, queryBuilder);
+ sasToken = queryBuilder.toString();
+ }
- final String encodedRenameSource = urlEncode(FORWARD_SLASH + this.getFileSystem() + source);
+ final String encodedRenameSource =
+ urlEncode(FORWARD_SLASH + this.getFileSystem() + source) + sasToken;
requestHeaders.add(new AbfsHttpHeader(X_MS_RENAME_SOURCE, encodedRenameSource));
requestHeaders.add(new AbfsHttpHeader(IF_NONE_MATCH, STAR));
final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder();
abfsUriQueryBuilder.addQuery(QUERY_PARAM_CONTINUATION, continuation);
+ if (authType == AuthType.SAS) {
Review comment:
this is replicated enough it should be its own method, e.g
```
maybeAppendSASToken(path, operation, querybuilder)
if (authType=SAS) {
appendSASTokenToQuery(path, SASTokenProvider.RENAME_SOURCE_OPERATION, queryBuilder);
}
```
and use wherever needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378961524
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/authentication/AuthorizationStatus.java
##########
@@ -0,0 +1,183 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.authentication;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResource;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult;
+
+import java.net.URI;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.HashMap;
+
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.DEFAULT_SAS_REFRESH_INTERVAL_BEFORE_EXPIRY;
+
+/**
+ * AuthorizationStatus maintains the status of Authorization and also SAS
+ * token if authorizer is providing it.
+ */
+public class AuthorizationStatus {
+ private boolean isAuthorized;
+ private HashMap<URI, SasTokenData> sasTokenMap;
+
+ public AuthorizationStatus() {
+ sasTokenMap = new HashMap<>();
+ }
+
+ /**
+ * Fetch the SAS token
+ *
+ * @return
+ */
+ public String getSasTokenQuery(URI storePathUri) {
+ if (sasTokenMap.containsKey(storePathUri)) {
+ SasTokenData sasTokenData = sasTokenMap.get(storePathUri);
+ if (isValidSas(sasTokenData)) {
+ return sasTokenData.sasToken;
+ }
+ }
+
+ return null;
+ }
+
+ /**
+ * Update authTokenMap
+ * Also update the refresh interval for each SAS token.
+ *
+ * @param authorizationResource
+ * @param authResult - Authorizer AuthorizationResult
+ */
+ public void setSasToken(AuthorizationResource[] authorizationResource,
+ AuthorizationResult authResult) throws AbfsAuthorizationException {
+
+ AuthorizationResourceResult[] resourceResult = authResult
+ .getAuthResourceResult();
+
+ int i = 0;
+ for (AuthorizationResourceResult singleResourceAuth : resourceResult) {
+ // First check if the requested resource matches the resource
+ // for which authToken is returned
+ AuthorizationResource authorizationRequestedForResource =
+ authorizationResource[i];
+ if ((singleResourceAuth == null) || (singleResourceAuth.storePathUri
+ == null) || (singleResourceAuth.authorizerAction == null)) {
+ throw new AbfsAuthorizationException("Invalid authorization response");
Review comment:
could a resource path go in here? Be useful later
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378954776
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java
##########
@@ -128,6 +134,46 @@ protected AbstractAbfsIntegrationTest() throws Exception {
}
+ public void loadAuthorizer() throws Exception {
+ this.authorizer = abfsConfig.getAbfsAuthorizer();
+
+ // if authorizerClass is configured, auto-creation of filesystem is disabled
+ if (authorizer == null) {
+ throw new InvalidConfigurationValueException(
+ "loadAuthorizer failed as " + "authorizer class is not configured");
+ }
+
+ // if authorizerClass is configured, auto-creation of filesystem is disabled
+ abfsConfig.setBoolean(AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION,
+ false);
+
+ // AbstractAbfsIntegrationTest always uses a new instance of FileSystem,
+ // need to disable that and force filesystem provided in test configs.
+ String[] authorityParts = authorityParts(
+ new java.net.URI(rawConfig.get(FS_AZURE_CONTRACT_TEST_URI)));
+ this.fileSystemName = authorityParts[0];
+
+ // Reset URL with configured filesystem
+ final String abfsUrl =
+ this.getFileSystemName() + "@" + this.getAccountName();
+ URI defaultUri = null;
+
+ try {
+ defaultUri = new URI(abfsScheme, abfsUrl, null, null, null);
+ } catch (Exception ex) {
Review comment:
no need to catch and wrap
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378949486
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -578,21 +587,30 @@ public AccessTokenProvider getTokenProvider() throws TokenAccessProviderExceptio
}
}
- public String getAbfsExternalAuthorizationClass() {
- return this.abfsExternalAuthorizationClass;
- }
+ public AbfsAuthorizer getAbfsAuthorizer() throws AzureBlobFileSystemException {
+ if (this.authorizer != null)
+ {
+ return this.authorizer;
+ }
- public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
- String authClassName = getAbfsExternalAuthorizationClass();
- AbfsAuthorizer authorizer = null;
+ String authClassName = this.abfsAuthorizerClass;
try {
if (authClassName != null && !authClassName.isEmpty()) {
@SuppressWarnings("unchecked")
Class<AbfsAuthorizer> authClass = (Class<AbfsAuthorizer>) rawConfig.getClassByName(authClassName);
- authorizer = authClass.getConstructor(new Class[] {Configuration.class}).newInstance(rawConfig);
+ this.authorizer =
+ authClass.getConstructor(new Class[] {Configuration.class}).newInstance(rawConfig);
LOG.trace("Initializing {}", authClassName);
- authorizer.init();
+ this.authorizer.init();
+ if ((this.authorizer.getAuthType() != AuthType.SAS) && (
+ this.authorizer.getAuthType() != AuthType.None)) {
+ throw new AbfsAuthorizationException(
Review comment:
why the nested exception?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378947799
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/AbfsAuthorizationException.java
##########
@@ -16,16 +16,12 @@
* limitations under the License.
*/
-package org.apache.hadoop.fs.azurebfs.extensions;
-
-import java.io.IOException;
+package org.apache.hadoop.fs.azurebfs.contracts.exceptions;
Review comment:
so this has moved. Any particular reason?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384882542
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -65,6 +66,21 @@ public void setup() throws Exception {
super.setup();
}
+ @Test
+ public void testSASTokenProviderInitializeException() throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+
+ final AzureBlobFileSystem testFs = new AzureBlobFileSystem();
+
+ MockSASTokenProvider.setThrowExceptionAtInit(true);
Review comment:
Updated PR to fix test failures. Re-ran tests and all passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382645295
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsUriQueryBuilder.java
##########
@@ -40,6 +41,10 @@ public void addQuery(final String name, final String value) {
}
}
+ public void setSASToken(final String sasToken) {
+ this.sasToken = sasToken;
Review comment:
is it a requirement that the token is unset? if so, add a Preconditions.checkState. if not, then not
similarly: any checks on token (not null/empty)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382641208
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -255,6 +283,11 @@ public AbfsRestOperation createPath(final String path, final boolean isFile, fin
final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder();
abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE, isFile ? FILE : DIRECTORY);
+ if (authType == AuthType.SAS) {
+ String operation = isFile ? SASTokenProvider.CREATEFILE_OPERATION : SASTokenProvider.MKDIR_OPERATION;
Review comment:
prefer
```
String operation = isFile
? SASTokenProvider.CREATEFILE_OPERATION
: SASTokenProvider.MKDIR_OPERATION;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-588688524
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 10s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 15s | trunk passed |
| +1 :green_heart: | compile | 0m 26s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 21s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 0m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 50s | trunk passed |
| -0 :warning: | patch | 1m 7s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 21s | the patch passed |
| +1 :green_heart: | javac | 0m 21s | the patch passed |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 37 new + 8 unchanged - 1 fixed = 45 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 24s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 13s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 63m 30s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b9d878c0655c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ec75071 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/8/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/8/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/8/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384890212
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -266,16 +295,23 @@ public AbfsRestOperation createPath(final String path, final boolean isFile, fin
return op;
}
- public AbfsRestOperation renamePath(final String source, final String destination, final String continuation)
+ public AbfsRestOperation renamePath(String source, final String destination, final String continuation)
throws AzureBlobFileSystemException {
final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
- final String encodedRenameSource = urlEncode(FORWARD_SLASH + this.getFileSystem() + source);
+ final AbfsUriQueryBuilder srcQueryBuilder = new AbfsUriQueryBuilder();
+ appendSASTokenToQuery(source, SASTokenProvider.RENAME_SOURCE_OPERATION, srcQueryBuilder);
+ String sasToken = srcQueryBuilder.toString();
+
+ final String encodedRenameSource =
+ urlEncode(FORWARD_SLASH + this.getFileSystem() + source) + sasToken;
+ LOG.trace("Rename source queryparam added {}", encodedRenameSource);
Review comment:
As per the recent Thomas's comment, will continue with the Auth check within appendSASTokenToQuery() method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378957220
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -36,331 +40,485 @@
import static org.apache.hadoop.fs.permission.AclEntryScope.ACCESS;
import static org.apache.hadoop.fs.permission.AclEntryType.GROUP;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
-import static org.junit.Assume.assumeTrue;
/**
* Test Perform Authorization Check operation
*/
-public class ITestAzureBlobFileSystemAuthorization extends AbstractAbfsIntegrationTest {
-
- private static final Path TEST_READ_ONLY_FILE_PATH_0 = new Path(TEST_READ_ONLY_FILE_0);
- private static final Path TEST_READ_ONLY_FOLDER_PATH = new Path(TEST_READ_ONLY_FOLDER);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_0 = new Path(TEST_WRITE_ONLY_FILE_0);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_1 = new Path(TEST_WRITE_ONLY_FILE_1);
- private static final Path TEST_READ_WRITE_FILE_PATH_0 = new Path(TEST_READ_WRITE_FILE_0);
- private static final Path TEST_READ_WRITE_FILE_PATH_1 = new Path(TEST_READ_WRITE_FILE_1);
- private static final Path TEST_WRITE_ONLY_FOLDER_PATH = new Path(TEST_WRITE_ONLY_FOLDER);
- private static final Path TEST_WRITE_THEN_READ_ONLY_PATH = new Path(TEST_WRITE_THEN_READ_ONLY);
- private static final String TEST_AUTHZ_CLASS = "org.apache.hadoop.fs.azurebfs.extensions.MockAbfsAuthorizer";
+public class ITestAzureBlobFileSystemAuthorization
+ extends AbstractAbfsIntegrationTest {
+
+ private static final String TEST_READ_ONLY_FILE_PATH_PREFIX_0 =
+ TEST_READ_ONLY_FILE_0;
+ private static final String TEST_READ_ONLY_FILE_PATH_PREFIX_1 =
+ TEST_READ_ONLY_FILE_1;
+ private static final String TEST_READ_ONLY_FOLDER_PATH_PREFIX =
+ TEST_READ_ONLY_FOLDER;
+ private static final String TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 =
+ TEST_WRITE_ONLY_FILE_0;
+ private static final String TEST_WRITE_ONLY_FILE_PATH_PREFIX_1 =
+ TEST_WRITE_ONLY_FILE_1;
+ private static final String TEST_READ_WRITE_FILE_PATH_PREFIX_0 =
+ TEST_READ_WRITE_FILE_0;
+ private static final String TEST_READ_WRITE_FILE_PATH_PREFIX_1 =
+ TEST_READ_WRITE_FILE_1;
+ private static final String TEST_WRITE_ONLY_FOLDER_PATH_PREFIX =
+ TEST_WRITE_ONLY_FOLDER;
+ private static final String TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX =
+ TEST_WRITE_THEN_READ_ONLY;
+ private static final String TEST_AUTHZ_CLASS =
+ "org.apache.hadoop.fs" + ".azurebfs.extensions.MockAbfsAuthorizer";
private static final String TEST_USER = UUID.randomUUID().toString();
private static final String TEST_GROUP = UUID.randomUUID().toString();
private static final String BAR = UUID.randomUUID().toString();
+ @Rule
+ public TestName name = new TestName();
public ITestAzureBlobFileSystemAuthorization() throws Exception {
}
@Override
public void setup() throws Exception {
- this.getConfiguration().set(ConfigurationKeys.ABFS_EXTERNAL_AUTHORIZATION_CLASS, TEST_AUTHZ_CLASS);
+ boolean isHNSEnabled = this.getConfiguration().getBoolean(
+ TestConfigurationKeys.FS_AZURE_TEST_NAMESPACE_ENABLED_ACCOUNT, false);
+ Assume.assumeTrue(isHNSEnabled == true);
+ this.getConfiguration().setAbfsAuthorizerClass(TEST_AUTHZ_CLASS);
+ loadAuthorizer();
super.setup();
}
@Test
public void testOpenFileWithInvalidPath() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- intercept(IllegalArgumentException.class,
- ()-> {
- fs.open(new Path("")).close();
+ intercept(IllegalArgumentException.class, () -> {
+ fs.open(new Path("")).close();
});
}
@Test
public void testOpenFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_THEN_READ_ONLY_PATH).close();
- fs.open(TEST_WRITE_THEN_READ_ONLY_PATH).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX + name.getMethodName());
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.WRITE_MODE);
+ fs.create(testFilePath).close();
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.READ_MODE);
+ fs.open(testFilePath).close();
}
@Test
public void testOpenFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.open(TEST_WRITE_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ fs.create(testFilePath).close();
+ intercept(AbfsAuthorizationException.class, () -> {
+ fs.open(testFilePath).close();
});
}
@Test
public void testCreateFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ fs.create(testFilePath).close();
}
@Test
public void testCreateFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.create(TEST_READ_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_READ_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ intercept(AbfsAuthorizationException.class, () -> {
Review comment:
might be good to see when the check fails: in the create or in the close?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384889428
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsUriQueryBuilder.java
##########
@@ -59,6 +64,20 @@ public String toString() {
throw new IllegalArgumentException("Query string param is not encode-able: " + entry.getKey() + "=" + entry.getValue());
}
}
+ // append SAS Token
+ if (sasToken != null) {
+ sasToken = sasToken.startsWith(AbfsHttpConstants.QUESTION_MARK)
+ ? sasToken.substring(1)
+ : sasToken;
Review comment:
Removed the handling of starting '?' in SAS Token query.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378943399
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -22,13 +22,21 @@
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URISyntaxException;
import java.net.URL;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.authentication.AuthorizationStatus;
Review comment:
think these should go next on one of existing azurebfs blocks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378952218
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -1131,20 +1142,38 @@ private void initializeClient(URI uri, String fileSystemName, String accountName
SharedKeyCredentials creds = null;
AccessTokenProvider tokenProvider = null;
- if (abfsConfiguration.getAuthType(accountName) == AuthType.SharedKey) {
- LOG.trace("Fetching SharedKey credentials");
- int dotIndex = accountName.indexOf(AbfsHttpConstants.DOT);
- if (dotIndex <= 0) {
- throw new InvalidUriException(
- uri.toString() + " - account name is not fully qualified.");
- }
- creds = new SharedKeyCredentials(accountName.substring(0, dotIndex),
+ AuthType authType = abfsConfiguration.getAuthType(accountName);
+ AbfsAuthorizer authorizer = abfsConfiguration.getAbfsAuthorizer();
+
+ // Create cred instance based on account level Auth config if
+ // - authorizer is not configured
+ // - or authroizer configured will not provide SAS
Review comment:
typo
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran edited a comment on issue #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran edited a comment on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-593460073
closing as it merged in. thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379374836
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsAuthorizerConstants.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidAbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperationType;
+
+/**
+ * Defines the string literals that need to be set to authorizer as action
+ */
+public class AbfsAuthorizerConstants {
+
+ public static final String RENAME_SOURCE_ACTION = "rename-source";
+ public static final String RENAME_DESTINATION_ACTION = "rename-destination";
+
+ public static final String CONCAT_SOURCE_ACTION = "concat-source";
+ public static final String CONCAT_TARGET_ACTION = "concat-target";
+
+ public static final String CHECKACCESS_ACTION_PREFIX_ACTION = "access-";
+
+ public static final String LISTSTATUS_ACTION = "liststatus";
+ public static final String DELETE_ACTION = "delete";
+ public static final String CREATEFILE_ACTION = "create";
+ public static final String MKDIR_ACTION = "mkdir";
+ public static final String GETACL_ACTION = "getaclstatus";
+ public static final String GETFILESTATUS_ACTION = "getfilestatus";
+ public static final String SETACL_ACTION = "setacl"; // Modify, removeacl,
+ // setacl
+ public static final String SETOWNER_ACTION = "setowner";
+ public static final String SETPERMISSION_ACTION = "setpermission";
+ public static final String APPEND_ACTION = "write";
+ public static final String READ_ACTION = "read";
+ public static final String EXECUTE_ACTION = "execute";
+
+ /**
+ * Converts AbfsRestOperation to Authorizer action
+ * @param opType
+ * @return
+ */
+ public static String getAction(AbfsRestOperationType opType)
+ throws InvalidAbfsRestOperationException {
+ switch (opType) {
+ case ListPaths:
+ return LISTSTATUS_ACTION;
+ case RenamePath:
+ return RENAME_DESTINATION_ACTION;
+ case GetAcl:
+ return GETACL_ACTION;
+ case GetPathStatus:
+ return GETFILESTATUS_ACTION;
+ case SetAcl:
+ return SETACL_ACTION;
+ case SetOwner:
+ return SETOWNER_ACTION;
+ case SetPermissions:
+ return SETPERMISSION_ACTION;
+ case Append:
+ case Flush:
+ return APPEND_ACTION;
+ case ReadFile:
+ return READ_ACTION;
+ case DeletePath:
+ return DELETE_ACTION;
+ case CreatePath:
+ return CREATEFILE_ACTION;
+ case Mkdir:
+ return MKDIR_ACTION;
+ default:
+ throw new InvalidAbfsRestOperationException(
Review comment:
Seems like the right place to check for invalid operation type. Retaining it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378948078
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -496,6 +497,14 @@ public boolean shouldTrackLatency() {
return this.trackLatency;
}
+ /**
+ * Return the accountName for the config generated
Review comment:
nit .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379623925
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -496,6 +497,14 @@ public boolean shouldTrackLatency() {
return this.trackLatency;
}
+ /**
+ * Return the accountName for the config generated
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379378008
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -294,18 +298,20 @@ private synchronized void writeCurrentBufferToService() throws IOException {
waitForTaskToComplete();
}
- final Future<Void> job = completionService.submit(new Callable<Void>() {
+ final Future<AuthorizationStatus> job = completionService.submit(new Callable<AuthorizationStatus>() {
Review comment:
Will retain current construct to support backporting actions to older hadoop versions still on Java 7.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r383185606
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -18,20 +18,23 @@
package org.apache.hadoop.fs.azurebfs;
+import java.io.*;
import java.util.Arrays;
import java.util.List;
import java.util.UUID;
+import org.apache.hadoop.fs.azurebfs.constants.*;
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379624778
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -1131,20 +1142,38 @@ private void initializeClient(URI uri, String fileSystemName, String accountName
SharedKeyCredentials creds = null;
AccessTokenProvider tokenProvider = null;
- if (abfsConfiguration.getAuthType(accountName) == AuthType.SharedKey) {
- LOG.trace("Fetching SharedKey credentials");
- int dotIndex = accountName.indexOf(AbfsHttpConstants.DOT);
- if (dotIndex <= 0) {
- throw new InvalidUriException(
- uri.toString() + " - account name is not fully qualified.");
- }
- creds = new SharedKeyCredentials(accountName.substring(0, dotIndex),
+ AuthType authType = abfsConfiguration.getAuthType(accountName);
+ AbfsAuthorizer authorizer = abfsConfiguration.getAbfsAuthorizer();
+
+ // Create cred instance based on account level Auth config if
+ // - authorizer is not configured
+ // - or authroizer configured will not provide SAS
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r383185517
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/SASGenerator.java
##########
@@ -0,0 +1,127 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.io.UnsupportedEncodingException;
+import java.time.format.DateTimeFormatter;
+import java.time.Instant;
+import java.time.ZoneId;
+import java.util.Locale;
+import javax.crypto.Mac;
+import javax.crypto.spec.SecretKeySpec;
+
+import org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants;
+import org.apache.hadoop.fs.azurebfs.services.AbfsUriQueryBuilder;
+
+/**
+ * Created by tmarq on 2/17/20.
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382510841
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/AbfsAuthorizationException.java
##########
@@ -16,16 +16,12 @@
* limitations under the License.
*/
-package org.apache.hadoop.fs.azurebfs.extensions;
-
-import java.io.IOException;
+package org.apache.hadoop.fs.azurebfs.contracts.exceptions;
Review comment:
Code is modified. Resolving this comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-589627591
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 23s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 11s | trunk passed |
| +1 :green_heart: | compile | 0m 27s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 31s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed |
| +0 :ok: | spotbugs | 0m 50s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 47s | trunk passed |
| -0 :warning: | patch | 1m 5s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 22s | the patch passed |
| +1 :green_heart: | javac | 0m 22s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 32 new + 8 unchanged - 1 fixed = 40 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 42s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 64m 14s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 7cf1d8653ae7 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 7f35676 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/10/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/10/testReport/ |
| Max. process+thread count | 310 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/10/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379621742
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/AbfsAuthorizer.java
##########
@@ -18,40 +18,50 @@
package org.apache.hadoop.fs.azurebfs.extensions;
-import java.io.IOException;
-
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizerUnhandledException;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+
+import java.io.IOException;
Review comment:
Updated as suggested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378960678
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/authentication/AuthorizationStatus.java
##########
@@ -0,0 +1,183 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.authentication;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResource;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult;
+
+import java.net.URI;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.HashMap;
+
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.DEFAULT_SAS_REFRESH_INTERVAL_BEFORE_EXPIRY;
+
+/**
+ * AuthorizationStatus maintains the status of Authorization and also SAS
+ * token if authorizer is providing it.
+ */
+public class AuthorizationStatus {
+ private boolean isAuthorized;
+ private HashMap<URI, SasTokenData> sasTokenMap;
Review comment:
Declare type as `Map<URI, SasTokenData>` and mark final.
What is the synchronization policy here if a set of sas tokens are being shared. Should it be some concurrent hash map?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-586260074
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 30m 10s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 5s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 12s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 50s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 48 new + 9 unchanged - 0 fixed = 57 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 9s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 52s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 16s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 92m 18s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a2f5c08256bb 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 954930e |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/4/testReport/ |
| Max. process+thread count | 360 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-589618887
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 19s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 9s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 9s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 1m 1s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 59s | trunk passed |
| -0 :warning: | patch | 1m 18s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 33 new + 8 unchanged - 1 fixed = 41 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 24s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 26s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 57s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 64m 57s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 97361ab0a33e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 7f35676 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/9/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/9/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/9/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384442045
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsUriQueryBuilder.java
##########
@@ -59,6 +64,20 @@ public String toString() {
throw new IllegalArgumentException("Query string param is not encode-able: " + entry.getKey() + "=" + entry.getValue());
}
}
+ // append SAS Token
+ if (sasToken != null) {
+ sasToken = sasToken.startsWith(AbfsHttpConstants.QUESTION_MARK)
+ ? sasToken.substring(1)
+ : sasToken;
Review comment:
Uribuilder toString() will auto add '?' at the start. Like in the case of the test SASGenerator code added to this PR, the SAS token generator would end up having a '?'. SAS token being a query URL it might be better to handle to presence or absence for '?' in the received token ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379627975
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -36,331 +40,485 @@
import static org.apache.hadoop.fs.permission.AclEntryScope.ACCESS;
import static org.apache.hadoop.fs.permission.AclEntryType.GROUP;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
-import static org.junit.Assume.assumeTrue;
/**
* Test Perform Authorization Check operation
*/
-public class ITestAzureBlobFileSystemAuthorization extends AbstractAbfsIntegrationTest {
-
- private static final Path TEST_READ_ONLY_FILE_PATH_0 = new Path(TEST_READ_ONLY_FILE_0);
- private static final Path TEST_READ_ONLY_FOLDER_PATH = new Path(TEST_READ_ONLY_FOLDER);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_0 = new Path(TEST_WRITE_ONLY_FILE_0);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_1 = new Path(TEST_WRITE_ONLY_FILE_1);
- private static final Path TEST_READ_WRITE_FILE_PATH_0 = new Path(TEST_READ_WRITE_FILE_0);
- private static final Path TEST_READ_WRITE_FILE_PATH_1 = new Path(TEST_READ_WRITE_FILE_1);
- private static final Path TEST_WRITE_ONLY_FOLDER_PATH = new Path(TEST_WRITE_ONLY_FOLDER);
- private static final Path TEST_WRITE_THEN_READ_ONLY_PATH = new Path(TEST_WRITE_THEN_READ_ONLY);
- private static final String TEST_AUTHZ_CLASS = "org.apache.hadoop.fs.azurebfs.extensions.MockAbfsAuthorizer";
+public class ITestAzureBlobFileSystemAuthorization
+ extends AbstractAbfsIntegrationTest {
+
+ private static final String TEST_READ_ONLY_FILE_PATH_PREFIX_0 =
+ TEST_READ_ONLY_FILE_0;
+ private static final String TEST_READ_ONLY_FILE_PATH_PREFIX_1 =
+ TEST_READ_ONLY_FILE_1;
+ private static final String TEST_READ_ONLY_FOLDER_PATH_PREFIX =
+ TEST_READ_ONLY_FOLDER;
+ private static final String TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 =
+ TEST_WRITE_ONLY_FILE_0;
+ private static final String TEST_WRITE_ONLY_FILE_PATH_PREFIX_1 =
+ TEST_WRITE_ONLY_FILE_1;
+ private static final String TEST_READ_WRITE_FILE_PATH_PREFIX_0 =
+ TEST_READ_WRITE_FILE_0;
+ private static final String TEST_READ_WRITE_FILE_PATH_PREFIX_1 =
+ TEST_READ_WRITE_FILE_1;
+ private static final String TEST_WRITE_ONLY_FOLDER_PATH_PREFIX =
+ TEST_WRITE_ONLY_FOLDER;
+ private static final String TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX =
+ TEST_WRITE_THEN_READ_ONLY;
+ private static final String TEST_AUTHZ_CLASS =
+ "org.apache.hadoop.fs" + ".azurebfs.extensions.MockAbfsAuthorizer";
private static final String TEST_USER = UUID.randomUUID().toString();
private static final String TEST_GROUP = UUID.randomUUID().toString();
private static final String BAR = UUID.randomUUID().toString();
+ @Rule
+ public TestName name = new TestName();
public ITestAzureBlobFileSystemAuthorization() throws Exception {
}
@Override
public void setup() throws Exception {
- this.getConfiguration().set(ConfigurationKeys.ABFS_EXTERNAL_AUTHORIZATION_CLASS, TEST_AUTHZ_CLASS);
+ boolean isHNSEnabled = this.getConfiguration().getBoolean(
+ TestConfigurationKeys.FS_AZURE_TEST_NAMESPACE_ENABLED_ACCOUNT, false);
+ Assume.assumeTrue(isHNSEnabled == true);
+ this.getConfiguration().setAbfsAuthorizerClass(TEST_AUTHZ_CLASS);
+ loadAuthorizer();
super.setup();
}
@Test
public void testOpenFileWithInvalidPath() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- intercept(IllegalArgumentException.class,
- ()-> {
- fs.open(new Path("")).close();
+ intercept(IllegalArgumentException.class, () -> {
+ fs.open(new Path("")).close();
});
}
@Test
public void testOpenFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_THEN_READ_ONLY_PATH).close();
- fs.open(TEST_WRITE_THEN_READ_ONLY_PATH).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX + name.getMethodName());
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.WRITE_MODE);
+ fs.create(testFilePath).close();
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.READ_MODE);
+ fs.open(testFilePath).close();
}
@Test
public void testOpenFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.open(TEST_WRITE_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ fs.create(testFilePath).close();
+ intercept(AbfsAuthorizationException.class, () -> {
+ fs.open(testFilePath).close();
});
}
@Test
public void testCreateFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ fs.create(testFilePath).close();
}
@Test
public void testCreateFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.create(TEST_READ_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_READ_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ intercept(AbfsAuthorizationException.class, () -> {
Review comment:
Its create, removed the additional close call.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379628733
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/authentication/AuthorizationStatus.java
##########
@@ -0,0 +1,183 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.authentication;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResource;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult;
+
+import java.net.URI;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.HashMap;
+
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.DEFAULT_SAS_REFRESH_INTERVAL_BEFORE_EXPIRY;
+
+/**
+ * AuthorizationStatus maintains the status of Authorization and also SAS
+ * token if authorizer is providing it.
+ */
+public class AuthorizationStatus {
+ private boolean isAuthorized;
+ private HashMap<URI, SasTokenData> sasTokenMap;
Review comment:
AuthorizationStatus instance is saved at a time by the Abfs inpt or output stream, which will be operating one I/O operation at a time.
As the instance is not shared at Abfs filesystem level across requests, concurrency doesnt need handling,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on issue #1842: HADOOP-16730 : ABFS:
Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-586471534
@steveloughran - Thanks for taking time to review this change. I have addressed almost all the comments and the ones I havent response is added in PR.
Re-ran the tests again with the current new changes and all the tests passed.
Can you please have a look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r381986150
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsUriQueryBuilder.java
##########
@@ -59,6 +64,20 @@ public String toString() {
throw new IllegalArgumentException("Query string param is not encode-able: " + entry.getKey() + "=" + entry.getValue());
}
}
+ // append SAS Token
+ if (sasToken != null) {
+ sasToken =
+ sasToken.startsWith(AbfsHttpConstants.QUESTION_MARK) ?
Review comment:
style nit, preferred layout is
```
predicate
? res1;
: res2;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-585379660
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 18s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 41s | trunk passed |
| +1 :green_heart: | compile | 0m 27s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 35s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 1m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 40 new + 9 unchanged - 0 fixed = 49 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 22s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 21s | hadoop-tools_hadoop-azure generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
| -1 :x: | findbugs | 0m 56s | hadoop-tools/hadoop-azure generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 64m 52s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Unused public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.authorizerAction In AuthorizationResourceResult.java |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.authToken At AuthorizationStatus.java:[line 78] |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.storePathUri At AuthorizationStatus.java:[line 91] |
| | org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult.getAuthResourceResult() may expose internal representation by returning AuthorizationResult.authResourceResult At AuthorizationResult.java:by returning AuthorizationResult.authResourceResult At AuthorizationResult.java:[line 40] |
| | org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult.setAuthResourceResult(AuthorizationResourceResult[]) may expose internal representation by storing an externally mutable object into AuthorizationResult.authResourceResult At AuthorizationResult.java:by storing an externally mutable object into AuthorizationResult.authResourceResult At AuthorizationResult.java:[line 45] |
| | Possible null pointer dereference of relativePath in org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(String, boolean, int, String) Dereferenced at AbfsClient.java:relativePath in org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(String, boolean, int, String) Dereferenced at AbfsClient.java:[line 225] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2cd01f55bef9 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / f09710b |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/2/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/2/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/2/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r381986574
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -39,6 +38,7 @@
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException;
Review comment:
next block please; imports are merge hell and we need to stay in control.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ThomasMarquardt commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
ThomasMarquardt commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-591747607
LGTM, +1
Tests run: 52, Failures: 0, Errors: 0, Skipped: 0
Tests run: 411, Failures: 0, Errors: 0, Skipped: 62
Tests run: 200, Failures: 0, Errors: 0, Skipped: 24
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379838208
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsAuthorizerConstants.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidAbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperationType;
+
+/**
+ * Defines the string literals that need to be set to authorizer as action
+ */
+public class AbfsAuthorizerConstants {
+
+ public static final String RENAME_SOURCE_ACTION = "rename-source";
+ public static final String RENAME_DESTINATION_ACTION = "rename-destination";
+
+ public static final String CONCAT_SOURCE_ACTION = "concat-source";
+ public static final String CONCAT_TARGET_ACTION = "concat-target";
+
+ public static final String CHECKACCESS_ACTION_PREFIX_ACTION = "access-";
+
+ public static final String LISTSTATUS_ACTION = "liststatus";
+ public static final String DELETE_ACTION = "delete";
+ public static final String CREATEFILE_ACTION = "create";
+ public static final String MKDIR_ACTION = "mkdir";
+ public static final String GETACL_ACTION = "getaclstatus";
+ public static final String GETFILESTATUS_ACTION = "getfilestatus";
+ public static final String SETACL_ACTION = "setacl"; // Modify, removeacl,
+ // setacl
+ public static final String SETOWNER_ACTION = "setowner";
+ public static final String SETPERMISSION_ACTION = "setpermission";
+ public static final String APPEND_ACTION = "write";
+ public static final String READ_ACTION = "read";
+ public static final String EXECUTE_ACTION = "execute";
+
+ /**
+ * Converts AbfsRestOperation to Authorizer action
+ * @param opType
+ * @return
+ */
+ public static String getAction(AbfsRestOperationType opType)
+ throws InvalidAbfsRestOperationException {
+ switch (opType) {
+ case ListPaths:
+ return LISTSTATUS_ACTION;
+ case RenamePath:
+ return RENAME_DESTINATION_ACTION;
+ case GetAcl:
+ return GETACL_ACTION;
+ case GetPathStatus:
+ return GETFILESTATUS_ACTION;
+ case SetAcl:
+ return SETACL_ACTION;
+ case SetOwner:
+ return SETOWNER_ACTION;
+ case SetPermissions:
+ return SETPERMISSION_ACTION;
+ case Append:
+ case Flush:
+ return APPEND_ACTION;
+ case ReadFile:
+ return READ_ACTION;
+ case DeletePath:
+ return DELETE_ACTION;
+ case CreatePath:
+ return CREATEFILE_ACTION;
+ case Mkdir:
+ return MKDIR_ACTION;
+ default:
+ throw new InvalidAbfsRestOperationException(
Review comment:
it was more the double exception I was referring to. Is this really needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on issue #1842: HADOOP-16730 : ABFS:
Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-590257367
@steveloughran - Have updated the PR with fixes for comments added. Kindly request you to take a look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r383183882
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -266,17 +299,29 @@ public AbfsRestOperation createPath(final String path, final boolean isFile, fin
return op;
}
- public AbfsRestOperation renamePath(final String source, final String destination, final String continuation)
+ public AbfsRestOperation renamePath(String source, final String destination, final String continuation)
throws AzureBlobFileSystemException {
final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
+ String sasToken = "";
+
+ if (authType == AuthType.SAS) {
+ final AbfsUriQueryBuilder queryBuilder = new AbfsUriQueryBuilder();
+ appendSASTokenToQuery(source, SASTokenProvider.RENAME_SOURCE_OPERATION, queryBuilder);
+ sasToken = queryBuilder.toString();
+ }
- final String encodedRenameSource = urlEncode(FORWARD_SLASH + this.getFileSystem() + source);
+ final String encodedRenameSource =
+ urlEncode(FORWARD_SLASH + this.getFileSystem() + source) + sasToken;
requestHeaders.add(new AbfsHttpHeader(X_MS_RENAME_SOURCE, encodedRenameSource));
requestHeaders.add(new AbfsHttpHeader(IF_NONE_MATCH, STAR));
final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder();
abfsUriQueryBuilder.addQuery(QUERY_PARAM_CONTINUATION, continuation);
+ if (authType == AuthType.SAS) {
Review comment:
Moved the authType check to appendSASToken method, though I have not renamed the method that adds the SAS token. Looks fine the way it is with the check for authType done within.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379623419
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -351,14 +358,21 @@ private synchronized void flushWrittenBytesToServiceInternal(final long offset,
AbfsPerfTracker tracker = client.getAbfsPerfTracker();
try (AbfsPerfInfo perfInfo = new AbfsPerfInfo(tracker,
"flushWrittenBytesToServiceInternal", "flush")) {
- AbfsRestOperation op = client.flush(path, offset, retainUncommitedData, isClose);
+ AbfsRestOperation op = client.flush(path, offset, retainUncommitedData,
+ isClose, this.authzStatus);
perfInfo.registerResult(op.getResult()).registerSuccess(true);
} catch (AzureBlobFileSystemException ex) {
if (ex instanceof AbfsRestOperationException) {
if (((AbfsRestOperationException) ex).getStatusCode() == HttpURLConnection.HTTP_NOT_FOUND) {
throw new FileNotFoundException(ex.getMessage());
}
}
+
+ if ((ex instanceof AbfsAuthorizationException)
Review comment:
Updated as suggested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379621067
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/authentication/AuthorizationStatus.java
##########
@@ -0,0 +1,183 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.authentication;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResource;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult;
+
+import java.net.URI;
Review comment:
Updated as suggested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379356660
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -604,9 +622,10 @@ public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
| NoSuchMethodException
| SecurityException
| AbfsAuthorizationException e) {
- throw new IOException(e);
+ throw new AbfsAuthorizationException("Unable to initialize "
Review comment:
Re-throwing as AbfsAuthorizerException with the actual exception as the innerexception. Expect to quickly narrow down the issue at authorizer level.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384442099
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/abfs.md
##########
@@ -626,7 +626,7 @@ points for third-parties to integrate their authentication and authorization
services into the ABFS client.
* `CustomDelegationTokenManager` : adds ability to issue Hadoop Delegation Tokens.
-* `AbfsAuthorizer` permits client-side authorization of file operations.
+* `SASTokenProvider` permits client-side authorization of file operations.
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378953565
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/extensions/MockAbfsAuthorizer.java
##########
@@ -41,47 +55,216 @@
public static final String TEST_WRITE_ONLY_FOLDER = "writeOnlyFolder";
public static final String TEST_READ_WRITE_FILE_0 = "readWriteFile0";
public static final String TEST_READ_WRITE_FILE_1 = "readWriteFile1";
- public static final String TEST_WRITE_THEN_READ_ONLY = "writeThenReadOnlyFile";
+ public static final String TEST_WRITE_THEN_READ_ONLY =
+ "writeThenReadOnlyFile";
+ private static final Set<String> apiAuthorizerActions = new HashSet<String>();
+ public String accountName;
private Configuration conf;
- private Set<Path> readOnlyPaths = new HashSet<Path>();
- private Set<Path> writeOnlyPaths = new HashSet<Path>();
- private Set<Path> readWritePaths = new HashSet<Path>();
- private int writeThenReadOnly = 0;
+ private Set<String> readOnlyPathsPrefixes = new HashSet<String>();
Review comment:
same on all of these; no need to be explicit on types any more
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-586260074
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 30m 10s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 5s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 12s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 50s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 48 new + 9 unchanged - 0 fixed = 57 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 9s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 52s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 16s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 92m 18s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a2f5c08256bb 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 954930e |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/4/testReport/ |
| Max. process+thread count | 360 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378953129
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/extensions/MockAbfsAuthorizer.java
##########
@@ -18,21 +18,35 @@
package org.apache.hadoop.fs.azurebfs.extensions;
-import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
Review comment:
keep under the java. imports
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378940654
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/AuthorizationResourceResult.java
##########
@@ -0,0 +1,31 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.extensions;
+
+import java.net.URI;
+
+/**
+ * AuthorizationResourceResult array will be included in the
+ * AuthorizationResult with results for each resource.
+ */
+public class AuthorizationResourceResult {
Review comment:
final class, final fields?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378962797
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsAuthorizerConstants.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidAbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperationType;
+
+/**
+ * Defines the string literals that need to be set to authorizer as action
+ */
+public class AbfsAuthorizerConstants {
+
+ public static final String RENAME_SOURCE_ACTION = "rename-source";
+ public static final String RENAME_DESTINATION_ACTION = "rename-destination";
+
+ public static final String CONCAT_SOURCE_ACTION = "concat-source";
+ public static final String CONCAT_TARGET_ACTION = "concat-target";
+
+ public static final String CHECKACCESS_ACTION_PREFIX_ACTION = "access-";
+
+ public static final String LISTSTATUS_ACTION = "liststatus";
+ public static final String DELETE_ACTION = "delete";
+ public static final String CREATEFILE_ACTION = "create";
+ public static final String MKDIR_ACTION = "mkdir";
+ public static final String GETACL_ACTION = "getaclstatus";
+ public static final String GETFILESTATUS_ACTION = "getfilestatus";
+ public static final String SETACL_ACTION = "setacl"; // Modify, removeacl,
Review comment:
comments here a bit confusing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on issue #1842: HADOOP-16730 : ABFS:
Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-585350897
Test results for account without namespace enabled (East US 2):
[INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Tests run: 415, Failures: 0, Errors: 0, Skipped: 244
[WARNING] Tests run: 194, Failures: 0, Errors: 0, Skipped: 128
Test results for account with namespace enabled (East US 2):
[ERROR] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Tests run: 415, Failures: 0, Errors: 0, Skipped: 36
[WARNING] Tests run: 194, Failures: 0, Errors: 0, Skipped: 128
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-591766974
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 11s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 37s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | trunk passed |
| +0 :ok: | spotbugs | 0m 54s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 51s | trunk passed |
| -0 :warning: | patch | 1m 9s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 4 new + 8 unchanged - 1 fixed = 12 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 18s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| -1 :x: | findbugs | 0m 58s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 23s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
| | | 59m 58s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Nullcheck of sasToken at line 613 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsClient.appendSASTokenToQuery(String, String, AbfsUriQueryBuilder) At AbfsClient.java:613 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsClient.appendSASTokenToQuery(String, String, AbfsUriQueryBuilder) At AbfsClient.java:[line 613] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 6c9ba47bae66 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 7dfa37e |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/14/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/14/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/14/testReport/ |
| Max. process+thread count | 454 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/14/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378963558
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -128,7 +128,8 @@ public static String accountProperty(String property, String account) {
public static final String FS_AZURE_ENABLE_DELEGATION_TOKEN = "fs.azure.enable.delegation.token";
public static final String FS_AZURE_DELEGATION_TOKEN_PROVIDER_TYPE = "fs.azure.delegation.token.provider.type";
- public static final String ABFS_EXTERNAL_AUTHORIZATION_CLASS = "abfs.external.authorization.class";
+ /** External Authorizer **/
Review comment:
nit: .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382645746
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsUriQueryBuilder.java
##########
@@ -59,6 +64,20 @@ public String toString() {
throw new IllegalArgumentException("Query string param is not encode-able: " + entry.getKey() + "=" + entry.getValue());
}
}
+ // append SAS Token
+ if (sasToken != null) {
+ sasToken = sasToken.startsWith(AbfsHttpConstants.QUESTION_MARK) ?
Review comment:
nit: move the ? down to the start ot the next line, just to make it slightly more visible
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378962275
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/authentication/SasTokenData.java
##########
@@ -0,0 +1,27 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.authentication;
+
+import java.time.Instant;
+
+public class SasTokenData {
Review comment:
`final`. And add some minimal javadoc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378963347
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsAuthorizerConstants.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidAbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperationType;
+
+/**
+ * Defines the string literals that need to be set to authorizer as action
+ */
+public class AbfsAuthorizerConstants {
+
+ public static final String RENAME_SOURCE_ACTION = "rename-source";
+ public static final String RENAME_DESTINATION_ACTION = "rename-destination";
+
+ public static final String CONCAT_SOURCE_ACTION = "concat-source";
+ public static final String CONCAT_TARGET_ACTION = "concat-target";
+
+ public static final String CHECKACCESS_ACTION_PREFIX_ACTION = "access-";
+
+ public static final String LISTSTATUS_ACTION = "liststatus";
+ public static final String DELETE_ACTION = "delete";
+ public static final String CREATEFILE_ACTION = "create";
+ public static final String MKDIR_ACTION = "mkdir";
+ public static final String GETACL_ACTION = "getaclstatus";
+ public static final String GETFILESTATUS_ACTION = "getfilestatus";
+ public static final String SETACL_ACTION = "setacl"; // Modify, removeacl,
+ // setacl
+ public static final String SETOWNER_ACTION = "setowner";
+ public static final String SETPERMISSION_ACTION = "setpermission";
+ public static final String APPEND_ACTION = "write";
+ public static final String READ_ACTION = "read";
+ public static final String EXECUTE_ACTION = "execute";
+
+ /**
+ * Converts AbfsRestOperation to Authorizer action
+ * @param opType
+ * @return
+ */
+ public static String getAction(AbfsRestOperationType opType)
+ throws InvalidAbfsRestOperationException {
+ switch (opType) {
+ case ListPaths:
+ return LISTSTATUS_ACTION;
+ case RenamePath:
+ return RENAME_DESTINATION_ACTION;
+ case GetAcl:
+ return GETACL_ACTION;
+ case GetPathStatus:
+ return GETFILESTATUS_ACTION;
+ case SetAcl:
+ return SETACL_ACTION;
+ case SetOwner:
+ return SETOWNER_ACTION;
+ case SetPermissions:
+ return SETPERMISSION_ACTION;
+ case Append:
+ case Flush:
+ return APPEND_ACTION;
+ case ReadFile:
+ return READ_ACTION;
+ case DeletePath:
+ return DELETE_ACTION;
+ case CreatePath:
+ return CREATEFILE_ACTION;
+ case Mkdir:
+ return MKDIR_ACTION;
+ default:
+ throw new InvalidAbfsRestOperationException(
+ new Exception("Unknown ABFS " + "Rest Opertation" + opType.name()));
Review comment:
+typo
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378954046
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsClient.java
##########
@@ -21,6 +21,7 @@
import java.net.URL;
import java.util.regex.Pattern;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
Review comment:
other import block
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r383997333
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/SASGenerator.java
##########
@@ -30,11 +30,12 @@
import org.apache.hadoop.fs.azurebfs.services.AbfsUriQueryBuilder;
/**
- * Created by tmarq on 2/17/20.
+ * Test container SAS generator
Review comment:
nit: add a . to keep javadoc happy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378942538
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/AuthorizationResult.java
##########
@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.extensions;
+
+/**
+ * AuthorizationResult will be returned by Authorizer with store path URI and
+ * supported Auth token if the AuthorizationResource is found authorized for
+ * specified action
Review comment:
add . then clarify that the get/set operators clone the arrays (but not the values)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r381985040
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -29,6 +30,8 @@
import java.util.Locale;
import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException;
+import org.apache.hadoop.fs.azurebfs.extensions.SASTokenProvider;
Review comment:
place in next block
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378948980
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -578,21 +587,30 @@ public AccessTokenProvider getTokenProvider() throws TokenAccessProviderExceptio
}
}
- public String getAbfsExternalAuthorizationClass() {
- return this.abfsExternalAuthorizationClass;
- }
+ public AbfsAuthorizer getAbfsAuthorizer() throws AzureBlobFileSystemException {
+ if (this.authorizer != null)
+ {
+ return this.authorizer;
+ }
- public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
- String authClassName = getAbfsExternalAuthorizationClass();
- AbfsAuthorizer authorizer = null;
+ String authClassName = this.abfsAuthorizerClass;
Review comment:
no need for this. everywhere; IDEs get to mark which are fields vs variables
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378950703
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -112,10 +113,13 @@ public void initialize(URI uri, Configuration configuration)
LOG.trace("AzureBlobFileSystemStore init complete");
final AbfsConfiguration abfsConfiguration = abfsStore.getAbfsConfiguration();
+ this.authorizer = abfsConfiguration.getAbfsAuthorizer();
this.setWorkingDirectory(this.getHomeDirectory());
- if (abfsConfiguration.getCreateRemoteFileSystemDuringInitialization()) {
+ // auto-creation of container is disabled when authorizer is configured
+ if (abfsConfiguration.getCreateRemoteFileSystemDuringInitialization() && (
Review comment:
style nit: put the newline before the && operator, so make it the start of th next line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-590282538
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 22s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 45s | trunk passed |
| +1 :green_heart: | compile | 0m 29s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 26s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 40s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 1m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 1s | trunk passed |
| -0 :warning: | patch | 1m 25s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed |
| +1 :green_heart: | javac | 0m 26s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 1 new + 8 unchanged - 1 fixed = 9 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 30s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 19s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | the patch passed |
| -1 :x: | findbugs | 1m 1s | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 18s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 68m 35s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Format-string method String.format(String, Object[]) called with format string "Failed to acquire a SAS token for {} on {} due to {}" wants 0 arguments but is given 3 in org.apache.hadoop.fs.azurebfs.services.AbfsClient.appendSASTokenToQuery(String, String, AbfsUriQueryBuilder) At AbfsClient.java:with format string "Failed to acquire a SAS token for {} on {} due to {}" wants 0 arguments but is given 3 in org.apache.hadoop.fs.azurebfs.services.AbfsClient.appendSASTokenToQuery(String, String, AbfsUriQueryBuilder) At AbfsClient.java:[line 614] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 7ce307fe2c60 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / b5698e0 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/11/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/11/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/11/testReport/ |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/11/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384066843
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -550,6 +595,30 @@ public AbfsRestOperation checkAccess(String path, String rwx)
return op;
}
+ /**
+ * If configured for SAS AuthType, appends SAS token to queryBuilder
+ * @param path
+ * @param operation
+ * @param queryBuilder
+ * @throws SASTokenProviderException
+ */
+ private void appendSASTokenToQuery(String path, String operation, AbfsUriQueryBuilder queryBuilder) throws SASTokenProviderException {
+ if (this.authType == AuthType.SAS) {
+ try {
+ LOG.trace("Fetch SAS token for {} on {}", operation, path);
Review comment:
log @ debug/trace is low cost as long as you aren't actually concatenating strings; the sole cost is to instantiate the "Fetch SAS token.." string to pass down a reference.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r383182474
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -57,25 +59,30 @@
private final SharedKeyCredentials sharedKeyCredentials;
private final String xMsVersion = "2018-11-09";
private final ExponentialRetryPolicy retryPolicy;
+ private final String accountName;
private final String filesystem;
+ private final AuthType authType;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382643634
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -550,6 +631,17 @@ public AbfsRestOperation checkAccess(String path, String rwx)
return op;
}
+ private void appendSASTokenToQuery(String path, String operation, AbfsUriQueryBuilder queryBuilder) throws SASTokenProviderException {
+ try {
+ LOG.trace("Fetch SAS token for {}", path);
Review comment:
add operation to these traces; it'll be handy in production
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379622231
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/AbfsAuthorizer.java
##########
@@ -18,40 +18,50 @@
package org.apache.hadoop.fs.azurebfs.extensions;
-import java.io.IOException;
-
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizerUnhandledException;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+
+import java.io.IOException;
/**
* Interface to support authorization in Azure Blob File System.
*/
@InterfaceAudience.LimitedPrivate("authorization-subsystems")
@InterfaceStability.Unstable
public interface AbfsAuthorizer {
-
/**
* Initialize authorizer for Azure Blob File System.
*
* @throws AbfsAuthorizationException if unable to initialize the authorizer.
* @throws IOException network problems or similar.
- * @throws IllegalArgumentException if the required parameters are not provided.
*/
- void init() throws AbfsAuthorizationException, IOException;
+ void init()
+ throws AbfsAuthorizationException, AbfsAuthorizerUnhandledException;
/**
- * Checks if the provided {@link FsAction} is allowed on the provided {@link Path}s.
+ * Get AuthType supported by Authorizer if Authorizer would be providing
+ * authentication token to ABFS server.
*
- * @param action the {@link FsAction} being requested on the provided {@link Path}s.
- * @param absolutePaths The absolute paths of the storage being accessed.
- * @return true if authorized, otherwise false.
- * @throws AbfsAuthorizationException on authorization failure.
- * @throws IOException network problems or similar.
- * @throws IllegalArgumentException if the required parameters are not provided.
+ * If Authorizer is not going to provide any auth tokens, return AuthType.None
+ * @return
*/
- boolean isAuthorized(FsAction action, Path... absolutePaths)
- throws AbfsAuthorizationException, IOException;
+ AuthType getAuthType();
-}
+ /**
+ * Checks if the provided {@link AuthorizationResource} is allowed to
+ * perform requested action.
+ *
+ * @param authorizationResource which contains the action and store path URI
+ * @return AuthorizationResult with store URI and Auth token if Authorizer
+ * is providing any.
+ * @throws AbfsAuthorizationException on authorization failure.
+ * @throws IOException network problems or
+ * similarITestAzureBlobFileSystemListStatus.
Review comment:
Sorry. Fixed the comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on issue #1842: HADOOP-16730 : ABFS:
Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-588568283
With namespace enabled account (East US 2)
`[INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0`
`[WARNING] Tests run: 407, Failures: 0, Errors: 0, Skipped: 32`
`[WARNING] Tests run: 194, Failures: 0, Errors: 0, Skipped: 24`
Without namespace enabled account (East US 2)
`[INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0`
`[WARNING] Tests run: 407, Failures: 0, Errors: 0, Skipped: 232`
`[WARNING] Tests run: 194, Failures: 0, Errors: 0, Skipped: 24`
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r383184005
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -550,6 +631,17 @@ public AbfsRestOperation checkAccess(String path, String rwx)
return op;
}
+ private void appendSASTokenToQuery(String path, String operation, AbfsUriQueryBuilder queryBuilder) throws SASTokenProviderException {
+ try {
+ LOG.trace("Fetch SAS token for {}", path);
+ String sasToken = sasTokenProvider.getSASToken(this.accountName, this.filesystem, path, operation);
+ queryBuilder.setSASToken(sasToken);
+ LOG.trace("SAS token fetch complete for {}", path);
+ } catch (Exception ex) {
+ throw new SASTokenProviderException("Failed to acquire a SAS token.", ex);
Review comment:
Added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379624400
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -233,6 +234,16 @@ String decodeAttribute(byte[] value) throws UnsupportedEncodingException {
public boolean getIsNamespaceEnabled() throws AzureBlobFileSystemException {
if (!isNamespaceEnabledSet) {
+ // If an authorizer is set, account needs to be HNS enabled. No need
+ // for server calls to check the same.
+ if (this.abfsConfiguration.getAbfsAuthorizer() != null)
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379625630
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsClient.java
##########
@@ -21,6 +21,7 @@
import java.net.URL;
import java.util.regex.Pattern;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on issue #1842: HADOOP-16730 : ABFS:
Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-588568589
@steveloughran @ThomasMarquardt - Have updated the review based on the recent suggestions form Thomas. Test results are as pasted above. Please review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378953303
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/extensions/MockAbfsAuthorizer.java
##########
@@ -41,47 +55,216 @@
public static final String TEST_WRITE_ONLY_FOLDER = "writeOnlyFolder";
public static final String TEST_READ_WRITE_FILE_0 = "readWriteFile0";
public static final String TEST_READ_WRITE_FILE_1 = "readWriteFile1";
- public static final String TEST_WRITE_THEN_READ_ONLY = "writeThenReadOnlyFile";
+ public static final String TEST_WRITE_THEN_READ_ONLY =
+ "writeThenReadOnlyFile";
+ private static final Set<String> apiAuthorizerActions = new HashSet<String>();
Review comment:
nit: can just use <>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379624955
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -1131,20 +1142,38 @@ private void initializeClient(URI uri, String fileSystemName, String accountName
SharedKeyCredentials creds = null;
AccessTokenProvider tokenProvider = null;
- if (abfsConfiguration.getAuthType(accountName) == AuthType.SharedKey) {
- LOG.trace("Fetching SharedKey credentials");
- int dotIndex = accountName.indexOf(AbfsHttpConstants.DOT);
- if (dotIndex <= 0) {
- throw new InvalidUriException(
- uri.toString() + " - account name is not fully qualified.");
- }
- creds = new SharedKeyCredentials(accountName.substring(0, dotIndex),
+ AuthType authType = abfsConfiguration.getAuthType(accountName);
+ AbfsAuthorizer authorizer = abfsConfiguration.getAbfsAuthorizer();
+
+ // Create cred instance based on account level Auth config if
+ // - authorizer is not configured
+ // - or authroizer configured will not provide SAS
+ if ((authorizer == null) || ((authorizer.getAuthType() != AuthType.SAS))) {
+ switch (authType) {
+ case SharedKey:
+ LOG.trace("Fetching SharedKey credentials");
+ int dotIndex = accountName.indexOf(AbfsHttpConstants.DOT);
+ if (dotIndex <= 0) {
+ throw new InvalidUriException(
+ uri.toString() + " - account name is not fully qualified.");
+ }
+ creds = new SharedKeyCredentials(accountName.substring(0, dotIndex),
abfsConfiguration.getStorageAccountKey());
- } else {
- LOG.trace("Fetching token provider");
- tokenProvider = abfsConfiguration.getTokenProvider();
- ExtensionHelper.bind(tokenProvider, uri,
- abfsConfiguration.getRawConfiguration());
+ break;
+ case Custom:
+ case OAuth:
+ LOG.trace("Fetching token provider");
+ tokenProvider = abfsConfiguration.getTokenProvider();
+ ExtensionHelper
+ .bind(tokenProvider, uri, abfsConfiguration.getRawConfiguration());
+ break;
+ case SAS:
+ throw new UnsupportedOperationException(
+ "There is no " + "Authorizer configured");
+ case None:
+ throw new UnsupportedOperationException(
+ "No authentication means " + "configured");
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378951847
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -233,6 +234,16 @@ String decodeAttribute(byte[] value) throws UnsupportedEncodingException {
public boolean getIsNamespaceEnabled() throws AzureBlobFileSystemException {
if (!isNamespaceEnabledSet) {
+ // If an authorizer is set, account needs to be HNS enabled. No need
+ // for server calls to check the same.
+ if (this.abfsConfiguration.getAbfsAuthorizer() != null)
Review comment:
style nit, put the { up with the if clause
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378940422
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/AuthorizationResource.java
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.extensions;
+
+import java.net.URI;
+
+/**
+ * AuthorizationResource with action and store path URI
+ */
+public class AuthorizationResource {
Review comment:
I assume this is just a little struct to pass around.
-will instances it be final? if so, make fields final and put in a constructor.
-if not, then at least mark the class as final
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on issue #1842: HADOOP-16730 : ABFS:
Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-591386525
@ThomasMarquardt - I have 2 open conversations for which I have added my queries. Kindly request you to provide your comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-591798802
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 23s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 51s | trunk passed |
| +1 :green_heart: | compile | 0m 29s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 11s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 48s | trunk passed |
| -0 :warning: | patch | 1m 9s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 4 new + 8 unchanged - 1 fixed = 12 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 37s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 55s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 63m 42s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux fb5f6c297517 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 7dfa37e |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/16/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/16/testReport/ |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/16/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379618345
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -22,13 +22,21 @@
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URISyntaxException;
import java.net.URL;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.authentication.AuthorizationStatus;
Review comment:
Moved as suggested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379620319
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -36,331 +40,485 @@
import static org.apache.hadoop.fs.permission.AclEntryScope.ACCESS;
import static org.apache.hadoop.fs.permission.AclEntryType.GROUP;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
-import static org.junit.Assume.assumeTrue;
/**
* Test Perform Authorization Check operation
*/
-public class ITestAzureBlobFileSystemAuthorization extends AbstractAbfsIntegrationTest {
-
- private static final Path TEST_READ_ONLY_FILE_PATH_0 = new Path(TEST_READ_ONLY_FILE_0);
- private static final Path TEST_READ_ONLY_FOLDER_PATH = new Path(TEST_READ_ONLY_FOLDER);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_0 = new Path(TEST_WRITE_ONLY_FILE_0);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_1 = new Path(TEST_WRITE_ONLY_FILE_1);
- private static final Path TEST_READ_WRITE_FILE_PATH_0 = new Path(TEST_READ_WRITE_FILE_0);
- private static final Path TEST_READ_WRITE_FILE_PATH_1 = new Path(TEST_READ_WRITE_FILE_1);
- private static final Path TEST_WRITE_ONLY_FOLDER_PATH = new Path(TEST_WRITE_ONLY_FOLDER);
- private static final Path TEST_WRITE_THEN_READ_ONLY_PATH = new Path(TEST_WRITE_THEN_READ_ONLY);
- private static final String TEST_AUTHZ_CLASS = "org.apache.hadoop.fs.azurebfs.extensions.MockAbfsAuthorizer";
+public class ITestAzureBlobFileSystemAuthorization
Review comment:
Refactored.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384437745
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -578,35 +585,37 @@ public AccessTokenProvider getTokenProvider() throws TokenAccessProviderExceptio
}
}
- public String getAbfsExternalAuthorizationClass() {
- return this.abfsExternalAuthorizationClass;
- }
-
- public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
- String authClassName = getAbfsExternalAuthorizationClass();
- AbfsAuthorizer authorizer = null;
+ public SASTokenProvider getSASTokenProvider() throws AzureBlobFileSystemException {
+ AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME, AuthType.SharedKey);
+ if (authType != AuthType.SAS) {
+ throw new SASTokenProviderException(String.format(
+ "Invalid auth type: %s is being used, expecting SAS", authType));
+ }
try {
- if (authClassName != null && !authClassName.isEmpty()) {
- @SuppressWarnings("unchecked")
- Class<AbfsAuthorizer> authClass = (Class<AbfsAuthorizer>) rawConfig.getClassByName(authClassName);
- authorizer = authClass.getConstructor(new Class[] {Configuration.class}).newInstance(rawConfig);
- LOG.trace("Initializing {}", authClassName);
- authorizer.init();
- LOG.trace("{} init complete", authClassName);
+ String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+ Class<? extends SASTokenProvider> sasTokenProviderClass =
+ getClass(configKey, null, SASTokenProvider.class);
+ if (sasTokenProviderClass == null) {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r383185313
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -1130,8 +1131,9 @@ private void initializeClient(URI uri, String fileSystemName, String accountName
SharedKeyCredentials creds = null;
AccessTokenProvider tokenProvider = null;
+ SASTokenProvider sasTokenProvider = null;
- if (abfsConfiguration.getAuthType(accountName) == AuthType.SharedKey) {
+ if (this.authType == AuthType.SharedKey) {
Review comment:
Removed this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384442216
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -578,35 +584,37 @@ public AccessTokenProvider getTokenProvider() throws TokenAccessProviderExceptio
}
}
- public String getAbfsExternalAuthorizationClass() {
- return this.abfsExternalAuthorizationClass;
- }
-
- public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
- String authClassName = getAbfsExternalAuthorizationClass();
- AbfsAuthorizer authorizer = null;
+ public SASTokenProvider getSASTokenProvider() throws AzureBlobFileSystemException {
+ AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME, AuthType.SharedKey);
+ if (authType != AuthType.SAS) {
+ throw new SASTokenProviderException(String.format(
Review comment:
Testcase added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ThomasMarquardt commented on a change in pull request
#1842: HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
ThomasMarquardt commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r383407748
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -578,35 +585,37 @@ public AccessTokenProvider getTokenProvider() throws TokenAccessProviderExceptio
}
}
- public String getAbfsExternalAuthorizationClass() {
- return this.abfsExternalAuthorizationClass;
- }
-
- public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
- String authClassName = getAbfsExternalAuthorizationClass();
- AbfsAuthorizer authorizer = null;
+ public SASTokenProvider getSASTokenProvider() throws AzureBlobFileSystemException {
+ AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME, AuthType.SharedKey);
+ if (authType != AuthType.SAS) {
+ throw new SASTokenProviderException(String.format(
+ "Invalid auth type: %s is being used, expecting SAS", authType));
+ }
try {
- if (authClassName != null && !authClassName.isEmpty()) {
- @SuppressWarnings("unchecked")
- Class<AbfsAuthorizer> authClass = (Class<AbfsAuthorizer>) rawConfig.getClassByName(authClassName);
- authorizer = authClass.getConstructor(new Class[] {Configuration.class}).newInstance(rawConfig);
- LOG.trace("Initializing {}", authClassName);
- authorizer.init();
- LOG.trace("{} init complete", authClassName);
+ String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+ Class<? extends SASTokenProvider> sasTokenProviderClass =
+ getClass(configKey, null, SASTokenProvider.class);
+ if (sasTokenProviderClass == null) {
Review comment:
How about using this:
Preconditions.checkArgument(
sasTokenProvider != null,
String.format("The configuration value for \"%s\" is invalid.", configKey));
It will throw the same exception and message that is currently in the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379628885
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/authentication/AuthorizationStatus.java
##########
@@ -0,0 +1,183 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.authentication;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResource;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult;
+
+import java.net.URI;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.HashMap;
+
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.DEFAULT_SAS_REFRESH_INTERVAL_BEFORE_EXPIRY;
+
+/**
+ * AuthorizationStatus maintains the status of Authorization and also SAS
+ * token if authorizer is providing it.
+ */
+public class AuthorizationStatus {
+ private boolean isAuthorized;
+ private HashMap<URI, SasTokenData> sasTokenMap;
+
+ public AuthorizationStatus() {
+ sasTokenMap = new HashMap<>();
+ }
+
+ /**
+ * Fetch the SAS token
+ *
+ * @return
+ */
+ public String getSasTokenQuery(URI storePathUri) {
+ if (sasTokenMap.containsKey(storePathUri)) {
+ SasTokenData sasTokenData = sasTokenMap.get(storePathUri);
+ if (isValidSas(sasTokenData)) {
+ return sasTokenData.sasToken;
+ }
+ }
+
+ return null;
+ }
+
+ /**
+ * Update authTokenMap
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382519485
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -578,35 +585,37 @@ public AccessTokenProvider getTokenProvider() throws TokenAccessProviderExceptio
}
}
- public String getAbfsExternalAuthorizationClass() {
- return this.abfsExternalAuthorizationClass;
- }
-
- public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
- String authClassName = getAbfsExternalAuthorizationClass();
- AbfsAuthorizer authorizer = null;
+ public SASTokenProvider getSASTokenProvider() throws AzureBlobFileSystemException {
+ AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME, AuthType.SharedKey);
+ if (authType != AuthType.SAS) {
+ throw new SASTokenProviderException(String.format(
+ "Invalid auth type: %s is being used, expecting SAS", authType));
+ }
try {
- if (authClassName != null && !authClassName.isEmpty()) {
- @SuppressWarnings("unchecked")
- Class<AbfsAuthorizer> authClass = (Class<AbfsAuthorizer>) rawConfig.getClassByName(authClassName);
- authorizer = authClass.getConstructor(new Class[] {Configuration.class}).newInstance(rawConfig);
- LOG.trace("Initializing {}", authClassName);
- authorizer.init();
- LOG.trace("{} init complete", authClassName);
+ String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+ Class<? extends SASTokenProvider> sasTokenProviderClass =
+ getClass(configKey, null, SASTokenProvider.class);
+ if (sasTokenProviderClass == null) {
Review comment:
Retaining the current code so that exception type thrown can be controlled.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379625106
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/extensions/MockAbfsAuthorizer.java
##########
@@ -18,21 +18,35 @@
package org.apache.hadoop.fs.azurebfs.extensions;
-import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r383182606
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -255,6 +283,11 @@ public AbfsRestOperation createPath(final String path, final boolean isFile, fin
final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder();
abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE, isFile ? FILE : DIRECTORY);
+ if (authType == AuthType.SAS) {
+ String operation = isFile ? SASTokenProvider.CREATEFILE_OPERATION : SASTokenProvider.MKDIR_OPERATION;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378946344
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -351,14 +358,21 @@ private synchronized void flushWrittenBytesToServiceInternal(final long offset,
AbfsPerfTracker tracker = client.getAbfsPerfTracker();
try (AbfsPerfInfo perfInfo = new AbfsPerfInfo(tracker,
"flushWrittenBytesToServiceInternal", "flush")) {
- AbfsRestOperation op = client.flush(path, offset, retainUncommitedData, isClose);
+ AbfsRestOperation op = client.flush(path, offset, retainUncommitedData,
+ isClose, this.authzStatus);
perfInfo.registerResult(op.getResult()).registerSuccess(true);
} catch (AzureBlobFileSystemException ex) {
if (ex instanceof AbfsRestOperationException) {
if (((AbfsRestOperationException) ex).getStatusCode() == HttpURLConnection.HTTP_NOT_FOUND) {
throw new FileNotFoundException(ex.getMessage());
}
}
+
+ if ((ex instanceof AbfsAuthorizationException)
Review comment:
this should really be done by separate catch clauses above line 364, e.g.
```
catch(AbfsAuthException | AbfsAuthUnhandledException ex) { throw ex; }
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382523121
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java
##########
@@ -38,8 +38,8 @@
import org.apache.hadoop.fs.azure.AzureNativeFileSystemStore;
import org.apache.hadoop.fs.azure.NativeAzureFileSystem;
import org.apache.hadoop.fs.azure.metrics.AzureFileSystemInstrumentation;
-import org.apache.hadoop.fs.azurebfs.constants.FileSystemUriSchemes;
-import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.constants.*;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378960842
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/authentication/AuthorizationStatus.java
##########
@@ -0,0 +1,183 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.authentication;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResource;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult;
+
+import java.net.URI;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.HashMap;
+
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.DEFAULT_SAS_REFRESH_INTERVAL_BEFORE_EXPIRY;
+
+/**
+ * AuthorizationStatus maintains the status of Authorization and also SAS
+ * token if authorizer is providing it.
+ */
+public class AuthorizationStatus {
+ private boolean isAuthorized;
+ private HashMap<URI, SasTokenData> sasTokenMap;
+
+ public AuthorizationStatus() {
+ sasTokenMap = new HashMap<>();
+ }
+
+ /**
+ * Fetch the SAS token
+ *
+ * @return
+ */
+ public String getSasTokenQuery(URI storePathUri) {
+ if (sasTokenMap.containsKey(storePathUri)) {
+ SasTokenData sasTokenData = sasTokenMap.get(storePathUri);
+ if (isValidSas(sasTokenData)) {
+ return sasTokenData.sasToken;
+ }
+ }
+
+ return null;
+ }
+
+ /**
+ * Update authTokenMap
Review comment:
nit: .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ThomasMarquardt commented on a change in pull request
#1842: HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
ThomasMarquardt commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384038589
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -266,16 +295,23 @@ public AbfsRestOperation createPath(final String path, final boolean isFile, fin
return op;
}
- public AbfsRestOperation renamePath(final String source, final String destination, final String continuation)
+ public AbfsRestOperation renamePath(String source, final String destination, final String continuation)
throws AzureBlobFileSystemException {
final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
- final String encodedRenameSource = urlEncode(FORWARD_SLASH + this.getFileSystem() + source);
+ final AbfsUriQueryBuilder srcQueryBuilder = new AbfsUriQueryBuilder();
+ appendSASTokenToQuery(source, SASTokenProvider.RENAME_SOURCE_OPERATION, srcQueryBuilder);
+ String sasToken = srcQueryBuilder.toString();
+
+ final String encodedRenameSource =
+ urlEncode(FORWARD_SLASH + this.getFileSystem() + source) + sasToken;
+ LOG.trace("Rename source queryparam added {}", encodedRenameSource);
Review comment:
We should only create the builder and append the sasToken when authType is AuthType.SAS. I don't think we should create throw-away objects. The driver should be lean and only allocate when necessary to reduce overall pressure on GC.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ThomasMarquardt commented on a change in pull request
#1842: HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
ThomasMarquardt commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384664330
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -266,16 +295,25 @@ public AbfsRestOperation createPath(final String path, final boolean isFile, fin
return op;
}
- public AbfsRestOperation renamePath(final String source, final String destination, final String continuation)
+ public AbfsRestOperation renamePath(String source, final String destination, final String continuation)
throws AzureBlobFileSystemException {
final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
- final String encodedRenameSource = urlEncode(FORWARD_SLASH + this.getFileSystem() + source);
+ final AbfsUriQueryBuilder srcQueryBuilder = new AbfsUriQueryBuilder();
+ appendSASTokenToQuery(source, SASTokenProvider.RENAME_SOURCE_OPERATION, srcQueryBuilder);
Review comment:
Ideally this is only done for AuthType:SAS, but ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379617782
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -128,7 +128,8 @@ public static String accountProperty(String property, String account) {
public static final String FS_AZURE_ENABLE_DELEGATION_TOKEN = "fs.azure.enable.delegation.token";
public static final String FS_AZURE_DELEGATION_TOKEN_PROVIDER_TYPE = "fs.azure.delegation.token.provider.type";
- public static final String ABFS_EXTERNAL_AUTHORIZATION_CLASS = "abfs.external.authorization.class";
+ /** External Authorizer **/
+ public static final String FS_AZURE_ABFS_AUTHORIZER = "fs.azure.abfs.authorizer";
Review comment:
There were no consumers for abfs.external.authorization.class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379622912
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -228,7 +230,9 @@ int readRemote(long position, byte[] b, int offset, int length) throws IOExcepti
final AbfsRestOperation op;
AbfsPerfTracker tracker = client.getAbfsPerfTracker();
try (AbfsPerfInfo perfInfo = new AbfsPerfInfo(tracker, "readRemote", "read")) {
- op = client.read(path, position, b, offset, length, tolerateOobAppends ? "*" : eTag);
+ op = client.read(path, position, b, offset, length, tolerateOobAppends
+ ? "*" : eTag, this.authzStatus);
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-588572835
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 19s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 0s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 28s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 54s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 52s | trunk passed |
| -0 :warning: | patch | 1m 9s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 36 new + 8 unchanged - 1 fixed = 44 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 22s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | the patch passed |
| +1 :green_heart: | findbugs | 0m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 18s | hadoop-azure in the patch passed. |
| -1 :x: | asflicense | 0m 32s | The patch generated 1 ASF License warnings. |
| | | 63m 57s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 47f03bdcfcbe 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ec75071 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/7/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/7/testReport/ |
| asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/7/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 321 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/7/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-585756697
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 17s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 43s | trunk passed |
| +1 :green_heart: | compile | 0m 29s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 31s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 47s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 21s | the patch passed |
| +1 :green_heart: | javac | 0m 21s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 36 new + 9 unchanged - 0 fixed = 45 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 13s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 19s | hadoop-tools_hadoop-azure generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
| -1 :x: | findbugs | 0m 57s | hadoop-tools/hadoop-azure generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 19s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 64m 35s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.authToken At AuthorizationStatus.java:[line 107] |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.authorizerAction At AuthorizationStatus.java:[line 80] |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.storePathUri At AuthorizationStatus.java:[line 80] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4f1e1cc00723 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / a98352c |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/3/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/3/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/3/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-586471959
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 13s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 53s | trunk passed |
| +1 :green_heart: | compile | 0m 27s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 29s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 48s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 22s | the patch passed |
| +1 :green_heart: | javac | 0m 22s | the patch passed |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 52 new + 9 unchanged - 0 fixed = 61 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 21s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 55s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 63m 42s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c6aab511882b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c75756f |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/5/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/5/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/5/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379640795
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##########
@@ -20,10 +20,21 @@
import java.io.IOException;
import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URISyntaxException;
import java.net.URL;
import java.net.UnknownHostException;
import java.util.List;
+import org.apache.hadoop.fs.azurebfs.authentication.AuthorizationStatus;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on issue #1842: HADOOP-16730 : ABFS:
Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-585734846
Reran the tests after the last fix for findbugs :
Test results for account without namespace enabled (East US 2):
`[INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0`
`[ERROR] Tests run: 414, Failures: 0, Errors: 0, Skipped: 243`
`[WARNING] Tests run: 194, Failures: 0, Errors: 0, Skipped: 128`
Test results for account with namespace enabled (East US 2):
`[INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0`
`[ERROR] Tests run: 414, Failures: 0, Errors: 0, Skipped: 36`
`[WARNING] Tests run: 194, Failures: 0, Errors: 0, Skipped: 128`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379624234
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -112,10 +113,13 @@ public void initialize(URI uri, Configuration configuration)
LOG.trace("AzureBlobFileSystemStore init complete");
final AbfsConfiguration abfsConfiguration = abfsStore.getAbfsConfiguration();
+ this.authorizer = abfsConfiguration.getAbfsAuthorizer();
this.setWorkingDirectory(this.getHomeDirectory());
- if (abfsConfiguration.getCreateRemoteFileSystemDuringInitialization()) {
+ // auto-creation of container is disabled when authorizer is configured
+ if (abfsConfiguration.getCreateRemoteFileSystemDuringInitialization() && (
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382644337
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -550,6 +631,17 @@ public AbfsRestOperation checkAccess(String path, String rwx)
return op;
}
+ private void appendSASTokenToQuery(String path, String operation, AbfsUriQueryBuilder queryBuilder) throws SASTokenProviderException {
+ try {
+ LOG.trace("Fetch SAS token for {}", path);
+ String sasToken = sasTokenProvider.getSASToken(this.accountName, this.filesystem, path, operation);
+ queryBuilder.setSASToken(sasToken);
+ LOG.trace("SAS token fetch complete for {}", path);
+ } catch (Exception ex) {
+ throw new SASTokenProviderException("Failed to acquire a SAS token.", ex);
Review comment:
Always preserve/add as much to the error string as you can. Especially as its all too easy for nested exceptions to not get logged on deep enough chains (i.e. hive)
add path and token to exception text, as well as ex.toString().
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ThomasMarquardt commented on a change in pull request
#1842: HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
ThomasMarquardt commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384040926
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsUriQueryBuilder.java
##########
@@ -59,6 +64,20 @@ public String toString() {
throw new IllegalArgumentException("Query string param is not encode-able: " + entry.getKey() + "=" + entry.getValue());
}
}
+ // append SAS Token
+ if (sasToken != null) {
+ sasToken = sasToken.startsWith(AbfsHttpConstants.QUESTION_MARK)
+ ? sasToken.substring(1)
+ : sasToken;
Review comment:
The interface should require that '?' be omitted from the SAS token. Allowing '?' means the plug-in will add it and then driver will remove it every time, which is wasteful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379624566
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -1131,20 +1142,38 @@ private void initializeClient(URI uri, String fileSystemName, String accountName
SharedKeyCredentials creds = null;
AccessTokenProvider tokenProvider = null;
- if (abfsConfiguration.getAuthType(accountName) == AuthType.SharedKey) {
- LOG.trace("Fetching SharedKey credentials");
- int dotIndex = accountName.indexOf(AbfsHttpConstants.DOT);
- if (dotIndex <= 0) {
- throw new InvalidUriException(
- uri.toString() + " - account name is not fully qualified.");
- }
- creds = new SharedKeyCredentials(accountName.substring(0, dotIndex),
+ AuthType authType = abfsConfiguration.getAuthType(accountName);
+ AbfsAuthorizer authorizer = abfsConfiguration.getAbfsAuthorizer();
+
+ // Create cred instance based on account level Auth config if
+ // - authorizer is not configured
+ // - or authroizer configured will not provide SAS
+ if ((authorizer == null) || ((authorizer.getAuthType() != AuthType.SAS))) {
+ switch (authType) {
+ case SharedKey:
+ LOG.trace("Fetching SharedKey credentials");
+ int dotIndex = accountName.indexOf(AbfsHttpConstants.DOT);
+ if (dotIndex <= 0) {
+ throw new InvalidUriException(
+ uri.toString() + " - account name is not fully qualified.");
+ }
+ creds = new SharedKeyCredentials(accountName.substring(0, dotIndex),
abfsConfiguration.getStorageAccountKey());
- } else {
- LOG.trace("Fetching token provider");
- tokenProvider = abfsConfiguration.getTokenProvider();
- ExtensionHelper.bind(tokenProvider, uri,
- abfsConfiguration.getRawConfiguration());
+ break;
+ case Custom:
+ case OAuth:
+ LOG.trace("Fetching token provider");
+ tokenProvider = abfsConfiguration.getTokenProvider();
+ ExtensionHelper
+ .bind(tokenProvider, uri, abfsConfiguration.getRawConfiguration());
+ break;
+ case SAS:
+ throw new UnsupportedOperationException(
+ "There is no " + "Authorizer configured");
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379625266
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/extensions/MockAbfsAuthorizer.java
##########
@@ -41,47 +55,216 @@
public static final String TEST_WRITE_ONLY_FOLDER = "writeOnlyFolder";
public static final String TEST_READ_WRITE_FILE_0 = "readWriteFile0";
public static final String TEST_READ_WRITE_FILE_1 = "readWriteFile1";
- public static final String TEST_WRITE_THEN_READ_ONLY = "writeThenReadOnlyFile";
+ public static final String TEST_WRITE_THEN_READ_ONLY =
+ "writeThenReadOnlyFile";
+ private static final Set<String> apiAuthorizerActions = new HashSet<String>();
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-591409358
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 17s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 25s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | trunk passed |
| -1 :x: | shadedclient | 16m 33s | branch has errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 45s | hadoop-azure in trunk failed. |
| +0 :ok: | spotbugs | 18m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| -1 :x: | findbugs | 0m 43s | hadoop-azure in trunk failed. |
| -0 :warning: | patch | 18m 38s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 22s | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 32s | hadoop-azure in the patch failed. |
| -1 :x: | javac | 0m 32s | hadoop-azure in the patch failed. |
| -0 :warning: | checkstyle | 0m 24s | The patch fails to run checkstyle in hadoop-azure |
| -1 :x: | mvnsite | 0m 27s | hadoop-azure in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 0m 27s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 22s | hadoop-azure in the patch failed. |
| -1 :x: | findbugs | 0m 21s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 7s | hadoop-azure in the patch failed. |
| +0 :ok: | asflicense | 0m 6s | ASF License check generated no output? |
| | | 48m 25s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 218428ed5c40 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / b5df1da |
| Default Java | 1.8.0_242 |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/13/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/13/artifact/out/branch-findbugs-hadoop-tools_hadoop-azure.txt |
| mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/13/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/13/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/13/artifact/out/patch-compile-hadoop-tools_hadoop-azure.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/13/artifact/out//home/jenkins/jenkins-slave/workspace/hadoop-multibranch_PR-1842/out/maven-patch-checkstyle-hadoop-tools_hadoop-azure.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/13/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/13/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/13/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/13/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/13/testReport/ |
| Max. process+thread count | 84 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/13/console |
| versions | git=2.7.4 maven=3.3.9 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-585164063
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 16s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 20s | trunk passed |
| +1 :green_heart: | compile | 0m 26s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 31s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 52s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 74 new + 9 unchanged - 0 fixed = 83 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 11s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 20s | hadoop-tools_hadoop-azure generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
| -1 :x: | findbugs | 0m 59s | hadoop-tools/hadoop-azure generated 23 new + 0 unchanged - 0 fixed = 23 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 19s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 63m 9s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Dead store to qualifiedPath in org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getXAttr(Path, String) At AzureBlobFileSystem.java:org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getXAttr(Path, String) At AzureBlobFileSystem.java:[line 690] |
| | Dead store to qualifiedPath in org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.setXAttr(Path, String, byte[], EnumSet) At AzureBlobFileSystem.java:org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.setXAttr(Path, String, byte[], EnumSet) At AzureBlobFileSystem.java:[line 655] |
| | Redundant nullcheck of authorizer, which is known to be non-null in org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(URI, String, String, boolean) Redundant null check at AzureBlobFileSystemStore.java:is known to be non-null in org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(URI, String, String, boolean) Redundant null check at AzureBlobFileSystemStore.java:[line 1151] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.APPEND_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 46] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.CREATEFILE_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 39] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.DELETE_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 38] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.GETACL_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 41] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.GETFILESTATUS_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 42] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.LISTSTATUS_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 37] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.MKDIR_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 40] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.READ_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 47] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.RENAME_DESTINATION_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 30] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.RENAME_SOURCE_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 29] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.SETACL_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 43] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.SETOWNER_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 44] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.SETPERMISSION_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 45] |
| | Unused public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.authorizerAction In AuthorizationResourceResult.java |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.authToken At AuthorizationStatus.java:[line 78] |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.storePathUri At AuthorizationStatus.java:[line 91] |
| | org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult.getAuthResourceResult() may expose internal representation by returning AuthorizationResult.authResourceResult At AuthorizationResult.java:by returning AuthorizationResult.authResourceResult At AuthorizationResult.java:[line 40] |
| | org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult.setAuthResourceResult(AuthorizationResourceResult[]) may expose internal representation by storing an externally mutable object into AuthorizationResult.authResourceResult At AuthorizationResult.java:by storing an externally mutable object into AuthorizationResult.authResourceResult At AuthorizationResult.java:[line 45] |
| | Nullcheck of relativePath at line 236 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(String, boolean, int, String) At AbfsClient.java:236 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(String, boolean, int, String) At AbfsClient.java:[line 224] |
| | Nullcheck of AbfsClient.authorizer at line 220 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(String, boolean, int, String) At AbfsClient.java:220 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(String, boolean, int, String) At AbfsClient.java:[line 215] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d1f540ccd624 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 9709afe |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/1/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/1/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/1/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r381987407
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java
##########
@@ -38,8 +38,8 @@
import org.apache.hadoop.fs.azure.AzureNativeFileSystemStore;
import org.apache.hadoop.fs.azure.NativeAzureFileSystem;
import org.apache.hadoop.fs.azure.metrics.AzureFileSystemInstrumentation;
-import org.apache.hadoop.fs.azurebfs.constants.FileSystemUriSchemes;
-import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.constants.*;
Review comment:
nit: revert to previous
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378952791
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -1131,20 +1142,38 @@ private void initializeClient(URI uri, String fileSystemName, String accountName
SharedKeyCredentials creds = null;
AccessTokenProvider tokenProvider = null;
- if (abfsConfiguration.getAuthType(accountName) == AuthType.SharedKey) {
- LOG.trace("Fetching SharedKey credentials");
- int dotIndex = accountName.indexOf(AbfsHttpConstants.DOT);
- if (dotIndex <= 0) {
- throw new InvalidUriException(
- uri.toString() + " - account name is not fully qualified.");
- }
- creds = new SharedKeyCredentials(accountName.substring(0, dotIndex),
+ AuthType authType = abfsConfiguration.getAuthType(accountName);
+ AbfsAuthorizer authorizer = abfsConfiguration.getAbfsAuthorizer();
+
+ // Create cred instance based on account level Auth config if
+ // - authorizer is not configured
+ // - or authroizer configured will not provide SAS
+ if ((authorizer == null) || ((authorizer.getAuthType() != AuthType.SAS))) {
+ switch (authType) {
+ case SharedKey:
+ LOG.trace("Fetching SharedKey credentials");
+ int dotIndex = accountName.indexOf(AbfsHttpConstants.DOT);
+ if (dotIndex <= 0) {
+ throw new InvalidUriException(
+ uri.toString() + " - account name is not fully qualified.");
+ }
+ creds = new SharedKeyCredentials(accountName.substring(0, dotIndex),
abfsConfiguration.getStorageAccountKey());
- } else {
- LOG.trace("Fetching token provider");
- tokenProvider = abfsConfiguration.getTokenProvider();
- ExtensionHelper.bind(tokenProvider, uri,
- abfsConfiguration.getRawConfiguration());
+ break;
+ case Custom:
+ case OAuth:
+ LOG.trace("Fetching token provider");
+ tokenProvider = abfsConfiguration.getTokenProvider();
+ ExtensionHelper
+ .bind(tokenProvider, uri, abfsConfiguration.getRawConfiguration());
+ break;
+ case SAS:
+ throw new UnsupportedOperationException(
+ "There is no " + "Authorizer configured");
+ case None:
+ throw new UnsupportedOperationException(
+ "No authentication means " + "configured");
Review comment:
nit: join the strings
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378937380
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsAuthorizerConstants.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidAbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperationType;
+
+/**
+ * Defines the string literals that need to be set to authorizer as action
+ */
+public class AbfsAuthorizerConstants {
+
+ public static final String RENAME_SOURCE_ACTION = "rename-source";
+ public static final String RENAME_DESTINATION_ACTION = "rename-destination";
+
+ public static final String CONCAT_SOURCE_ACTION = "concat-source";
+ public static final String CONCAT_TARGET_ACTION = "concat-target";
+
+ public static final String CHECKACCESS_ACTION_PREFIX_ACTION = "access-";
+
+ public static final String LISTSTATUS_ACTION = "liststatus";
+ public static final String DELETE_ACTION = "delete";
+ public static final String CREATEFILE_ACTION = "create";
+ public static final String MKDIR_ACTION = "mkdir";
+ public static final String GETACL_ACTION = "getaclstatus";
+ public static final String GETFILESTATUS_ACTION = "getfilestatus";
+ public static final String SETACL_ACTION = "setacl"; // Modify, removeacl,
+ // setacl
+ public static final String SETOWNER_ACTION = "setowner";
+ public static final String SETPERMISSION_ACTION = "setpermission";
+ public static final String APPEND_ACTION = "write";
+ public static final String READ_ACTION = "read";
+ public static final String EXECUTE_ACTION = "execute";
+
+ /**
+ * Converts AbfsRestOperation to Authorizer action
+ * @param opType
+ * @return
+ */
+ public static String getAction(AbfsRestOperationType opType)
+ throws InvalidAbfsRestOperationException {
+ switch (opType) {
+ case ListPaths:
+ return LISTSTATUS_ACTION;
+ case RenamePath:
+ return RENAME_DESTINATION_ACTION;
+ case GetAcl:
+ return GETACL_ACTION;
+ case GetPathStatus:
+ return GETFILESTATUS_ACTION;
+ case SetAcl:
+ return SETACL_ACTION;
+ case SetOwner:
+ return SETOWNER_ACTION;
+ case SetPermissions:
+ return SETPERMISSION_ACTION;
+ case Append:
+ case Flush:
+ return APPEND_ACTION;
+ case ReadFile:
+ return READ_ACTION;
+ case DeletePath:
+ return DELETE_ACTION;
+ case CreatePath:
+ return CREATEFILE_ACTION;
+ case Mkdir:
+ return MKDIR_ACTION;
+ default:
+ throw new InvalidAbfsRestOperationException(
Review comment:
bit inelegant. Any way to avoid?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379625885
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java
##########
@@ -128,6 +134,46 @@ protected AbstractAbfsIntegrationTest() throws Exception {
}
+ public void loadAuthorizer() throws Exception {
+ this.authorizer = abfsConfig.getAbfsAuthorizer();
+
+ // if authorizerClass is configured, auto-creation of filesystem is disabled
+ if (authorizer == null) {
+ throw new InvalidConfigurationValueException(
+ "loadAuthorizer failed as " + "authorizer class is not configured");
+ }
+
+ // if authorizerClass is configured, auto-creation of filesystem is disabled
+ abfsConfig.setBoolean(AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION,
+ false);
+
+ // AbstractAbfsIntegrationTest always uses a new instance of FileSystem,
+ // need to disable that and force filesystem provided in test configs.
+ String[] authorityParts = authorityParts(
+ new java.net.URI(rawConfig.get(FS_AZURE_CONTRACT_TEST_URI)));
+ this.fileSystemName = authorityParts[0];
+
+ // Reset URL with configured filesystem
+ final String abfsUrl =
+ this.getFileSystemName() + "@" + this.getAccountName();
+ URI defaultUri = null;
+
+ try {
+ defaultUri = new URI(abfsScheme, abfsUrl, null, null, null);
+ } catch (Exception ex) {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379628733
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/authentication/AuthorizationStatus.java
##########
@@ -0,0 +1,183 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.authentication;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResource;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult;
+import org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult;
+
+import java.net.URI;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.HashMap;
+
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.DEFAULT_SAS_REFRESH_INTERVAL_BEFORE_EXPIRY;
+
+/**
+ * AuthorizationStatus maintains the status of Authorization and also SAS
+ * token if authorizer is providing it.
+ */
+public class AuthorizationStatus {
+ private boolean isAuthorized;
+ private HashMap<URI, SasTokenData> sasTokenMap;
Review comment:
AuthorizationStatus instance represents a single request response from authorizer and a copy is saved only Abfs input or output stream, which will be operating one I/O operation at a time.
As the instance is not shared at Abfs filesystem level across requests, concurrency doesnt need handling,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378943886
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java
##########
@@ -228,7 +230,9 @@ int readRemote(long position, byte[] b, int offset, int length) throws IOExcepti
final AbfsRestOperation op;
AbfsPerfTracker tracker = client.getAbfsPerfTracker();
try (AbfsPerfInfo perfInfo = new AbfsPerfInfo(tracker, "readRemote", "read")) {
- op = client.read(path, position, b, offset, length, tolerateOobAppends ? "*" : eTag);
+ op = client.read(path, position, b, offset, length, tolerateOobAppends
+ ? "*" : eTag, this.authzStatus);
Review comment:
no need to use this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378955696
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -18,15 +18,19 @@
package org.apache.hadoop.fs.azurebfs;
+import org.junit.Assume;
Review comment:
nit: leave org.junit imports where they are; add the new ones by it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-591826871
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 8 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 42s | trunk passed |
| +1 :green_heart: | compile | 0m 30s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 32s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | trunk passed |
| +0 :ok: | spotbugs | 0m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
| -0 :warning: | patch | 1m 9s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed |
| +1 :green_heart: | compile | 0m 22s | the patch passed |
| +1 :green_heart: | javac | 0m 22s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 4 new + 8 unchanged - 1 fixed = 12 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 39s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed |
| +1 :green_heart: | findbugs | 1m 4s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
| | | 66m 38s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 5031b32c98fb 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 7dfa37e |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/17/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/17/testReport/ |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/17/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384438052
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/SASGenerator.java
##########
@@ -30,11 +30,12 @@
import org.apache.hadoop.fs.azurebfs.services.AbfsUriQueryBuilder;
/**
- * Created by tmarq on 2/17/20.
+ * Test container SAS generator
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ThomasMarquardt commented on a change in pull request
#1842: HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
ThomasMarquardt commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384738532
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -65,6 +66,21 @@ public void setup() throws Exception {
super.setup();
}
+ @Test
+ public void testSASTokenProviderInitializeException() throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+
+ final AzureBlobFileSystem testFs = new AzureBlobFileSystem();
+
+ MockSASTokenProvider.setThrowExceptionAtInit(true);
Review comment:
I am seeing failures when running the tests
**mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify**
[ERROR] Errors:
[ERROR] ITestAbfsIdentityTransformer.<init>:66 » NullPointer
[ERROR] ITestAbfsIdentityTransformer.<init>:66 » NullPointer
[ERROR] ITestAbfsIdentityTransformer.<init>:66 » NullPointer
[ERROR] ITestAbfsIdentityTransformer.<init>:66 » NullPointer
[ERROR] ITestAbfsIdentityTransformer.<init>:66 » NullPointer
[ERROR] ITestAbfsIdentityTransformer.<init>:66 » NullPointer
[ERROR] ITestAzureBlobFileSystemAuthorization.testRenameAuthorized:125->runTest:241-
[ERROR] Tests run: 409, Failures: 0, Errors: 7, Skipped: 32
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382648840
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -1130,8 +1131,9 @@ private void initializeClient(URI uri, String fileSystemName, String accountName
SharedKeyCredentials creds = null;
AccessTokenProvider tokenProvider = null;
+ SASTokenProvider sasTokenProvider = null;
- if (abfsConfiguration.getAuthType(accountName) == AuthType.SharedKey) {
+ if (this.authType == AuthType.SharedKey) {
Review comment:
we don't normally use this. for field refs on the basis that IDEs mark things differently anyway. But if you wan to be consistent with the existin g code, stay with it. Though its actually a bit mixed (abfsConfiguration is not qualified).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382639992
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/SASTokenProvider.java
##########
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.extensions;
+
+import java.io.IOException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.AccessControlException;
+
+/**
+ * Interface to support SAS authorization.
+ */
+@InterfaceAudience.LimitedPrivate("authorization-subsystems")
+@InterfaceStability.Unstable
+public interface SASTokenProvider {
+
+ public static final String CONCAT_SOURCE_OPERATION = "concat-source";
Review comment:
you don't need `public static final` in interfaces; this is implicit. hence the checkstyle complaints
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-586471959
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 13s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 53s | trunk passed |
| +1 :green_heart: | compile | 0m 27s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 29s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 48s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 22s | the patch passed |
| +1 :green_heart: | javac | 0m 22s | the patch passed |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 52 new + 9 unchanged - 0 fixed = 61 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 21s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 55s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 63m 42s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c6aab511882b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c75756f |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/5/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/5/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/5/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382647018
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -578,35 +585,37 @@ public AccessTokenProvider getTokenProvider() throws TokenAccessProviderExceptio
}
}
- public String getAbfsExternalAuthorizationClass() {
- return this.abfsExternalAuthorizationClass;
- }
-
- public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
- String authClassName = getAbfsExternalAuthorizationClass();
- AbfsAuthorizer authorizer = null;
+ public SASTokenProvider getSASTokenProvider() throws AzureBlobFileSystemException {
+ AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME, AuthType.SharedKey);
+ if (authType != AuthType.SAS) {
+ throw new SASTokenProviderException(String.format(
+ "Invalid auth type: %s is being used, expecting SAS", authType));
+ }
try {
- if (authClassName != null && !authClassName.isEmpty()) {
- @SuppressWarnings("unchecked")
- Class<AbfsAuthorizer> authClass = (Class<AbfsAuthorizer>) rawConfig.getClassByName(authClassName);
- authorizer = authClass.getConstructor(new Class[] {Configuration.class}).newInstance(rawConfig);
- LOG.trace("Initializing {}", authClassName);
- authorizer.init();
- LOG.trace("{} init complete", authClassName);
+ String configKey = FS_AZURE_SAS_TOKEN_PROVIDER_TYPE;
+ Class<? extends SASTokenProvider> sasTokenProviderClass =
+ getClass(configKey, null, SASTokenProvider.class);
+ if (sasTokenProviderClass == null) {
+ throw new IllegalArgumentException(
+ String.format("The configuration value for \"%s\" is invalid.", configKey));
}
- } catch (
- IllegalAccessException
- | InstantiationException
- | ClassNotFoundException
- | IllegalArgumentException
- | InvocationTargetException
- | NoSuchMethodException
- | SecurityException
- | AbfsAuthorizationException e) {
- throw new IOException(e);
+
+ SASTokenProvider sasTokenProvider = ReflectionUtils
+ .newInstance(sasTokenProviderClass, rawConfig);
+ if (sasTokenProvider == null) {
+ throw new IllegalArgumentException("Failed to initialize " + sasTokenProviderClass);
+ }
+
+ LOG.trace("Initializing {}", sasTokenProviderClass.getName());
+ sasTokenProvider.initialize(rawConfig, accountName);
+ LOG.trace("{} init complete", sasTokenProviderClass.getName());
+ return sasTokenProvider;
+ } catch(IllegalArgumentException e) {
+ throw e;
+ } catch (Exception e) {
Review comment:
this is the change which probably leaes those imports unused -just cut the now obsolete imports.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: Hadoop 16730: Add
Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: Hadoop 16730: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-585164063
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 16s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 20s | trunk passed |
| +1 :green_heart: | compile | 0m 26s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 31s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 52s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 74 new + 9 unchanged - 0 fixed = 83 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 11s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 20s | hadoop-tools_hadoop-azure generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
| -1 :x: | findbugs | 0m 59s | hadoop-tools/hadoop-azure generated 23 new + 0 unchanged - 0 fixed = 23 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 19s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 63m 9s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Dead store to qualifiedPath in org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getXAttr(Path, String) At AzureBlobFileSystem.java:org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getXAttr(Path, String) At AzureBlobFileSystem.java:[line 690] |
| | Dead store to qualifiedPath in org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.setXAttr(Path, String, byte[], EnumSet) At AzureBlobFileSystem.java:org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.setXAttr(Path, String, byte[], EnumSet) At AzureBlobFileSystem.java:[line 655] |
| | Redundant nullcheck of authorizer, which is known to be non-null in org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(URI, String, String, boolean) Redundant null check at AzureBlobFileSystemStore.java:is known to be non-null in org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(URI, String, String, boolean) Redundant null check at AzureBlobFileSystemStore.java:[line 1151] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.APPEND_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 46] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.CREATEFILE_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 39] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.DELETE_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 38] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.GETACL_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 41] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.GETFILESTATUS_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 42] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.LISTSTATUS_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 37] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.MKDIR_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 40] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.READ_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 47] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.RENAME_DESTINATION_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 30] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.RENAME_SOURCE_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 29] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.SETACL_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 43] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.SETOWNER_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 44] |
| | org.apache.hadoop.fs.azurebfs.constants.AbfsAuthorizerConstants.SETPERMISSION_ACTION isn't final but should be At AbfsAuthorizerConstants.java:be At AbfsAuthorizerConstants.java:[line 45] |
| | Unused public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.authorizerAction In AuthorizationResourceResult.java |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.authToken At AuthorizationStatus.java:[line 78] |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.storePathUri At AuthorizationStatus.java:[line 91] |
| | org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult.getAuthResourceResult() may expose internal representation by returning AuthorizationResult.authResourceResult At AuthorizationResult.java:by returning AuthorizationResult.authResourceResult At AuthorizationResult.java:[line 40] |
| | org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult.setAuthResourceResult(AuthorizationResourceResult[]) may expose internal representation by storing an externally mutable object into AuthorizationResult.authResourceResult At AuthorizationResult.java:by storing an externally mutable object into AuthorizationResult.authResourceResult At AuthorizationResult.java:[line 45] |
| | Nullcheck of relativePath at line 236 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(String, boolean, int, String) At AbfsClient.java:236 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(String, boolean, int, String) At AbfsClient.java:[line 224] |
| | Nullcheck of AbfsClient.authorizer at line 220 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(String, boolean, int, String) At AbfsClient.java:220 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(String, boolean, int, String) At AbfsClient.java:[line 215] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d1f540ccd624 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 9709afe |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/1/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/1/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/1/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378963898
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java
##########
@@ -128,7 +128,8 @@ public static String accountProperty(String property, String account) {
public static final String FS_AZURE_ENABLE_DELEGATION_TOKEN = "fs.azure.enable.delegation.token";
public static final String FS_AZURE_DELEGATION_TOKEN_PROVIDER_TYPE = "fs.azure.delegation.token.provider.type";
- public static final String ABFS_EXTERNAL_AUTHORIZATION_CLASS = "abfs.external.authorization.class";
+ /** External Authorizer **/
+ public static final String FS_AZURE_ABFS_AUTHORIZER = "fs.azure.abfs.authorizer";
Review comment:
Now, this is going to be backwards incompatible. Which happens in the code anyway. Is everyone happy with this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384890291
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -266,16 +295,25 @@ public AbfsRestOperation createPath(final String path, final boolean isFile, fin
return op;
}
- public AbfsRestOperation renamePath(final String source, final String destination, final String continuation)
+ public AbfsRestOperation renamePath(String source, final String destination, final String continuation)
throws AzureBlobFileSystemException {
final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
- final String encodedRenameSource = urlEncode(FORWARD_SLASH + this.getFileSystem() + source);
+ final AbfsUriQueryBuilder srcQueryBuilder = new AbfsUriQueryBuilder();
+ appendSASTokenToQuery(source, SASTokenProvider.RENAME_SOURCE_OPERATION, srcQueryBuilder);
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378946735
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##########
@@ -20,10 +20,21 @@
import java.io.IOException;
import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URISyntaxException;
import java.net.URL;
import java.net.UnknownHostException;
import java.util.List;
+import org.apache.hadoop.fs.azurebfs.authentication.AuthorizationStatus;
Review comment:
move to org.apache block
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378939629
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/AuthorizationResource.java
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.extensions;
+
+import java.net.URI;
+
+/**
+ * AuthorizationResource with action and store path URI
Review comment:
nit: Add "."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r381987083
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/SASGenerator.java
##########
@@ -0,0 +1,126 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import javax.crypto.Mac;
+import javax.crypto.spec.SecretKeySpec;
+import java.io.UnsupportedEncodingException;
+import java.time.*;
Review comment:
these should all be expanded
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378944296
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -106,7 +110,7 @@ public AbfsOutputStream(
10L,
TimeUnit.SECONDS,
new LinkedBlockingQueue<>());
- this.completionService = new ExecutorCompletionService<>(this.threadExecutor);
+ this.completionService = new ExecutorCompletionService<AuthorizationStatus>(this.threadExecutor);
Review comment:
explicit type not needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r383181656
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/SASTokenProvider.java
##########
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.extensions;
+
+import java.io.IOException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.AccessControlException;
+
+/**
+ * Interface to support SAS authorization.
+ */
+@InterfaceAudience.LimitedPrivate("authorization-subsystems")
+@InterfaceStability.Unstable
+public interface SASTokenProvider {
+
+ public static final String CONCAT_SOURCE_OPERATION = "concat-source";
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382523369
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
##########
@@ -39,6 +38,7 @@
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382510512
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -604,9 +622,10 @@ public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
| NoSuchMethodException
| SecurityException
| AbfsAuthorizationException e) {
- throw new IOException(e);
+ throw new AbfsAuthorizationException("Unable to initialize "
Review comment:
Code is modified. Resolving this comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379639612
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/AbfsAuthorizerUnhandledException.java
##########
@@ -0,0 +1,30 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.contracts.exceptions;
+
+/**
+ * Exception for all unhandled errors while communicating with authorizer
+ */
+public class AbfsAuthorizerUnhandledException
+ extends AzureBlobFileSystemException {
+ public AbfsAuthorizerUnhandledException(Exception innerException) {
+ super("An unhandled Authorizer request processing exception",
Review comment:
Updated the exception message
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-588572835
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 19s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 0s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 28s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed |
| +0 :ok: | spotbugs | 0m 54s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 52s | trunk passed |
| -0 :warning: | patch | 1m 9s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 36 new + 8 unchanged - 1 fixed = 44 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 22s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | the patch passed |
| +1 :green_heart: | findbugs | 0m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 18s | hadoop-azure in the patch passed. |
| -1 :x: | asflicense | 0m 32s | The patch generated 1 ASF License warnings. |
| | | 63m 57s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 47f03bdcfcbe 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ec75071 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/7/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/7/testReport/ |
| asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/7/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 321 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/7/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379623593
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/UriUtils.java
##########
@@ -18,8 +18,17 @@
package org.apache.hadoop.fs.azurebfs.utils;
+import org.apache.http.client.utils.URIBuilder;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382512112
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsUriQueryBuilder.java
##########
@@ -59,6 +64,20 @@ public String toString() {
throw new IllegalArgumentException("Query string param is not encode-able: " + entry.getKey() + "=" + entry.getValue());
}
}
+ // append SAS Token
+ if (sasToken != null) {
+ sasToken =
+ sasToken.startsWith(AbfsHttpConstants.QUESTION_MARK) ?
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382510394
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -578,21 +587,30 @@ public AccessTokenProvider getTokenProvider() throws TokenAccessProviderExceptio
}
}
- public String getAbfsExternalAuthorizationClass() {
- return this.abfsExternalAuthorizationClass;
- }
+ public AbfsAuthorizer getAbfsAuthorizer() throws AzureBlobFileSystemException {
+ if (this.authorizer != null)
+ {
+ return this.authorizer;
+ }
- public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
- String authClassName = getAbfsExternalAuthorizationClass();
- AbfsAuthorizer authorizer = null;
+ String authClassName = this.abfsAuthorizerClass;
try {
if (authClassName != null && !authClassName.isEmpty()) {
@SuppressWarnings("unchecked")
Class<AbfsAuthorizer> authClass = (Class<AbfsAuthorizer>) rawConfig.getClassByName(authClassName);
- authorizer = authClass.getConstructor(new Class[] {Configuration.class}).newInstance(rawConfig);
+ this.authorizer =
+ authClass.getConstructor(new Class[] {Configuration.class}).newInstance(rawConfig);
LOG.trace("Initializing {}", authClassName);
- authorizer.init();
+ this.authorizer.init();
+ if ((this.authorizer.getAuthType() != AuthType.SAS) && (
+ this.authorizer.getAuthType() != AuthType.None)) {
+ throw new AbfsAuthorizationException(
Review comment:
Code is modified. Resolving this comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379618171
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/AuthorizationResourceResult.java
##########
@@ -0,0 +1,31 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.extensions;
+
+import java.net.URI;
+
+/**
+ * AuthorizationResourceResult array will be included in the
+ * AuthorizationResult with results for each resource.
+ */
+public class AuthorizationResourceResult {
Review comment:
Fields made final.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379356145
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -578,21 +587,30 @@ public AccessTokenProvider getTokenProvider() throws TokenAccessProviderExceptio
}
}
- public String getAbfsExternalAuthorizationClass() {
- return this.abfsExternalAuthorizationClass;
- }
+ public AbfsAuthorizer getAbfsAuthorizer() throws AzureBlobFileSystemException {
+ if (this.authorizer != null)
+ {
+ return this.authorizer;
+ }
- public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
- String authClassName = getAbfsExternalAuthorizationClass();
- AbfsAuthorizer authorizer = null;
+ String authClassName = this.abfsAuthorizerClass;
try {
if (authClassName != null && !authClassName.isEmpty()) {
@SuppressWarnings("unchecked")
Class<AbfsAuthorizer> authClass = (Class<AbfsAuthorizer>) rawConfig.getClassByName(authClassName);
- authorizer = authClass.getConstructor(new Class[] {Configuration.class}).newInstance(rawConfig);
+ this.authorizer =
+ authClass.getConstructor(new Class[] {Configuration.class}).newInstance(rawConfig);
LOG.trace("Initializing {}", authClassName);
- authorizer.init();
+ this.authorizer.init();
+ if ((this.authorizer.getAuthType() != AuthType.SAS) && (
+ this.authorizer.getAuthType() != AuthType.None)) {
+ throw new AbfsAuthorizationException(
Review comment:
Currently the only supported AuthType for an external Authorizer is SAS. Throwing the exception if AuthType is set to any other such as OAuth or SharedKey right after the authorizer init.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378938001
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/AbfsAuthorizer.java
##########
@@ -18,40 +18,50 @@
package org.apache.hadoop.fs.azurebfs.extensions;
-import java.io.IOException;
-
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizerUnhandledException;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+
+import java.io.IOException;
Review comment:
please don't move imports around, even when they are in the wrong place -makes backporting/merging harder
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379639316
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/authentication/SasTokenData.java
##########
@@ -0,0 +1,27 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.authentication;
+
+import java.time.Instant;
+
+public class SasTokenData {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-590907288
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 16s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 51s | trunk passed |
| +1 :green_heart: | compile | 0m 29s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 19s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 0m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
| -0 :warning: | patch | 1m 7s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 1 new + 8 unchanged - 1 fixed = 9 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 24s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 32s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 63m 31s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 9702b2d3081f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / dda00d3 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/12/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/12/testReport/ |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/12/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379618015
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/AuthorizationResource.java
##########
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.extensions;
+
+import java.net.URI;
+
+/**
+ * AuthorizationResource with action and store path URI
+ */
+public class AuthorizationResource {
Review comment:
Made the fields final as suggested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ThomasMarquardt commented on a change in pull request
#1842: HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
ThomasMarquardt commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384055634
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -578,35 +584,37 @@ public AccessTokenProvider getTokenProvider() throws TokenAccessProviderExceptio
}
}
- public String getAbfsExternalAuthorizationClass() {
- return this.abfsExternalAuthorizationClass;
- }
-
- public AbfsAuthorizer getAbfsAuthorizer() throws IOException {
- String authClassName = getAbfsExternalAuthorizationClass();
- AbfsAuthorizer authorizer = null;
+ public SASTokenProvider getSASTokenProvider() throws AzureBlobFileSystemException {
+ AuthType authType = getEnum(FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME, AuthType.SharedKey);
+ if (authType != AuthType.SAS) {
+ throw new SASTokenProviderException(String.format(
Review comment:
Within reason, we should add a test case for the error handling when SASTokenProvider fails to load or fails to initialize. Not sure how difficult it is to test, so we could do this manually if needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379623215
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -106,7 +110,7 @@ public AbfsOutputStream(
10L,
TimeUnit.SECONDS,
new LinkedBlockingQueue<>());
- this.completionService = new ExecutorCompletionService<>(this.threadExecutor);
+ this.completionService = new ExecutorCompletionService<AuthorizationStatus>(this.threadExecutor);
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382510065
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsAuthorizerConstants.java
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.constants;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidAbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperationType;
+
+/**
+ * Defines the string literals that need to be set to authorizer as action
+ */
+public class AbfsAuthorizerConstants {
+
+ public static final String RENAME_SOURCE_ACTION = "rename-source";
+ public static final String RENAME_DESTINATION_ACTION = "rename-destination";
+
+ public static final String CONCAT_SOURCE_ACTION = "concat-source";
+ public static final String CONCAT_TARGET_ACTION = "concat-target";
+
+ public static final String CHECKACCESS_ACTION_PREFIX_ACTION = "access-";
+
+ public static final String LISTSTATUS_ACTION = "liststatus";
+ public static final String DELETE_ACTION = "delete";
+ public static final String CREATEFILE_ACTION = "create";
+ public static final String MKDIR_ACTION = "mkdir";
+ public static final String GETACL_ACTION = "getaclstatus";
+ public static final String GETFILESTATUS_ACTION = "getfilestatus";
+ public static final String SETACL_ACTION = "setacl"; // Modify, removeacl,
+ // setacl
+ public static final String SETOWNER_ACTION = "setowner";
+ public static final String SETPERMISSION_ACTION = "setpermission";
+ public static final String APPEND_ACTION = "write";
+ public static final String READ_ACTION = "read";
+ public static final String EXECUTE_ACTION = "execute";
+
+ /**
+ * Converts AbfsRestOperation to Authorizer action
+ * @param opType
+ * @return
+ */
+ public static String getAction(AbfsRestOperationType opType)
+ throws InvalidAbfsRestOperationException {
+ switch (opType) {
+ case ListPaths:
+ return LISTSTATUS_ACTION;
+ case RenamePath:
+ return RENAME_DESTINATION_ACTION;
+ case GetAcl:
+ return GETACL_ACTION;
+ case GetPathStatus:
+ return GETFILESTATUS_ACTION;
+ case SetAcl:
+ return SETACL_ACTION;
+ case SetOwner:
+ return SETOWNER_ACTION;
+ case SetPermissions:
+ return SETPERMISSION_ACTION;
+ case Append:
+ case Flush:
+ return APPEND_ACTION;
+ case ReadFile:
+ return READ_ACTION;
+ case DeletePath:
+ return DELETE_ACTION;
+ case CreatePath:
+ return CREATEFILE_ACTION;
+ case Mkdir:
+ return MKDIR_ACTION;
+ default:
+ throw new InvalidAbfsRestOperationException(
Review comment:
Code has changed since this version. Resolving the comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379625337
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/extensions/MockAbfsAuthorizer.java
##########
@@ -41,47 +55,216 @@
public static final String TEST_WRITE_ONLY_FOLDER = "writeOnlyFolder";
public static final String TEST_READ_WRITE_FILE_0 = "readWriteFile0";
public static final String TEST_READ_WRITE_FILE_1 = "readWriteFile1";
- public static final String TEST_WRITE_THEN_READ_ONLY = "writeThenReadOnlyFile";
+ public static final String TEST_WRITE_THEN_READ_ONLY =
+ "writeThenReadOnlyFile";
+ private static final Set<String> apiAuthorizerActions = new HashSet<String>();
+ public String accountName;
private Configuration conf;
- private Set<Path> readOnlyPaths = new HashSet<Path>();
- private Set<Path> writeOnlyPaths = new HashSet<Path>();
- private Set<Path> readWritePaths = new HashSet<Path>();
- private int writeThenReadOnly = 0;
+ private Set<String> readOnlyPathsPrefixes = new HashSet<String>();
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-585756697
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 17s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 43s | trunk passed |
| +1 :green_heart: | compile | 0m 29s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 31s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 47s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 21s | the patch passed |
| +1 :green_heart: | javac | 0m 21s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 36 new + 9 unchanged - 0 fixed = 45 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 13s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 19s | hadoop-tools_hadoop-azure generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
| -1 :x: | findbugs | 0m 57s | hadoop-tools/hadoop-azure generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 19s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
| | | 64m 35s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.authToken At AuthorizationStatus.java:[line 107] |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.authorizerAction At AuthorizationStatus.java:[line 80] |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.storePathUri At AuthorizationStatus.java:[line 80] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4f1e1cc00723 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / a98352c |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/3/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/3/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/3/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r384439390
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##########
@@ -266,16 +295,23 @@ public AbfsRestOperation createPath(final String path, final boolean isFile, fin
return op;
}
- public AbfsRestOperation renamePath(final String source, final String destination, final String continuation)
+ public AbfsRestOperation renamePath(String source, final String destination, final String continuation)
throws AzureBlobFileSystemException {
final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
- final String encodedRenameSource = urlEncode(FORWARD_SLASH + this.getFileSystem() + source);
+ final AbfsUriQueryBuilder srcQueryBuilder = new AbfsUriQueryBuilder();
+ appendSASTokenToQuery(source, SASTokenProvider.RENAME_SOURCE_OPERATION, srcQueryBuilder);
+ String sasToken = srcQueryBuilder.toString();
+
+ final String encodedRenameSource =
+ urlEncode(FORWARD_SLASH + this.getFileSystem() + source) + sasToken;
+ LOG.trace("Rename source queryparam added {}", encodedRenameSource);
Review comment:
Are you suggesting we move the AuthType == SAS check back to the API methods ? It was moved to appendSASTokenToQuery() as it was a repeating code across API methods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-585379660
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 18s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 41s | trunk passed |
| +1 :green_heart: | compile | 0m 27s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 35s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 1m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 40 new + 9 unchanged - 0 fixed = 49 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 22s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 21s | hadoop-tools_hadoop-azure generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
| -1 :x: | findbugs | 0m 56s | hadoop-tools/hadoop-azure generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 64m 52s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-tools/hadoop-azure |
| | Unused public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.authorizerAction In AuthorizationResourceResult.java |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.authToken At AuthorizationStatus.java:[line 78] |
| | Unwritten public or protected field:org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResourceResult.storePathUri At AuthorizationStatus.java:[line 91] |
| | org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult.getAuthResourceResult() may expose internal representation by returning AuthorizationResult.authResourceResult At AuthorizationResult.java:by returning AuthorizationResult.authResourceResult At AuthorizationResult.java:[line 40] |
| | org.apache.hadoop.fs.azurebfs.extensions.AuthorizationResult.setAuthResourceResult(AuthorizationResourceResult[]) may expose internal representation by storing an externally mutable object into AuthorizationResult.authResourceResult At AuthorizationResult.java:by storing an externally mutable object into AuthorizationResult.authResourceResult At AuthorizationResult.java:[line 45] |
| | Possible null pointer dereference of relativePath in org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(String, boolean, int, String) Dereferenced at AbfsClient.java:relativePath in org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(String, boolean, int, String) Dereferenced at AbfsClient.java:[line 225] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2cd01f55bef9 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / f09710b |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/2/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/2/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/2/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r379641302
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -36,331 +40,485 @@
import static org.apache.hadoop.fs.permission.AclEntryScope.ACCESS;
import static org.apache.hadoop.fs.permission.AclEntryType.GROUP;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
-import static org.junit.Assume.assumeTrue;
/**
* Test Perform Authorization Check operation
*/
-public class ITestAzureBlobFileSystemAuthorization extends AbstractAbfsIntegrationTest {
-
- private static final Path TEST_READ_ONLY_FILE_PATH_0 = new Path(TEST_READ_ONLY_FILE_0);
- private static final Path TEST_READ_ONLY_FOLDER_PATH = new Path(TEST_READ_ONLY_FOLDER);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_0 = new Path(TEST_WRITE_ONLY_FILE_0);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_1 = new Path(TEST_WRITE_ONLY_FILE_1);
- private static final Path TEST_READ_WRITE_FILE_PATH_0 = new Path(TEST_READ_WRITE_FILE_0);
- private static final Path TEST_READ_WRITE_FILE_PATH_1 = new Path(TEST_READ_WRITE_FILE_1);
- private static final Path TEST_WRITE_ONLY_FOLDER_PATH = new Path(TEST_WRITE_ONLY_FOLDER);
- private static final Path TEST_WRITE_THEN_READ_ONLY_PATH = new Path(TEST_WRITE_THEN_READ_ONLY);
- private static final String TEST_AUTHZ_CLASS = "org.apache.hadoop.fs.azurebfs.extensions.MockAbfsAuthorizer";
+public class ITestAzureBlobFileSystemAuthorization
+ extends AbstractAbfsIntegrationTest {
+
+ private static final String TEST_READ_ONLY_FILE_PATH_PREFIX_0 =
+ TEST_READ_ONLY_FILE_0;
+ private static final String TEST_READ_ONLY_FILE_PATH_PREFIX_1 =
+ TEST_READ_ONLY_FILE_1;
+ private static final String TEST_READ_ONLY_FOLDER_PATH_PREFIX =
+ TEST_READ_ONLY_FOLDER;
+ private static final String TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 =
+ TEST_WRITE_ONLY_FILE_0;
+ private static final String TEST_WRITE_ONLY_FILE_PATH_PREFIX_1 =
+ TEST_WRITE_ONLY_FILE_1;
+ private static final String TEST_READ_WRITE_FILE_PATH_PREFIX_0 =
+ TEST_READ_WRITE_FILE_0;
+ private static final String TEST_READ_WRITE_FILE_PATH_PREFIX_1 =
+ TEST_READ_WRITE_FILE_1;
+ private static final String TEST_WRITE_ONLY_FOLDER_PATH_PREFIX =
+ TEST_WRITE_ONLY_FOLDER;
+ private static final String TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX =
+ TEST_WRITE_THEN_READ_ONLY;
+ private static final String TEST_AUTHZ_CLASS =
+ "org.apache.hadoop.fs" + ".azurebfs.extensions.MockAbfsAuthorizer";
private static final String TEST_USER = UUID.randomUUID().toString();
private static final String TEST_GROUP = UUID.randomUUID().toString();
private static final String BAR = UUID.randomUUID().toString();
+ @Rule
+ public TestName name = new TestName();
public ITestAzureBlobFileSystemAuthorization() throws Exception {
}
@Override
public void setup() throws Exception {
- this.getConfiguration().set(ConfigurationKeys.ABFS_EXTERNAL_AUTHORIZATION_CLASS, TEST_AUTHZ_CLASS);
+ boolean isHNSEnabled = this.getConfiguration().getBoolean(
+ TestConfigurationKeys.FS_AZURE_TEST_NAMESPACE_ENABLED_ACCOUNT, false);
+ Assume.assumeTrue(isHNSEnabled == true);
+ this.getConfiguration().setAbfsAuthorizerClass(TEST_AUTHZ_CLASS);
+ loadAuthorizer();
super.setup();
}
@Test
public void testOpenFileWithInvalidPath() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- intercept(IllegalArgumentException.class,
- ()-> {
- fs.open(new Path("")).close();
+ intercept(IllegalArgumentException.class, () -> {
+ fs.open(new Path("")).close();
});
}
@Test
public void testOpenFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_THEN_READ_ONLY_PATH).close();
- fs.open(TEST_WRITE_THEN_READ_ONLY_PATH).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX + name.getMethodName());
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.WRITE_MODE);
+ fs.create(testFilePath).close();
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.READ_MODE);
+ fs.open(testFilePath).close();
}
@Test
public void testOpenFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.open(TEST_WRITE_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ fs.create(testFilePath).close();
+ intercept(AbfsAuthorizationException.class, () -> {
+ fs.open(testFilePath).close();
});
}
@Test
public void testCreateFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ fs.create(testFilePath).close();
}
@Test
public void testCreateFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.create(TEST_READ_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_READ_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ intercept(AbfsAuthorizationException.class, () -> {
+ fs.create(testFilePath).close();
});
}
@Test
public void testAppendFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
- fs.append(TEST_WRITE_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_READ_WRITE_FILE_PATH_PREFIX_0 + name.getMethodName());
+ fs.create(testFilePath).close();
+ fs.append(testFilePath).close();
}
@Test
public void testAppendFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_THEN_READ_ONLY_PATH).close();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.append(TEST_WRITE_THEN_READ_ONLY_PATH).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX + name.getMethodName());
+
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.WRITE_MODE);
+ fs.create(testFilePath).close();
+
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.READ_MODE);
+ intercept(AbfsAuthorizationException.class, () -> {
+ fs.append(testFilePath).close();
});
}
@Test
- public void testRenameAuthorized() throws Exception {
+ public void testRenameSourceUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.rename(TEST_READ_WRITE_FILE_PATH_0, TEST_READ_WRITE_FILE_PATH_1);
+ Path testSrcFilePath = new Path(
+ TEST_READ_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ Path testDestFilePath = new Path(
+ TEST_READ_WRITE_FILE_PATH_PREFIX_0 + name.getMethodName());
+
+ intercept(AbfsAuthorizationException.class, () -> {
+ fs.rename(testSrcFilePath, testDestFilePath);
+ });
}
@Test
- public void testRenameUnauthorized() throws Exception {
+ public void testRenameDestUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.rename(TEST_WRITE_ONLY_FILE_PATH_0, TEST_WRITE_ONLY_FILE_PATH_1);
+ Path testSrcFilePath = new Path(
+ TEST_READ_WRITE_FILE_PATH_PREFIX_0 + name.getMethodName());
+ Path testDestFilePath = new Path(
+ TEST_READ_ONLY_FILE_PATH_PREFIX_1 + name.getMethodName());
+
+ intercept(AbfsAuthorizationException.class, () -> {
+ fs.rename(testSrcFilePath, testDestFilePath);
});
}
@Test
public void testDeleteFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
- fs.delete(TEST_WRITE_ONLY_FILE_PATH_0, false);
+ Path testFilePath = new Path(
+ TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ fs.create(testFilePath).close();
+ fs.delete(testFilePath, false);
}
@Test
public void testDeleteFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_THEN_READ_ONLY_PATH).close();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.delete(TEST_WRITE_THEN_READ_ONLY_PATH, false);
+ Path testFilePath = new Path(
+ TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX + name.getMethodName());
+
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
Review comment:
Refactored
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382512241
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/SASGenerator.java
##########
@@ -0,0 +1,126 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import javax.crypto.Mac;
+import javax.crypto.spec.SecretKeySpec;
+import java.io.UnsupportedEncodingException;
+import java.time.*;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378958189
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -36,331 +40,485 @@
import static org.apache.hadoop.fs.permission.AclEntryScope.ACCESS;
import static org.apache.hadoop.fs.permission.AclEntryType.GROUP;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
-import static org.junit.Assume.assumeTrue;
/**
* Test Perform Authorization Check operation
*/
-public class ITestAzureBlobFileSystemAuthorization extends AbstractAbfsIntegrationTest {
-
- private static final Path TEST_READ_ONLY_FILE_PATH_0 = new Path(TEST_READ_ONLY_FILE_0);
- private static final Path TEST_READ_ONLY_FOLDER_PATH = new Path(TEST_READ_ONLY_FOLDER);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_0 = new Path(TEST_WRITE_ONLY_FILE_0);
- private static final Path TEST_WRITE_ONLY_FILE_PATH_1 = new Path(TEST_WRITE_ONLY_FILE_1);
- private static final Path TEST_READ_WRITE_FILE_PATH_0 = new Path(TEST_READ_WRITE_FILE_0);
- private static final Path TEST_READ_WRITE_FILE_PATH_1 = new Path(TEST_READ_WRITE_FILE_1);
- private static final Path TEST_WRITE_ONLY_FOLDER_PATH = new Path(TEST_WRITE_ONLY_FOLDER);
- private static final Path TEST_WRITE_THEN_READ_ONLY_PATH = new Path(TEST_WRITE_THEN_READ_ONLY);
- private static final String TEST_AUTHZ_CLASS = "org.apache.hadoop.fs.azurebfs.extensions.MockAbfsAuthorizer";
+public class ITestAzureBlobFileSystemAuthorization
+ extends AbstractAbfsIntegrationTest {
+
+ private static final String TEST_READ_ONLY_FILE_PATH_PREFIX_0 =
+ TEST_READ_ONLY_FILE_0;
+ private static final String TEST_READ_ONLY_FILE_PATH_PREFIX_1 =
+ TEST_READ_ONLY_FILE_1;
+ private static final String TEST_READ_ONLY_FOLDER_PATH_PREFIX =
+ TEST_READ_ONLY_FOLDER;
+ private static final String TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 =
+ TEST_WRITE_ONLY_FILE_0;
+ private static final String TEST_WRITE_ONLY_FILE_PATH_PREFIX_1 =
+ TEST_WRITE_ONLY_FILE_1;
+ private static final String TEST_READ_WRITE_FILE_PATH_PREFIX_0 =
+ TEST_READ_WRITE_FILE_0;
+ private static final String TEST_READ_WRITE_FILE_PATH_PREFIX_1 =
+ TEST_READ_WRITE_FILE_1;
+ private static final String TEST_WRITE_ONLY_FOLDER_PATH_PREFIX =
+ TEST_WRITE_ONLY_FOLDER;
+ private static final String TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX =
+ TEST_WRITE_THEN_READ_ONLY;
+ private static final String TEST_AUTHZ_CLASS =
+ "org.apache.hadoop.fs" + ".azurebfs.extensions.MockAbfsAuthorizer";
private static final String TEST_USER = UUID.randomUUID().toString();
private static final String TEST_GROUP = UUID.randomUUID().toString();
private static final String BAR = UUID.randomUUID().toString();
+ @Rule
+ public TestName name = new TestName();
public ITestAzureBlobFileSystemAuthorization() throws Exception {
}
@Override
public void setup() throws Exception {
- this.getConfiguration().set(ConfigurationKeys.ABFS_EXTERNAL_AUTHORIZATION_CLASS, TEST_AUTHZ_CLASS);
+ boolean isHNSEnabled = this.getConfiguration().getBoolean(
+ TestConfigurationKeys.FS_AZURE_TEST_NAMESPACE_ENABLED_ACCOUNT, false);
+ Assume.assumeTrue(isHNSEnabled == true);
+ this.getConfiguration().setAbfsAuthorizerClass(TEST_AUTHZ_CLASS);
+ loadAuthorizer();
super.setup();
}
@Test
public void testOpenFileWithInvalidPath() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- intercept(IllegalArgumentException.class,
- ()-> {
- fs.open(new Path("")).close();
+ intercept(IllegalArgumentException.class, () -> {
+ fs.open(new Path("")).close();
});
}
@Test
public void testOpenFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_THEN_READ_ONLY_PATH).close();
- fs.open(TEST_WRITE_THEN_READ_ONLY_PATH).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX + name.getMethodName());
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.WRITE_MODE);
+ fs.create(testFilePath).close();
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.READ_MODE);
+ fs.open(testFilePath).close();
}
@Test
public void testOpenFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.open(TEST_WRITE_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ fs.create(testFilePath).close();
+ intercept(AbfsAuthorizationException.class, () -> {
+ fs.open(testFilePath).close();
});
}
@Test
public void testCreateFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ fs.create(testFilePath).close();
}
@Test
public void testCreateFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.create(TEST_READ_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_READ_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ intercept(AbfsAuthorizationException.class, () -> {
+ fs.create(testFilePath).close();
});
}
@Test
public void testAppendFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
- fs.append(TEST_WRITE_ONLY_FILE_PATH_0).close();
+ Path testFilePath = new Path(
+ TEST_READ_WRITE_FILE_PATH_PREFIX_0 + name.getMethodName());
+ fs.create(testFilePath).close();
+ fs.append(testFilePath).close();
}
@Test
public void testAppendFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_THEN_READ_ONLY_PATH).close();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.append(TEST_WRITE_THEN_READ_ONLY_PATH).close();
+ Path testFilePath = new Path(
+ TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX + name.getMethodName());
+
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.WRITE_MODE);
+ fs.create(testFilePath).close();
+
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
+ .setwriteThenReadOnly(testFilePath.getName(), WriteReadMode.READ_MODE);
+ intercept(AbfsAuthorizationException.class, () -> {
+ fs.append(testFilePath).close();
});
}
@Test
- public void testRenameAuthorized() throws Exception {
+ public void testRenameSourceUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.rename(TEST_READ_WRITE_FILE_PATH_0, TEST_READ_WRITE_FILE_PATH_1);
+ Path testSrcFilePath = new Path(
+ TEST_READ_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ Path testDestFilePath = new Path(
+ TEST_READ_WRITE_FILE_PATH_PREFIX_0 + name.getMethodName());
+
+ intercept(AbfsAuthorizationException.class, () -> {
+ fs.rename(testSrcFilePath, testDestFilePath);
+ });
}
@Test
- public void testRenameUnauthorized() throws Exception {
+ public void testRenameDestUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.rename(TEST_WRITE_ONLY_FILE_PATH_0, TEST_WRITE_ONLY_FILE_PATH_1);
+ Path testSrcFilePath = new Path(
+ TEST_READ_WRITE_FILE_PATH_PREFIX_0 + name.getMethodName());
+ Path testDestFilePath = new Path(
+ TEST_READ_ONLY_FILE_PATH_PREFIX_1 + name.getMethodName());
+
+ intercept(AbfsAuthorizationException.class, () -> {
+ fs.rename(testSrcFilePath, testDestFilePath);
});
}
@Test
public void testDeleteFileAuthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_ONLY_FILE_PATH_0).close();
- fs.delete(TEST_WRITE_ONLY_FILE_PATH_0, false);
+ Path testFilePath = new Path(
+ TEST_WRITE_ONLY_FILE_PATH_PREFIX_0 + name.getMethodName());
+ fs.create(testFilePath).close();
+ fs.delete(testFilePath, false);
}
@Test
public void testDeleteFileUnauthorized() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
- fs.create(TEST_WRITE_THEN_READ_ONLY_PATH).close();
- intercept(AbfsAuthorizationException.class,
- ()-> {
- fs.delete(TEST_WRITE_THEN_READ_ONLY_PATH, false);
+ Path testFilePath = new Path(
+ TEST_WRITE_THEN_READ_ONLY_PATH_PREFIX + name.getMethodName());
+
+ ((MockAbfsAuthorizer) fs.getAbfsStore().getAuthorizer())
Review comment:
pull this out into `getMockAuthorizer() {return getFileSystem().getAbfsStore.getAuthorizer()}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378964859
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/AbfsAuthorizerUnhandledException.java
##########
@@ -0,0 +1,30 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.contracts.exceptions;
+
+/**
+ * Exception for all unhandled errors while communicating with authorizer
+ */
+public class AbfsAuthorizerUnhandledException
+ extends AzureBlobFileSystemException {
+ public AbfsAuthorizerUnhandledException(Exception innerException) {
+ super("An unhandled Authorizer request processing exception",
Review comment:
I wouldn't know how to interpret that message when I got a support call with it in the stack trace
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378938678
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/AbfsAuthorizer.java
##########
@@ -18,40 +18,50 @@
package org.apache.hadoop.fs.azurebfs.extensions;
-import java.io.IOException;
-
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsAuthorizerUnhandledException;
+import org.apache.hadoop.fs.azurebfs.services.AuthType;
+
+import java.io.IOException;
/**
* Interface to support authorization in Azure Blob File System.
*/
@InterfaceAudience.LimitedPrivate("authorization-subsystems")
@InterfaceStability.Unstable
public interface AbfsAuthorizer {
-
/**
* Initialize authorizer for Azure Blob File System.
*
* @throws AbfsAuthorizationException if unable to initialize the authorizer.
* @throws IOException network problems or similar.
- * @throws IllegalArgumentException if the required parameters are not provided.
*/
- void init() throws AbfsAuthorizationException, IOException;
+ void init()
+ throws AbfsAuthorizationException, AbfsAuthorizerUnhandledException;
/**
- * Checks if the provided {@link FsAction} is allowed on the provided {@link Path}s.
+ * Get AuthType supported by Authorizer if Authorizer would be providing
+ * authentication token to ABFS server.
*
- * @param action the {@link FsAction} being requested on the provided {@link Path}s.
- * @param absolutePaths The absolute paths of the storage being accessed.
- * @return true if authorized, otherwise false.
- * @throws AbfsAuthorizationException on authorization failure.
- * @throws IOException network problems or similar.
- * @throws IllegalArgumentException if the required parameters are not provided.
+ * If Authorizer is not going to provide any auth tokens, return AuthType.None
+ * @return
*/
- boolean isAuthorized(FsAction action, Path... absolutePaths)
- throws AbfsAuthorizationException, IOException;
+ AuthType getAuthType();
-}
+ /**
+ * Checks if the provided {@link AuthorizationResource} is allowed to
+ * perform requested action.
+ *
+ * @param authorizationResource which contains the action and store path URI
+ * @return AuthorizationResult with store URI and Auth token if Authorizer
+ * is providing any.
+ * @throws AbfsAuthorizationException on authorization failure.
+ * @throws IOException network problems or
+ * similarITestAzureBlobFileSystemListStatus.
Review comment:
??
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382510986
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##########
@@ -294,18 +298,20 @@ private synchronized void writeCurrentBufferToService() throws IOException {
waitForTaskToComplete();
}
- final Future<Void> job = completionService.submit(new Callable<Void>() {
+ final Future<AuthorizationStatus> job = completionService.submit(new Callable<AuthorizationStatus>() {
Review comment:
Code is modified. Resolving this comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r382650778
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemAuthorization.java
##########
@@ -18,20 +18,23 @@
package org.apache.hadoop.fs.azurebfs;
+import java.io.*;
import java.util.Arrays;
import java.util.List;
import java.util.UUID;
+import org.apache.hadoop.fs.azurebfs.constants.*;
Review comment:
all into the org.apache.block. .* is only allowed for static imports..if that is what these are then move down there as import static entries. Otherwise: expand
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
snvijaya commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r383185095
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsUriQueryBuilder.java
##########
@@ -40,6 +41,10 @@ public void addQuery(final String name, final String value) {
}
}
+ public void setSASToken(final String sasToken) {
+ this.sasToken = sasToken;
Review comment:
Didn't modify as the QueryBuilder will be used in a state where SasToken need not be mandatorily set. If set, will be used by ToString().
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1842: HADOOP-16730 :
ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#issuecomment-586491014
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 20s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 43s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 34s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed |
| +0 :ok: | spotbugs | 0m 49s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 47s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 21s | the patch passed |
| +1 :green_heart: | javac | 0m 21s | the patch passed |
| -0 :warning: | checkstyle | 0m 15s | hadoop-tools/hadoop-azure: The patch generated 51 new + 9 unchanged - 0 fixed = 60 total (was 9) |
| +1 :green_heart: | mvnsite | 0m 24s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 27s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | the patch passed |
| +1 :green_heart: | findbugs | 0m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 16s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 63m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1842 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2a32b1a470e7 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c75756f |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/6/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/6/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1842/6/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378952710
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -1131,20 +1142,38 @@ private void initializeClient(URI uri, String fileSystemName, String accountName
SharedKeyCredentials creds = null;
AccessTokenProvider tokenProvider = null;
- if (abfsConfiguration.getAuthType(accountName) == AuthType.SharedKey) {
- LOG.trace("Fetching SharedKey credentials");
- int dotIndex = accountName.indexOf(AbfsHttpConstants.DOT);
- if (dotIndex <= 0) {
- throw new InvalidUriException(
- uri.toString() + " - account name is not fully qualified.");
- }
- creds = new SharedKeyCredentials(accountName.substring(0, dotIndex),
+ AuthType authType = abfsConfiguration.getAuthType(accountName);
+ AbfsAuthorizer authorizer = abfsConfiguration.getAbfsAuthorizer();
+
+ // Create cred instance based on account level Auth config if
+ // - authorizer is not configured
+ // - or authroizer configured will not provide SAS
+ if ((authorizer == null) || ((authorizer.getAuthType() != AuthType.SAS))) {
+ switch (authType) {
+ case SharedKey:
+ LOG.trace("Fetching SharedKey credentials");
+ int dotIndex = accountName.indexOf(AbfsHttpConstants.DOT);
+ if (dotIndex <= 0) {
+ throw new InvalidUriException(
+ uri.toString() + " - account name is not fully qualified.");
+ }
+ creds = new SharedKeyCredentials(accountName.substring(0, dotIndex),
abfsConfiguration.getStorageAccountKey());
- } else {
- LOG.trace("Fetching token provider");
- tokenProvider = abfsConfiguration.getTokenProvider();
- ExtensionHelper.bind(tokenProvider, uri,
- abfsConfiguration.getRawConfiguration());
+ break;
+ case Custom:
+ case OAuth:
+ LOG.trace("Fetching token provider");
+ tokenProvider = abfsConfiguration.getTokenProvider();
+ ExtensionHelper
+ .bind(tokenProvider, uri, abfsConfiguration.getRawConfiguration());
+ break;
+ case SAS:
+ throw new UnsupportedOperationException(
+ "There is no " + "Authorizer configured");
Review comment:
nit: join the strings
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1842:
HADOOP-16730 : ABFS: Add Authorizer Interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1842: HADOOP-16730 : ABFS: Add Authorizer Interface
URL: https://github.com/apache/hadoop/pull/1842#discussion_r378964568
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/AbfsAuthorizerUnhandledException.java
##########
@@ -0,0 +1,30 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.contracts.exceptions;
+
+/**
+ * Exception for all unhandled errors while communicating with authorizer
Review comment:
nit: .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org