You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by tm...@apache.org on 2019/08/28 22:44:17 UTC
[hadoop] branch branch-2 updated (8e8c16d -> f56972b)
This is an automated email from the ASF dual-hosted git repository.
tmarquardt pushed a change to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 8e8c16d YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06).
new e5bb849 HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream.
new 50953c5 HADOOP-16269. ABFS: add listFileStatus with StartFrom.
new 605c812 HADOOP-16251. ABFS: add FSMainOperationsBaseTest
new 67fe539 Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest"
new 42816bf HADOOP-16376. ABFS: Override access() to no-op.
new 8044415 HADOOP-16404. ABFS default blocksize change(256MB from 512MB)
new 1376f2e HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC.
new 6cb7462 HADOOP-16315. ABFS: transform full UPN for named user in AclStatus
new f56972b HADOOP-16460: ABFS: fix for Sever Name Indication (SNI)
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
hadoop-project/pom.xml | 2 +-
.../hadoop/fs/azurebfs/AzureBlobFileSystem.java | 23 +++-
.../fs/azurebfs/AzureBlobFileSystemStore.java | 126 +++++++++++++++--
.../fs/azurebfs/constants/AbfsHttpConstants.java | 9 ++
.../constants/FileSystemConfigurations.java | 2 +-
.../fs/azurebfs/oauth2/IdentityTransformer.java | 75 ++++++++--
.../fs/azurebfs/services/AbfsOutputStream.java | 17 ++-
.../org/apache/hadoop/fs/azurebfs/utils/CRC64.java | 60 ++++++++
.../fs/azurebfs/AbstractAbfsIntegrationTest.java | 7 +-
.../fs/azurebfs/ITestAbfsIdentityTransformer.java | 58 +++++++-
.../ITestAzureBlobFileSystemFileStatus.java | 18 +++
.../ITestAzureBlobFileSystemMainOperation.java | 71 ++++++++++
...zureBlobFileSystemStoreListStatusWithRange.java | 151 +++++++++++++++++++++
.../TestAbfsConfigurationFieldsValidation.java | 6 +
.../apache/hadoop/fs/azurebfs/TestAbfsCrc64.java | 26 ++--
15 files changed, 601 insertions(+), 50 deletions(-)
create mode 100644 hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/CRC64.java
create mode 100644 hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemMainOperation.java
create mode 100644 hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemStoreListStatusWithRange.java
copy hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/TestJUnitSetup.java => hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsCrc64.java (59%)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 08/09: HADOOP-16315. ABFS: transform full UPN for named
user in AclStatus
Posted by tm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tmarquardt pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 6cb74629ec7f2a8f11afe5abd5c6587ee071d7aa
Author: Da Zhou <da...@microsoft.com>
AuthorDate: Fri Aug 9 12:37:27 2019 +0100
HADOOP-16315. ABFS: transform full UPN for named user in AclStatus
Contributed by Da Zhou
Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
---
.../fs/azurebfs/AzureBlobFileSystemStore.java | 17 ++---
.../fs/azurebfs/oauth2/IdentityTransformer.java | 75 +++++++++++++++++++---
.../fs/azurebfs/ITestAbfsIdentityTransformer.java | 58 ++++++++++++++++-
3 files changed, 131 insertions(+), 19 deletions(-)
diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
index 27ba202..2694565 100644
--- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
+++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
@@ -721,8 +721,8 @@ public class AzureBlobFileSystemStore {
path.toString(),
AclEntry.aclSpecToString(aclSpec));
- final List<AclEntry> transformedAclEntries = identityTransformer.transformAclEntriesForSetRequest(aclSpec);
- final Map<String, String> modifyAclEntries = AbfsAclHelper.deserializeAclSpec(AclEntry.aclSpecToString(transformedAclEntries));
+ identityTransformer.transformAclEntriesForSetRequest(aclSpec);
+ final Map<String, String> modifyAclEntries = AbfsAclHelper.deserializeAclSpec(AclEntry.aclSpecToString(aclSpec));
boolean useUpn = AbfsAclHelper.isUpnFormatAclEntries(modifyAclEntries);
final AbfsRestOperation op = client.getAclStatus(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path, true), useUpn);
@@ -748,8 +748,8 @@ public class AzureBlobFileSystemStore {
path.toString(),
AclEntry.aclSpecToString(aclSpec));
- final List<AclEntry> transformedAclEntries = identityTransformer.transformAclEntriesForSetRequest(aclSpec);
- final Map<String, String> removeAclEntries = AbfsAclHelper.deserializeAclSpec(AclEntry.aclSpecToString(transformedAclEntries));
+ identityTransformer.transformAclEntriesForSetRequest(aclSpec);
+ final Map<String, String> removeAclEntries = AbfsAclHelper.deserializeAclSpec(AclEntry.aclSpecToString(aclSpec));
boolean isUpnFormat = AbfsAclHelper.isUpnFormatAclEntries(removeAclEntries);
final AbfsRestOperation op = client.getAclStatus(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path, true), isUpnFormat);
@@ -827,8 +827,8 @@ public class AzureBlobFileSystemStore {
path.toString(),
AclEntry.aclSpecToString(aclSpec));
- final List<AclEntry> transformedAclEntries = identityTransformer.transformAclEntriesForSetRequest(aclSpec);
- final Map<String, String> aclEntries = AbfsAclHelper.deserializeAclSpec(AclEntry.aclSpecToString(transformedAclEntries));
+ identityTransformer.transformAclEntriesForSetRequest(aclSpec);
+ final Map<String, String> aclEntries = AbfsAclHelper.deserializeAclSpec(AclEntry.aclSpecToString(aclSpec));
final boolean isUpnFormat = AbfsAclHelper.isUpnFormatAclEntries(aclEntries);
final AbfsRestOperation op = client.getAclStatus(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path, true), isUpnFormat);
@@ -867,7 +867,8 @@ public class AzureBlobFileSystemStore {
final String permissions = result.getResponseHeader(HttpHeaderConfigurations.X_MS_PERMISSIONS);
final String aclSpecString = op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_ACL);
- final List<AclEntry> processedAclEntries = AclEntry.parseAclSpec(AbfsAclHelper.processAclString(aclSpecString), true);
+ final List<AclEntry> aclEntries = AclEntry.parseAclSpec(AbfsAclHelper.processAclString(aclSpecString), true);
+ identityTransformer.transformAclEntriesForGetRequest(aclEntries, userName, primaryUserGroup);
final FsPermission fsPermission = permissions == null ? new AbfsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL)
: AbfsPermission.valueOf(permissions);
@@ -877,7 +878,7 @@ public class AzureBlobFileSystemStore {
aclStatusBuilder.setPermission(fsPermission);
aclStatusBuilder.stickyBit(fsPermission.getStickyBit());
- aclStatusBuilder.addEntries(processedAclEntries);
+ aclStatusBuilder.addEntries(aclEntries);
return aclStatusBuilder.build();
}
diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/IdentityTransformer.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/IdentityTransformer.java
index 343b233..6844afb 100644
--- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/IdentityTransformer.java
+++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/IdentityTransformer.java
@@ -81,6 +81,7 @@ public class IdentityTransformer {
* Perform identity transformation for the Get request results in AzureBlobFileSystemStore:
* getFileStatus(), listStatus(), getAclStatus().
* Input originalIdentity can be one of the following:
+ * <pre>
* 1. $superuser:
* by default it will be transformed to local user/group, this can be disabled by setting
* "fs.azure.identity.transformer.skip.superuser.replacement" to true.
@@ -93,7 +94,7 @@ public class IdentityTransformer {
* 3. User principal name (UPN):
* can be transformed to a short name(localIdentity) if originalIdentity is owner name, and
* "fs.azure.identity.transformer.enable.short.name" is enabled.
- *
+ * </pre>
* @param originalIdentity the original user or group in the get request results: FileStatus, AclStatus.
* @param isUserName indicate whether the input originalIdentity is an owner name or owning group name.
* @param localIdentity the local user or group, should be parsed from UserGroupInformation.
@@ -134,7 +135,7 @@ public class IdentityTransformer {
* Perform Identity transformation when setting owner on a path.
* There are four possible input:
* 1.short name; 2.$superuser; 3.Fully qualified name; 4. principal id.
- *
+ * <pre>
* short name could be transformed to:
* - A service principal id or $superuser, if short name belongs a daemon service
* stated in substitution list AND "fs.azure.identity.transformer.service.principal.id"
@@ -142,7 +143,7 @@ public class IdentityTransformer {
* - Fully qualified name, if "fs.azure.identity.transformer.domain.name" is set in configuration.
*
* $superuser, fully qualified name and principalId should not be transformed.
- *
+ * </pre>
* @param userOrGroup the user or group to be set as owner.
* @return user or group after transformation.
* */
@@ -168,21 +169,21 @@ public class IdentityTransformer {
* Perform Identity transformation when calling setAcl(),removeAclEntries() and modifyAclEntries()
* If the AclEntry type is a user or group, and its name is one of the following:
* 1.short name; 2.$superuser; 3.Fully qualified name; 4. principal id.
+ * <pre>
* Short name could be transformed to:
* - A service principal id or $superuser, if short name belongs a daemon service
* stated in substitution list AND "fs.azure.identity.transformer.service.principal.id"
* is set with $superuser or a principal id.
* - A fully qualified name, if the AclEntry type is User AND if "fs.azure.identity.transformer.domain.name"
- * is set in configuration. This is to make the behavior consistent with HDI.
+ * is set in configuration. This is to make the behavior consistent with HDI.
*
* $superuser, fully qualified name and principal id should not be transformed.
- *
+ * </pre>
* @param aclEntries list of AclEntry
- * @return list of AclEntry after the identity transformation.
* */
- public List<AclEntry> transformAclEntriesForSetRequest(final List<AclEntry> aclEntries) {
+ public void transformAclEntriesForSetRequest(final List<AclEntry> aclEntries) {
if (skipUserIdentityReplacement) {
- return aclEntries;
+ return;
}
for (int i = 0; i < aclEntries.size(); i++) {
@@ -218,7 +219,63 @@ public class IdentityTransformer {
// Replace the original AclEntry
aclEntries.set(i, aclEntryBuilder.build());
}
- return aclEntries;
+ }
+
+ /**
+ * Perform Identity transformation when calling GetAclStatus()
+ * If the AclEntry type is a user or group, and its name is one of the following:
+ * <pre>
+ * 1. $superuser:
+ * by default it will be transformed to local user/group, this can be disabled by setting
+ * "fs.azure.identity.transformer.skip.superuser.replacement" to true.
+ *
+ * 2. User principal id:
+ * can be transformed to localUser/localGroup, if this principal id matches the principal id set in
+ * "fs.azure.identity.transformer.service.principal.id" and localIdentity is stated in
+ * "fs.azure.identity.transformer.service.principal.substitution.list"
+ *
+ * 3. User principal name (UPN):
+ * can be transformed to a short name(local identity) if originalIdentity is owner name, and
+ * "fs.azure.identity.transformer.enable.short.name" is enabled.
+ * </pre>
+ * @param aclEntries list of AclEntry
+ * @param localUser local user name
+ * @param localGroup local primary group
+ * */
+ public void transformAclEntriesForGetRequest(final List<AclEntry> aclEntries, String localUser, String localGroup) {
+ if (skipUserIdentityReplacement) {
+ return;
+ }
+
+ for (int i = 0; i < aclEntries.size(); i++) {
+ AclEntry aclEntry = aclEntries.get(i);
+ String name = aclEntry.getName();
+ String transformedName = name;
+ if (name == null || name.isEmpty() || aclEntry.getType().equals(AclEntryType.OTHER) || aclEntry.getType().equals(AclEntryType.MASK)) {
+ continue;
+ }
+
+ // when type of aclEntry is user or group
+ if (aclEntry.getType().equals(AclEntryType.USER)) {
+ transformedName = transformIdentityForGetRequest(name, true, localUser);
+ } else if (aclEntry.getType().equals(AclEntryType.GROUP)) {
+ transformedName = transformIdentityForGetRequest(name, false, localGroup);
+ }
+
+ // Avoid unnecessary new AclEntry allocation
+ if (transformedName.equals(name)) {
+ continue;
+ }
+
+ AclEntry.Builder aclEntryBuilder = new AclEntry.Builder();
+ aclEntryBuilder.setType(aclEntry.getType());
+ aclEntryBuilder.setName(transformedName);
+ aclEntryBuilder.setScope(aclEntry.getScope());
+ aclEntryBuilder.setPermission(aclEntry.getPermission());
+
+ // Replace the original AclEntry
+ aclEntries.set(i, aclEntryBuilder.build());
+ }
}
/**
diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsIdentityTransformer.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsIdentityTransformer.java
index 2316d6b..a31524f 100644
--- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsIdentityTransformer.java
+++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsIdentityTransformer.java
@@ -249,9 +249,13 @@ public class ITestAbfsIdentityTransformer extends AbstractAbfsScaleTest{
aclEntry(DEFAULT, MASK, ALL) // to make the behavior consistent with HDI.
);
+ // make a copy
+ List<AclEntry> aclEntries = Lists.newArrayList(aclEntriesToBeTransformed);
+
// Default config should not change the identities
IdentityTransformer identityTransformer = getTransformerWithDefaultIdentityConfig(config);
- checkAclEntriesList(aclEntriesToBeTransformed, identityTransformer.transformAclEntriesForSetRequest(aclEntriesToBeTransformed));
+ identityTransformer.transformAclEntriesForSetRequest(aclEntries);
+ checkAclEntriesList(aclEntriesToBeTransformed, aclEntries);
resetIdentityConfig(config);
// With config
@@ -261,6 +265,8 @@ public class ITestAbfsIdentityTransformer extends AbstractAbfsScaleTest{
config.set(FS_AZURE_OVERRIDE_OWNER_SP, SERVICE_PRINCIPAL_ID);
identityTransformer = getTransformerWithCustomizedIdentityConfig(config);
+ identityTransformer.transformAclEntriesForSetRequest(aclEntries);
+
// expected acl entries
List<AclEntry> expectedAclEntries = Lists.newArrayList(
aclEntry(ACCESS, USER, SERVICE_PRINCIPAL_ID, ALL),
@@ -274,8 +280,56 @@ public class ITestAbfsIdentityTransformer extends AbstractAbfsScaleTest{
aclEntry(DEFAULT, MASK, ALL)
);
- checkAclEntriesList(identityTransformer.transformAclEntriesForSetRequest(aclEntriesToBeTransformed), expectedAclEntries);
+ checkAclEntriesList(aclEntries, expectedAclEntries);
+ }
+
+ @Test
+ public void transformAclEntriesForGetRequest() throws IOException {
+ Configuration config = this.getRawConfiguration();
+ resetIdentityConfig(config);
+
+ List<AclEntry> aclEntriesToBeTransformed = Lists.newArrayList(
+ aclEntry(ACCESS, USER, FULLY_QUALIFIED_NAME, ALL),
+ aclEntry(DEFAULT, USER, SUPER_USER, ALL),
+ aclEntry(DEFAULT, USER, SERVICE_PRINCIPAL_ID, ALL),
+ aclEntry(DEFAULT, USER, SHORT_NAME, ALL),
+ aclEntry(DEFAULT, GROUP, SHORT_NAME, ALL),
+ aclEntry(DEFAULT, OTHER, ALL),
+ aclEntry(DEFAULT, MASK, ALL)
+ );
+
+ // make a copy
+ List<AclEntry> aclEntries = Lists.newArrayList(aclEntriesToBeTransformed);
+
+ // Default config should not change the identities
+ IdentityTransformer identityTransformer = getTransformerWithDefaultIdentityConfig(config);
+ identityTransformer.transformAclEntriesForGetRequest(aclEntries, localUser, localGroup);
+ checkAclEntriesList(aclEntriesToBeTransformed, aclEntries);
+
+ resetIdentityConfig(config);
+ // With config
+ config.set(FS_AZURE_OVERRIDE_OWNER_SP_LIST, localUser + ",a,b,c,d");
+ config.setBoolean(FS_AZURE_FILE_OWNER_ENABLE_SHORTNAME, true);
+ config.set(FS_AZURE_FILE_OWNER_DOMAINNAME, DOMAIN);
+ config.set(FS_AZURE_OVERRIDE_OWNER_SP, SERVICE_PRINCIPAL_ID);
+ identityTransformer = getTransformerWithCustomizedIdentityConfig(config);
+
+ // make a copy
+ aclEntries = Lists.newArrayList(aclEntriesToBeTransformed);
+ identityTransformer.transformAclEntriesForGetRequest(aclEntries, localUser, localGroup);
+
+ // expected acl entries
+ List<AclEntry> expectedAclEntries = Lists.newArrayList(
+ aclEntry(ACCESS, USER, SHORT_NAME, ALL), // Full UPN should be transformed to shortName
+ aclEntry(DEFAULT, USER, localUser, ALL), // $SuperUser should be transformed to shortName
+ aclEntry(DEFAULT, USER, localUser, ALL), // principal Id should be transformed to local user name
+ aclEntry(DEFAULT, USER, SHORT_NAME, ALL),
+ aclEntry(DEFAULT, GROUP, SHORT_NAME, ALL),
+ aclEntry(DEFAULT, OTHER, ALL),
+ aclEntry(DEFAULT, MASK, ALL)
+ );
+ checkAclEntriesList(aclEntries, expectedAclEntries);
}
private void resetIdentityConfig(Configuration config) {
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 03/09: HADOOP-16251. ABFS: add FSMainOperationsBaseTest
Posted by tm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tmarquardt pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 605c812749324c63a664ce2f91bc0b18968d84b1
Author: DadanielZ <zh...@microsoft.com>
AuthorDate: Tue Apr 30 19:38:48 2019 +0000
HADOOP-16251. ABFS: add FSMainOperationsBaseTest
Author: Da Zhou
---
.../ITestAzureBlobFileSystemMainOperation.java | 78 ++++++++++++++++++++++
1 file changed, 78 insertions(+)
diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemMainOperation.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemMainOperation.java
new file mode 100644
index 0000000..41abfe8
--- /dev/null
+++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemMainOperation.java
@@ -0,0 +1,78 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import org.junit.Ignore;
+
+import org.apache.hadoop.fs.FSMainOperationsBaseTest;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.azurebfs.contract.ABFSContractTestBinding;
+
+/**
+ * Test AzureBlobFileSystem main operations.
+ * */
+public class ITestAzureBlobFileSystemMainOperation extends FSMainOperationsBaseTest {
+
+ private static final String TEST_ROOT_DIR =
+ "/tmp/TestAzureBlobFileSystemMainOperations";
+
+ private final ABFSContractTestBinding binding;
+
+ public ITestAzureBlobFileSystemMainOperation () throws Exception {
+ super(TEST_ROOT_DIR);
+ // Note: There are shared resources in this test suite (eg: "test/new/newfile")
+ // To make sure this test suite can be ran in parallel, different containers
+ // will be used for each test.
+ binding = new ABFSContractTestBinding(false);
+ }
+
+ @Override
+ public void setUp() throws Exception {
+ binding.setup();
+ fSys = binding.getFileSystem();
+ }
+
+ @Override
+ public void tearDown() throws Exception {
+ // Note: Because "tearDown()" is called during the testing,
+ // here we should not call binding.tearDown() to destroy the container.
+ // Instead we should remove the test containers manually with
+ // AbfsTestUtils.
+ super.tearDown();
+ }
+
+ @Override
+ protected FileSystem createFileSystem() throws Exception {
+ return fSys;
+ }
+
+ @Override
+ @Ignore("Permission check for getFileInfo doesn't match the HdfsPermissionsGuide")
+ public void testListStatusThrowsExceptionForUnreadableDir() {
+ // Permission Checks:
+ // https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html
+ }
+
+ @Override
+ @Ignore("Permission check for getFileInfo doesn't match the HdfsPermissionsGuide")
+ public void testGlobStatusThrowsExceptionForUnreadableDir() {
+ // Permission Checks:
+ // https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 01/09: HADOOP-16242. ABFS: add bufferpool to
AbfsOutputStream.
Posted by tm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tmarquardt pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit e5bb8498df2b22495ae58829e7125d1377cc765a
Author: Da Zhou <da...@microsoft.com>
AuthorDate: Mon Apr 29 13:27:28 2019 +0100
HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream.
Contributed by Da Zhou.
---
.../hadoop/fs/azurebfs/services/AbfsOutputStream.java | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
index 5764bcb..679f22e 100644
--- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
+++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
@@ -23,6 +23,7 @@ import java.io.IOException;
import java.io.InterruptedIOException;
import java.io.OutputStream;
import java.net.HttpURLConnection;
+import java.nio.ByteBuffer;
import java.util.Locale;
import java.util.concurrent.ConcurrentLinkedDeque;
import java.util.concurrent.LinkedBlockingQueue;
@@ -37,6 +38,7 @@ import com.google.common.base.Preconditions;
import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.io.ElasticByteBufferPool;
import org.apache.hadoop.fs.FSExceptionMessages;
import org.apache.hadoop.fs.StreamCapabilities;
import org.apache.hadoop.fs.Syncable;
@@ -64,6 +66,15 @@ public class AbfsOutputStream extends OutputStream implements Syncable, StreamCa
private final ThreadPoolExecutor threadExecutor;
private final ExecutorCompletionService<Void> completionService;
+ /**
+ * Queue storing buffers with the size of the Azure block ready for
+ * reuse. The pool allows reusing the blocks instead of allocating new
+ * blocks. After the data is sent to the service, the buffer is returned
+ * back to the queue
+ */
+ private final ElasticByteBufferPool byteBufferPool
+ = new ElasticByteBufferPool();
+
public AbfsOutputStream(
final AbfsClient client,
final String path,
@@ -78,7 +89,7 @@ public class AbfsOutputStream extends OutputStream implements Syncable, StreamCa
this.lastError = null;
this.lastFlushOffset = 0;
this.bufferSize = bufferSize;
- this.buffer = new byte[bufferSize];
+ this.buffer = byteBufferPool.getBuffer(false, bufferSize).array();
this.bufferIndex = 0;
this.writeOperations = new ConcurrentLinkedDeque<>();
@@ -268,8 +279,7 @@ public class AbfsOutputStream extends OutputStream implements Syncable, StreamCa
final byte[] bytes = buffer;
final int bytesLength = bufferIndex;
-
- buffer = new byte[bufferSize];
+ buffer = byteBufferPool.getBuffer(false, bufferSize).array();
bufferIndex = 0;
final long offset = position;
position += bytesLength;
@@ -283,6 +293,7 @@ public class AbfsOutputStream extends OutputStream implements Syncable, StreamCa
public Void call() throws Exception {
client.append(path, offset, bytes, 0,
bytesLength);
+ byteBufferPool.putBuffer(ByteBuffer.wrap(bytes));
return null;
}
});
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 06/09: HADOOP-16404. ABFS default blocksize change(256MB
from 512MB)
Posted by tm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tmarquardt pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 8044415ea1936c0dec4dba8b429c32a96e567e9a
Author: Arun Singh <Ar...@microsoft.com>
AuthorDate: Fri Jul 19 20:21:28 2019 -0700
HADOOP-16404. ABFS default blocksize change(256MB from 512MB)
Contributed by: Arun Singh
---
.../hadoop/fs/azurebfs/constants/FileSystemConfigurations.java | 2 +-
.../hadoop/fs/azurebfs/TestAbfsConfigurationFieldsValidation.java | 6 ++++++
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java
index 9744307..a2a0064 100644
--- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java
+++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java
@@ -44,7 +44,7 @@ public final class FileSystemConfigurations {
public static final int DEFAULT_READ_BUFFER_SIZE = 4 * ONE_MB; // 4 MB
public static final int MIN_BUFFER_SIZE = 16 * ONE_KB; // 16 KB
public static final int MAX_BUFFER_SIZE = 100 * ONE_MB; // 100 MB
- public static final long MAX_AZURE_BLOCK_SIZE = 512 * 1024 * 1024L;
+ public static final long MAX_AZURE_BLOCK_SIZE = 256 * 1024 * 1024L; // changing default abfs blocksize to 256MB
public static final String AZURE_BLOCK_LOCATION_HOST_DEFAULT = "localhost";
public static final int MAX_CONCURRENT_READ_THREADS = 12;
diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsConfigurationFieldsValidation.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsConfigurationFieldsValidation.java
index a78602b..2a65263 100644
--- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsConfigurationFieldsValidation.java
+++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsConfigurationFieldsValidation.java
@@ -144,6 +144,12 @@ public class TestAbfsConfigurationFieldsValidation {
}
@Test
+ public void testConfigBlockSizeInitialized() throws Exception {
+ // test the block size annotated field has been initialized in the constructor
+ assertEquals(MAX_AZURE_BLOCK_SIZE, abfsConfiguration.getAzureBlockSize());
+ }
+
+ @Test
public void testGetAccountKey() throws Exception {
String accountKey = abfsConfiguration.getStorageAccountKey();
assertEquals(this.encodedAccountKey, accountKey);
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 05/09: HADOOP-16376. ABFS: Override access() to no-op.
Posted by tm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tmarquardt pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 42816bf698f7c312dce18ff1749e64598973f09d
Author: Da Zhou <da...@microsoft.com>
AuthorDate: Sun Jun 16 19:20:46 2019 +0100
HADOOP-16376. ABFS: Override access() to no-op.
Contributed by Da Zhou.
Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0
---
.../hadoop/fs/azurebfs/AzureBlobFileSystem.java | 23 ++++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
index e321e9e..1663ed9 100644
--- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
+++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java
@@ -38,12 +38,12 @@ import java.util.concurrent.Future;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
-import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
-import org.apache.hadoop.fs.azurebfs.services.AbfsClientThrottlingIntercept;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.commons.lang3.ArrayUtils;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClient;
+import org.apache.hadoop.fs.azurebfs.services.AbfsClientThrottlingIntercept;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.BlockLocation;
@@ -70,6 +70,7 @@ import org.apache.hadoop.fs.permission.AclEntry;
import org.apache.hadoop.fs.permission.AclStatus;
import org.apache.hadoop.fs.permission.FsAction;
import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.security.AccessControlException;
import org.apache.hadoop.security.token.Token;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.util.Progressable;
@@ -839,6 +840,24 @@ public class AzureBlobFileSystem extends FileSystem {
}
}
+ /**
+ * Checks if the user can access a path. The mode specifies which access
+ * checks to perform. If the requested permissions are granted, then the
+ * method returns normally. If access is denied, then the method throws an
+ * {@link AccessControlException}.
+ *
+ * @param path Path to check
+ * @param mode type of access to check
+ * @throws AccessControlException if access is denied
+ * @throws java.io.FileNotFoundException if the path does not exist
+ * @throws IOException see specific implementation
+ */
+ @Override
+ public void access(final Path path, FsAction mode) throws IOException {
+ // TODO: make it no-op to unblock hive permission issue for now.
+ // Will add a long term fix similar to the implementation in AdlFileSystem.
+ }
+
private FileStatus tryGetFileStatus(final Path f) {
try {
return getFileStatus(f);
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 02/09: HADOOP-16269. ABFS: add listFileStatus with
StartFrom.
Posted by tm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tmarquardt pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 50953c586b1a96e46119fdb6a3366309da389985
Author: Da Zhou <da...@microsoft.com>
AuthorDate: Wed May 8 17:20:46 2019 +0100
HADOOP-16269. ABFS: add listFileStatus with StartFrom.
Author: Da Zhou
---
.../fs/azurebfs/AzureBlobFileSystemStore.java | 107 ++++++++++++++-
.../fs/azurebfs/constants/AbfsHttpConstants.java | 9 ++
.../org/apache/hadoop/fs/azurebfs/utils/CRC64.java | 60 ++++++++
.../fs/azurebfs/AbstractAbfsIntegrationTest.java | 7 +-
...zureBlobFileSystemStoreListStatusWithRange.java | 151 +++++++++++++++++++++
.../apache/hadoop/fs/azurebfs/TestAbfsCrc64.java | 38 ++++++
6 files changed, 363 insertions(+), 9 deletions(-)
diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
index 4d11563..f36153d 100644
--- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
+++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
@@ -31,6 +31,7 @@ import java.nio.charset.CharacterCodingException;
import java.nio.charset.Charset;
import java.nio.charset.CharsetDecoder;
import java.nio.charset.CharsetEncoder;
+import java.nio.charset.StandardCharsets;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
@@ -46,6 +47,7 @@ import java.util.Set;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
@@ -79,6 +81,7 @@ import org.apache.hadoop.fs.azurebfs.services.AuthType;
import org.apache.hadoop.fs.azurebfs.services.ExponentialRetryPolicy;
import org.apache.hadoop.fs.azurebfs.services.SharedKeyCredentials;
import org.apache.hadoop.fs.azurebfs.utils.Base64;
+import org.apache.hadoop.fs.azurebfs.utils.CRC64;
import org.apache.hadoop.fs.azurebfs.utils.UriUtils;
import org.apache.hadoop.fs.permission.AclEntry;
import org.apache.hadoop.fs.permission.AclStatus;
@@ -89,7 +92,17 @@ import org.apache.http.client.utils.URIBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.CHAR_EQUALS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.CHAR_FORWARD_SLASH;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.CHAR_HYPHEN;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.CHAR_PLUS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.CHAR_STAR;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.CHAR_UNDERSCORE;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.SINGLE_WHITE_SPACE;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.TOKEN_VERSION;
import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.AZURE_ABFS_ENDPOINT;
+
/**
* Provides the bridging logic between Hadoop's abstract filesystem and Azure Storage.
*/
@@ -103,6 +116,7 @@ public class AzureBlobFileSystemStore {
private String userName;
private String primaryUserGroup;
private static final String DATE_TIME_PATTERN = "E, dd MMM yyyy HH:mm:ss 'GMT'";
+ private static final String TOKEN_DATE_PATTERN = "yyyy-MM-dd'T'HH:mm:ss.SSSSSSS'Z'";
private static final String XMS_PROPERTIES_ENCODING = "ISO-8859-1";
private static final int LIST_MAX_RESULTS = 500;
@@ -514,15 +528,43 @@ public class AzureBlobFileSystemStore {
eTag);
}
+ /**
+ * @param path The list path.
+ * @return the entries in the path.
+ * */
public FileStatus[] listStatus(final Path path) throws IOException {
- LOG.debug("listStatus filesystem: {} path: {}",
+ return listStatus(path, null);
+ }
+
+ /**
+ * @param path Path the list path.
+ * @param startFrom the entry name that list results should start with.
+ * For example, if folder "/folder" contains four files: "afile", "bfile", "hfile", "ifile".
+ * Then listStatus(Path("/folder"), "hfile") will return "/folder/hfile" and "folder/ifile"
+ * Notice that if startFrom is a non-existent entry name, then the list response contains
+ * all entries after this non-existent entry in lexical order:
+ * listStatus(Path("/folder"), "cfile") will return "/folder/hfile" and "/folder/ifile".
+ *
+ * @return the entries in the path start from "startFrom" in lexical order.
+ * */
+ @InterfaceStability.Unstable
+ public FileStatus[] listStatus(final Path path, final String startFrom) throws IOException {
+ LOG.debug("listStatus filesystem: {} path: {}, startFrom: {}",
client.getFileSystem(),
- path);
+ path,
+ startFrom);
- String relativePath = path.isRoot() ? AbfsHttpConstants.EMPTY_STRING : getRelativePath(path);
+ final String relativePath = path.isRoot() ? AbfsHttpConstants.EMPTY_STRING : getRelativePath(path);
String continuation = null;
- ArrayList<FileStatus> fileStatuses = new ArrayList<>();
+ // generate continuation token if a valid startFrom is provided.
+ if (startFrom != null && !startFrom.isEmpty()) {
+ continuation = getIsNamespaceEnabled()
+ ? generateContinuationTokenForXns(startFrom)
+ : generateContinuationTokenForNonXns(path.isRoot() ? ROOT_PATH : relativePath, startFrom);
+ }
+
+ ArrayList<FileStatus> fileStatuses = new ArrayList<>();
do {
AbfsRestOperation op = client.listPath(relativePath, false, LIST_MAX_RESULTS, continuation);
continuation = op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_CONTINUATION);
@@ -575,6 +617,61 @@ public class AzureBlobFileSystemStore {
return fileStatuses.toArray(new FileStatus[fileStatuses.size()]);
}
+ // generate continuation token for xns account
+ private String generateContinuationTokenForXns(final String firstEntryName) {
+ Preconditions.checkArgument(!Strings.isNullOrEmpty(firstEntryName)
+ && !firstEntryName.startsWith(AbfsHttpConstants.ROOT_PATH),
+ "startFrom must be a dir/file name and it can not be a full path");
+
+ StringBuilder sb = new StringBuilder();
+ sb.append(firstEntryName).append("#$").append("0");
+
+ CRC64 crc64 = new CRC64();
+ StringBuilder token = new StringBuilder();
+ token.append(crc64.compute(sb.toString().getBytes(StandardCharsets.UTF_8)))
+ .append(SINGLE_WHITE_SPACE)
+ .append("0")
+ .append(SINGLE_WHITE_SPACE)
+ .append(firstEntryName);
+
+ return Base64.encode(token.toString().getBytes(StandardCharsets.UTF_8));
+ }
+
+ // generate continuation token for non-xns account
+ private String generateContinuationTokenForNonXns(final String path, final String firstEntryName) {
+ Preconditions.checkArgument(!Strings.isNullOrEmpty(firstEntryName)
+ && !firstEntryName.startsWith(AbfsHttpConstants.ROOT_PATH),
+ "startFrom must be a dir/file name and it can not be a full path");
+
+ // Notice: non-xns continuation token requires full path (first "/" is not included) for startFrom
+ final String startFrom = (path.isEmpty() || path.equals(ROOT_PATH))
+ ? firstEntryName
+ : path + ROOT_PATH + firstEntryName;
+
+ SimpleDateFormat simpleDateFormat = new SimpleDateFormat(TOKEN_DATE_PATTERN, Locale.US);
+ String date = simpleDateFormat.format(new Date());
+ String token = String.format("%06d!%s!%06d!%s!%06d!%s!",
+ path.length(), path, startFrom.length(), startFrom, date.length(), date);
+ String base64EncodedToken = Base64.encode(token.getBytes(StandardCharsets.UTF_8));
+
+ StringBuilder encodedTokenBuilder = new StringBuilder(base64EncodedToken.length() + 5);
+ encodedTokenBuilder.append(String.format("%s!%d!", TOKEN_VERSION, base64EncodedToken.length()));
+
+ for (int i = 0; i < base64EncodedToken.length(); i++) {
+ char current = base64EncodedToken.charAt(i);
+ if (CHAR_FORWARD_SLASH == current) {
+ current = CHAR_UNDERSCORE;
+ } else if (CHAR_PLUS == current) {
+ current = CHAR_STAR;
+ } else if (CHAR_EQUALS == current) {
+ current = CHAR_HYPHEN;
+ }
+ encodedTokenBuilder.append(current);
+ }
+
+ return encodedTokenBuilder.toString();
+ }
+
public void setOwner(final Path path, final String owner, final String group) throws
AzureBlobFileSystemException {
if (!getIsNamespaceEnabled()) {
@@ -990,7 +1087,7 @@ public class AzureBlobFileSystemStore {
FileStatus other = (FileStatus) obj;
- if (!other.equals(this)) {// compare the path
+ if (!this.getPath().equals(other.getPath())) {// compare the path
return false;
}
diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java
index 1f35854..e85c7f0 100644
--- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java
+++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java
@@ -39,6 +39,7 @@ public final class AbfsHttpConstants {
public static final String GET_ACCESS_CONTROL = "getAccessControl";
public static final String GET_STATUS = "getStatus";
public static final String DEFAULT_TIMEOUT = "90";
+ public static final String TOKEN_VERSION = "2";
public static final String JAVA_VERSION = "java.version";
public static final String OS_NAME = "os.name";
@@ -91,5 +92,13 @@ public final class AbfsHttpConstants {
public static final String PERMISSION_FORMAT = "%04d";
public static final String SUPER_USER = "$superuser";
+ public static final char CHAR_FORWARD_SLASH = '/';
+ public static final char CHAR_EXCLAMATION_POINT = '!';
+ public static final char CHAR_UNDERSCORE = '_';
+ public static final char CHAR_HYPHEN = '-';
+ public static final char CHAR_EQUALS = '=';
+ public static final char CHAR_STAR = '*';
+ public static final char CHAR_PLUS = '+';
+
private AbfsHttpConstants() {}
}
diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/CRC64.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/CRC64.java
new file mode 100644
index 0000000..9790744
--- /dev/null
+++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/CRC64.java
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.azurebfs.utils;
+
+/**
+ * CRC64 implementation for AzureBlobFileSystem.
+ */
+public class CRC64 {
+
+ private static final long POLY = 0x9a6c9329ac4bc9b5L;
+ private static final int TABLE_LENGTH = 256;
+ private static final long[] TABLE = new long[TABLE_LENGTH];
+
+ private long value = -1;
+
+ /**
+ * @param input byte arrays.
+ * @return long value of the CRC-64 checksum of the data.
+ * */
+ public long compute(byte[] input) {
+ init();
+ for (int i = 0; i < input.length; i++) {
+ value = TABLE[(input[i] ^ (int) value) & 0xFF] ^ (value >>> 8);
+ }
+ return ~value;
+ }
+
+ /*
+ * Initialize a table constructed from POLY (0x9a6c9329ac4bc9b5L).
+ * */
+ private void init() {
+ value = -1;
+ for (int n = 0; n < TABLE_LENGTH; ++n) {
+ long crc = n;
+ for (int i = 0; i < 8; ++i) {
+ if ((crc & 1) == 1) {
+ crc = (crc >>> 1) ^ POLY;
+ } else {
+ crc >>>= 1;
+ }
+ }
+ TABLE[n] = crc;
+ }
+ }
+}
diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java
index cb9549d..04be4f4 100644
--- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java
+++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java
@@ -24,7 +24,6 @@ import java.util.Hashtable;
import java.util.UUID;
import java.util.concurrent.Callable;
-import com.google.common.base.Preconditions;
import org.junit.After;
import org.junit.Before;
import org.slf4j.Logger;
@@ -210,9 +209,9 @@ public abstract class AbstractAbfsIntegrationTest extends
* @throws IOException failure during create/init.
*/
public AzureBlobFileSystem createFileSystem() throws IOException {
- Preconditions.checkState(abfs == null,
- "existing ABFS instance exists: %s", abfs);
- abfs = (AzureBlobFileSystem) FileSystem.newInstance(rawConfig);
+ if (abfs == null) {
+ abfs = (AzureBlobFileSystem) FileSystem.newInstance(rawConfig);
+ }
return abfs;
}
diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemStoreListStatusWithRange.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemStoreListStatusWithRange.java
new file mode 100644
index 0000000..849bb6b
--- /dev/null
+++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemStoreListStatusWithRange.java
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Test AzureBlobFileSystemStore listStatus with startFrom.
+ * */
+@RunWith(Parameterized.class)
+public class ITestAzureBlobFileSystemStoreListStatusWithRange extends
+ AbstractAbfsIntegrationTest {
+ private static final boolean SUCCEED = true;
+ private static final boolean FAIL = false;
+ private static final String[] SORTED_ENTRY_NAMES = {"1_folder", "A0", "D01", "a+", "c0", "name5"};
+
+ private AzureBlobFileSystemStore store;
+ private AzureBlobFileSystem fs;
+
+ @Parameterized.Parameter
+ public String path;
+
+ /**
+ * A valid startFrom for listFileStatus with range is a non-fully qualified dir/file name
+ * */
+ @Parameterized.Parameter(1)
+ public String startFrom;
+
+ @Parameterized.Parameter(2)
+ public int expectedStartIndexInArray;
+
+ @Parameterized.Parameter(3)
+ public boolean expectedResult;
+
+ @Parameterized.Parameters(name = "Testing path \"{0}\", startFrom: \"{1}\", Expecting result : {3}") // Test path
+ public static Iterable<Object[]> params() {
+ return Arrays.asList(
+ new Object[][]{
+ // case 0: list in root, without range
+ {"/", null, 0, SUCCEED},
+
+ // case 1: list in the root, start from the second file
+ {"/", SORTED_ENTRY_NAMES[1], 1, SUCCEED},
+
+ // case 2: list in the root, invalid startFrom
+ {"/", "/", -1, FAIL},
+
+ // case 3: list in non-root level, valid startFrom : dir name
+ {"/" + SORTED_ENTRY_NAMES[2], SORTED_ENTRY_NAMES[1], 1, SUCCEED},
+
+ // case 4: list in non-root level, valid startFrom : file name
+ {"/" + SORTED_ENTRY_NAMES[2], SORTED_ENTRY_NAMES[2], 2, SUCCEED},
+
+ // case 5: list in non root level, invalid startFrom
+ {"/" + SORTED_ENTRY_NAMES[2], "/" + SORTED_ENTRY_NAMES[3], -1, FAIL},
+
+ // case 6: list using non existent startFrom, startFrom is smaller than the entries in lexical order
+ // expecting return all entries
+ {"/" + SORTED_ENTRY_NAMES[2], "0-non-existent", 0, SUCCEED},
+
+ // case 7: list using non existent startFrom, startFrom is larger than the entries in lexical order
+ // expecting return 0 entries
+ {"/" + SORTED_ENTRY_NAMES[2], "z-non-existent", -1, SUCCEED},
+
+ // case 8: list using non existent startFrom, startFrom is in the range
+ {"/" + SORTED_ENTRY_NAMES[2], "A1", 2, SUCCEED}
+ });
+ }
+
+ public ITestAzureBlobFileSystemStoreListStatusWithRange() throws Exception {
+ super();
+ if (this.getFileSystem() == null) {
+ super.createFileSystem();
+ }
+ fs = this.getFileSystem();
+ store = fs.getAbfsStore();
+ prepareTestFiles();
+ // Sort the names for verification, ABFS service should return the results in order.
+ Arrays.sort(SORTED_ENTRY_NAMES);
+ }
+
+ @Test
+ public void testListWithRange() throws IOException {
+ try {
+ FileStatus[] listResult = store.listStatus(new Path(path), startFrom);
+ if (!expectedResult) {
+ Assert.fail("Excepting failure with IllegalArgumentException");
+ }
+ verifyFileStatus(listResult, new Path(path), expectedStartIndexInArray);
+ } catch (IllegalArgumentException ex) {
+ if (expectedResult) {
+ Assert.fail("Excepting success");
+ }
+ }
+ }
+
+ // compare the file status
+ private void verifyFileStatus(FileStatus[] listResult, Path parentPath, int startIndexInSortedName) throws IOException {
+ if (startIndexInSortedName == -1) {
+ Assert.assertEquals("Expected empty FileStatus array", 0, listResult.length);
+ return;
+ }
+
+ FileStatus[] allFileStatuses = fs.listStatus(parentPath);
+ Assert.assertEquals("number of dir/file doesn't match",
+ SORTED_ENTRY_NAMES.length, allFileStatuses.length);
+ int indexInResult = 0;
+ for (int index = startIndexInSortedName; index < SORTED_ENTRY_NAMES.length; index++) {
+ Assert.assertEquals("fileStatus doesn't match", allFileStatuses[index], listResult[indexInResult++]);
+ }
+ }
+
+ private void prepareTestFiles() throws IOException {
+ final AzureBlobFileSystem fs = getFileSystem();
+ // created 2 level file structures
+ for (String levelOneFolder : SORTED_ENTRY_NAMES) {
+ Path levelOnePath = new Path("/" + levelOneFolder);
+ Assert.assertTrue(fs.mkdirs(levelOnePath));
+ for (String fileName : SORTED_ENTRY_NAMES) {
+ Path filePath = new Path(levelOnePath, fileName);
+ ContractTestUtils.touch(fs, filePath);
+ ContractTestUtils.assertIsFile(fs, filePath);
+ }
+ }
+ }
+}
diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsCrc64.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsCrc64.java
new file mode 100644
index 0000000..ab39750
--- /dev/null
+++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsCrc64.java
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.azurebfs;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.apache.hadoop.fs.azurebfs.utils.CRC64;
+/**
+ * Test for Crc64 in AzureBlobFileSystem, notice that ABFS CRC64 has its own polynomial.
+ * */
+public class TestAbfsCrc64 {
+
+ @Test
+ public void tesCrc64Compute() {
+ CRC64 crc64 = new CRC64();
+ final String[] testStr = {"#$", "dir_2_ac83abee", "dir_42_976df1f5"};
+ final String[] expected = {"f91f7e6a837dbfa8", "203f9fefc38ae97b", "cc0d56eafe58a855"};
+ for (int i = 0; i < testStr.length; i++) {
+ Assert.assertEquals(expected[i], Long.toHexString(crc64.compute(testStr[i].getBytes())));
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 09/09: HADOOP-16460: ABFS: fix for Sever Name Indication
(SNI)
Posted by tm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tmarquardt pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit f56972bfe1c79c11a13a88284ca5286709967b4a
Author: Sneha Vijayarajan <sn...@microsoft.com>
AuthorDate: Tue Jul 30 15:18:15 2019 +0000
HADOOP-16460: ABFS: fix for Sever Name Indication (SNI)
Contributed by Sneha Vijayarajan <sn...@microsoft.com>
---
hadoop-project/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 4c0464a..f437ec3 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -1121,7 +1121,7 @@
<dependency>
<groupId>org.wildfly.openssl</groupId>
<artifactId>wildfly-openssl</artifactId>
- <version>1.0.4.Final</version>
+ <version>1.0.7.Final</version>
</dependency>
<dependency>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 07/09: HADOOP-16479. ABFS FileStatus.getModificationTime
returns localized time instead of UTC.
Posted by tm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tmarquardt pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 1376f2ed3dc26b98cb9c92b10797ad0f99377b58
Author: bilaharith <th...@gmail.com>
AuthorDate: Thu Aug 8 19:08:04 2019 +0100
HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC.
Contributed by Bilahari T H
Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0
---
.../hadoop/fs/azurebfs/AzureBlobFileSystemStore.java | 2 +-
.../azurebfs/ITestAzureBlobFileSystemFileStatus.java | 18 ++++++++++++++++++
2 files changed, 19 insertions(+), 1 deletion(-)
diff --git a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
index f36153d..27ba202 100644
--- a/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
+++ b/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
@@ -115,7 +115,7 @@ public class AzureBlobFileSystemStore {
private URI uri;
private String userName;
private String primaryUserGroup;
- private static final String DATE_TIME_PATTERN = "E, dd MMM yyyy HH:mm:ss 'GMT'";
+ private static final String DATE_TIME_PATTERN = "E, dd MMM yyyy HH:mm:ss z";
private static final String TOKEN_DATE_PATTERN = "yyyy-MM-dd'T'HH:mm:ss.SSSSSSS'Z'";
private static final String XMS_PROPERTIES_ENCODING = "ISO-8859-1";
private static final int LIST_MAX_RESULTS = 500;
diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemFileStatus.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemFileStatus.java
index f514696..421fa9a 100644
--- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemFileStatus.java
+++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemFileStatus.java
@@ -122,4 +122,22 @@ public class ITestAzureBlobFileSystemFileStatus extends
assertEquals(pathWithHost2.getName(), fileStatus2.getPath().getName());
}
+ @Test
+ public void testLastModifiedTime() throws IOException {
+ AzureBlobFileSystem fs = this.getFileSystem();
+ Path testFilePath = new Path("childfile1.txt");
+ long createStartTime = System.currentTimeMillis();
+ long minCreateStartTime = (createStartTime / 1000) * 1000 - 1;
+ // Dividing and multiplying by 1000 to make last 3 digits 0.
+ // It is observed that modification time is returned with last 3
+ // digits 0 always.
+ fs.create(testFilePath);
+ long createEndTime = System.currentTimeMillis();
+ FileStatus fStat = fs.getFileStatus(testFilePath);
+ long lastModifiedTime = fStat.getModificationTime();
+ assertTrue("lastModifiedTime should be after minCreateStartTime",
+ minCreateStartTime < lastModifiedTime);
+ assertTrue("lastModifiedTime should be before createEndTime",
+ createEndTime > lastModifiedTime);
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 04/09: Revert "HADOOP-16251. ABFS: add
FSMainOperationsBaseTest"
Posted by tm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tmarquardt pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 67fe5398fd8400287e4eeaf360cffa4e8788447e
Author: Aaron Fabbri <fa...@apache.org>
AuthorDate: Fri May 10 13:55:56 2019 -0700
Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest"
This reverts commit 7c2d7c05a9a9cc981674f97cc3825e917a17b1f7.
Git Commit name and email were incorrect. Will re-commit.
---
.../hadoop/fs/azurebfs/ITestAzureBlobFileSystemMainOperation.java | 7 -------
1 file changed, 7 deletions(-)
diff --git a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemMainOperation.java b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemMainOperation.java
index 41abfe8..38682b3 100644
--- a/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemMainOperation.java
+++ b/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemMainOperation.java
@@ -68,11 +68,4 @@ public class ITestAzureBlobFileSystemMainOperation extends FSMainOperationsBaseT
// Permission Checks:
// https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html
}
-
- @Override
- @Ignore("Permission check for getFileInfo doesn't match the HdfsPermissionsGuide")
- public void testGlobStatusThrowsExceptionForUnreadableDir() {
- // Permission Checks:
- // https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html
- }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org