You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/08/03 04:25:04 UTC
[GitHub] [hadoop] JohnZZGithub opened a new pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
JohnZZGithub opened a new pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185
Differential Revision: https://phabricator.twitter.biz/D526212
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-687687926
Thanks for the comments!
> > I guess most caller of getMountPoints wants to traverse all the file systems to do some operation. E.g. setVerifyChecksum(). We didn't see issues on our internal Yarn + HDFS and Yarn + GCS clusters. The usage pattern includes but not limited to MR, Spark, Presto, Vertica loading and etc. But it's possible that some users might rely on these APIs.
>
> In YarnClient seems to be collecting tokens from all DelegationTokenIssuer.
> in DelegationTokenIssuer#collectDelegationTokens
>
> ```
> // Now collect the tokens from the children.
> final DelegationTokenIssuer[] ancillary =
> issuer.getAdditionalTokenIssuers();
> if (ancillary != null) {
> for (DelegationTokenIssuer subIssuer : ancillary) {
> collectDelegationTokens(subIssuer, renewer, credentials, tokens);
> }
> }
> ```
>
> If you look here issuer is current fs and it's trying to get additionalTokenIssuers.
> The default implementation of getDelegationTokenIssuers at FileSystem.java is simply getting all ChildFileSystems.
>
> ```
> @InterfaceAudience.Private
> @Override
> public DelegationTokenIssuer[] getAdditionalTokenIssuers()
> throws IOException {
> return getChildFileSystems();
> }
> ```
>
> This will get all the child file systems available. Currently the implementation is getChildFileSystems in ViewFileSystem is like below:
>
> ```
> @Override
> public FileSystem[] getChildFileSystems() {
> List<InodeTree.MountPoint<FileSystem>> mountPoints =
> fsState.getMountPoints();
> Set<FileSystem> children = new HashSet<FileSystem>();
> for (InodeTree.MountPoint<FileSystem> mountPoint : mountPoints) {
> FileSystem targetFs = mountPoint.target.targetFileSystem;
> children.addAll(Arrays.asList(targetFs.getChildFileSystems()));
> }
>
> if (fsState.isRootInternalDir() && fsState.getRootFallbackLink() != null) {
> children.addAll(Arrays.asList(
> fsState.getRootFallbackLink().targetFileSystem
> .getChildFileSystems()));
> }
> return children.toArray(new FileSystem[]{});
> }
> ```
>
> It's iterating over mount points available getting all targetFileSystems. In the case of REGEX based mount points, we will not have any childFileSystems available via getChildFileSystems call.
> We also implemented ViewDistributedFileSystem to provide hdfs specific API compatibility. Here also we used getChildFileSystems for some APIs.
>
> > Returning a MountPint with special FileSystem for Regex Mount points. We could cache the initialized fileSystem under the regex mountpoint and perform the operation. For filesystems that might appear in the future, we could cache the past calls from callers and try to apply it or just not support it.
>
> I am thinking that, how about adding the resolved mount points from RegxBased to MountPoints list? So, that when user calls getMounts, it will simply return whatever mount points so far inited. How many unique mount points could be there in total with Regx based in practice (resolved mappings)? We should document that, with RegEX based mount points, getMountPoints will return only currently resolved mount points.
Totally make sense. I agree it won't work well with delegation tokens now. A few context here, the regex mount point feature was built more than 2 years ago. It was before Delegation Tokens was introduced (HADOOP-14556). The version we are using is also before HADOOP-14556. I agree we need to do more work to support delegation tokens.
+1 on document it. Internally, we are using a mixed model of regex mount points and normal mount points. We mainly use regex mount points for GCS buckets. The difference from HDFS is we could use limited user namespaces to represent /user. But it's hard to do it for cloud storage as we will have many more buckets. We could make it clear to users the pros and cons of regex mount points.
> > We did see an issue with addDelegationTokens in the secure Hadoop cluster. But the problem we met is not all normal mountpoints are secure. So the API caused a problem when it tries to initialize all children's file systems. We took a workaround by making it path-based. As for getDelegationTokens, I guess the problem is similar. We didn't see issues because it's not used. Could we make it path based too?
>
> Certainly we can make it uri path based. However users need to make use of it and it could be a long term improvement because users would not change immediately to new APIs what we introduce now. It will take longer time for upstream projects to change.
>
> > Could we make the inner cache a thread-safe structure and track all the opened file systems under regex mount points?
>
> Let's target to solve this problem first. Yes, I think maintaining initialized fs-es in InnerCache could help to close fileSystems correctly. Let's make it to thread-safe and add opened fs-es there.
Thanks, let me fix it first. Do you think it makes sense to fix the close problem first and fix other issues in other PRs with subtasks?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r485332518
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##########
@@ -217,6 +239,7 @@ public Path getMountedOnPath() {
Path homeDir = null;
private boolean enableInnerCache = false;
private InnerCache cache;
+ private boolean evictCacheOnClose = false;
Review comment:
Do you see any issue if we make it true? If no issues, we can simply clean it on close right instead of having another config?
Seems like this is an improvement to existing code. If you want, I am ok to file small JIRA and fix this cleanup thing.( I am assuming it's not necessarily needed with this.)
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestRegexMountPoint.java
##########
@@ -0,0 +1,160 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test Regex Mount Point.
+ */
+public class TestRegexMountPoint {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(TestRegexMountPoint.class.getName());
+
+ private InodeTree inodeTree;
+ private Configuration conf;
+
+ class TestRegexMountPointFileSystem {
+ public URI getUri() {
+ return uri;
+ }
+
+ private URI uri;
+
+ TestRegexMountPointFileSystem(URI uri) {
+ String uriStr = uri == null ? "null" : uri.toString();
+ LOGGER.info("Create TestRegexMountPointFileSystem Via URI:" + uriStr);
+ this.uri = uri;
+ }
+ }
+
+ @Before
+ public void setUp() throws Exception {
+ conf = new Configuration();
+ ConfigUtil.addLink(conf, TestRegexMountPoint.class.getName(), "/mnt",
+ URI.create("file:///"));
+
+ inodeTree = new InodeTree<TestRegexMountPointFileSystem>(conf,
+ TestRegexMountPoint.class.getName(), null, false) {
+ @Override
+ protected TestRegexMountPointFileSystem getTargetFileSystem(
+ final URI uri) {
+ return new TestRegexMountPointFileSystem(uri);
+ }
+
+ @Override
+ protected TestRegexMountPointFileSystem getTargetFileSystem(
+ final INodeDir<TestRegexMountPointFileSystem> dir) {
+ return new TestRegexMountPointFileSystem(null);
+ }
+
+ @Override
+ protected TestRegexMountPointFileSystem getTargetFileSystem(
+ final String settings, final URI[] mergeFsURIList) {
+ return new TestRegexMountPointFileSystem(null);
+ }
+ };
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ inodeTree = null;
+ }
+
+ @Test
+ public void testGetVarListInString() throws IOException {
+ String srcRegex = "/(\\w+)";
+ String target = "/$0/${1}/$1/${2}/${2}";
+ RegexMountPoint regexMountPoint =
+ new RegexMountPoint(inodeTree, srcRegex, target, null);
+ regexMountPoint.initialize();
+ Map<String, Set<String>> varMap = regexMountPoint.getVarInDestPathMap();
+ Assert.assertEquals(varMap.size(), 3);
+ Assert.assertEquals(varMap.get("0").size(), 1);
+ Assert.assertTrue(varMap.get("0").contains("$0"));
+ Assert.assertEquals(varMap.get("1").size(), 2);
+ Assert.assertTrue(varMap.get("1").contains("${1}"));
+ Assert.assertTrue(varMap.get("1").contains("$1"));
+ Assert.assertEquals(varMap.get("2").size(), 1);
+ Assert.assertTrue(varMap.get("2").contains("${2}"));
+ }
+
+ @Test
+ public void testResolve() throws IOException {
+ String regexStr = "^/user/(?<username>\\w+)";
+ String dstPathStr = "/namenode1/testResolve/$username";
+ String settingsStr = null;
+ RegexMountPoint regexMountPoint =
+ new RegexMountPoint(inodeTree, regexStr, dstPathStr, settingsStr);
+ regexMountPoint.initialize();
+ InodeTree.ResolveResult resolveResult =
+ regexMountPoint.resolve("/user/hadoop/file1", true);
+ Assert.assertEquals(resolveResult.kind, InodeTree.ResultKind.EXTERNAL_DIR);
+ Assert.assertTrue(
+ resolveResult.targetFileSystem
+ instanceof TestRegexMountPointFileSystem);
+ Assert.assertTrue(resolveResult.resolvedPath.equals("/user/hadoop"));
+ Assert.assertTrue(
+ resolveResult.targetFileSystem
+ instanceof TestRegexMountPointFileSystem);
+ Assert.assertTrue(
+ ((TestRegexMountPointFileSystem) resolveResult.targetFileSystem)
+ .getUri().toString().equals("/namenode1/testResolve/hadoop"));
+ Assert.assertTrue(resolveResult.remainingPath.toString().equals("/file1"));
+ }
+
+ @Test
+ public void testResolveWithInterceptor() throws IOException {
+ String regexStr = "^/user/(?<username>\\w+)";
+ String dstPathStr = "/namenode1/testResolve/$username";
+ // Replace "_" with "-"
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ new RegexMountPointResolvedDstPathReplaceInterceptor("_", "-");
+ // replaceresolvedpath:_:-
+ String settingsStr = interceptor.serializeToString();
+ RegexMountPoint regexMountPoint =
+ new RegexMountPoint(inodeTree, regexStr, dstPathStr, settingsStr);
+ regexMountPoint.initialize();
+ InodeTree.ResolveResult resolveResult =
+ regexMountPoint.resolve("/user/hadoop_user1/file_index", true);
+ Assert.assertEquals(resolveResult.kind, InodeTree.ResultKind.EXTERNAL_DIR);
+ Assert.assertTrue(
+ resolveResult.targetFileSystem
+ instanceof TestRegexMountPointFileSystem);
+ Assert.assertTrue(resolveResult.resolvedPath.equals("/user/hadoop_user1"));
Review comment:
Still this asserts can use assertEquals method? Please use them all places.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
##########
@@ -366,6 +366,69 @@ Don't want to change scheme or difficult to copy mount-table configurations to a
Please refer to the [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html)
+Regex Pattern Based Mount Points
+--------------------------------
+
+The view file system mount points were a Key-Value based mapping system. It is not friendly for user cases which mapping config could be abstracted to rules. E.g. Users want to provide a GCS bucket per user and there might be thousands of users in total. The old key-value based approach won't work well for several reasons:
+
+1. The mount table is used by FileSystem clients. There's a cost to spread the config to all clients and we should avoid it if possible. The [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html) could help the distribution by central mount table management. But the mount table still have to be updated on every change. The change could be greatly avoided if provide a rule-based mount table.
+
+2. The client have to understand all the KVs in the mount table. This is not ideal when the mountable grows to thousands of items. E.g. thousands of file systems might be initialized even users only need one. And the config itself will become bloated at scale.
+
+### Understand the Difference
+
+In the key-value based mount table, view file system treats every mount point as a partition. There's several file system APIs which will lead to operation on all partitions. E.g. there's an HDFS cluster with multiple mount. Users want to run “hadoop fs -put file viewfs://hdfs.namenode.apache.org/tmp/” cmd to copy data from local disk to our HDFS cluster. The cmd will trigger ViewFileSystem to call setVerifyChecksum() method which will initialize the file system for every mount point.
+For a regex-base rule mount table entry, we couldn't know what's corresponding path until parsing. So the regex based mount table entry will be ignored on such cases. The file system (ChRootedFileSystem) will be created upon accessing. But the underlying file system will be cached by inner cache of ViewFileSystem.
Review comment:
>For a regex-base rule mount table entry, we couldn't know what's corresponding path until parsing.
Whatever we know should be added to mountPoints? So, that getMountPoints will return known fs-es? It may be a good idea to add Java doc on API level.
I am ok to have this in followup JIRA to cover all of this aspects.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkRegex.java
##########
@@ -0,0 +1,473 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.FsConstants;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSNNTopology;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPoint.INTERCEPTOR_INTERNAL_SEP;
+import static org.junit.Assert.assertSame;
+
+/**
+ * Test linkRegex node type for view file system.
+ */
+public class TestViewFileSystemLinkRegex extends ViewFileSystemBaseTest {
+ public static final Logger LOGGER =
+ LoggerFactory.getLogger(TestViewFileSystemLinkRegex.class);
+
+ private static FileSystem fsDefault;
+ private static MiniDFSCluster cluster;
+ private static Configuration clusterConfig;
+ private static final int NAME_SPACES_COUNT = 3;
+ private static final int DATA_NODES_COUNT = 3;
+ private static final int FS_INDEX_DEFAULT = 0;
+ private static final FileSystem[] FS_HDFS = new FileSystem[NAME_SPACES_COUNT];
+ private static final String CLUSTER_NAME =
+ "TestViewFileSystemLinkRegexCluster";
+ private static final File TEST_DIR = GenericTestUtils
+ .getTestDir(TestViewFileSystemLinkRegex.class.getSimpleName());
+ private static final String TEST_BASE_PATH =
+ "/tmp/TestViewFileSystemLinkRegex";
+
+ @Override
+ protected FileSystemTestHelper createFileSystemHelper() {
+ return new FileSystemTestHelper(TEST_BASE_PATH);
+ }
+
+ @BeforeClass
+ public static void clusterSetupAtBeginning() throws IOException {
+ SupportsBlocks = true;
+ clusterConfig = ViewFileSystemTestSetup.createConfig();
+ clusterConfig.setBoolean(
+ DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY,
+ true);
+ cluster = new MiniDFSCluster.Builder(clusterConfig).nnTopology(
+ MiniDFSNNTopology.simpleFederatedTopology(NAME_SPACES_COUNT))
+ .numDataNodes(DATA_NODES_COUNT).build();
+ cluster.waitClusterUp();
+
+ for (int i = 0; i < NAME_SPACES_COUNT; i++) {
+ FS_HDFS[i] = cluster.getFileSystem(i);
+ }
+ fsDefault = FS_HDFS[FS_INDEX_DEFAULT];
+ }
+
+ @AfterClass
+ public static void clusterShutdownAtEnd() throws Exception {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+
+ @Override
+ @Before
+ public void setUp() throws Exception {
+ fsTarget = fsDefault;
+ super.setUp();
+ }
+
+ /**
+ * Override this so that we don't set the targetTestRoot to any path under the
+ * root of the FS, and so that we don't try to delete the test dir, but rather
+ * only its contents.
+ */
+ @Override
+ void initializeTargetTestRoot() throws IOException {
+ targetTestRoot = fsDefault.makeQualified(new Path("/"));
+ for (FileStatus status : fsDefault.listStatus(targetTestRoot)) {
+ fsDefault.delete(status.getPath(), true);
+ }
+ }
+
+ @Override
+ void setupMountPoints() {
+ super.setupMountPoints();
+ }
+
+ @Override
+ int getExpectedDelegationTokenCount() {
+ return 1; // all point to the same fs so 1 unique token
+ }
+
+ @Override
+ int getExpectedDelegationTokenCountWithCredentials() {
+ return 1;
+ }
+
+ public String buildReplaceInterceptorSettingString(String srcRegex,
+ String replaceString) {
+ return
+ RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + INTERCEPTOR_INTERNAL_SEP + srcRegex + INTERCEPTOR_INTERNAL_SEP
+ + replaceString;
+ }
+
+ public String linkInterceptorSettings(
+ List<String> interceptorSettingStrList) {
+ StringBuilder stringBuilder = new StringBuilder();
+ int listSize = interceptorSettingStrList.size();
+ for (int i = 0; i < listSize; ++i) {
+ stringBuilder.append(interceptorSettingStrList.get(i));
+ if (i < listSize - 1) {
+ stringBuilder.append(RegexMountPoint.INTERCEPTOR_SEP);
+ }
+ }
+ return stringBuilder.toString();
+ }
+
+ private void createDirWithChildren(
+ FileSystem fileSystem, Path dir, List<Path> childrenFiles)
+ throws IOException {
+ Assert.assertTrue(fileSystem.mkdirs(dir));
+ int index = 0;
+ for (Path childFile : childrenFiles) {
+ createFile(fileSystem, childFile, index, true);
+ }
+ }
+
+ private void createFile(
+ FileSystem fileSystem, Path file, int dataLenToWrite, boolean overwrite)
+ throws IOException {
+ FSDataOutputStream outputStream = null;
+ try {
+ outputStream = fileSystem.create(file, overwrite);
+ for (int i = 0; i < dataLenToWrite; ++i) {
+ outputStream.writeByte(i);
+ }
+ outputStream.close();
+ } finally {
+ if (outputStream != null) {
+ outputStream.close();
+ }
+ }
+ }
+
+ private void createDirWithChildren(
+ FileSystem fileSystem, Path dir, int childrenFilesCnt)
+ throws IOException {
+ List<Path> childrenFiles = new ArrayList<>(childrenFilesCnt);
+ for (int i = 0; i < childrenFilesCnt; ++i) {
+ childrenFiles.add(new Path(dir, "file" + i));
+ }
+ createDirWithChildren(fileSystem, dir, childrenFiles);
+ }
+
+ /**
+ * The function used to test regex mountpoints.
+ * @param config - get mountable config from this conf
+ * @param regexStr - the src path regex expression that applies to this config
+ * @param dstPathStr - the string of target path
+ * @param interceptorSettings - the serialized interceptor string to be
+ * applied while resolving the mapping
+ * @param dirPathBeforeMountPoint - the src path user passed in to be mapped.
+ * @param expectedResolveResult - the expected path after resolve
+ * dirPathBeforeMountPoint via regex mountpint.
+ * @param childrenFilesCnt - the child files under dirPathBeforeMountPoint to
+ * be created
+ * @throws IOException
+ * @throws URISyntaxException
+ */
+ private void testRegexMountpoint(
+ Configuration config,
+ String regexStr,
+ String dstPathStr,
+ String interceptorSettings,
+ Path dirPathBeforeMountPoint,
+ Path expectedResolveResult,
+ int childrenFilesCnt)
+ throws IOException, URISyntaxException {
+ FileSystem vfs = null;
+ try {
+ // Set up test env
+ createDirWithChildren(
+ fsTarget, expectedResolveResult, childrenFilesCnt);
+ ConfigUtil.addLinkRegex(
+ config, CLUSTER_NAME, regexStr, dstPathStr, interceptorSettings);
+
+ // Asserts
+ URI viewFsUri = new URI(
+ FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ vfs = FileSystem.get(viewFsUri, config);
Review comment:
small suggestion here:
Move above lines out of try block. Then use try(FileSystem vfs = = FileSystem.get(viewFsUri, config)){
}
This should close automatically after block. So, we can remove finally block below?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
##########
@@ -86,12 +86,21 @@
*/
String CONFIG_VIEWFS_LINK_MERGE_SLASH = "linkMergeSlash";
+ /**
+ * Config variable for specifying a regex link which uses regular expressions
+ * as source and target could use group captured in src.
+ * E.g. (^/(?<firstDir>\\w+), /prefix-${firstDir}) =>
+ * (/path1/file1 => /prefix-path1/file1)
+ */
+ String CONFIG_VIEWFS_LINK_REGEX = "linkRegex";
+
FsPermission PERMISSION_555 = new FsPermission((short) 0555);
String CONFIG_VIEWFS_RENAME_STRATEGY = "fs.viewfs.rename.strategy";
/**
* Enable ViewFileSystem to cache all children filesystems in inner cache.
Review comment:
Is the below comment is still valid?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r481531453
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/RegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.regex.PatternSyntaxException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Implementation of RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+class RegexMountPointResolvedDstPathReplaceInterceptor
+ implements RegexMountPointInterceptor {
+
+ private String srcRegexString;
+ private String replaceString;
+ private Pattern srcRegexPattern;
+
+ RegexMountPointResolvedDstPathReplaceInterceptor(String srcRegex,
+ String replaceString) {
+ this.srcRegexString = srcRegex;
+ this.replaceString = replaceString;
+ this.srcRegexPattern = null;
+ }
+
+ public String getSrcRegexString() {
+ return srcRegexString;
+ }
+
+ public String getReplaceString() {
+ return replaceString;
+ }
+
+ public Pattern getSrcRegexPattern() {
+ return srcRegexPattern;
+ }
+
+ @Override
+ public void initialize() throws IOException {
+ try {
+ srcRegexPattern = Pattern.compile(srcRegexString);
+ } catch (PatternSyntaxException ex) {
+ throw new IOException(
+ "Initialize interceptor failed, srcRegx:" + srcRegexString, ex);
+ }
+ }
+
+ /**
+ * Intercept source before resolution.
Review comment:
Sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-690147488
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 28m 24s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 14s | trunk passed |
| +1 :green_heart: | compile | 20m 45s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 17m 51s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 52s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 51s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 53s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 35s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 2m 59s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 31s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 5s | the patch passed |
| +1 :green_heart: | compile | 19m 36s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 19m 36s | the patch passed |
| +1 :green_heart: | compile | 18m 31s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 18m 31s | the patch passed |
| -0 :warning: | checkstyle | 2m 52s | root: The patch generated 1 new + 182 unchanged - 1 fixed = 183 total (was 183) |
| +1 :green_heart: | mvnsite | 2m 51s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 16s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 29s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 6s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 5m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 48s | hadoop-common in the patch passed. |
| -1 :x: | unit | 106m 51s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. |
| | | 322m 34s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestCrcCorruption |
| | hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.TestViewDistributedFileSystem |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy |
| | hadoop.hdfs.TestBlocksScheduledCounter |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/22/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 16f6f77a6cb6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e5fe3262702 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/22/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/22/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/22/testReport/ |
| Max. process+thread count | 3857 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/22/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-689963635
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 38s | trunk passed |
| +1 :green_heart: | compile | 27m 43s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 22m 48s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 34s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 50s | trunk passed |
| +1 :green_heart: | shadedclient | 22m 43s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 32s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 1s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 45s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 6m 13s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 59s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 9s | the patch passed |
| +1 :green_heart: | compile | 22m 11s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 22m 11s | the patch passed |
| +1 :green_heart: | compile | 21m 10s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 21m 10s | the patch passed |
| -0 :warning: | checkstyle | 3m 33s | root: The patch generated 1 new + 182 unchanged - 1 fixed = 183 total (was 183) |
| +1 :green_heart: | mvnsite | 3m 14s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 49s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 46s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 6m 55s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 10m 49s | hadoop-common in the patch passed. |
| -1 :x: | unit | 128m 6s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 23s | The patch does not generate ASF License warnings. |
| | | 351m 30s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/21/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 0d714f4a57d4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e5fe3262702 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/21/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/21/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/21/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/21/testReport/ |
| Max. process+thread count | 2736 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/21/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r481543446
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/RegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.regex.PatternSyntaxException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Implementation of RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+class RegexMountPointResolvedDstPathReplaceInterceptor
+ implements RegexMountPointInterceptor {
+
+ private String srcRegexString;
+ private String replaceString;
+ private Pattern srcRegexPattern;
+
+ RegexMountPointResolvedDstPathReplaceInterceptor(String srcRegex,
+ String replaceString) {
+ this.srcRegexString = srcRegex;
+ this.replaceString = replaceString;
+ this.srcRegexPattern = null;
+ }
+
+ public String getSrcRegexString() {
+ return srcRegexString;
+ }
+
+ public String getReplaceString() {
+ return replaceString;
+ }
+
+ public Pattern getSrcRegexPattern() {
+ return srcRegexPattern;
+ }
+
+ @Override
+ public void initialize() throws IOException {
+ try {
+ srcRegexPattern = Pattern.compile(srcRegexString);
+ } catch (PatternSyntaxException ex) {
+ throw new IOException(
+ "Initialize interceptor failed, srcRegx:" + srcRegexString, ex);
+ }
+ }
+
+ /**
+ * Intercept source before resolution.
+ *
+ * @param source
+ * @return
+ */
+ @Override
+ public String interceptSource(String source) {
+ return source;
+ }
+
+ /**
+ * Intercept resolved path, e.g.
+ * Mount point /^(\\w+)/, ${1}.hadoop.net
+ * If incoming path is /user1/home/tmp/job1,
+ * then the resolved path str will be user1.
+ *
+ * @return intercepted string
+ */
+ @Override public String interceptResolvedDestPathStr(
+ String parsedDestPathStr) {
+ Matcher matcher = srcRegexPattern.matcher(parsedDestPathStr);
+ return matcher.replaceAll(replaceString);
+ }
+
+ /**
+ * Intercept remaining path.
+ *
+ * @return intercepted path
+ */
+ @Override public Path interceptRemainingPath(Path remainingPath) {
Review comment:
This is a great question.
Here's an example:
/user/$userName => s3://$userName.apache.org
/user/hadoop/dir1/dir2/dir3/file3 will be mapped to s3://user.hadoop.apche.org/dir1/dir2/dir3/file3
interceptSource(/user/hadoop/) take sourcePath and won't know which part will be replaced in the final path.
After mapping, the dest will be s3://user.hadoop.apche.org/dir1/dir2/dir3/file3
interceptRemainingPath(dir1/dir2/dir3/file3) could enable users to do process the remaining part (dir1/dir2/dir3/file3) after the mounted path. E.g. take care of bad characters.
But I'm good to remove it if we feel it's cleaner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao merged pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao merged pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-676797592
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 1s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 9m 46s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 37s | trunk passed |
| +1 :green_heart: | compile | 19m 15s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 44s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 44s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 57s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 9s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 9s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 14s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 27s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 6s | the patch passed |
| +1 :green_heart: | compile | 19m 41s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 19m 41s | the patch passed |
| +1 :green_heart: | compile | 18m 33s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 18m 33s | the patch passed |
| -0 :warning: | checkstyle | 2m 43s | root: The patch generated 2 new + 182 unchanged - 1 fixed = 184 total (was 183) |
| +1 :green_heart: | mvnsite | 2m 53s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 48s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 38s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 13s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 5m 36s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 57s | hadoop-common in the patch passed. |
| -1 :x: | unit | 119m 56s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 4s | The patch does not generate ASF License warnings. |
| | | 309m 58s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux e923a46d6648 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9b9f7ea16a2 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/10/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/10/testReport/ |
| Max. process+thread count | 3649 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/10/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-689822943
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 30s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 23s | trunk passed |
| +1 :green_heart: | compile | 21m 17s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 18m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 44s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 57s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 24s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 43s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 12s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 25s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 57s | the patch passed |
| +1 :green_heart: | compile | 18m 46s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 18m 46s | the patch passed |
| +1 :green_heart: | compile | 16m 50s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 16m 50s | the patch passed |
| -0 :warning: | checkstyle | 2m 46s | root: The patch generated 2 new + 182 unchanged - 1 fixed = 184 total (was 183) |
| +1 :green_heart: | mvnsite | 2m 55s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 23s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 40s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 16s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 5m 39s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 43s | hadoop-common in the patch passed. |
| -1 :x: | unit | 96m 58s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. |
| | | 284m 1s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.TestDFSOutputStream |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/20/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux d70552001d55 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 85119267be7 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/20/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/20/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/20/testReport/ |
| Max. process+thread count | 3867 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/20/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r471892967
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
##########
@@ -1430,4 +1432,49 @@ public void testGetContentSummaryWithFileInLocalFS() throws Exception {
summaryAfter.getLength());
}
}
+
+ @Test
+ public void testMountPointCache() throws Exception {
+ conf.setInt(Constants.CONFIG_VIEWFS_PATH_RESOLUTION_CACHE_CAPACITY, 1);
+ conf.setBoolean("fs.viewfs.impl.disable.cache", true);
Review comment:
One more thing to clarify, I guess this config is for per schema level cache. Regex mount point is OK with it. Mount table is not good inner cache inside ViewFileSystem.java.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r467176661
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##########
@@ -486,84 +506,113 @@ protected InodeTree(final Configuration config, final String viewName,
final UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
for (Entry<String, String> si : config) {
final String key = si.getKey();
- if (key.startsWith(mountTablePrefix)) {
- gotMountTableEntry = true;
- LinkType linkType;
- String src = key.substring(mountTablePrefix.length());
- String settings = null;
- if (src.startsWith(linkPrefix)) {
- src = src.substring(linkPrefix.length());
- if (src.equals(SlashPath.toString())) {
- throw new UnsupportedFileSystemException("Unexpected mount table "
- + "link entry '" + key + "'. Use "
- + Constants.CONFIG_VIEWFS_LINK_MERGE_SLASH + " instead!");
- }
- linkType = LinkType.SINGLE;
- } else if (src.startsWith(linkFallbackPrefix)) {
- if (src.length() != linkFallbackPrefix.length()) {
- throw new IOException("ViewFs: Mount points initialization error." +
- " Invalid " + Constants.CONFIG_VIEWFS_LINK_FALLBACK +
- " entry in config: " + src);
- }
- linkType = LinkType.SINGLE_FALLBACK;
- } else if (src.startsWith(linkMergePrefix)) { // A merge link
- src = src.substring(linkMergePrefix.length());
- linkType = LinkType.MERGE;
- } else if (src.startsWith(linkMergeSlashPrefix)) {
- // This is a LinkMergeSlash entry. This entry should
- // not have any additional source path.
- if (src.length() != linkMergeSlashPrefix.length()) {
- throw new IOException("ViewFs: Mount points initialization error." +
- " Invalid " + Constants.CONFIG_VIEWFS_LINK_MERGE_SLASH +
- " entry in config: " + src);
- }
- linkType = LinkType.MERGE_SLASH;
- } else if (src.startsWith(Constants.CONFIG_VIEWFS_LINK_NFLY)) {
- // prefix.settings.src
- src = src.substring(Constants.CONFIG_VIEWFS_LINK_NFLY.length() + 1);
- // settings.src
- settings = src.substring(0, src.indexOf('.'));
- // settings
-
- // settings.src
- src = src.substring(settings.length() + 1);
- // src
-
- linkType = LinkType.NFLY;
- } else if (src.startsWith(Constants.CONFIG_VIEWFS_HOMEDIR)) {
- // ignore - we set home dir from config
- continue;
- } else {
- throw new IOException("ViewFs: Cannot initialize: Invalid entry in " +
- "Mount table in config: " + src);
- }
+ if (!key.startsWith(mountTablePrefix)) {
Review comment:
It looks complex. But the major change here is simple.
Before:
if (key.startsWith(mountTablePrefix)) {
// hundred line of code
}
After:
if (!key.startsWith(mountTablePrefix)) {
continue
}
// hundred line of code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r481883237
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/RegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.regex.PatternSyntaxException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Implementation of RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+class RegexMountPointResolvedDstPathReplaceInterceptor
+ implements RegexMountPointInterceptor {
+
+ private String srcRegexString;
+ private String replaceString;
+ private Pattern srcRegexPattern;
+
+ RegexMountPointResolvedDstPathReplaceInterceptor(String srcRegex,
+ String replaceString) {
+ this.srcRegexString = srcRegex;
+ this.replaceString = replaceString;
+ this.srcRegexPattern = null;
+ }
+
+ public String getSrcRegexString() {
+ return srcRegexString;
+ }
+
+ public String getReplaceString() {
+ return replaceString;
+ }
+
+ public Pattern getSrcRegexPattern() {
+ return srcRegexPattern;
+ }
+
+ @Override
+ public void initialize() throws IOException {
+ try {
+ srcRegexPattern = Pattern.compile(srcRegexString);
+ } catch (PatternSyntaxException ex) {
+ throw new IOException(
+ "Initialize interceptor failed, srcRegx:" + srcRegexString, ex);
+ }
+ }
+
+ /**
+ * Intercept source before resolution.
+ *
+ * @param source
+ * @return
+ */
+ @Override
+ public String interceptSource(String source) {
+ return source;
+ }
+
+ /**
+ * Intercept resolved path, e.g.
+ * Mount point /^(\\w+)/, ${1}.hadoop.net
+ * If incoming path is /user1/home/tmp/job1,
+ * then the resolved path str will be user1.
+ *
+ * @return intercepted string
+ */
+ @Override public String interceptResolvedDestPathStr(
+ String parsedDestPathStr) {
+ Matcher matcher = srcRegexPattern.matcher(parsedDestPathStr);
+ return matcher.replaceAll(replaceString);
+ }
+
+ /**
+ * Intercept remaining path.
+ *
+ * @return intercepted path
+ */
+ @Override public Path interceptRemainingPath(Path remainingPath) {
Review comment:
The splitting of remaining part is kind implementation details I feel. User may not clearly have idea on that, unless they are very advanced users.
My question was what if we use same API for remaining part also? in ur example interceptSource((dir1/dir2/dir3/file3)). What will happen here? Intercepter are just for replacing some pattern with other right. Let me know if I am missing.
Thanks for your efforts on fixing other comments..
Also please take a look at check-style warnings reported in latest report.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r472360734
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##########
@@ -226,7 +239,14 @@ void addLink(final String pathComponent, final INodeLink<T> link)
* Config prefix: fs.viewfs.mounttable.<mnt_tbl_name>.linkNfly
* Refer: {@link Constants#CONFIG_VIEWFS_LINK_NFLY}
*/
- NFLY;
+ NFLY,
+ /**
+ * Link entry which source are regex exrepssions and target refer matched
+ * group from source
+ * Config prefix: fs.viewfs.mounttable.<mnt_tbl_name>.linkMerge
Review comment:
Yes, nice catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-690866308
@umamaheswararao Thanks a lot!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-668872388
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 19s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 17s | root in trunk failed. |
| +1 :green_heart: | compile | 23m 19s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 18m 16s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 3m 3s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 58s | trunk passed |
| +1 :green_heart: | shadedclient | 22m 11s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 58s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 2m 54s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 3m 21s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 34s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 5s | the patch passed |
| +1 :green_heart: | compile | 21m 11s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 21m 11s | the patch passed |
| +1 :green_heart: | compile | 17m 35s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 17m 35s | the patch passed |
| -0 :warning: | checkstyle | 2m 51s | root: The patch generated 2 new + 92 unchanged - 1 fixed = 94 total (was 93) |
| +1 :green_heart: | mvnsite | 2m 46s | the patch passed |
| -1 :x: | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 15m 36s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 58s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 2m 49s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 5m 43s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 9s | hadoop-common in the patch passed. |
| -1 :x: | unit | 119m 22s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 56s | The patch does not generate ASF License warnings. |
| | | 282m 13s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 5e99c9f0c2cc 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 0277856738a |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/3/artifact/out/branch-mvninstall-root.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/3/artifact/out/diff-checkstyle-root.txt |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/3/artifact/out/whitespace-eol.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/3/testReport/ |
| Max. process+thread count | 3224 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-690863729
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r471825828
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
##########
@@ -166,6 +166,41 @@ public static void addLinkNfly(final Configuration conf, final String src,
addLinkNfly(conf, getDefaultMountTableName(conf), src, null, targets);
}
+
+ /**
+ * Add a LinkRegex to the config for the specified mount table.
Review comment:
Make sense, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-685233867
HI @JohnZZGithub, I think currently this branch is away from 45commits. It may be a good idea to rebase once on trunk? Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-668980590
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 30m 21s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 2s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 58s | trunk passed |
| +1 :green_heart: | compile | 19m 10s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 41s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 2m 37s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 53s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 6s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 14s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 9s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 3m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 16s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 57s | the patch passed |
| +1 :green_heart: | compile | 18m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 18m 33s | the patch passed |
| +1 :green_heart: | compile | 16m 41s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 16m 41s | the patch passed |
| -0 :warning: | checkstyle | 2m 42s | root: The patch generated 1 new + 92 unchanged - 1 fixed = 93 total (was 93) |
| +1 :green_heart: | mvnsite | 2m 56s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 5s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 14s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 8s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 5m 33s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 57s | hadoop-common in the patch passed. |
| -1 :x: | unit | 115m 40s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 3s | The patch does not generate ASF License warnings. |
| | | 315m 19s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 730c28fdf701 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ed3ab4b87d9 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/4/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/4/testReport/ |
| Max. process+thread count | 3394 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/4/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r485953722
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
##########
@@ -366,6 +366,69 @@ Don't want to change scheme or difficult to copy mount-table configurations to a
Please refer to the [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html)
+Regex Pattern Based Mount Points
+--------------------------------
+
+The view file system mount points were a Key-Value based mapping system. It is not friendly for user cases which mapping config could be abstracted to rules. E.g. Users want to provide a GCS bucket per user and there might be thousands of users in total. The old key-value based approach won't work well for several reasons:
+
+1. The mount table is used by FileSystem clients. There's a cost to spread the config to all clients and we should avoid it if possible. The [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html) could help the distribution by central mount table management. But the mount table still have to be updated on every change. The change could be greatly avoided if provide a rule-based mount table.
+
+2. The client have to understand all the KVs in the mount table. This is not ideal when the mountable grows to thousands of items. E.g. thousands of file systems might be initialized even users only need one. And the config itself will become bloated at scale.
+
+### Understand the Difference
+
+In the key-value based mount table, view file system treats every mount point as a partition. There's several file system APIs which will lead to operation on all partitions. E.g. there's an HDFS cluster with multiple mount. Users want to run “hadoop fs -put file viewfs://hdfs.namenode.apache.org/tmp/” cmd to copy data from local disk to our HDFS cluster. The cmd will trigger ViewFileSystem to call setVerifyChecksum() method which will initialize the file system for every mount point.
+For a regex-base rule mount table entry, we couldn't know what's corresponding path until parsing. So the regex based mount table entry will be ignored on such cases. The file system (ChRootedFileSystem) will be created upon accessing. But the underlying file system will be cached by inner cache of ViewFileSystem.
Review comment:
Thanks for having separate JIRA. It make sense to me. !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-675345218
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 1s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 21s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 29s | trunk passed |
| +1 :green_heart: | compile | 19m 13s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 44s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 46s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 57s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 11s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 39s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 8s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 19s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 3s | the patch passed |
| +1 :green_heart: | compile | 20m 27s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 20m 27s | the patch passed |
| +1 :green_heart: | compile | 17m 20s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 17m 20s | the patch passed |
| -0 :warning: | checkstyle | 2m 46s | root: The patch generated 14 new + 182 unchanged - 1 fixed = 196 total (was 183) |
| +1 :green_heart: | mvnsite | 2m 55s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 4s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 39s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 13s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 5m 37s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 54s | hadoop-common in the patch passed. |
| -1 :x: | unit | 119m 11s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 3s | The patch does not generate ASF License warnings. |
| | | 302m 10s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestDecommissionWithBackoffMonitor |
| | hadoop.hdfs.TestStripedFileAppend |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestMultipleNNPortQOP |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 829e31a8d922 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fefacf2578e |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/7/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/7/testReport/ |
| Max. process+thread count | 3923 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/7/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub edited a comment on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub edited a comment on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-686183399
Thanks @umamaheswararao
@umamaheswararao Thanks for the comments. Please see the reply inline.
> Hi @JohnZZGithub, I got few other points to discuss.
>
> 1. We have exposed getMountPoints API. It seems we can't return any mount points from REGEX based because you would not know until you got src paths to resolve and find real target fs. What should we do for this API?
It's a great question. I guess most caller of getMountPoints wants to traverse all the file systems to do some operation. E.g. setVerifyChecksum(). We didn't see issues on our internal Yarn + HDFS and Yarn + GCS clusters. The usage pattern includes but not limited to MR, Spark, Presto, Vertica loading and etc. But it's possible that some users might rely on these APIs. I could see two options forward:
1. Returning a MountPint with special FileSystem for Regex Mount points. We could cache the initialized fileSystem under the regex mountpoint and perform the operation. For filesystems that might appear in the future, we could cache the past calls from callers and try to apply it or just not support it.
2. We could indicate that we don't support such APIs for regex mount points.
And to extend the topic a little bit, this kind of ViewFileSystem API (API which tries to visit all file systems) caused several problems for us. E.g. setVerifyChecksum() initialized a file system for a mount point users didn't want to use it all. And the initialization of the file system will fail as it requires credentials during initialization. Users don't have it as it never means to visit the mount point. We developed a LazyChRootedFileSystem on top of every target system (not public) to do lazy initialization for path-based APIs. But it's hard to tackle APIs without path passed in. So to summarize, we see cases users want to avoid these non-path based API to trigger actions on every child file system. In the meantime, some users(though rare in our scenarios) might want to use these APIs applied to all children's filesystems. I feel it's hard to satisfy both needs.
> 2. Other API is getDelegationTokenIssuers. Applications like YARN uses this API to get all child fs delegation tokens. This also will not work for REGEX based mount points.
We did see an issue with addDelegationTokens in the secure Hadoop cluster. But the problem we met is not all normal mountpoints are secure. So the API caused a problem when it tries to initialize all children's file systems. We took a workaround by making it path-based. As for getDelegationTokens, I guess the problem is similar. We didn't see issues because it's not used. Could we make it path based too? Or we could take the approach stated in problem one.
> 3. Other question is how this child filesystem objects gets closed. There was an issue with [ViewFileSystem#close | https://issues.apache.org/jira/browse/HADOOP-15565 ]. I would like to know how that get addressed in this case as don't keep anything in InnerCache.
Could we make the inner cache a thread-safe structure and track all the opened file systems under regex mount points?
These are really great points, thanks a lot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-675689087
@umamaheswararao thanks for the review. I update the diff based on the comments. There might be still some issues regarding docs and comments. I will continue to watch it. Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r471830630
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
##########
@@ -1430,4 +1432,49 @@ public void testGetContentSummaryWithFileInLocalFS() throws Exception {
summaryAfter.getLength());
}
}
+
+ @Test
+ public void testMountPointCache() throws Exception {
+ conf.setInt(Constants.CONFIG_VIEWFS_PATH_RESOLUTION_CACHE_CAPACITY, 1);
+ conf.setBoolean("fs.viewfs.impl.disable.cache", true);
Review comment:
@umamaheswararao This is a good point. Let me add some preconditions check.
BTW, now the inner cache assumes every filesystem is created while InodeTree is constructed and never changed. Do you think it's reasonable to change it to a concurrent hash map?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-689477953
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 26s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 8s | trunk passed |
| +1 :green_heart: | compile | 19m 39s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 52s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 48s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 9s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 53s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 39s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 17s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 20s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 57s | the patch passed |
| +1 :green_heart: | compile | 18m 55s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 18m 55s | the patch passed |
| +1 :green_heart: | compile | 16m 41s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 16m 41s | the patch passed |
| -0 :warning: | checkstyle | 2m 45s | root: The patch generated 15 new + 182 unchanged - 1 fixed = 197 total (was 183) |
| +1 :green_heart: | mvnsite | 2m 59s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 9s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 40s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 17s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 5m 37s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 31s | hadoop-common in the patch passed. |
| -1 :x: | unit | 94m 9s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. |
| | | 276m 11s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestGetFileChecksum |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/18/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux a4477cf31c7c 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0d855159f09 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/18/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/18/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/18/testReport/ |
| Max. process+thread count | 4454 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/18/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r485346542
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestRegexMountPoint.java
##########
@@ -0,0 +1,160 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test Regex Mount Point.
+ */
+public class TestRegexMountPoint {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(TestRegexMountPoint.class.getName());
+
+ private InodeTree inodeTree;
+ private Configuration conf;
+
+ class TestRegexMountPointFileSystem {
+ public URI getUri() {
+ return uri;
+ }
+
+ private URI uri;
+
+ TestRegexMountPointFileSystem(URI uri) {
+ String uriStr = uri == null ? "null" : uri.toString();
+ LOGGER.info("Create TestRegexMountPointFileSystem Via URI:" + uriStr);
+ this.uri = uri;
+ }
+ }
+
+ @Before
+ public void setUp() throws Exception {
+ conf = new Configuration();
+ ConfigUtil.addLink(conf, TestRegexMountPoint.class.getName(), "/mnt",
+ URI.create("file:///"));
+
+ inodeTree = new InodeTree<TestRegexMountPointFileSystem>(conf,
+ TestRegexMountPoint.class.getName(), null, false) {
+ @Override
+ protected TestRegexMountPointFileSystem getTargetFileSystem(
+ final URI uri) {
+ return new TestRegexMountPointFileSystem(uri);
+ }
+
+ @Override
+ protected TestRegexMountPointFileSystem getTargetFileSystem(
+ final INodeDir<TestRegexMountPointFileSystem> dir) {
+ return new TestRegexMountPointFileSystem(null);
+ }
+
+ @Override
+ protected TestRegexMountPointFileSystem getTargetFileSystem(
+ final String settings, final URI[] mergeFsURIList) {
+ return new TestRegexMountPointFileSystem(null);
+ }
+ };
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ inodeTree = null;
+ }
+
+ @Test
+ public void testGetVarListInString() throws IOException {
+ String srcRegex = "/(\\w+)";
+ String target = "/$0/${1}/$1/${2}/${2}";
+ RegexMountPoint regexMountPoint =
+ new RegexMountPoint(inodeTree, srcRegex, target, null);
+ regexMountPoint.initialize();
+ Map<String, Set<String>> varMap = regexMountPoint.getVarInDestPathMap();
+ Assert.assertEquals(varMap.size(), 3);
+ Assert.assertEquals(varMap.get("0").size(), 1);
+ Assert.assertTrue(varMap.get("0").contains("$0"));
+ Assert.assertEquals(varMap.get("1").size(), 2);
+ Assert.assertTrue(varMap.get("1").contains("${1}"));
+ Assert.assertTrue(varMap.get("1").contains("$1"));
+ Assert.assertEquals(varMap.get("2").size(), 1);
+ Assert.assertTrue(varMap.get("2").contains("${2}"));
+ }
+
+ @Test
+ public void testResolve() throws IOException {
+ String regexStr = "^/user/(?<username>\\w+)";
+ String dstPathStr = "/namenode1/testResolve/$username";
+ String settingsStr = null;
+ RegexMountPoint regexMountPoint =
+ new RegexMountPoint(inodeTree, regexStr, dstPathStr, settingsStr);
+ regexMountPoint.initialize();
+ InodeTree.ResolveResult resolveResult =
+ regexMountPoint.resolve("/user/hadoop/file1", true);
+ Assert.assertEquals(resolveResult.kind, InodeTree.ResultKind.EXTERNAL_DIR);
+ Assert.assertTrue(
+ resolveResult.targetFileSystem
+ instanceof TestRegexMountPointFileSystem);
+ Assert.assertTrue(resolveResult.resolvedPath.equals("/user/hadoop"));
+ Assert.assertTrue(
+ resolveResult.targetFileSystem
+ instanceof TestRegexMountPointFileSystem);
+ Assert.assertTrue(
+ ((TestRegexMountPointFileSystem) resolveResult.targetFileSystem)
+ .getUri().toString().equals("/namenode1/testResolve/hadoop"));
+ Assert.assertTrue(resolveResult.remainingPath.toString().equals("/file1"));
+ }
+
+ @Test
+ public void testResolveWithInterceptor() throws IOException {
+ String regexStr = "^/user/(?<username>\\w+)";
+ String dstPathStr = "/namenode1/testResolve/$username";
+ // Replace "_" with "-"
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ new RegexMountPointResolvedDstPathReplaceInterceptor("_", "-");
+ // replaceresolvedpath:_:-
+ String settingsStr = interceptor.serializeToString();
+ RegexMountPoint regexMountPoint =
+ new RegexMountPoint(inodeTree, regexStr, dstPathStr, settingsStr);
+ regexMountPoint.initialize();
+ InodeTree.ResolveResult resolveResult =
+ regexMountPoint.resolve("/user/hadoop_user1/file_index", true);
+ Assert.assertEquals(resolveResult.kind, InodeTree.ResultKind.EXTERNAL_DIR);
+ Assert.assertTrue(
+ resolveResult.targetFileSystem
+ instanceof TestRegexMountPointFileSystem);
+ Assert.assertTrue(resolveResult.resolvedPath.equals("/user/hadoop_user1"));
Review comment:
Nice catch, let me clean them up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-668314560
@templedf It will be great if you could help with the review, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r471830882
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
##########
@@ -366,6 +366,82 @@ Don't want to change scheme or difficult to copy mount-table configurations to a
Please refer to the [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html)
+Regex Pattern Based Mount Points
+--------------------------------
+
+The view file system mount points were a Key-Value based mapping system. It is not friendly for user cases which mapping config could be abstracted to rules. E.g. Users want to provide a GCS bucket per user and there might be thousands of users in total. The old key-value based approach won't work well for several reasons:
+
+1. The mount table is used by FileSystem clients. There's a cost to spread the config to all clients and we should avoid it if possible. The [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html) could help the distribution by central mount table management. But the mount table still have to be updated on every change. The change could be greatly avoided if provide a rule-based mount table..
+
+2. The client have to understand all the KVs in the mount table. This is not ideal when the mountable grows to thousands of items. E.g. thousands of file systems might be initialized even users only need one. And the config itself will become bloated at scale.
Review comment:
I didn't realize that there is a jira. This is great.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-671693591
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 6s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 11m 4s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 5s | trunk passed |
| +1 :green_heart: | compile | 23m 39s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 19m 20s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 0s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 3s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 25s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 57s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 2s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 35s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 6m 0s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 10s | the patch passed |
| +1 :green_heart: | compile | 23m 13s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 23m 13s | the patch passed |
| +1 :green_heart: | compile | 20m 33s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 20m 33s | the patch passed |
| -0 :warning: | checkstyle | 3m 33s | root: The patch generated 1 new + 92 unchanged - 1 fixed = 93 total (was 93) |
| +1 :green_heart: | mvnsite | 3m 1s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 16s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 59s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 2s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 6m 24s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 26s | hadoop-common in the patch passed. |
| -1 :x: | unit | 113m 48s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | The patch does not generate ASF License warnings. |
| | | 325m 28s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.TestPread |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.TestErasureCodeBenchmarkThroughput |
| | hadoop.hdfs.web.TestWebHDFS |
| | hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy |
| | hadoop.hdfs.TestFileCorruption |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux f54b9ef146d8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 5e0f8797790 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/6/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/6/testReport/ |
| Max. process+thread count | 4061 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/6/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao merged pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao merged pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-671589516
Just did a rebase and updated the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-689082045
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 56s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 18s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 36s | trunk passed |
| +1 :green_heart: | compile | 23m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 19m 47s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 3s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 1s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 55s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 31s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 27s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 29s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 52s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 11s | the patch passed |
| +1 :green_heart: | compile | 23m 34s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 23m 34s | the patch passed |
| +1 :green_heart: | compile | 19m 49s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 19m 49s | the patch passed |
| -0 :warning: | checkstyle | 2m 56s | root: The patch generated 5 new + 182 unchanged - 1 fixed = 187 total (was 183) |
| +1 :green_heart: | mvnsite | 3m 0s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 28s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 30s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 16s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 6m 42s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 33s | hadoop-common in the patch passed. |
| -1 :x: | unit | 97m 51s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. |
| | | 304m 7s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.server.namenode.TestProtectedDirectories |
| | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestDatanodeRegistration |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.server.namenode.TestReencryption |
| | hadoop.hdfs.TestFileChecksum |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux d118fba1e4d6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0d855159f09 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/16/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/16/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/16/testReport/ |
| Max. process+thread count | 3695 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/16/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r480836939
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
##########
@@ -86,12 +86,21 @@
*/
String CONFIG_VIEWFS_LINK_MERGE_SLASH = "linkMergeSlash";
+ /**
+ * Config variable for specifying a regex link which uses regular expressions
+ * as source and target could use group captured in src.
+ * E.g. (^/(?<firstDir>\\w+), /prefix-${firstDir}) =>
+ * (/path1/file1 => /prefix-path1/file1)
+ */
+ String CONFIG_VIEWFS_LINK_REGEX = "linkRegex";
+
FsPermission PERMISSION_555 = new FsPermission((short) 0555);
String CONFIG_VIEWFS_RENAME_STRATEGY = "fs.viewfs.rename.strategy";
/**
* Enable ViewFileSystem to cache all children filesystems in inner cache.
Review comment:
below comment can be corrected? regex base mount point will not use caching now.
Otherwise people could confuse and may tend to disable. IIUC, even if they enable this, RegexBasedMountPoints will continue to work right?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/RegexMountPoint.java
##########
@@ -0,0 +1,306 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.regex.PatternSyntaxException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.util.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.viewfs.InodeTree.SlashPath;
+
+/**
+ * Regex mount point is build to implement regex based mount point.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+class RegexMountPoint<T> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(RegexMountPoint.class.getName());
+
+ private InodeTree inodeTree;
+ private String srcPathRegex;
+ private Pattern srcPattern;
+ private String dstPath;
+ private String interceptorSettingsString;
+ private List<RegexMountPointInterceptor> interceptorList;
+
+ public static final String SETTING_SRCREGEX_SEP = "#.";
+ public static final char INTERCEPTOR_SEP = ';';
+ public static final char INTERCEPTOR_INTERNAL_SEP = ':';
+ // ${var},$var
+ public static final Pattern VAR_PATTERN_IN_DEST =
+ Pattern.compile("\\$((\\{\\w+\\})|(\\w+))");
+
+ // Same var might have different representations.
+ // e.g.
+ // key => $key or key = > ${key}
+ private Map<String, Set<String>> varInDestPathMap;
+
+ public Map<String, Set<String>> getVarInDestPathMap() {
+ return varInDestPathMap;
+ }
+
+ RegexMountPoint(InodeTree inodeTree, String sourcePathRegex,
+ String destPath, String settingsStr) {
+ this.inodeTree = inodeTree;
+ this.srcPathRegex = sourcePathRegex;
+ this.dstPath = destPath;
+ this.interceptorSettingsString = settingsStr;
+ this.interceptorList = new ArrayList<>();
+ }
+
+ /**
+ * Initialize regex mount point.
+ *
+ * @throws IOException
+ */
+ public void initialize() throws IOException {
+ try {
+ srcPattern = Pattern.compile(srcPathRegex);
+ } catch (PatternSyntaxException ex) {
+ throw new IOException(
+ "Failed to initialized mount point due to bad src path regex:"
+ + srcPathRegex + ", dstPath:" + dstPath, ex);
+ }
+ varInDestPathMap = getVarListInString(dstPath);
+ initializeInterceptors();
+ }
+
+ private void initializeInterceptors() throws IOException {
+ if (interceptorSettingsString == null
+ || interceptorSettingsString.isEmpty()) {
+ return;
+ }
+ String[] interceptorStrArray =
+ StringUtils.split(interceptorSettingsString, INTERCEPTOR_SEP);
+ for (String interceptorStr : interceptorStrArray) {
+ RegexMountPointInterceptor interceptor =
+ RegexMountPointInterceptorFactory.create(interceptorStr);
+ if (interceptor == null) {
+ throw new IOException(
+ "Illegal settings String " + interceptorSettingsString);
+ }
+ interceptor.initialize();
+ interceptorList.add(interceptor);
+ }
+ }
+
+ /**
+ * Get $var1 and $var2 style variables in string.
+ *
+ * @param input - the string to be process.
+ * @return
+ */
+ public static Map<String, Set<String>> getVarListInString(String input) {
+ Map<String, Set<String>> varMap = new HashMap<>();
+ Matcher matcher = VAR_PATTERN_IN_DEST.matcher(input);
+ while (matcher.find()) {
+ // $var or ${var}
+ String varName = matcher.group(0);
+ // var or {var}
+ String strippedVarName = matcher.group(1);
+ if (strippedVarName.startsWith("{")) {
+ // {varName} = > varName
+ strippedVarName =
+ strippedVarName.substring(1, strippedVarName.length() - 1);
+ }
+ varMap.putIfAbsent(strippedVarName, new HashSet<>());
+ varMap.get(strippedVarName).add(varName);
+ }
+ return varMap;
+ }
+
+ public String getSrcPathRegex() {
+ return srcPathRegex;
+ }
+
+ public Pattern getSrcPattern() {
+ return srcPattern;
+ }
+
+ public String getDstPath() {
+ return dstPath;
+ }
+
+ public static Pattern getVarPatternInDest() {
+ return VAR_PATTERN_IN_DEST;
+ }
+
+ /**
+ * Get resolved path from regex mount points.
+ * E.g. link: ^/user/(?<username>\\w+) => s3://$user.apache.com/_${user}
+ * srcPath: is /user/hadoop/dir1
+ * resolveLastComponent: true
+ * then return value is s3://hadoop.apache.com/_hadoop
+ * @param srcPath - the src path to resolve
+ * @param resolveLastComponent - whether resolve the path after last `/`
+ * @return mapped path of the mount point.
+ */
+ public InodeTree.ResolveResult<T> resolve(final String srcPath,
+ final boolean resolveLastComponent) {
+ String pathStrToResolve = getPathToResolve(srcPath, resolveLastComponent);
+ for (RegexMountPointInterceptor interceptor : interceptorList) {
+ pathStrToResolve = interceptor.interceptSource(pathStrToResolve);
+ }
+ LOGGER.debug("Path to resolve:" + pathStrToResolve + ", srcPattern:"
+ + getSrcPathRegex());
+ Matcher srcMatcher = getSrcPattern().matcher(pathStrToResolve);
+ String parsedDestPath = getDstPath();
+ int mappedCount = 0;
+ String resolvedPathStr = "";
+ while (srcMatcher.find()) {
+ resolvedPathStr = pathStrToResolve.substring(0, srcMatcher.end());
+ Map<String, Set<String>> varMap = getVarInDestPathMap();
+ for (Map.Entry<String, Set<String>> entry : varMap.entrySet()) {
+ String regexGroupNameOrIndexStr = entry.getKey();
+ Set<String> groupRepresentationStrSetInDest = entry.getValue();
+ parsedDestPath = replaceRegexCaptureGroupInPath(
+ parsedDestPath, srcMatcher,
+ regexGroupNameOrIndexStr, groupRepresentationStrSetInDest);
+ }
+ ++mappedCount;
+ }
+ if (0 == mappedCount) {
+ return null;
+ }
+ Path remainingPath = getRemainingPathStr(srcPath, resolvedPathStr);
+ for (RegexMountPointInterceptor interceptor : interceptorList) {
+ parsedDestPath = interceptor.interceptResolvedDestPathStr(parsedDestPath);
+ remainingPath =
+ interceptor.interceptRemainingPath(remainingPath);
+ }
+ InodeTree.ResolveResult resolveResult = inodeTree
+ .buildResolveResultForRegexMountPoint(InodeTree.ResultKind.EXTERNAL_DIR,
+ resolvedPathStr, parsedDestPath, remainingPath);
+ return resolveResult;
+ }
+
+ private Path getRemainingPathStr(
+ String srcPath,
+ String resolvedPathStr) {
+ String remainingPathStr = srcPath.substring(resolvedPathStr.length());
+ if (!remainingPathStr.startsWith("/")) {
+ remainingPathStr = "/" + remainingPathStr;
+ }
+ return new Path(remainingPathStr);
+ }
+
+ private String getPathToResolve(
+ String srcPath, boolean resolveLastComponent) {
+ if (resolveLastComponent) {
+ return srcPath;
+ }
+ int lastSlashIndex = srcPath.lastIndexOf(SlashPath.toString());
+ if (lastSlashIndex == -1) {
+ return null;
+ }
+ return srcPath.substring(0, lastSlashIndex);
+ }
+
+ /**
+ * Use capture group named regexGroupNameOrIndexStr in mather to replace
+ * parsedDestPath.
+ * E.g. link: ^/user/(?<username>\\w+) => s3://$user.apache.com/_${user}
+ * srcMatcher is from /user/hadoop.
+ * Then the params will be like following.
+ * parsedDestPath: s3://$user.apache.com/_${user},
+ * regexGroupNameOrIndexStr: user
+ * groupRepresentationStrSetInDest: {user:$user; user:${user}}
+ * return value will be s3://hadoop.apache.com/_hadoop
+ * @param parsedDestPath
+ * @param srcMatcher
+ * @param regexGroupNameOrIndexStr
+ * @param groupRepresentationStrSetInDest
+ * @return return parsedDestPath while ${var},$var replaced or
+ * parsedDestPath nothing found.
+ */
+ private String replaceRegexCaptureGroupInPath(
+ String parsedDestPath,
+ Matcher srcMatcher,
+ String regexGroupNameOrIndexStr,
+ Set<String> groupRepresentationStrSetInDest) {
+ String groupValue = getRegexGroupValueFromMather(
+ srcMatcher, regexGroupNameOrIndexStr);
+ if (groupValue == null) {
+ return parsedDestPath;
+ }
+ for (String varName : groupRepresentationStrSetInDest) {
+ parsedDestPath = parsedDestPath.replace(varName, groupValue);
+ LOGGER.debug("parsedDestPath value is:" + parsedDestPath);
+ }
+ return parsedDestPath;
+ }
+
+ /**
+ * Get matched capture group value from regex matched string. E.g.
+ * Regex: ^/user/(?<username>\\w+), regexGroupNameOrIndexStr: userName
+ * then /user/hadoop should return hadoop while call
+ * getRegexGroupValueFromMather(matcher, usersName)
+ * or getRegexGroupValueFromMather(matcher, 1)
+ *
+ * @param srcMatcher - the matcher to be use
+ * @param regexGroupNameOrIndexStr - the regex group name or index
+ * @return - Null if no matched group named regexGroupNameOrIndexStr found.
+ */
+ private String getRegexGroupValueFromMather(
+ Matcher srcMatcher, String regexGroupNameOrIndexStr) {
+ if (regexGroupNameOrIndexStr.matches("\\d+")) {
+ // group index
+ int groupIndex = Integer.parseUnsignedInt(regexGroupNameOrIndexStr);
+ if (groupIndex >= 0 && groupIndex <= srcMatcher.groupCount()) {
+ return srcMatcher.group(groupIndex);
+ }
+ } else {
+ // named group in regex
+ return srcMatcher.group(regexGroupNameOrIndexStr);
+ }
+ return null;
+ }
+
+ /**
Review comment:
Below method seems like unused method? Please removed it.
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestRegexMountPoint.java
##########
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Test Regex Mount Point.
+ */
+public class TestRegexMountPoint {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(TestRegexMountPoint.class.getName());
+
+ private InodeTree inodeTree;
+ private Configuration conf;
+
+ class TestRegexMountPointFileSystem {
+ public URI getUri() {
+ return uri;
+ }
+
+ private URI uri;
+
+ TestRegexMountPointFileSystem(URI uri) {
+ String uriStr = uri == null ? "null" : uri.toString();
+ LOGGER.info("Create TestRegexMountPointFileSystem Via URI:" + uriStr);
+ this.uri = uri;
+ }
+ }
+
+ @Before
+ public void setUp() throws Exception {
+ conf = new Configuration();
+ ConfigUtil.addLink(conf, TestRegexMountPoint.class.getName(), "/mnt",
+ URI.create("file:///"));
+
+ inodeTree = new InodeTree<TestRegexMountPointFileSystem>(conf,
+ TestRegexMountPoint.class.getName(), null, false) {
+ @Override
+ protected TestRegexMountPointFileSystem getTargetFileSystem(
+ final URI uri) {
+ return new TestRegexMountPointFileSystem(uri);
+ }
+
+ @Override
+ protected TestRegexMountPointFileSystem getTargetFileSystem(
+ final URI uri, boolean enableCache) {
+ return new TestRegexMountPointFileSystem(uri);
+ }
+
+ @Override
+ protected TestRegexMountPointFileSystem getTargetFileSystem(
+ final INodeDir<TestRegexMountPointFileSystem> dir) {
+ return new TestRegexMountPointFileSystem(null);
+ }
+
+ @Override
+ protected TestRegexMountPointFileSystem getTargetFileSystem(
+ final String settings, final URI[] mergeFsURIList) {
+ return new TestRegexMountPointFileSystem(null);
+ }
+ };
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ inodeTree = null;
+ }
+
+ @Test
+ public void testGetVarListInString() throws IOException {
+ String srcRegex = "/(\\w+)";
+ String target = "/$0/${1}/$1/${2}/${2}";
+ RegexMountPoint regexMountPoint =
+ new RegexMountPoint(inodeTree, srcRegex, target, null);
+ regexMountPoint.initialize();
+ Map<String, Set<String>> varMap = regexMountPoint.getVarInDestPathMap();
+ Assert.assertEquals(varMap.size(), 3);
+ Assert.assertEquals(varMap.get("0").size(), 1);
+ Assert.assertTrue(varMap.get("0").contains("$0"));
+ Assert.assertEquals(varMap.get("1").size(), 2);
+ Assert.assertTrue(varMap.get("1").contains("${1}"));
+ Assert.assertTrue(varMap.get("1").contains("$1"));
+ Assert.assertEquals(varMap.get("2").size(), 1);
+ Assert.assertTrue(varMap.get("2").contains("${2}"));
+ }
+
+ @Test
+ public void testResolve() throws IOException {
+ String regexStr = "^/user/(?<username>\\w+)";
+ String dstPathStr = "/namenode1/testResolve/$username";
+ String settingsStr = null;
+ RegexMountPoint regexMountPoint =
+ new RegexMountPoint(inodeTree, regexStr, dstPathStr, settingsStr);
+ regexMountPoint.initialize();
+ InodeTree.ResolveResult resolveResult =
+ regexMountPoint.resolve("/user/hadoop/file1", true);
+ Assert.assertEquals(resolveResult.kind, InodeTree.ResultKind.EXTERNAL_DIR);
+ Assert.assertTrue(
+ resolveResult.targetFileSystem
+ instanceof TestRegexMountPointFileSystem);
+ Assert.assertTrue(resolveResult.resolvedPath.equals("/user/hadoop"));
+ Assert.assertTrue(
+ resolveResult.targetFileSystem
+ instanceof TestRegexMountPointFileSystem);
+ Assert.assertTrue(
Review comment:
You may want to use assertEquals ? The advantage would be that, when assertion fails it will tell you what's mismatching. There are lot of asserts like this, please change if possible.
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestRegexMountPointInterceptorFactory.java
##########
@@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * Test Regex Mount Point Interceptor Factory.
+ */
+public class TestRegexMountPointInterceptorFactory {
+
+ @Test
+ public void testCreateNormalCase() {
+ String replaceInterceptorStr =
+ RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + Character.toString(RegexMountPoint.INTERCEPTOR_INTERNAL_SEP)
+ + "src" + Character
+ .toString(RegexMountPoint.INTERCEPTOR_INTERNAL_SEP) + "replace";
+ RegexMountPointInterceptor interceptor =
+ RegexMountPointInterceptorFactory.create(replaceInterceptorStr);
+ Assert.assertTrue(
+ interceptor
+ instanceof RegexMountPointResolvedDstPathReplaceInterceptor);
+ }
+
+ @Test
+ public void testCreateBadCase() {
+ String replaceInterceptorStr =
+ RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + "___" + Character
+ .toString(RegexMountPoint.INTERCEPTOR_INTERNAL_SEP) + "src"
+ + Character.toString(RegexMountPoint.INTERCEPTOR_INTERNAL_SEP)
+ + "replace";
+ RegexMountPointInterceptor interceptor =
+ RegexMountPointInterceptorFactory.create(replaceInterceptorStr);
Review comment:
you can use assertNull
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/RegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.regex.PatternSyntaxException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Implementation of RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+class RegexMountPointResolvedDstPathReplaceInterceptor
+ implements RegexMountPointInterceptor {
+
+ private String srcRegexString;
+ private String replaceString;
+ private Pattern srcRegexPattern;
+
+ RegexMountPointResolvedDstPathReplaceInterceptor(String srcRegex,
+ String replaceString) {
+ this.srcRegexString = srcRegex;
+ this.replaceString = replaceString;
+ this.srcRegexPattern = null;
+ }
+
+ public String getSrcRegexString() {
+ return srcRegexString;
+ }
+
+ public String getReplaceString() {
+ return replaceString;
+ }
+
+ public Pattern getSrcRegexPattern() {
+ return srcRegexPattern;
+ }
+
+ @Override
+ public void initialize() throws IOException {
+ try {
+ srcRegexPattern = Pattern.compile(srcRegexString);
+ } catch (PatternSyntaxException ex) {
+ throw new IOException(
+ "Initialize interceptor failed, srcRegx:" + srcRegexString, ex);
+ }
+ }
+
+ /**
+ * Intercept source before resolution.
+ *
+ * @param source
+ * @return
+ */
+ @Override
+ public String interceptSource(String source) {
+ return source;
+ }
+
+ /**
+ * Intercept resolved path, e.g.
+ * Mount point /^(\\w+)/, ${1}.hadoop.net
+ * If incoming path is /user1/home/tmp/job1,
+ * then the resolved path str will be user1.
+ *
+ * @return intercepted string
+ */
+ @Override public String interceptResolvedDestPathStr(
+ String parsedDestPathStr) {
+ Matcher matcher = srcRegexPattern.matcher(parsedDestPathStr);
+ return matcher.replaceAll(replaceString);
+ }
+
+ /**
+ * Intercept remaining path.
+ *
Review comment:
same a interceptSrc doc.
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestRegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Test RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+public class TestRegexMountPointResolvedDstPathReplaceInterceptor {
+
+ public String createSerializedString(String regex, String replaceString) {
+ return REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + regex
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + replaceString;
+ }
+
+ @Test
+ public void testDeserializeFromStringNormalCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertTrue(interceptor.getSrcRegexString().equals(srcRegex));
+ Assert.assertTrue(interceptor.getReplaceString().equals(replaceString));
+ Assert.assertTrue(interceptor.getSrcRegexPattern() == null);
+ interceptor.initialize();
+ Assert.assertTrue(
+ interceptor.getSrcRegexPattern().toString().equals(srcRegex));
+ }
+
+ @Test
+ public void testDeserializeFromStringBadCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ serializedString = serializedString + ":ddd";
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertEquals(interceptor, null);
+ }
+
+ @Test
+ public void testSerialization() {
+ String srcRegex = "word1";
+ String replaceString = "word2";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ new RegexMountPointResolvedDstPathReplaceInterceptor(srcRegex,
+ replaceString);
+ Assert.assertEquals(interceptor.serializeToString(), serializedString);
+ }
+
+ @Test
Review comment:
What are you asserting in this case?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/RegexMountPointInterceptorFactory.java
##########
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * The interceptor factory used to create RegexMountPoint interceptors.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+final class RegexMountPointInterceptorFactory {
+
+ private RegexMountPointInterceptorFactory() {
+
+ }
+
+ /**
+ * interceptorSettingsString string should be like ${type}:${string},
+ * e.g. replaceresolveddstpath:word1,word2.
+ *
+ * @param interceptorSettingsString
Review comment:
you may want to remove the below return ? or specify what it's returning.
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestRegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Test RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+public class TestRegexMountPointResolvedDstPathReplaceInterceptor {
+
+ public String createSerializedString(String regex, String replaceString) {
+ return REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + regex
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + replaceString;
+ }
+
+ @Test
+ public void testDeserializeFromStringNormalCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertTrue(interceptor.getSrcRegexString().equals(srcRegex));
+ Assert.assertTrue(interceptor.getReplaceString().equals(replaceString));
+ Assert.assertTrue(interceptor.getSrcRegexPattern() == null);
+ interceptor.initialize();
+ Assert.assertTrue(
+ interceptor.getSrcRegexPattern().toString().equals(srcRegex));
+ }
+
+ @Test
+ public void testDeserializeFromStringBadCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ serializedString = serializedString + ":ddd";
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertEquals(interceptor, null);
+ }
+
+ @Test
+ public void testSerialization() {
+ String srcRegex = "word1";
+ String replaceString = "word2";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ new RegexMountPointResolvedDstPathReplaceInterceptor(srcRegex,
+ replaceString);
+ Assert.assertEquals(interceptor.serializeToString(), serializedString);
+ }
+
+ @Test
+ public void testInterceptSource() {
+ String srcRegex = "word1";
+ String replaceString = "word2";
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ new RegexMountPointResolvedDstPathReplaceInterceptor(srcRegex,
+ replaceString);
+ String sourcePath = "/a/b/l3/dd";
+ sourcePath = interceptor.interceptSource(sourcePath);
+ }
+
+ @Test
+ public void testInterceptResolve() throws IOException {
+ String pathAfterResolution = "/user-hadoop";
+ Path remainingPath = new Path("/ad-data");
+
+ String srcRegex = "hadoop";
+ String replaceString = "hdfs";
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ new RegexMountPointResolvedDstPathReplaceInterceptor(srcRegex,
+ replaceString);
+ interceptor.initialize();
+ Assert.assertTrue(
Review comment:
You want to use assertEquals instead?
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestRegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Test RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+public class TestRegexMountPointResolvedDstPathReplaceInterceptor {
+
+ public String createSerializedString(String regex, String replaceString) {
+ return REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + regex
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + replaceString;
+ }
+
+ @Test
+ public void testDeserializeFromStringNormalCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertTrue(interceptor.getSrcRegexString().equals(srcRegex));
+ Assert.assertTrue(interceptor.getReplaceString().equals(replaceString));
+ Assert.assertTrue(interceptor.getSrcRegexPattern() == null);
+ interceptor.initialize();
+ Assert.assertTrue(
Review comment:
You may want to use assertEquals directly?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/RegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.regex.PatternSyntaxException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Implementation of RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+class RegexMountPointResolvedDstPathReplaceInterceptor
+ implements RegexMountPointInterceptor {
+
+ private String srcRegexString;
+ private String replaceString;
+ private Pattern srcRegexPattern;
+
+ RegexMountPointResolvedDstPathReplaceInterceptor(String srcRegex,
+ String replaceString) {
+ this.srcRegexString = srcRegex;
+ this.replaceString = replaceString;
+ this.srcRegexPattern = null;
+ }
+
+ public String getSrcRegexString() {
+ return srcRegexString;
+ }
+
+ public String getReplaceString() {
+ return replaceString;
+ }
+
+ public Pattern getSrcRegexPattern() {
+ return srcRegexPattern;
+ }
+
+ @Override
+ public void initialize() throws IOException {
+ try {
+ srcRegexPattern = Pattern.compile(srcRegexString);
+ } catch (PatternSyntaxException ex) {
+ throw new IOException(
+ "Initialize interceptor failed, srcRegx:" + srcRegexString, ex);
+ }
+ }
+
+ /**
+ * Intercept source before resolution.
+ *
+ * @param source
+ * @return
+ */
+ @Override
+ public String interceptSource(String source) {
+ return source;
+ }
+
+ /**
+ * Intercept resolved path, e.g.
+ * Mount point /^(\\w+)/, ${1}.hadoop.net
+ * If incoming path is /user1/home/tmp/job1,
+ * then the resolved path str will be user1.
+ *
+ * @return intercepted string
+ */
+ @Override public String interceptResolvedDestPathStr(
+ String parsedDestPathStr) {
+ Matcher matcher = srcRegexPattern.matcher(parsedDestPathStr);
+ return matcher.replaceAll(replaceString);
+ }
+
+ /**
+ * Intercept remaining path.
+ *
+ * @return intercepted path
+ */
+ @Override public Path interceptRemainingPath(Path remainingPath) {
Review comment:
shouldn't it interceptSrc and remainingPath follow same pattern to intercept as they both split from src path only?
Whats the issue if I use same interceptSource method? Do we have some concerns? Could you explain me if I miss something here?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/RegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.regex.PatternSyntaxException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Implementation of RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+class RegexMountPointResolvedDstPathReplaceInterceptor
+ implements RegexMountPointInterceptor {
+
+ private String srcRegexString;
+ private String replaceString;
+ private Pattern srcRegexPattern;
+
+ RegexMountPointResolvedDstPathReplaceInterceptor(String srcRegex,
+ String replaceString) {
+ this.srcRegexString = srcRegex;
+ this.replaceString = replaceString;
+ this.srcRegexPattern = null;
+ }
+
+ public String getSrcRegexString() {
+ return srcRegexString;
+ }
+
+ public String getReplaceString() {
+ return replaceString;
+ }
+
+ public Pattern getSrcRegexPattern() {
+ return srcRegexPattern;
+ }
+
+ @Override
+ public void initialize() throws IOException {
+ try {
+ srcRegexPattern = Pattern.compile(srcRegexString);
+ } catch (PatternSyntaxException ex) {
+ throw new IOException(
+ "Initialize interceptor failed, srcRegx:" + srcRegexString, ex);
+ }
+ }
+
+ /**
+ * Intercept source before resolution.
Review comment:
Seems like we don't do anything here. Could you please add that in Javadoc. Say src will not be intercepted in this impl, it's only for dst?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r486063312
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
##########
@@ -166,6 +166,42 @@ public static void addLinkNfly(final Configuration conf, final String src,
addLinkNfly(conf, getDefaultMountTableName(conf), src, null, targets);
}
+
+ /**
+ * Add a LinkRegex to the config for the specified mount table.
+ * @param conf - get mountable config from this conf
+ * @param mountTableName - the mountable name of the regex config item
+ * @param srcRegex - the src path regex expression that applies to this config
+ * @param targetStr - the string of target path
+ */
Review comment:
This method not used anywhere?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-670807166
The UTs failure are not related to the change. Here're some references:
https://issues.apache.org/jira/browse/HADOOP-15891
Design doc: https://issues.apache.org/jira/secure/attachment/12946315/HDFS-13948_%20Regex%20Link%20Type%20In%20Mount%20Table-v1.pdf
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r486071356
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
##########
@@ -166,6 +166,42 @@ public static void addLinkNfly(final Configuration conf, final String src,
addLinkNfly(conf, getDefaultMountTableName(conf), src, null, targets);
}
+
+ /**
+ * Add a LinkRegex to the config for the specified mount table.
+ * @param conf - get mountable config from this conf
+ * @param mountTableName - the mountable name of the regex config item
+ * @param srcRegex - the src path regex expression that applies to this config
+ * @param targetStr - the string of target path
+ */
Review comment:
Removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-689240050
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 21s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 6s | trunk passed |
| +1 :green_heart: | compile | 19m 32s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 59s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 45s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 59s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 34s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 12s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 24s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 58s | the patch passed |
| +1 :green_heart: | compile | 18m 49s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 18m 49s | the patch passed |
| +1 :green_heart: | compile | 16m 53s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 16m 53s | the patch passed |
| -0 :warning: | checkstyle | 2m 44s | root: The patch generated 1 new + 182 unchanged - 1 fixed = 183 total (was 183) |
| +1 :green_heart: | mvnsite | 2m 57s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 12s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 41s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 13s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 5m 39s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 29s | hadoop-common in the patch passed. |
| -1 :x: | unit | 94m 24s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. |
| | | 276m 23s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.TestMultipleNNPortQOP |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.TestFileChecksum |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 45b80267deee 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0d855159f09 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/17/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/17/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/17/testReport/ |
| Max. process+thread count | 4127 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/17/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r481539041
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestRegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Test RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+public class TestRegexMountPointResolvedDstPathReplaceInterceptor {
+
+ public String createSerializedString(String regex, String replaceString) {
+ return REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + regex
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + replaceString;
+ }
+
+ @Test
+ public void testDeserializeFromStringNormalCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertTrue(interceptor.getSrcRegexString().equals(srcRegex));
+ Assert.assertTrue(interceptor.getReplaceString().equals(replaceString));
+ Assert.assertTrue(interceptor.getSrcRegexPattern() == null);
+ interceptor.initialize();
+ Assert.assertTrue(
+ interceptor.getSrcRegexPattern().toString().equals(srcRegex));
+ }
+
+ @Test
+ public void testDeserializeFromStringBadCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ serializedString = serializedString + ":ddd";
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertEquals(interceptor, null);
+ }
+
+ @Test
+ public void testSerialization() {
+ String srcRegex = "word1";
+ String replaceString = "word2";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ new RegexMountPointResolvedDstPathReplaceInterceptor(srcRegex,
+ replaceString);
+ Assert.assertEquals(interceptor.serializeToString(), serializedString);
+ }
+
+ @Test
Review comment:
I mean we just need to assert I think. Ex: Test saying "InterceptSource". SO, just validate what it should return. In this case, it will return same string. It may not have much value now, but if some one changes it, it can just catch it.
Currently this test will never fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-668464142
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 34s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 53s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 14s | trunk passed |
| +1 :green_heart: | compile | 21m 44s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 18m 24s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 2m 36s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 55s | trunk passed |
| +1 :green_heart: | shadedclient | 19m 41s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 8s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 13s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 3m 15s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 29s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 0s | the patch passed |
| +1 :green_heart: | compile | 21m 43s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 21m 43s | the patch passed |
| +1 :green_heart: | compile | 18m 25s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 18m 25s | the patch passed |
| -0 :warning: | checkstyle | 2m 44s | root: The patch generated 3 new + 92 unchanged - 1 fixed = 95 total (was 93) |
| -1 :x: | mvnsite | 1m 20s | hadoop-hdfs in the patch failed. |
| -1 :x: | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 14m 14s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 4s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| -1 :x: | findbugs | 1m 23s | hadoop-hdfs in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 42s | hadoop-common in the patch passed. |
| -1 :x: | unit | 1m 14s | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 0m 49s | The patch does not generate ASF License warnings. |
| | | 178m 36s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 13f23983b2cf 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ab2b3df2de1 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/2/artifact/out/diff-checkstyle-root.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/2/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/2/artifact/out/whitespace-eol.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/2/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/2/testReport/ |
| Max. process+thread count | 3464 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-685248748
@umamaheswararao Thanks. I did a rebase and updated the patch.
Created https://issues.apache.org/jira/browse/HADOOP-17238 to track the mount points resolution cache problem.
As for `porting into FileContext`, do you mean fix the APIs which are not implemented in ViewFileSystem so FileContext could find the right path?
E.g. createMultipartUploader api.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-685959366
```
As for porting into FileContext, do you mean fix the APIs which are not implemented in ViewFileSystem so FileContext could find the right path?
E.g. createMultipartUploader api.
```
For the FileContext related, we have almost same class implemented ViewFs.java. Whatever we built in ViewFileSystem.java should also be copied or implemented there in ViewFs. Whoever uses AbstractFileSystem interface, they will be able to use it.
Ex: https://issues.apache.org/jira/browse/HDFS-15329
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-689877639
Great work @JohnZZGithub.
Thanks a lot for your hard work in this.
+1 pending jenkins clean report.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-668288316
@liuml07 Thanks, this is a feature adopted inside our company for almost two years. The code is almost as same as our internal branch except I removed some refactored code to make it easier to review. Seemed like the rebase caused some UT failures. Let me fix the UTs and update the user guide.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-690866308
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r481350228
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/RegexMountPoint.java
##########
@@ -0,0 +1,306 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.regex.PatternSyntaxException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.util.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.viewfs.InodeTree.SlashPath;
+
+/**
+ * Regex mount point is build to implement regex based mount point.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+class RegexMountPoint<T> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(RegexMountPoint.class.getName());
+
+ private InodeTree inodeTree;
+ private String srcPathRegex;
+ private Pattern srcPattern;
+ private String dstPath;
+ private String interceptorSettingsString;
+ private List<RegexMountPointInterceptor> interceptorList;
+
+ public static final String SETTING_SRCREGEX_SEP = "#.";
+ public static final char INTERCEPTOR_SEP = ';';
+ public static final char INTERCEPTOR_INTERNAL_SEP = ':';
+ // ${var},$var
+ public static final Pattern VAR_PATTERN_IN_DEST =
+ Pattern.compile("\\$((\\{\\w+\\})|(\\w+))");
+
+ // Same var might have different representations.
+ // e.g.
+ // key => $key or key = > ${key}
+ private Map<String, Set<String>> varInDestPathMap;
+
+ public Map<String, Set<String>> getVarInDestPathMap() {
+ return varInDestPathMap;
+ }
+
+ RegexMountPoint(InodeTree inodeTree, String sourcePathRegex,
+ String destPath, String settingsStr) {
+ this.inodeTree = inodeTree;
+ this.srcPathRegex = sourcePathRegex;
+ this.dstPath = destPath;
+ this.interceptorSettingsString = settingsStr;
+ this.interceptorList = new ArrayList<>();
+ }
+
+ /**
+ * Initialize regex mount point.
+ *
+ * @throws IOException
+ */
+ public void initialize() throws IOException {
+ try {
+ srcPattern = Pattern.compile(srcPathRegex);
+ } catch (PatternSyntaxException ex) {
+ throw new IOException(
+ "Failed to initialized mount point due to bad src path regex:"
+ + srcPathRegex + ", dstPath:" + dstPath, ex);
+ }
+ varInDestPathMap = getVarListInString(dstPath);
+ initializeInterceptors();
+ }
+
+ private void initializeInterceptors() throws IOException {
+ if (interceptorSettingsString == null
+ || interceptorSettingsString.isEmpty()) {
+ return;
+ }
+ String[] interceptorStrArray =
+ StringUtils.split(interceptorSettingsString, INTERCEPTOR_SEP);
+ for (String interceptorStr : interceptorStrArray) {
+ RegexMountPointInterceptor interceptor =
+ RegexMountPointInterceptorFactory.create(interceptorStr);
+ if (interceptor == null) {
+ throw new IOException(
+ "Illegal settings String " + interceptorSettingsString);
+ }
+ interceptor.initialize();
+ interceptorList.add(interceptor);
+ }
+ }
+
+ /**
+ * Get $var1 and $var2 style variables in string.
+ *
+ * @param input - the string to be process.
+ * @return
+ */
+ public static Map<String, Set<String>> getVarListInString(String input) {
+ Map<String, Set<String>> varMap = new HashMap<>();
+ Matcher matcher = VAR_PATTERN_IN_DEST.matcher(input);
+ while (matcher.find()) {
+ // $var or ${var}
+ String varName = matcher.group(0);
+ // var or {var}
+ String strippedVarName = matcher.group(1);
+ if (strippedVarName.startsWith("{")) {
+ // {varName} = > varName
+ strippedVarName =
+ strippedVarName.substring(1, strippedVarName.length() - 1);
+ }
+ varMap.putIfAbsent(strippedVarName, new HashSet<>());
+ varMap.get(strippedVarName).add(varName);
+ }
+ return varMap;
+ }
+
+ public String getSrcPathRegex() {
+ return srcPathRegex;
+ }
+
+ public Pattern getSrcPattern() {
+ return srcPattern;
+ }
+
+ public String getDstPath() {
+ return dstPath;
+ }
+
+ public static Pattern getVarPatternInDest() {
+ return VAR_PATTERN_IN_DEST;
+ }
+
+ /**
+ * Get resolved path from regex mount points.
+ * E.g. link: ^/user/(?<username>\\w+) => s3://$user.apache.com/_${user}
+ * srcPath: is /user/hadoop/dir1
+ * resolveLastComponent: true
+ * then return value is s3://hadoop.apache.com/_hadoop
+ * @param srcPath - the src path to resolve
+ * @param resolveLastComponent - whether resolve the path after last `/`
+ * @return mapped path of the mount point.
+ */
+ public InodeTree.ResolveResult<T> resolve(final String srcPath,
+ final boolean resolveLastComponent) {
+ String pathStrToResolve = getPathToResolve(srcPath, resolveLastComponent);
+ for (RegexMountPointInterceptor interceptor : interceptorList) {
+ pathStrToResolve = interceptor.interceptSource(pathStrToResolve);
+ }
+ LOGGER.debug("Path to resolve:" + pathStrToResolve + ", srcPattern:"
+ + getSrcPathRegex());
+ Matcher srcMatcher = getSrcPattern().matcher(pathStrToResolve);
+ String parsedDestPath = getDstPath();
+ int mappedCount = 0;
+ String resolvedPathStr = "";
+ while (srcMatcher.find()) {
+ resolvedPathStr = pathStrToResolve.substring(0, srcMatcher.end());
+ Map<String, Set<String>> varMap = getVarInDestPathMap();
+ for (Map.Entry<String, Set<String>> entry : varMap.entrySet()) {
+ String regexGroupNameOrIndexStr = entry.getKey();
+ Set<String> groupRepresentationStrSetInDest = entry.getValue();
+ parsedDestPath = replaceRegexCaptureGroupInPath(
+ parsedDestPath, srcMatcher,
+ regexGroupNameOrIndexStr, groupRepresentationStrSetInDest);
+ }
+ ++mappedCount;
+ }
+ if (0 == mappedCount) {
+ return null;
+ }
+ Path remainingPath = getRemainingPathStr(srcPath, resolvedPathStr);
+ for (RegexMountPointInterceptor interceptor : interceptorList) {
+ parsedDestPath = interceptor.interceptResolvedDestPathStr(parsedDestPath);
+ remainingPath =
+ interceptor.interceptRemainingPath(remainingPath);
+ }
+ InodeTree.ResolveResult resolveResult = inodeTree
+ .buildResolveResultForRegexMountPoint(InodeTree.ResultKind.EXTERNAL_DIR,
+ resolvedPathStr, parsedDestPath, remainingPath);
+ return resolveResult;
+ }
+
+ private Path getRemainingPathStr(
+ String srcPath,
+ String resolvedPathStr) {
+ String remainingPathStr = srcPath.substring(resolvedPathStr.length());
+ if (!remainingPathStr.startsWith("/")) {
+ remainingPathStr = "/" + remainingPathStr;
+ }
+ return new Path(remainingPathStr);
+ }
+
+ private String getPathToResolve(
+ String srcPath, boolean resolveLastComponent) {
+ if (resolveLastComponent) {
+ return srcPath;
+ }
+ int lastSlashIndex = srcPath.lastIndexOf(SlashPath.toString());
+ if (lastSlashIndex == -1) {
+ return null;
+ }
+ return srcPath.substring(0, lastSlashIndex);
+ }
+
+ /**
+ * Use capture group named regexGroupNameOrIndexStr in mather to replace
+ * parsedDestPath.
+ * E.g. link: ^/user/(?<username>\\w+) => s3://$user.apache.com/_${user}
+ * srcMatcher is from /user/hadoop.
+ * Then the params will be like following.
+ * parsedDestPath: s3://$user.apache.com/_${user},
+ * regexGroupNameOrIndexStr: user
+ * groupRepresentationStrSetInDest: {user:$user; user:${user}}
+ * return value will be s3://hadoop.apache.com/_hadoop
+ * @param parsedDestPath
+ * @param srcMatcher
+ * @param regexGroupNameOrIndexStr
+ * @param groupRepresentationStrSetInDest
+ * @return return parsedDestPath while ${var},$var replaced or
+ * parsedDestPath nothing found.
+ */
+ private String replaceRegexCaptureGroupInPath(
+ String parsedDestPath,
+ Matcher srcMatcher,
+ String regexGroupNameOrIndexStr,
+ Set<String> groupRepresentationStrSetInDest) {
+ String groupValue = getRegexGroupValueFromMather(
+ srcMatcher, regexGroupNameOrIndexStr);
+ if (groupValue == null) {
+ return parsedDestPath;
+ }
+ for (String varName : groupRepresentationStrSetInDest) {
+ parsedDestPath = parsedDestPath.replace(varName, groupValue);
+ LOGGER.debug("parsedDestPath value is:" + parsedDestPath);
+ }
+ return parsedDestPath;
+ }
+
+ /**
+ * Get matched capture group value from regex matched string. E.g.
+ * Regex: ^/user/(?<username>\\w+), regexGroupNameOrIndexStr: userName
+ * then /user/hadoop should return hadoop while call
+ * getRegexGroupValueFromMather(matcher, usersName)
+ * or getRegexGroupValueFromMather(matcher, 1)
+ *
+ * @param srcMatcher - the matcher to be use
+ * @param regexGroupNameOrIndexStr - the regex group name or index
+ * @return - Null if no matched group named regexGroupNameOrIndexStr found.
+ */
+ private String getRegexGroupValueFromMather(
+ Matcher srcMatcher, String regexGroupNameOrIndexStr) {
+ if (regexGroupNameOrIndexStr.matches("\\d+")) {
+ // group index
+ int groupIndex = Integer.parseUnsignedInt(regexGroupNameOrIndexStr);
+ if (groupIndex >= 0 && groupIndex <= srcMatcher.groupCount()) {
+ return srcMatcher.group(groupIndex);
+ }
+ } else {
+ // named group in regex
+ return srcMatcher.group(regexGroupNameOrIndexStr);
+ }
+ return null;
+ }
+
+ /**
Review comment:
Good catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub edited a comment on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub edited a comment on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-686183399
Thanks @umamaheswararao
@umamaheswararao Thanks for the comments. Please see the reply inline.
> Hi @JohnZZGithub, I got few other points to discuss.
>
> 1. We have exposed getMountPoints API. It seems we can't return any mount points from REGEX based because you would not know until you got src paths to resolve and find real target fs. What should we do for this API?
It's a great question. I guess most caller of getMountPoints wants to traverse all the file systems to do some operation. E.g. setVerifyChecksum(). We didn't see issues on our internal Yarn + HDFS and Yarn + GCS clusters. The usage pattern includes but not limited to MR, Spark, Presto, Vertica loading and etc. But it's possible that some users might rely on these APIs. I could see two options forward:
1. Returning a MountPint with special FileSystem for Regex Mount points. We could cache the initialized fileSystem under the regex mountpoint and perform the operation. For filesystems that might appear in the future, we could cache the past calls from callers and try to apply it or just not support it.
2. We could indicate that we don't support such APIs for regex mount points.
And to extend the topic a little bit, this kind of ViewFileSystem API (API which tries to visit all file systems) caused several problems for us. E.g. setVerifyChecksum() initialized a file system for a mount point users didn't want to use it all. And the initialization of the file system will fail as it requires credentials during initialization. Users don't have it as it never means to visit the mount point. We developed a LazyChRootedFileSystem on top of every target system (not public) to do lazy initialization for path-based APIs. But it's hard to tackle APIs without path passed in. So to summarize, we see cases users want to avoid these non-path based API to trigger actions on every child file system. In the meantime, some users(though rare in our scenarios) might want to use these APIs applied to all children's filesystems. I feel it's hard to satisfy both needs.
> 2. Other API is getDelegationTokenIssuers. Applications like YARN uses this API to get all child fs delegation tokens. This also will not work for REGEX based mount points.
We did see an issue with addDelegationTokens in the secure Hadoop cluster. But the problem we met is not all normal mountpoints are secure. So the API caused a problem when it tries to initialize all children's file systems. We took a workaround by making it path-based. As for getDelegationTokens, I guess the problem is similar. We didn't see issues because it's not used. Could we make it path based too? Or we could take the approach stated in problem one.
> 3. Other question is how this child filesystem objects gets closed. There was an issue with [ViewFileSystem#close | https://issues.apache.org/jira/browse/HADOOP-15565 ]. I would like to know how that get addressed in this case as don't keep anything in InnerCache.
Could we make the inner cache a thread-safe structure and track all the opened file systems under regex mount points?
These are really great points, thanks a lot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r467176661
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##########
@@ -486,84 +506,113 @@ protected InodeTree(final Configuration config, final String viewName,
final UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
for (Entry<String, String> si : config) {
final String key = si.getKey();
- if (key.startsWith(mountTablePrefix)) {
- gotMountTableEntry = true;
- LinkType linkType;
- String src = key.substring(mountTablePrefix.length());
- String settings = null;
- if (src.startsWith(linkPrefix)) {
- src = src.substring(linkPrefix.length());
- if (src.equals(SlashPath.toString())) {
- throw new UnsupportedFileSystemException("Unexpected mount table "
- + "link entry '" + key + "'. Use "
- + Constants.CONFIG_VIEWFS_LINK_MERGE_SLASH + " instead!");
- }
- linkType = LinkType.SINGLE;
- } else if (src.startsWith(linkFallbackPrefix)) {
- if (src.length() != linkFallbackPrefix.length()) {
- throw new IOException("ViewFs: Mount points initialization error." +
- " Invalid " + Constants.CONFIG_VIEWFS_LINK_FALLBACK +
- " entry in config: " + src);
- }
- linkType = LinkType.SINGLE_FALLBACK;
- } else if (src.startsWith(linkMergePrefix)) { // A merge link
- src = src.substring(linkMergePrefix.length());
- linkType = LinkType.MERGE;
- } else if (src.startsWith(linkMergeSlashPrefix)) {
- // This is a LinkMergeSlash entry. This entry should
- // not have any additional source path.
- if (src.length() != linkMergeSlashPrefix.length()) {
- throw new IOException("ViewFs: Mount points initialization error." +
- " Invalid " + Constants.CONFIG_VIEWFS_LINK_MERGE_SLASH +
- " entry in config: " + src);
- }
- linkType = LinkType.MERGE_SLASH;
- } else if (src.startsWith(Constants.CONFIG_VIEWFS_LINK_NFLY)) {
- // prefix.settings.src
- src = src.substring(Constants.CONFIG_VIEWFS_LINK_NFLY.length() + 1);
- // settings.src
- settings = src.substring(0, src.indexOf('.'));
- // settings
-
- // settings.src
- src = src.substring(settings.length() + 1);
- // src
-
- linkType = LinkType.NFLY;
- } else if (src.startsWith(Constants.CONFIG_VIEWFS_HOMEDIR)) {
- // ignore - we set home dir from config
- continue;
- } else {
- throw new IOException("ViewFs: Cannot initialize: Invalid entry in " +
- "Mount table in config: " + src);
- }
+ if (!key.startsWith(mountTablePrefix)) {
Review comment:
It looks complex. But the change here is simple.
Before:
if (key.startsWith(mountTablePrefix)) {
// hundred line of code
}
After:
if (!key.startsWith(mountTablePrefix)) {
continue
}
// hundred line of code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-690180254
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 2s | trunk passed |
| +1 :green_heart: | compile | 21m 11s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 18m 0s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 44s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 1s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 17s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 35s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 15s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 21s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 38s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 3s | the patch passed |
| +1 :green_heart: | compile | 20m 12s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 20m 12s | the patch passed |
| +1 :green_heart: | compile | 18m 8s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 18m 8s | the patch passed |
| -0 :warning: | checkstyle | 3m 0s | root: The patch generated 1 new + 182 unchanged - 1 fixed = 183 total (was 183) |
| +1 :green_heart: | mvnsite | 3m 10s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 37s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 43s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 18s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 6m 47s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 20s | hadoop-common in the patch passed. |
| -1 :x: | unit | 103m 58s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 8s | The patch does not generate ASF License warnings. |
| | | 299m 6s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.TestErasureCodingMultipleRacks |
| | hadoop.hdfs.server.namenode.TestAddStripedBlocks |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocksWithRandomECPolicy |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.namenode.TestRefreshBlockPlacementPolicy |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/23/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 2bce93b11ee9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e5fe3262702 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/23/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/23/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/23/testReport/ |
| Max. process+thread count | 3867 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/23/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-690016227
@umamaheswararao Totally make sense, updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-690866308
@umamaheswararao Thanks a lot!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-690863729
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r486070892
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
##########
@@ -166,6 +166,42 @@ public static void addLinkNfly(final Configuration conf, final String src,
addLinkNfly(conf, getDefaultMountTableName(conf), src, null, targets);
}
+
+ /**
+ * Add a LinkRegex to the config for the specified mount table.
+ * @param conf - get mountable config from this conf
+ * @param mountTableName - the mountable name of the regex config item
+ * @param srcRegex - the src path regex expression that applies to this config
+ * @param targetStr - the string of target path
+ */
Review comment:
Good catch, I guess the next addLinkRegex is used but not this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-687645245
> I guess most caller of getMountPoints wants to traverse all the file systems to do some operation. E.g. setVerifyChecksum(). We didn't see issues on our internal Yarn + HDFS and Yarn + GCS clusters. The usage pattern includes but not limited to MR, Spark, Presto, Vertica loading and etc. But it's possible that some users might rely on these APIs.
In YarnClient seems to be collecting tokens from all DelegationTokenIssuer.
in DelegationTokenIssuer#collectDelegationTokens
```
// Now collect the tokens from the children.
final DelegationTokenIssuer[] ancillary =
issuer.getAdditionalTokenIssuers();
if (ancillary != null) {
for (DelegationTokenIssuer subIssuer : ancillary) {
collectDelegationTokens(subIssuer, renewer, credentials, tokens);
}
}
```
If you look here issuer is current fs and it's trying to get additionalTokenIssuers.
The default implementation of getDelegationTokenIssuers at FileSystem.java is simply getting all ChildFileSystems.
```
@InterfaceAudience.Private
@Override
public DelegationTokenIssuer[] getAdditionalTokenIssuers()
throws IOException {
return getChildFileSystems();
}
```
This will get all the child file systems available. Currently the implementation is getChildFileSystems in ViewFileSystem is like below:
```
@Override
public FileSystem[] getChildFileSystems() {
List<InodeTree.MountPoint<FileSystem>> mountPoints =
fsState.getMountPoints();
Set<FileSystem> children = new HashSet<FileSystem>();
for (InodeTree.MountPoint<FileSystem> mountPoint : mountPoints) {
FileSystem targetFs = mountPoint.target.targetFileSystem;
children.addAll(Arrays.asList(targetFs.getChildFileSystems()));
}
if (fsState.isRootInternalDir() && fsState.getRootFallbackLink() != null) {
children.addAll(Arrays.asList(
fsState.getRootFallbackLink().targetFileSystem
.getChildFileSystems()));
}
return children.toArray(new FileSystem[]{});
}
```
It's iterating over mount points available getting all targetFileSystems. In the case of REGEX based mount points, we will not have any childFileSystems available via getChildFileSystems call.
We also implemented ViewDistributedFileSystem to provide hdfs specific API compatibility. Here also we used getChildFileSystems for some APIs.
>Returning a MountPint with special FileSystem for Regex Mount points. We could cache the initialized fileSystem under the regex mountpoint and perform the operation. For filesystems that might appear in the future, we could cache the past calls from callers and try to apply it or just not support it.
I am thinking that, how about adding the resolved mount points from RegxBased to MountPoints list? So, that when user calls getMounts, it will simply return whatever mountPonts so far inited. How many unique mount points could be there in total with Regx based in practice (resolved mappings)? We should document that, with RegEX based mount points, getMountPoints will return only currently resolved mount points.
> We did see an issue with addDelegationTokens in the secure Hadoop cluster. But the problem we met is not all normal mountpoints are secure. So the API caused a problem when it tries to initialize all children's file systems. We took a workaround by making it path-based. As for getDelegationTokens, I guess the problem is similar. We didn't see issues because it's not used. Could we make it path based too?
Certainly we can make it uri path based. However users need to make use of it and it could be a long term improvement because users would not change immediately to new APIs what we introduce now. It will take longer time for upstream projects to change.
>Could we make the inner cache a thread-safe structure and track all the opened file systems under regex mount points?
Let's target to solve this problem first. Yes, I think maintaining initialized fs-es in InnerCache could help to close fileSystems correctly. Let's make it to thread-safe and add opened fs-es there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r481521111
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestRegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Test RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+public class TestRegexMountPointResolvedDstPathReplaceInterceptor {
+
+ public String createSerializedString(String regex, String replaceString) {
+ return REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + regex
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + replaceString;
+ }
+
+ @Test
+ public void testDeserializeFromStringNormalCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertTrue(interceptor.getSrcRegexString().equals(srcRegex));
+ Assert.assertTrue(interceptor.getReplaceString().equals(replaceString));
+ Assert.assertTrue(interceptor.getSrcRegexPattern() == null);
+ interceptor.initialize();
+ Assert.assertTrue(
+ interceptor.getSrcRegexPattern().toString().equals(srcRegex));
+ }
+
+ @Test
+ public void testDeserializeFromStringBadCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ serializedString = serializedString + ":ddd";
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertEquals(interceptor, null);
+ }
+
+ @Test
+ public void testSerialization() {
+ String srcRegex = "word1";
+ String replaceString = "word2";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ new RegexMountPointResolvedDstPathReplaceInterceptor(srcRegex,
+ replaceString);
+ Assert.assertEquals(interceptor.serializeToString(), serializedString);
+ }
+
+ @Test
Review comment:
I meant to have a safegurad when users want to change the interceptor settings format. But I'm OK to remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-685704174
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 21s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 11s | trunk passed |
| +1 :green_heart: | compile | 19m 36s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 57s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 49s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 1s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 32s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 41s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 12s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 24s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 58s | the patch passed |
| +1 :green_heart: | compile | 18m 49s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 18m 49s | the patch passed |
| +1 :green_heart: | compile | 16m 56s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 16m 56s | the patch passed |
| -0 :warning: | checkstyle | 2m 43s | root: The patch generated 1 new + 182 unchanged - 1 fixed = 183 total (was 183) |
| +1 :green_heart: | mvnsite | 2m 58s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 6s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 40s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 11s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 5m 40s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 12m 4s | hadoop-common in the patch passed. |
| -1 :x: | unit | 95m 18s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 7s | The patch does not generate ASF License warnings. |
| | | 280m 3s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestDataTransferKeepalive |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 931ca8bebf37 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0207f5cf461 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/13/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/13/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/13/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/13/testReport/ |
| Max. process+thread count | 4186 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/13/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-689975586
@umamaheswararao Thanks a lot!
There's one checkstyle violation. However, I guess it's not introduced by the patch. Yetus recognized a long function name which is unchanged in the path.
> ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java:487: protected InodeTree(final Configuration config, final String viewName,:3: Method length is 191 lines (max allowed is 150). [MethodLength]
As for the failure UTs, I guess they are not related to the patch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r471892058
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkRegex.java
##########
@@ -0,0 +1,320 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.FsConstants;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSNNTopology;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE;
+import static org.apache.hadoop.fs.viewfs.RegexMountPoint.INTERCEPTOR_INTERNAL_SEP;
+
+/**
+ * Test linkRegex node type for view file system.
+ */
+public class TestViewFileSystemLinkRegex extends ViewFileSystemBaseTest {
+ public static final Logger LOGGER =
+ LoggerFactory.getLogger(TestViewFileSystemLinkRegex.class);
+
+ private static FileSystem fsDefault;
+ private static MiniDFSCluster cluster;
+ private static Configuration clusterConfig;
+ private static final int NAME_SPACES_COUNT = 3;
+ private static final int DATA_NODES_COUNT = 3;
+ private static final int FS_INDEX_DEFAULT = 0;
+ private static final FileSystem[] FS_HDFS = new FileSystem[NAME_SPACES_COUNT];
+ private static final String CLUSTER_NAME =
+ "TestViewFileSystemLinkRegexCluster";
+ private static final File TEST_DIR = GenericTestUtils
+ .getTestDir(TestViewFileSystemLinkRegex.class.getSimpleName());
+ private static final String TEST_BASE_PATH =
+ "/tmp/TestViewFileSystemLinkRegex";
+
+ @Override
+ protected FileSystemTestHelper createFileSystemHelper() {
+ return new FileSystemTestHelper(TEST_BASE_PATH);
+ }
+
+ @BeforeClass
+ public static void clusterSetupAtBeginning() throws IOException {
+ SupportsBlocks = true;
+ clusterConfig = ViewFileSystemTestSetup.createConfig();
+ clusterConfig.setBoolean(
+ DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY,
+ true);
+ cluster = new MiniDFSCluster.Builder(clusterConfig).nnTopology(
+ MiniDFSNNTopology.simpleFederatedTopology(NAME_SPACES_COUNT))
+ .numDataNodes(DATA_NODES_COUNT).build();
+ cluster.waitClusterUp();
+
+ for (int i = 0; i < NAME_SPACES_COUNT; i++) {
+ FS_HDFS[i] = cluster.getFileSystem(i);
+ }
+ fsDefault = FS_HDFS[FS_INDEX_DEFAULT];
+ }
+
+ @AfterClass
+ public static void clusterShutdownAtEnd() throws Exception {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+
+ @Override
+ @Before
+ public void setUp() throws Exception {
+ fsTarget = fsDefault;
+ super.setUp();
+ }
+
+ /**
+ * Override this so that we don't set the targetTestRoot to any path under the
+ * root of the FS, and so that we don't try to delete the test dir, but rather
+ * only its contents.
+ */
+ @Override
+ void initializeTargetTestRoot() throws IOException {
+ targetTestRoot = fsDefault.makeQualified(new Path("/"));
+ for (FileStatus status : fsDefault.listStatus(targetTestRoot)) {
+ fsDefault.delete(status.getPath(), true);
+ }
+ }
+
+ @Override
+ void setupMountPoints() {
+ super.setupMountPoints();
+ }
+
+ @Override
+ int getExpectedDelegationTokenCount() {
+ return 1; // all point to the same fs so 1 unique token
+ }
+
+ @Override
+ int getExpectedDelegationTokenCountWithCredentials() {
+ return 1;
+ }
+
+ public String buildReplaceInterceptorSettingString(String srcRegex,
+ String replaceString) {
+ return
+ RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + INTERCEPTOR_INTERNAL_SEP + srcRegex + INTERCEPTOR_INTERNAL_SEP
+ + replaceString;
+ }
+
+ public String linkInterceptorSettings(
+ List<String> interceptorSettingStrList) {
+ StringBuilder stringBuilder = new StringBuilder();
+ int listSize = interceptorSettingStrList.size();
+ for (int i = 0; i < listSize; ++i) {
+ stringBuilder.append(interceptorSettingStrList.get(i));
+ if (i < listSize - 1) {
+ stringBuilder.append(RegexMountPoint.INTERCEPTOR_SEP);
+ }
+ }
+ return stringBuilder.toString();
+ }
+
+ @Test
+ public void testConfLinkRegexIndexMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ // (^/(\w+),/targetTestRoot/$1)
+ // => /targetTestRoot/testConfLinkRegexIndexMapping1
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/(\\w+)";
+ String dstPathStr = targetTestRoot + "$1";
+ Path srcPath = new Path("/testConfLinkRegexIndexMapping1");
+ Path expectedResolveResult =
+ new Path(dstPathStr.replace("$1", "testConfLinkRegexIndexMapping1"));
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ // Test ${1} format
+ // ^/(\w+, /targetTestRoot/${1})
+ // => /targetTestRoot/testConfLinkRegexIndexMapping2
+ dstPathStr = targetTestRoot + "${1}";
+ srcPath = new Path("/testConfLinkRegexIndexMapping2");
+ expectedResolveResult =
+ new Path(dstPathStr.replace("${1}", "testConfLinkRegexIndexMapping2"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping3/file1
+ dstPathStr = targetTestRoot + "$1";
+ srcPath = new Path("/testConfLinkRegexIndexMapping3/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping3/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1/)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping4/file1
+ dstPathStr = targetTestRoot + "$1/";
+ srcPath = new Path("/testConfLinkRegexIndexMapping4/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping4/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+ }
+
+ @Test
+ public void testConfLinkRegexNamedGroupMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ // ^/(?<firstDir>\\w+) = > /targetTestRoot/$firstDir
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/(?<firstDir>\\w+)";
+ String dstPathStr = targetTestRoot + "$firstDir";
+ Path srcPath = new Path("/testConfLinkRegexNamedGroupMapping1");
+ Path expectedResolveResult = new Path(
+ dstPathStr.replace("$firstDir", "testConfLinkRegexNamedGroupMapping1"));
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ // Test ${1} format
+ dstPathStr = targetTestRoot + "${firstDir}";
+ srcPath = new Path("/testConfLinkRegexNamedGroupMapping2");
+ expectedResolveResult = new Path(dstPathStr
+ .replace("${firstDir}", "testConfLinkRegexNamedGroupMapping2"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+ }
+
+ @Test
+ public void testConfLinkRegexFixedDestMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/\\w+";
+ String dstPathStr =
+ targetTestRoot + "testConfLinkRegexFixedDestMappingFile";
+ Path expectedResolveResult = new Path(dstPathStr);
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(
+ expectedResolveResult.equals(vfs.resolvePath(new Path("/misc1"))));
+ Assert.assertTrue(
+ expectedResolveResult.equals(vfs.resolvePath(new Path("/misc2"))));
+ }
+
+ @Test
+ public void testConfLinkRegexWithSingleInterceptor() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/user/(?<username>\\w+)";
+ String dstPathStr = targetTestRoot + "$username";
+ // Replace "_" with "-"
+ String settingString = buildReplaceInterceptorSettingString("_", "-");
+ Path srcPath = new Path("/user/hadoop_user1/hadoop_file1");
+ Path expectedResolveResult =
+ new Path(targetTestRoot, "hadoop-user1/hadoop_file1");
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil
+ .addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, settingString);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+ }
+
Review comment:
Sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao merged pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao merged pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-686183399
Thanks @umamaheswararao
@umamaheswararao Thanks for the comments. Please see the reply inline.
> Hi @JohnZZGithub, I got few other points to discuss.
>
> 1. We have exposed getMountPoints API. It seems we can't return any mount points from REGEX based because you would not know until you got src paths to resolve and find real target fs. What should we do for this API?
It's a great question. I guess most caller of getMountPoints wants to traverse all the file systems to do some operation. E.g. setVerifyChecksum(). We didn't see issues on our internal Yarn + HDFS and Yarn + GCS clusters. The usage pattern includes but not limited to MR, Spark, Presto, Vertica loading and etc. But it's possible that some users might rely on these APIs. I could see two options forward:
1. Returning a MountPint with special FileSystem for Regex Mount points. We could cache the initialized fileSystem under the regex mountpoint and perform the operation. For filesystems that might appear in the future, we could cache the past calls from callers and try to apply it or just not support it.
2. We could indicate that we don't support such APIs for regex mount points.
And to extend the topic a little bit, this kind of ViewFileSystem API (API which tries to visit all file systems) caused several problems for us. E.g. setVerifyChecksum() initialized a file system for a mount point users didn't want to use it all. And the initialization of the file system will fail as it requires credentials during initialization. Users don't have it as it never means to visit the mount point. We developed a LazyChRootedFileSystem on top of every target system (not public) to do lazy initialization for path-based APIs. But it's hard to tackle APIs without path passed in. So to summarize, we see cases users want to avoid these non-path based API to trigger actions on every child file system. In the meantime, some users(though rare in our scenarios) might want to use these APIs applied to all children's filesystems. I feel it's hard to satisfy both needs.
> 2. Other API is getDelegationTokenIssuers. Applications like YARN uses this API to get all child fs delegation tokens. This also will not work for REGEX based mount points.
We did see an issue with addDelegationTokens in the secure Hadoop cluster. But the problem we met is not all normal mountpoints are secure. So the API caused a problem when it tries to initialize all children's file systems. We took a workaround by making it path-based. As for getDelegationTokens, I guess the problem is similar. We didn't see issues because it's not used. Could we make it path based too? Or we could take the approach stated in problem one.
> 3. Other question is how this child filesystem objects gets closed. There was an issue with [ViewFileSystem#close | https://issues.apache.org/jira/browse/HADOOP-15565 ]. I would like to know how that get addressed in this case as don't keep anything in InnerCache.
Could we make the inner cache a thread-safe structure and track all the opened file systems under regex mount points?
These are really great points, thanks a lot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r485346954
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
##########
@@ -86,12 +86,21 @@
*/
String CONFIG_VIEWFS_LINK_MERGE_SLASH = "linkMergeSlash";
+ /**
+ * Config variable for specifying a regex link which uses regular expressions
+ * as source and target could use group captured in src.
+ * E.g. (^/(?<firstDir>\\w+), /prefix-${firstDir}) =>
+ * (/path1/file1 => /prefix-path1/file1)
+ */
+ String CONFIG_VIEWFS_LINK_REGEX = "linkRegex";
+
FsPermission PERMISSION_555 = new FsPermission((short) 0555);
String CONFIG_VIEWFS_RENAME_STRATEGY = "fs.viewfs.rename.strategy";
/**
* Enable ViewFileSystem to cache all children filesystems in inner cache.
Review comment:
Nice catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r471866865
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkRegex.java
##########
@@ -0,0 +1,320 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.FsConstants;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSNNTopology;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE;
+import static org.apache.hadoop.fs.viewfs.RegexMountPoint.INTERCEPTOR_INTERNAL_SEP;
+
+/**
+ * Test linkRegex node type for view file system.
+ */
+public class TestViewFileSystemLinkRegex extends ViewFileSystemBaseTest {
+ public static final Logger LOGGER =
+ LoggerFactory.getLogger(TestViewFileSystemLinkRegex.class);
+
+ private static FileSystem fsDefault;
+ private static MiniDFSCluster cluster;
+ private static Configuration clusterConfig;
+ private static final int NAME_SPACES_COUNT = 3;
+ private static final int DATA_NODES_COUNT = 3;
+ private static final int FS_INDEX_DEFAULT = 0;
+ private static final FileSystem[] FS_HDFS = new FileSystem[NAME_SPACES_COUNT];
+ private static final String CLUSTER_NAME =
+ "TestViewFileSystemLinkRegexCluster";
+ private static final File TEST_DIR = GenericTestUtils
+ .getTestDir(TestViewFileSystemLinkRegex.class.getSimpleName());
+ private static final String TEST_BASE_PATH =
+ "/tmp/TestViewFileSystemLinkRegex";
+
+ @Override
+ protected FileSystemTestHelper createFileSystemHelper() {
+ return new FileSystemTestHelper(TEST_BASE_PATH);
+ }
+
+ @BeforeClass
+ public static void clusterSetupAtBeginning() throws IOException {
+ SupportsBlocks = true;
+ clusterConfig = ViewFileSystemTestSetup.createConfig();
+ clusterConfig.setBoolean(
+ DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY,
+ true);
+ cluster = new MiniDFSCluster.Builder(clusterConfig).nnTopology(
+ MiniDFSNNTopology.simpleFederatedTopology(NAME_SPACES_COUNT))
+ .numDataNodes(DATA_NODES_COUNT).build();
+ cluster.waitClusterUp();
+
+ for (int i = 0; i < NAME_SPACES_COUNT; i++) {
+ FS_HDFS[i] = cluster.getFileSystem(i);
+ }
+ fsDefault = FS_HDFS[FS_INDEX_DEFAULT];
+ }
+
+ @AfterClass
+ public static void clusterShutdownAtEnd() throws Exception {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+
+ @Override
+ @Before
+ public void setUp() throws Exception {
+ fsTarget = fsDefault;
+ super.setUp();
+ }
+
+ /**
+ * Override this so that we don't set the targetTestRoot to any path under the
+ * root of the FS, and so that we don't try to delete the test dir, but rather
+ * only its contents.
+ */
+ @Override
+ void initializeTargetTestRoot() throws IOException {
+ targetTestRoot = fsDefault.makeQualified(new Path("/"));
+ for (FileStatus status : fsDefault.listStatus(targetTestRoot)) {
+ fsDefault.delete(status.getPath(), true);
+ }
+ }
+
+ @Override
+ void setupMountPoints() {
+ super.setupMountPoints();
+ }
+
+ @Override
+ int getExpectedDelegationTokenCount() {
+ return 1; // all point to the same fs so 1 unique token
+ }
+
+ @Override
+ int getExpectedDelegationTokenCountWithCredentials() {
+ return 1;
+ }
+
+ public String buildReplaceInterceptorSettingString(String srcRegex,
+ String replaceString) {
+ return
+ RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + INTERCEPTOR_INTERNAL_SEP + srcRegex + INTERCEPTOR_INTERNAL_SEP
+ + replaceString;
+ }
+
+ public String linkInterceptorSettings(
+ List<String> interceptorSettingStrList) {
+ StringBuilder stringBuilder = new StringBuilder();
+ int listSize = interceptorSettingStrList.size();
+ for (int i = 0; i < listSize; ++i) {
+ stringBuilder.append(interceptorSettingStrList.get(i));
+ if (i < listSize - 1) {
+ stringBuilder.append(RegexMountPoint.INTERCEPTOR_SEP);
+ }
+ }
+ return stringBuilder.toString();
+ }
+
+ @Test
+ public void testConfLinkRegexIndexMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ // (^/(\w+),/targetTestRoot/$1)
+ // => /targetTestRoot/testConfLinkRegexIndexMapping1
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/(\\w+)";
+ String dstPathStr = targetTestRoot + "$1";
+ Path srcPath = new Path("/testConfLinkRegexIndexMapping1");
+ Path expectedResolveResult =
+ new Path(dstPathStr.replace("$1", "testConfLinkRegexIndexMapping1"));
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ // Test ${1} format
+ // ^/(\w+, /targetTestRoot/${1})
+ // => /targetTestRoot/testConfLinkRegexIndexMapping2
+ dstPathStr = targetTestRoot + "${1}";
+ srcPath = new Path("/testConfLinkRegexIndexMapping2");
+ expectedResolveResult =
+ new Path(dstPathStr.replace("${1}", "testConfLinkRegexIndexMapping2"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping3/file1
+ dstPathStr = targetTestRoot + "$1";
+ srcPath = new Path("/testConfLinkRegexIndexMapping3/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping3/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1/)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping4/file1
+ dstPathStr = targetTestRoot + "$1/";
+ srcPath = new Path("/testConfLinkRegexIndexMapping4/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping4/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+ }
+
+ @Test
+ public void testConfLinkRegexNamedGroupMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ // ^/(?<firstDir>\\w+) = > /targetTestRoot/$firstDir
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/(?<firstDir>\\w+)";
+ String dstPathStr = targetTestRoot + "$firstDir";
+ Path srcPath = new Path("/testConfLinkRegexNamedGroupMapping1");
+ Path expectedResolveResult = new Path(
+ dstPathStr.replace("$firstDir", "testConfLinkRegexNamedGroupMapping1"));
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
Review comment:
Good point
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkRegex.java
##########
@@ -0,0 +1,320 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.FsConstants;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSNNTopology;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE;
+import static org.apache.hadoop.fs.viewfs.RegexMountPoint.INTERCEPTOR_INTERNAL_SEP;
+
+/**
+ * Test linkRegex node type for view file system.
+ */
+public class TestViewFileSystemLinkRegex extends ViewFileSystemBaseTest {
+ public static final Logger LOGGER =
+ LoggerFactory.getLogger(TestViewFileSystemLinkRegex.class);
+
+ private static FileSystem fsDefault;
+ private static MiniDFSCluster cluster;
+ private static Configuration clusterConfig;
+ private static final int NAME_SPACES_COUNT = 3;
+ private static final int DATA_NODES_COUNT = 3;
+ private static final int FS_INDEX_DEFAULT = 0;
+ private static final FileSystem[] FS_HDFS = new FileSystem[NAME_SPACES_COUNT];
+ private static final String CLUSTER_NAME =
+ "TestViewFileSystemLinkRegexCluster";
+ private static final File TEST_DIR = GenericTestUtils
+ .getTestDir(TestViewFileSystemLinkRegex.class.getSimpleName());
+ private static final String TEST_BASE_PATH =
+ "/tmp/TestViewFileSystemLinkRegex";
+
+ @Override
+ protected FileSystemTestHelper createFileSystemHelper() {
+ return new FileSystemTestHelper(TEST_BASE_PATH);
+ }
+
+ @BeforeClass
+ public static void clusterSetupAtBeginning() throws IOException {
+ SupportsBlocks = true;
+ clusterConfig = ViewFileSystemTestSetup.createConfig();
+ clusterConfig.setBoolean(
+ DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY,
+ true);
+ cluster = new MiniDFSCluster.Builder(clusterConfig).nnTopology(
+ MiniDFSNNTopology.simpleFederatedTopology(NAME_SPACES_COUNT))
+ .numDataNodes(DATA_NODES_COUNT).build();
+ cluster.waitClusterUp();
+
+ for (int i = 0; i < NAME_SPACES_COUNT; i++) {
+ FS_HDFS[i] = cluster.getFileSystem(i);
+ }
+ fsDefault = FS_HDFS[FS_INDEX_DEFAULT];
+ }
+
+ @AfterClass
+ public static void clusterShutdownAtEnd() throws Exception {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+
+ @Override
+ @Before
+ public void setUp() throws Exception {
+ fsTarget = fsDefault;
+ super.setUp();
+ }
+
+ /**
+ * Override this so that we don't set the targetTestRoot to any path under the
+ * root of the FS, and so that we don't try to delete the test dir, but rather
+ * only its contents.
+ */
+ @Override
+ void initializeTargetTestRoot() throws IOException {
+ targetTestRoot = fsDefault.makeQualified(new Path("/"));
+ for (FileStatus status : fsDefault.listStatus(targetTestRoot)) {
+ fsDefault.delete(status.getPath(), true);
+ }
+ }
+
+ @Override
+ void setupMountPoints() {
+ super.setupMountPoints();
+ }
+
+ @Override
+ int getExpectedDelegationTokenCount() {
+ return 1; // all point to the same fs so 1 unique token
+ }
+
+ @Override
+ int getExpectedDelegationTokenCountWithCredentials() {
+ return 1;
+ }
+
+ public String buildReplaceInterceptorSettingString(String srcRegex,
+ String replaceString) {
+ return
+ RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + INTERCEPTOR_INTERNAL_SEP + srcRegex + INTERCEPTOR_INTERNAL_SEP
+ + replaceString;
+ }
+
+ public String linkInterceptorSettings(
+ List<String> interceptorSettingStrList) {
+ StringBuilder stringBuilder = new StringBuilder();
+ int listSize = interceptorSettingStrList.size();
+ for (int i = 0; i < listSize; ++i) {
+ stringBuilder.append(interceptorSettingStrList.get(i));
+ if (i < listSize - 1) {
+ stringBuilder.append(RegexMountPoint.INTERCEPTOR_SEP);
+ }
+ }
+ return stringBuilder.toString();
+ }
+
+ @Test
+ public void testConfLinkRegexIndexMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ // (^/(\w+),/targetTestRoot/$1)
+ // => /targetTestRoot/testConfLinkRegexIndexMapping1
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/(\\w+)";
+ String dstPathStr = targetTestRoot + "$1";
+ Path srcPath = new Path("/testConfLinkRegexIndexMapping1");
+ Path expectedResolveResult =
+ new Path(dstPathStr.replace("$1", "testConfLinkRegexIndexMapping1"));
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ // Test ${1} format
+ // ^/(\w+, /targetTestRoot/${1})
+ // => /targetTestRoot/testConfLinkRegexIndexMapping2
+ dstPathStr = targetTestRoot + "${1}";
+ srcPath = new Path("/testConfLinkRegexIndexMapping2");
+ expectedResolveResult =
+ new Path(dstPathStr.replace("${1}", "testConfLinkRegexIndexMapping2"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping3/file1
+ dstPathStr = targetTestRoot + "$1";
+ srcPath = new Path("/testConfLinkRegexIndexMapping3/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping3/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
Review comment:
Good point
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-671776607
I will review it in a day or two, Thanks
BTW, you may need the similar changes in ViewFs.java as well, I think nfly also missed there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-679845408
Thanks a lot @JohnZZGithub for the update. Sorry for the delay. Last week we were on engineering holidays. Could not spend time on this.
I will review this soon, Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-690863729
+1 on the latest patch. Thanks again for your great work @JohnZZGithub
Test failures are unrelated.
I think we are 3 line crossed for a warning. Looks like it's not worth minor refactor for sake of that unless we do full refactoring of that method. Please file separate issue to refactor into much cleaner with smaller methods. I will proceed to commit the current patch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-675348714
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 29s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 47s | trunk passed |
| +1 :green_heart: | compile | 19m 28s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 50s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 46s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 57s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 19s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 16s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 15s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 27s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 59s | the patch passed |
| +1 :green_heart: | compile | 18m 50s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 18m 50s | the patch passed |
| +1 :green_heart: | compile | 16m 46s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 16m 46s | the patch passed |
| -0 :warning: | checkstyle | 2m 44s | root: The patch generated 14 new + 182 unchanged - 1 fixed = 196 total (was 183) |
| +1 :green_heart: | mvnsite | 2m 53s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 15s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 39s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 13s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 5m 37s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 42s | hadoop-common in the patch passed. |
| -1 :x: | unit | 95m 52s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. |
| | | 277m 14s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux ff8f13716174 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fefacf2578e |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/8/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/8/testReport/ |
| Max. process+thread count | 4403 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/8/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r484092172
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestRegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Test RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+public class TestRegexMountPointResolvedDstPathReplaceInterceptor {
+
+ public String createSerializedString(String regex, String replaceString) {
+ return REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + regex
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + replaceString;
+ }
+
+ @Test
+ public void testDeserializeFromStringNormalCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertTrue(interceptor.getSrcRegexString().equals(srcRegex));
+ Assert.assertTrue(interceptor.getReplaceString().equals(replaceString));
+ Assert.assertTrue(interceptor.getSrcRegexPattern() == null);
+ interceptor.initialize();
+ Assert.assertTrue(
+ interceptor.getSrcRegexPattern().toString().equals(srcRegex));
+ }
+
+ @Test
+ public void testDeserializeFromStringBadCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ serializedString = serializedString + ":ddd";
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertEquals(interceptor, null);
+ }
+
+ @Test
+ public void testSerialization() {
+ String srcRegex = "word1";
+ String replaceString = "word2";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ new RegexMountPointResolvedDstPathReplaceInterceptor(srcRegex,
+ replaceString);
+ Assert.assertEquals(interceptor.serializeToString(), serializedString);
+ }
+
+ @Test
Review comment:
No worries. Thanks for addressing. :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-675876739
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 28s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 29s | trunk passed |
| +1 :green_heart: | compile | 20m 47s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 20m 39s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 53s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 47s | trunk passed |
| +1 :green_heart: | shadedclient | 22m 16s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 33s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 2m 58s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 20s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 6m 8s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 3s | the patch passed |
| +1 :green_heart: | compile | 21m 15s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 21m 15s | the patch passed |
| +1 :green_heart: | compile | 18m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 18m 24s | the patch passed |
| -0 :warning: | checkstyle | 2m 53s | root: The patch generated 15 new + 182 unchanged - 1 fixed = 197 total (was 183) |
| +1 :green_heart: | mvnsite | 2m 46s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 34s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 25s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 0s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 5m 55s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 50s | hadoop-common in the patch passed. |
| -1 :x: | unit | 109m 58s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 53s | The patch does not generate ASF License warnings. |
| | | 306m 54s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 51b05eaeb1dd 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b65e43fe386 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/9/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/9/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/9/testReport/ |
| Max. process+thread count | 3243 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/9/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-690001235
@umamaheswararao Sure. In fact, I did a major refactor in our internal repo to make it switch case-based. (https://github.com/apache/hadoop/pull/424/files#diff-69fd14ba63365b6a428bf7142c463990R511) However, I didn't put it here as it's harder to review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r484013414
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestRegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Test RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+public class TestRegexMountPointResolvedDstPathReplaceInterceptor {
+
+ public String createSerializedString(String regex, String replaceString) {
+ return REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + regex
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + replaceString;
+ }
+
+ @Test
+ public void testDeserializeFromStringNormalCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertTrue(interceptor.getSrcRegexString().equals(srcRegex));
+ Assert.assertTrue(interceptor.getReplaceString().equals(replaceString));
+ Assert.assertTrue(interceptor.getSrcRegexPattern() == null);
+ interceptor.initialize();
+ Assert.assertTrue(
+ interceptor.getSrcRegexPattern().toString().equals(srcRegex));
+ }
+
+ @Test
+ public void testDeserializeFromStringBadCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ serializedString = serializedString + ":ddd";
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertEquals(interceptor, null);
+ }
+
+ @Test
+ public void testSerialization() {
+ String srcRegex = "word1";
+ String replaceString = "word2";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ new RegexMountPointResolvedDstPathReplaceInterceptor(srcRegex,
+ replaceString);
+ Assert.assertEquals(interceptor.serializeToString(), serializedString);
+ }
+
+ @Test
Review comment:
Ah, gotcha. I thought the comment was for testSerialization().
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r481532882
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/RegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.regex.PatternSyntaxException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.Path;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Implementation of RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+class RegexMountPointResolvedDstPathReplaceInterceptor
+ implements RegexMountPointInterceptor {
+
+ private String srcRegexString;
+ private String replaceString;
+ private Pattern srcRegexPattern;
+
+ RegexMountPointResolvedDstPathReplaceInterceptor(String srcRegex,
+ String replaceString) {
+ this.srcRegexString = srcRegex;
+ this.replaceString = replaceString;
+ this.srcRegexPattern = null;
+ }
+
+ public String getSrcRegexString() {
+ return srcRegexString;
+ }
+
+ public String getReplaceString() {
+ return replaceString;
+ }
+
+ public Pattern getSrcRegexPattern() {
+ return srcRegexPattern;
+ }
+
+ @Override
+ public void initialize() throws IOException {
+ try {
+ srcRegexPattern = Pattern.compile(srcRegexString);
+ } catch (PatternSyntaxException ex) {
+ throw new IOException(
+ "Initialize interceptor failed, srcRegx:" + srcRegexString, ex);
+ }
+ }
+
+ /**
+ * Intercept source before resolution.
+ *
+ * @param source
+ * @return
+ */
+ @Override
+ public String interceptSource(String source) {
+ return source;
+ }
+
+ /**
+ * Intercept resolved path, e.g.
+ * Mount point /^(\\w+)/, ${1}.hadoop.net
+ * If incoming path is /user1/home/tmp/job1,
+ * then the resolved path str will be user1.
+ *
+ * @return intercepted string
+ */
+ @Override public String interceptResolvedDestPathStr(
+ String parsedDestPathStr) {
+ Matcher matcher = srcRegexPattern.matcher(parsedDestPathStr);
+ return matcher.replaceAll(replaceString);
+ }
+
+ /**
+ * Intercept remaining path.
+ *
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao merged pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao merged pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-690012492
Thanks @JohnZZGithub I agree it's always better to have refactoring patch separated to make the reviews easier.
In this case, we need to do minor refactors to follow the coding rules. I would be happy to review your refactoring patch later if you would like to contribute. Thanks :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-690863729
+1 on the latest patch. Thanks again for your great work @JohnZZGithub
Test failures are unrelated.
I think we are 3 line crossed for a warning. Looks like it's not worth minor refactor for sake of that unless we do full refactoring of that method. Please file separate issue to refactor into much cleaner with smaller methods. I will proceed to commit the current patch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r485346458
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##########
@@ -217,6 +239,7 @@ public Path getMountedOnPath() {
Path homeDir = null;
private boolean enableInnerCache = false;
private InnerCache cache;
+ private boolean evictCacheOnClose = false;
Review comment:
@umamaheswararao Thanks for the kindness. I didn't see a problem with making it true. Just mean to be more cautious, let me remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-687765095
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 28m 21s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 1s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 24s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 32s | trunk passed |
| +1 :green_heart: | compile | 20m 14s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 55s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 52s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 57s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 41s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 41s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 14s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 11s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 25s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 57s | the patch passed |
| +1 :green_heart: | compile | 18m 46s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 18m 46s | the patch passed |
| +1 :green_heart: | compile | 16m 47s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 16m 47s | the patch passed |
| -0 :warning: | checkstyle | 2m 42s | root: The patch generated 5 new + 182 unchanged - 1 fixed = 187 total (was 183) |
| +1 :green_heart: | mvnsite | 2m 54s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 9s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 41s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 14s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 5m 38s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 29s | hadoop-common in the patch passed. |
| -1 :x: | unit | 99m 13s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. |
| | | 311m 6s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestGetFileChecksum |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 2a8252a0d357 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1841a5bb03f |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/15/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/15/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/15/testReport/ |
| Max. process+thread count | 4529 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/15/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-685957079
Hi @JohnZZGithub, I got few other points to discuss.
1. We have exposed getMountPoints API. It seems we can't return any mount points from REGEX based because you would not know until you got src paths to resolve and find real target fs. What should we do for this API?
2. Other API is getDelegationTokenIssuers. Applications like YARN uses this API to get all child fs delegation tokens. This also will not work for REGEX based mount points.
3. Other question is how this child filesystem objects gets closed. There was an issue with [ViewFileSystem#close | https://issues.apache.org/jira/browse/HADOOP-15565 ]. I would like to know how that get addressed in this case as don't keep anything in InnerCache.
Sorry for bringing this points late, but they just got popped up in my mind.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-667912969
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 31s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 2s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 17s | trunk passed |
| +1 :green_heart: | compile | 19m 27s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 39s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 2m 45s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 57s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 3s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 14s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 9s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 3m 8s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 18s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 58s | the patch passed |
| +1 :green_heart: | compile | 18m 40s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 18m 40s | the patch passed |
| +1 :green_heart: | compile | 16m 37s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 16m 37s | the patch passed |
| -0 :warning: | checkstyle | 2m 45s | root: The patch generated 4 new + 92 unchanged - 1 fixed = 96 total (was 93) |
| +1 :green_heart: | mvnsite | 2m 56s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 30s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 9s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 5s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 5m 55s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 26s | hadoop-common in the patch passed. |
| -1 :x: | unit | 95m 25s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 1s | The patch does not generate ASF License warnings. |
| | | 266m 55s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.fs.viewfs.TestViewFileSystemLinkRegex |
| | hadoop.hdfs.TestMultipleNNPortQOP |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f1c0e7c98309 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c40cbc57fa2 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/1/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/1/testReport/ |
| Max. process+thread count | 4396 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r471831712
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
##########
@@ -366,6 +366,82 @@ Don't want to change scheme or difficult to copy mount-table configurations to a
Please refer to the [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html)
+Regex Pattern Based Mount Points
+--------------------------------
+
+The view file system mount points were a Key-Value based mapping system. It is not friendly for user cases which mapping config could be abstracted to rules. E.g. Users want to provide a GCS bucket per user and there might be thousands of users in total. The old key-value based approach won't work well for several reasons:
+
+1. The mount table is used by FileSystem clients. There's a cost to spread the config to all clients and we should avoid it if possible. The [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html) could help the distribution by central mount table management. But the mount table still have to be updated on every change. The change could be greatly avoided if provide a rule-based mount table..
+
+2. The client have to understand all the KVs in the mount table. This is not ideal when the mountable grows to thousands of items. E.g. thousands of file systems might be initialized even users only need one. And the config itself will become bloated at scale.
+
+### Understand the Difference
+
+In the key-value based mount table, view file system treats every mount point as a partition. There's several file system APIs which will lead to operation on all partitions. E.g. there's an HDFS cluster with multiple mount. Users want to run “hadoop fs -put file viewfs://hdfs.namenode.apache.org/tmp/” cmd to copy data from local disk to our HDFS cluster. The cmd will trigger ViewFileSystem to call setVerifyChecksum() method which will initialize the file system for every mount point.
+For a regex-base rule mount table entry, we couldn't know what's corresponding path until parsing. So the regex based mount table entry will be ignored on such cases and the file system will be created upon accessing. The inner cache of ViewFs is also not available for regex-base mount points now as it assumes target file system doesn't change after viewfs initialization. Please disable it if you want to use regex-base mount table. We also need to change the rename strategy to SAME_FILESYSTEM_ACROSS_MOUNTPOINT for the same reason.
+```xml
+<property>
+ <name>fs.viewfs.enable.inner.cache</name>
+ <value>false</value>
Review comment:
Good call.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r485356795
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
##########
@@ -366,6 +366,69 @@ Don't want to change scheme or difficult to copy mount-table configurations to a
Please refer to the [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html)
+Regex Pattern Based Mount Points
+--------------------------------
+
+The view file system mount points were a Key-Value based mapping system. It is not friendly for user cases which mapping config could be abstracted to rules. E.g. Users want to provide a GCS bucket per user and there might be thousands of users in total. The old key-value based approach won't work well for several reasons:
+
+1. The mount table is used by FileSystem clients. There's a cost to spread the config to all clients and we should avoid it if possible. The [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html) could help the distribution by central mount table management. But the mount table still have to be updated on every change. The change could be greatly avoided if provide a rule-based mount table.
+
+2. The client have to understand all the KVs in the mount table. This is not ideal when the mountable grows to thousands of items. E.g. thousands of file systems might be initialized even users only need one. And the config itself will become bloated at scale.
+
+### Understand the Difference
+
+In the key-value based mount table, view file system treats every mount point as a partition. There's several file system APIs which will lead to operation on all partitions. E.g. there's an HDFS cluster with multiple mount. Users want to run “hadoop fs -put file viewfs://hdfs.namenode.apache.org/tmp/” cmd to copy data from local disk to our HDFS cluster. The cmd will trigger ViewFileSystem to call setVerifyChecksum() method which will initialize the file system for every mount point.
+For a regex-base rule mount table entry, we couldn't know what's corresponding path until parsing. So the regex based mount table entry will be ignored on such cases. The file system (ChRootedFileSystem) will be created upon accessing. But the underlying file system will be cached by inner cache of ViewFileSystem.
Review comment:
Good idea. I guess this patch didn't add parsed fs to mount point yet. Maybe it's better when we modify the code and doc at the same time. Created https://issues.apache.org/jira/browse/HADOOP-17247 to track the issue. Does it make sense? Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-689992391
I think the following new lines added into this method
```
} else if (src.startsWith(Constants.CONFIG_VIEWFS_LINK_REGEX)) {
final String target = si.getValue();
String linkKeyPath = null;
final String linkRegexPrefix = Constants.CONFIG_VIEWFS_LINK_REGEX + ".";
// settings#.linkKey
String settingsAndLinkKeyPath = src.substring(linkRegexPrefix.length());
int settingLinkKeySepIndex = settingsAndLinkKeyPath
.indexOf(RegexMountPoint.SETTING_SRCREGEX_SEP);
if (settingLinkKeySepIndex == -1) {
// There's no settings
linkKeyPath = settingsAndLinkKeyPath;
settings = null;
} else {
// settings#.linkKey style configuration
// settings from settings#.linkKey
settings =
settingsAndLinkKeyPath.substring(0, settingLinkKeySepIndex);
// linkKeyPath
linkKeyPath = settingsAndLinkKeyPath.substring(
settings.length() + RegexMountPoint.SETTING_SRCREGEX_SEP
.length());
}
linkType = LinkType.REGEX;
linkEntries.add(
new LinkEntry(linkKeyPath, target, linkType, settings, ugi,
config));
continue;
}
.......
.......
case REGEX:
LOGGER.info("Add regex mount point:" + le.getSrc()
+ ", target:" + le.getTarget()
+ ", interceptor settings:" + le.getSettings());
RegexMountPoint regexMountPoint =
new RegexMountPoint<T>(
this, le.getSrc(), le.getTarget(), le.getSettings());
regexMountPoint.initialize();
regexMountPointList.add(regexMountPoint);
continue;
```
Probably we can extract this logics to separate methods.
I understand it needs multiple values needs to get from extarcted method, they are settings, linkType etc. Probably you can create LinkEntry in method and return LinkEntry. From there you can set linkType, settings etc.
Check if they are possible to reduce lines in that method.
move the following code to a small check method. There are two places we are doing same checks.
```
if (src.length() != linkMergeSlashPrefix.length()) {
throw new IOException("ViewFs: Mount points initialization error." +
" Invalid " + Constants.CONFIG_VIEWFS_LINK_MERGE_SLASH +
" entry in config: " + src);
}
```
At end of the method we have some logics for handle no mount points code. IF we need we could extract that to handleNoMountPoints method or so.
Check if this refactoring can make checkstyle happy and code can look cleaner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao edited a comment on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao edited a comment on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-689992391
I think the following new lines added into this method. Thats the reason it is reporting longer method warning..
```
} else if (src.startsWith(Constants.CONFIG_VIEWFS_LINK_REGEX)) {
final String target = si.getValue();
String linkKeyPath = null;
final String linkRegexPrefix = Constants.CONFIG_VIEWFS_LINK_REGEX + ".";
// settings#.linkKey
String settingsAndLinkKeyPath = src.substring(linkRegexPrefix.length());
int settingLinkKeySepIndex = settingsAndLinkKeyPath
.indexOf(RegexMountPoint.SETTING_SRCREGEX_SEP);
if (settingLinkKeySepIndex == -1) {
// There's no settings
linkKeyPath = settingsAndLinkKeyPath;
settings = null;
} else {
// settings#.linkKey style configuration
// settings from settings#.linkKey
settings =
settingsAndLinkKeyPath.substring(0, settingLinkKeySepIndex);
// linkKeyPath
linkKeyPath = settingsAndLinkKeyPath.substring(
settings.length() + RegexMountPoint.SETTING_SRCREGEX_SEP
.length());
}
linkType = LinkType.REGEX;
linkEntries.add(
new LinkEntry(linkKeyPath, target, linkType, settings, ugi,
config));
continue;
}
.......
.......
case REGEX:
LOGGER.info("Add regex mount point:" + le.getSrc()
+ ", target:" + le.getTarget()
+ ", interceptor settings:" + le.getSettings());
RegexMountPoint regexMountPoint =
new RegexMountPoint<T>(
this, le.getSrc(), le.getTarget(), le.getSettings());
regexMountPoint.initialize();
regexMountPointList.add(regexMountPoint);
continue;
```
Probably we can extract this logics to separate methods.
I understand it needs multiple values needs to get from extarcted method, they are settings, linkType etc. Probably you can create LinkEntry in method and return LinkEntry. From there you can set linkType, settings etc.
Check if they are possible to reduce lines in that method.
move the following code to a small check method. There are two places we are doing same checks.
```
if (src.length() != linkMergeSlashPrefix.length()) {
throw new IOException("ViewFs: Mount points initialization error." +
" Invalid " + Constants.CONFIG_VIEWFS_LINK_MERGE_SLASH +
" entry in config: " + src);
}
```
At end of the method we have some logics for handle no mount points code. IF we need we could extract that to handleNoMountPoints method or so.
Check if this refactoring can make checkstyle happy and code can look cleaner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r485346691
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkRegex.java
##########
@@ -0,0 +1,473 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.FsConstants;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSNNTopology;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPoint.INTERCEPTOR_INTERNAL_SEP;
+import static org.junit.Assert.assertSame;
+
+/**
+ * Test linkRegex node type for view file system.
+ */
+public class TestViewFileSystemLinkRegex extends ViewFileSystemBaseTest {
+ public static final Logger LOGGER =
+ LoggerFactory.getLogger(TestViewFileSystemLinkRegex.class);
+
+ private static FileSystem fsDefault;
+ private static MiniDFSCluster cluster;
+ private static Configuration clusterConfig;
+ private static final int NAME_SPACES_COUNT = 3;
+ private static final int DATA_NODES_COUNT = 3;
+ private static final int FS_INDEX_DEFAULT = 0;
+ private static final FileSystem[] FS_HDFS = new FileSystem[NAME_SPACES_COUNT];
+ private static final String CLUSTER_NAME =
+ "TestViewFileSystemLinkRegexCluster";
+ private static final File TEST_DIR = GenericTestUtils
+ .getTestDir(TestViewFileSystemLinkRegex.class.getSimpleName());
+ private static final String TEST_BASE_PATH =
+ "/tmp/TestViewFileSystemLinkRegex";
+
+ @Override
+ protected FileSystemTestHelper createFileSystemHelper() {
+ return new FileSystemTestHelper(TEST_BASE_PATH);
+ }
+
+ @BeforeClass
+ public static void clusterSetupAtBeginning() throws IOException {
+ SupportsBlocks = true;
+ clusterConfig = ViewFileSystemTestSetup.createConfig();
+ clusterConfig.setBoolean(
+ DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY,
+ true);
+ cluster = new MiniDFSCluster.Builder(clusterConfig).nnTopology(
+ MiniDFSNNTopology.simpleFederatedTopology(NAME_SPACES_COUNT))
+ .numDataNodes(DATA_NODES_COUNT).build();
+ cluster.waitClusterUp();
+
+ for (int i = 0; i < NAME_SPACES_COUNT; i++) {
+ FS_HDFS[i] = cluster.getFileSystem(i);
+ }
+ fsDefault = FS_HDFS[FS_INDEX_DEFAULT];
+ }
+
+ @AfterClass
+ public static void clusterShutdownAtEnd() throws Exception {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+
+ @Override
+ @Before
+ public void setUp() throws Exception {
+ fsTarget = fsDefault;
+ super.setUp();
+ }
+
+ /**
+ * Override this so that we don't set the targetTestRoot to any path under the
+ * root of the FS, and so that we don't try to delete the test dir, but rather
+ * only its contents.
+ */
+ @Override
+ void initializeTargetTestRoot() throws IOException {
+ targetTestRoot = fsDefault.makeQualified(new Path("/"));
+ for (FileStatus status : fsDefault.listStatus(targetTestRoot)) {
+ fsDefault.delete(status.getPath(), true);
+ }
+ }
+
+ @Override
+ void setupMountPoints() {
+ super.setupMountPoints();
+ }
+
+ @Override
+ int getExpectedDelegationTokenCount() {
+ return 1; // all point to the same fs so 1 unique token
+ }
+
+ @Override
+ int getExpectedDelegationTokenCountWithCredentials() {
+ return 1;
+ }
+
+ public String buildReplaceInterceptorSettingString(String srcRegex,
+ String replaceString) {
+ return
+ RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + INTERCEPTOR_INTERNAL_SEP + srcRegex + INTERCEPTOR_INTERNAL_SEP
+ + replaceString;
+ }
+
+ public String linkInterceptorSettings(
+ List<String> interceptorSettingStrList) {
+ StringBuilder stringBuilder = new StringBuilder();
+ int listSize = interceptorSettingStrList.size();
+ for (int i = 0; i < listSize; ++i) {
+ stringBuilder.append(interceptorSettingStrList.get(i));
+ if (i < listSize - 1) {
+ stringBuilder.append(RegexMountPoint.INTERCEPTOR_SEP);
+ }
+ }
+ return stringBuilder.toString();
+ }
+
+ private void createDirWithChildren(
+ FileSystem fileSystem, Path dir, List<Path> childrenFiles)
+ throws IOException {
+ Assert.assertTrue(fileSystem.mkdirs(dir));
+ int index = 0;
+ for (Path childFile : childrenFiles) {
+ createFile(fileSystem, childFile, index, true);
+ }
+ }
+
+ private void createFile(
+ FileSystem fileSystem, Path file, int dataLenToWrite, boolean overwrite)
+ throws IOException {
+ FSDataOutputStream outputStream = null;
+ try {
+ outputStream = fileSystem.create(file, overwrite);
+ for (int i = 0; i < dataLenToWrite; ++i) {
+ outputStream.writeByte(i);
+ }
+ outputStream.close();
+ } finally {
+ if (outputStream != null) {
+ outputStream.close();
+ }
+ }
+ }
+
+ private void createDirWithChildren(
+ FileSystem fileSystem, Path dir, int childrenFilesCnt)
+ throws IOException {
+ List<Path> childrenFiles = new ArrayList<>(childrenFilesCnt);
+ for (int i = 0; i < childrenFilesCnt; ++i) {
+ childrenFiles.add(new Path(dir, "file" + i));
+ }
+ createDirWithChildren(fileSystem, dir, childrenFiles);
+ }
+
+ /**
+ * The function used to test regex mountpoints.
+ * @param config - get mountable config from this conf
+ * @param regexStr - the src path regex expression that applies to this config
+ * @param dstPathStr - the string of target path
+ * @param interceptorSettings - the serialized interceptor string to be
+ * applied while resolving the mapping
+ * @param dirPathBeforeMountPoint - the src path user passed in to be mapped.
+ * @param expectedResolveResult - the expected path after resolve
+ * dirPathBeforeMountPoint via regex mountpint.
+ * @param childrenFilesCnt - the child files under dirPathBeforeMountPoint to
+ * be created
+ * @throws IOException
+ * @throws URISyntaxException
+ */
+ private void testRegexMountpoint(
+ Configuration config,
+ String regexStr,
+ String dstPathStr,
+ String interceptorSettings,
+ Path dirPathBeforeMountPoint,
+ Path expectedResolveResult,
+ int childrenFilesCnt)
+ throws IOException, URISyntaxException {
+ FileSystem vfs = null;
+ try {
+ // Set up test env
+ createDirWithChildren(
+ fsTarget, expectedResolveResult, childrenFilesCnt);
+ ConfigUtil.addLinkRegex(
+ config, CLUSTER_NAME, regexStr, dstPathStr, interceptorSettings);
+
+ // Asserts
+ URI viewFsUri = new URI(
+ FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ vfs = FileSystem.get(viewFsUri, config);
Review comment:
Good call
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r485338866
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
##########
@@ -366,6 +366,69 @@ Don't want to change scheme or difficult to copy mount-table configurations to a
Please refer to the [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html)
+Regex Pattern Based Mount Points
+--------------------------------
+
+The view file system mount points were a Key-Value based mapping system. It is not friendly for user cases which mapping config could be abstracted to rules. E.g. Users want to provide a GCS bucket per user and there might be thousands of users in total. The old key-value based approach won't work well for several reasons:
+
+1. The mount table is used by FileSystem clients. There's a cost to spread the config to all clients and we should avoid it if possible. The [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html) could help the distribution by central mount table management. But the mount table still have to be updated on every change. The change could be greatly avoided if provide a rule-based mount table.
+
+2. The client have to understand all the KVs in the mount table. This is not ideal when the mountable grows to thousands of items. E.g. thousands of file systems might be initialized even users only need one. And the config itself will become bloated at scale.
+
+### Understand the Difference
+
+In the key-value based mount table, view file system treats every mount point as a partition. There's several file system APIs which will lead to operation on all partitions. E.g. there's an HDFS cluster with multiple mount. Users want to run “hadoop fs -put file viewfs://hdfs.namenode.apache.org/tmp/” cmd to copy data from local disk to our HDFS cluster. The cmd will trigger ViewFileSystem to call setVerifyChecksum() method which will initialize the file system for every mount point.
+For a regex-base rule mount table entry, we couldn't know what's corresponding path until parsing. So the regex based mount table entry will be ignored on such cases. The file system (ChRootedFileSystem) will be created upon accessing. But the underlying file system will be cached by inner cache of ViewFileSystem.
Review comment:
>For a regex-base rule mount table entry, we couldn't know what's corresponding path until parsing.
Whatever we know should be added to mountPoints? So, that getMountPoints will return known fs-es? It may be a good idea to add Java doc on API level.
I am ok to have this in followup JIRA to cover all of this aspects.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r481328912
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/Constants.java
##########
@@ -86,12 +86,21 @@
*/
String CONFIG_VIEWFS_LINK_MERGE_SLASH = "linkMergeSlash";
+ /**
+ * Config variable for specifying a regex link which uses regular expressions
+ * as source and target could use group captured in src.
+ * E.g. (^/(?<firstDir>\\w+), /prefix-${firstDir}) =>
+ * (/path1/file1 => /prefix-path1/file1)
+ */
+ String CONFIG_VIEWFS_LINK_REGEX = "linkRegex";
+
FsPermission PERMISSION_555 = new FsPermission((short) 0555);
String CONFIG_VIEWFS_RENAME_STRATEGY = "fs.viewfs.rename.strategy";
/**
* Enable ViewFileSystem to cache all children filesystems in inner cache.
Review comment:
Good cache, I should clarify it. ViewFileSystem will work if people enable inner cache. However, regex based mounts won't be cache.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r472385404
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##########
@@ -646,102 +714,222 @@ boolean isInternalDir() {
}
/**
- * Resolve the pathname p relative to root InodeDir
+ * Resolve the pathname p relative to root InodeDir.
* @param p - input path
* @param resolveLastComponent
* @return ResolveResult which allows further resolution of the remaining path
* @throws FileNotFoundException
*/
ResolveResult<T> resolve(final String p, final boolean resolveLastComponent)
throws FileNotFoundException {
- String[] path = breakIntoPathComponents(p);
- if (path.length <= 1) { // special case for when path is "/"
- T targetFs = root.isInternalDir() ?
- getRootDir().getInternalDirFs() : getRootLink().getTargetFileSystem();
- ResolveResult<T> res = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
- targetFs, root.fullPath, SlashPath);
- return res;
- }
+ ResolveResult<T> resolveResult = null;
+ resolveResult = getResolveResultFromCache(p, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
+ }
+
+ try {
+ String[] path = breakIntoPathComponents(p);
+ if (path.length <= 1) { // special case for when path is "/"
+ T targetFs = root.isInternalDir() ?
+ getRootDir().getInternalDirFs()
+ : getRootLink().getTargetFileSystem();
+ resolveResult = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
+ targetFs, root.fullPath, SlashPath);
+ return resolveResult;
+ }
- /**
- * linkMergeSlash has been configured. The root of this mount table has
- * been linked to the root directory of a file system.
- * The first non-slash path component should be name of the mount table.
- */
- if (root.isLink()) {
- Path remainingPath;
- StringBuilder remainingPathStr = new StringBuilder();
- // ignore first slash
- for (int i = 1; i < path.length; i++) {
- remainingPathStr.append("/").append(path[i]);
+ /**
+ * linkMergeSlash has been configured. The root of this mount table has
+ * been linked to the root directory of a file system.
+ * The first non-slash path component should be name of the mount table.
+ */
+ if (root.isLink()) {
+ Path remainingPath;
+ StringBuilder remainingPathStr = new StringBuilder();
+ // ignore first slash
+ for (int i = 1; i < path.length; i++) {
+ remainingPathStr.append("/").append(path[i]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ getRootLink().getTargetFileSystem(), root.fullPath, remainingPath);
+ return resolveResult;
}
- remainingPath = new Path(remainingPathStr.toString());
- ResolveResult<T> res = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- getRootLink().getTargetFileSystem(), root.fullPath, remainingPath);
- return res;
- }
- Preconditions.checkState(root.isInternalDir());
- INodeDir<T> curInode = getRootDir();
+ Preconditions.checkState(root.isInternalDir());
+ INodeDir<T> curInode = getRootDir();
- int i;
- // ignore first slash
- for (i = 1; i < path.length - (resolveLastComponent ? 0 : 1); i++) {
- INode<T> nextInode = curInode.resolveInternal(path[i]);
- if (nextInode == null) {
- if (hasFallbackLink()) {
- return new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- getRootFallbackLink().getTargetFileSystem(),
- root.fullPath, new Path(p));
- } else {
- StringBuilder failedAt = new StringBuilder(path[0]);
- for (int j = 1; j <= i; ++j) {
- failedAt.append('/').append(path[j]);
+ // Try to resolve path in the regex mount point
+ resolveResult = tryResolveInRegexMountpoint(p, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
+ }
+
+ int i;
+ // ignore first slash
+ for (i = 1; i < path.length - (resolveLastComponent ? 0 : 1); i++) {
+ INode<T> nextInode = curInode.resolveInternal(path[i]);
+ if (nextInode == null) {
+ if (hasFallbackLink()) {
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ getRootFallbackLink().getTargetFileSystem(), root.fullPath,
+ new Path(p));
+ return resolveResult;
+ } else {
+ StringBuilder failedAt = new StringBuilder(path[0]);
+ for (int j = 1; j <= i; ++j) {
+ failedAt.append('/').append(path[j]);
+ }
+ throw (new FileNotFoundException(
+ "File/Directory does not exist: " + failedAt.toString()));
}
- throw (new FileNotFoundException(
- "File/Directory does not exist: " + failedAt.toString()));
}
- }
- if (nextInode.isLink()) {
- final INodeLink<T> link = (INodeLink<T>) nextInode;
- final Path remainingPath;
- if (i >= path.length - 1) {
- remainingPath = SlashPath;
- } else {
- StringBuilder remainingPathStr = new StringBuilder("/" + path[i + 1]);
- for (int j = i + 2; j < path.length; ++j) {
- remainingPathStr.append('/').append(path[j]);
+ if (nextInode.isLink()) {
+ final INodeLink<T> link = (INodeLink<T>) nextInode;
+ final Path remainingPath;
+ if (i >= path.length - 1) {
+ remainingPath = SlashPath;
+ } else {
+ StringBuilder remainingPathStr =
+ new StringBuilder("/" + path[i + 1]);
+ for (int j = i + 2; j < path.length; ++j) {
+ remainingPathStr.append('/').append(path[j]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
}
- remainingPath = new Path(remainingPathStr.toString());
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ link.getTargetFileSystem(), nextInode.fullPath, remainingPath);
+ return resolveResult;
+ } else if (nextInode.isInternalDir()) {
+ curInode = (INodeDir<T>) nextInode;
}
- final ResolveResult<T> res =
- new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- link.getTargetFileSystem(), nextInode.fullPath, remainingPath);
- return res;
- } else if (nextInode.isInternalDir()) {
- curInode = (INodeDir<T>) nextInode;
+ }
+
+ // We have resolved to an internal dir in mount table.
+ Path remainingPath;
+ if (resolveLastComponent) {
+ remainingPath = SlashPath;
+ } else {
+ // note we have taken care of when path is "/" above
+ // for internal dirs rem-path does not start with / since the lookup
+ // that follows will do a children.get(remaningPath) and will have to
+ // strip-out the initial /
+ StringBuilder remainingPathStr = new StringBuilder("/" + path[i]);
+ for (int j = i + 1; j < path.length; ++j) {
+ remainingPathStr.append('/').append(path[j]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
+ }
+ resolveResult = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
+ curInode.getInternalDirFs(), curInode.fullPath, remainingPath);
+ return resolveResult;
+ } finally {
+ if (pathResolutionCacheCapacity > 0 && resolveResult != null) {
+ addResolveResultToCache(p, resolveLastComponent, resolveResult);
}
}
+ }
- // We have resolved to an internal dir in mount table.
- Path remainingPath;
- if (resolveLastComponent) {
- remainingPath = SlashPath;
- } else {
- // note we have taken care of when path is "/" above
- // for internal dirs rem-path does not start with / since the lookup
- // that follows will do a children.get(remaningPath) and will have to
- // strip-out the initial /
- StringBuilder remainingPathStr = new StringBuilder("/" + path[i]);
- for (int j = i + 1; j < path.length; ++j) {
- remainingPathStr.append('/').append(path[j]);
+ /**
+ * Walk through all regex mount points to see
+ * whether the path match any regex expressions.
+ *
+ * @param srcPath
+ * @param resolveLastComponent
+ * @return
+ */
+ protected ResolveResult<T> tryResolveInRegexMountpoint(final String srcPath,
+ final boolean resolveLastComponent) {
+ for (RegexMountPoint regexMountPoint : regexMountPointList) {
+ ResolveResult resolveResult =
+ regexMountPoint.resolve(srcPath, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
}
- remainingPath = new Path(remainingPathStr.toString());
}
- final ResolveResult<T> res =
- new ResolveResult<T>(ResultKind.INTERNAL_DIR,
- curInode.getInternalDirFs(), curInode.fullPath, remainingPath);
- return res;
+ return null;
+ }
+
+ /**
+ * Build resolve result return to caller.
+ *
+ * @param resultKind
+ * @param resolvedPathStr
+ * @param targetOfResolvedPathStr
+ * @param remainingPath
+ * @return
+ */
+ protected ResolveResult<T> buildResolveResultForRegexMountPoint(
+ ResultKind resultKind, String resolvedPathStr,
+ String targetOfResolvedPathStr, Path remainingPath) {
+ try {
+ T targetFs = getTargetFileSystem(new URI(targetOfResolvedPathStr));
+ return new ResolveResult<T>(resultKind, targetFs, resolvedPathStr,
+ remainingPath);
+ } catch (IOException ex) {
+ LOGGER.error(String.format(
+ "Got Exception while build resolve result."
+ + " ResultKind:%s, resolvedPathStr:%s,"
+ + " targetOfResolvedPathStr:%s, remainingPath:%s,"
+ + " will return null.",
+ resultKind, resolvedPathStr, targetOfResolvedPathStr, remainingPath),
+ ex);
+ return null;
+ } catch (URISyntaxException uex) {
+ LOGGER.error(String.format(
+ "Got Exception while build resolve result."
+ + " ResultKind:%s, resolvedPathStr:%s,"
+ + " targetOfResolvedPathStr:%s, remainingPath:%s,"
+ + " will return null.",
+ resultKind, resolvedPathStr, targetOfResolvedPathStr, remainingPath),
+ uex);
+ return null;
+ }
+ }
+
+ /**
+ * Return resolution cache capacity.
+ *
+ * @return
+ */
+ public int getPathResolutionCacheCapacity() {
+ return pathResolutionCacheCapacity;
+ }
+
Review comment:
Sure, make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-669606794
Here're the UT failed. I didn't see them related. But I could double-check. Thanks
[ERROR] Failures:
[ERROR] TestBPOfferService.testMissBlocksWhenReregister:350
[ERROR] TestDataNodeErasureCodingMetrics.testFullBlock:97->doTest:205 Wrongly computed block reconstruction work
[ERROR] TestNameNodeRetryCacheMetrics.testRetryCacheMetrics:95->checkMetrics:103 CacheHit expected:<2> but was:<0>
[ERROR] Errors:
[ERROR] TestHDFSContractMultipartUploader>AbstractContractMultipartUploaderTest.testConcurrentUploads:815 ? IllegalArgument
[ERROR] TestBlockTokenWithDFSStriped.testRead:92->TestBlockTokenWithDFS.doTestRead:508->isBlockTokenExpired:139->TestBlockTokenWithDFS.isBlockTokenExpired:633 ? NullPointer
[ERROR] TestBootstrapStandby.testSuccessfulBaseCase:130->restartNameNodesFromIndex:342 ? Bind
[ERROR] TestExternalStoragePolicySatisfier.testChooseInSameDatanodeWithONESSDShouldNotChooseIfNoSpace:1064 ? Timeout
[INFO]
[ERROR] Tests run: 6500, Failures: 3, Errors: 4, Skipped: 23
[INFO]
[ERROR] There are test failures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-690866308
@umamaheswararao Thanks a lot!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-677413520
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 26m 25s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 44s | trunk passed |
| +1 :green_heart: | compile | 19m 25s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 17m 6s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 56s | trunk passed |
| +1 :green_heart: | mvnsite | 2m 49s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 1s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 38s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 6s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 57s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 6m 45s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 22s | the patch passed |
| +1 :green_heart: | compile | 24m 54s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 24m 54s | the patch passed |
| +1 :green_heart: | compile | 21m 58s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 21m 58s | the patch passed |
| -0 :warning: | checkstyle | 3m 37s | root: The patch generated 1 new + 182 unchanged - 1 fixed = 183 total (was 183) |
| +1 :green_heart: | mvnsite | 3m 23s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 13s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 35s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 7m 10s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 53s | hadoop-common in the patch passed. |
| -1 :x: | unit | 111m 10s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 7s | The patch does not generate ASF License warnings. |
| | | 338m 11s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestDFSInotifyEventInputStream |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot |
| | hadoop.hdfs.TestReadStripedFileWithDNFailure |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.TestDatanodeDeath |
| | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.TestStripedFileAppend |
| | hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.TestDFSStorageStateRecovery |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 9aaa4225ec86 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9b9f7ea16a2 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/11/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/11/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/11/testReport/ |
| Max. process+thread count | 4141 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/11/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-670285487
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 20s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 49s | trunk passed |
| +1 :green_heart: | compile | 21m 52s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 18m 47s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 2m 44s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 12s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 14s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 11s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 12s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 3m 20s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 39s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 9s | the patch passed |
| +1 :green_heart: | compile | 21m 12s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 21m 12s | the patch passed |
| +1 :green_heart: | compile | 19m 7s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 19m 7s | the patch passed |
| -0 :warning: | checkstyle | 3m 20s | root: The patch generated 1 new + 92 unchanged - 1 fixed = 93 total (was 93) |
| +1 :green_heart: | mvnsite | 3m 11s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 54s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 14s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 21s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 6m 4s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 58s | hadoop-common in the patch passed. |
| -1 :x: | unit | 95m 7s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. |
| | | 287m 8s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestMultipleNNPortQOP |
| | hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux 0367d0a0fe35 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 1d5ccc790bd |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/5/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/5/testReport/ |
| Max. process+thread count | 4090 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/5/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-685399208
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 28m 35s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 26s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 2s | trunk passed |
| +1 :green_heart: | compile | 19m 34s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 57s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 45s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 0s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 44s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 11s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 11s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 24s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 59s | the patch passed |
| +1 :green_heart: | compile | 18m 51s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 18m 51s | the patch passed |
| +1 :green_heart: | compile | 16m 52s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 16m 52s | the patch passed |
| -0 :warning: | checkstyle | 2m 43s | root: The patch generated 2 new + 182 unchanged - 1 fixed = 184 total (was 183) |
| +1 :green_heart: | mvnsite | 2m 57s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 16s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 40s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 17s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 5m 34s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 45s | hadoop-common in the patch passed. |
| -1 :x: | unit | 98m 27s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 9s | The patch does not generate ASF License warnings. |
| | | 309m 3s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux ffcb3de8fafd 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c7dab2b23e4 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/12/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/12/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/12/testReport/ |
| Max. process+thread count | 4253 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/12/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-687743536
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 28m 21s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 22s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 7s | trunk passed |
| +1 :green_heart: | compile | 19m 33s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 47s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 50s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 0s | trunk passed |
| +1 :green_heart: | shadedclient | 20m 27s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 43s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 9s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 5m 21s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 57s | the patch passed |
| +1 :green_heart: | compile | 18m 54s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 18m 54s | the patch passed |
| +1 :green_heart: | compile | 17m 20s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 17m 20s | the patch passed |
| -0 :warning: | checkstyle | 2m 43s | root: The patch generated 2 new + 182 unchanged - 1 fixed = 184 total (was 183) |
| +1 :green_heart: | mvnsite | 2m 55s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 21s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 41s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 14s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 5m 38s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 45s | hadoop-common in the patch passed. |
| -1 :x: | unit | 97m 55s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 7s | The patch does not generate ASF License warnings. |
| | | 308m 22s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2185 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux bdb50cea41e2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1841a5bb03f |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/14/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/14/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/14/testReport/ |
| Max. process+thread count | 4314 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2185/14/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-685028507
@umamaheswararao Sure, thanks a ton!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
liuml07 commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-668274867
This seems an interesting feature. Do we need to update the `ViewFs.md` user guide?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-684493223
@JohnZZGithub Thanks a lot for the update. Changes looks great. I added few more nits/minors to address before we commit. ( sorry for coming late on review )
Could you also file JIRAs for porting into FileContext and cache improvement?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-680179907
@umamaheswararao No worries, thanks a lot!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] umamaheswararao commented on a change in pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#discussion_r471268265
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java
##########
@@ -166,6 +166,41 @@ public static void addLinkNfly(final Configuration conf, final String src,
addLinkNfly(conf, getDefaultMountTableName(conf), src, null, targets);
}
+
+ /**
+ * Add a LinkRegex to the config for the specified mount table.
Review comment:
Below params javadoc does not help anything I think. If you want to add, please add description( most of that params seems self explanatory ), otherwise just remove params part.
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##########
@@ -646,102 +714,222 @@ boolean isInternalDir() {
}
/**
- * Resolve the pathname p relative to root InodeDir
+ * Resolve the pathname p relative to root InodeDir.
* @param p - input path
* @param resolveLastComponent
* @return ResolveResult which allows further resolution of the remaining path
* @throws FileNotFoundException
*/
ResolveResult<T> resolve(final String p, final boolean resolveLastComponent)
throws FileNotFoundException {
- String[] path = breakIntoPathComponents(p);
- if (path.length <= 1) { // special case for when path is "/"
- T targetFs = root.isInternalDir() ?
- getRootDir().getInternalDirFs() : getRootLink().getTargetFileSystem();
- ResolveResult<T> res = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
- targetFs, root.fullPath, SlashPath);
- return res;
- }
+ ResolveResult<T> resolveResult = null;
+ resolveResult = getResolveResultFromCache(p, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
+ }
+
+ try {
+ String[] path = breakIntoPathComponents(p);
+ if (path.length <= 1) { // special case for when path is "/"
+ T targetFs = root.isInternalDir() ?
+ getRootDir().getInternalDirFs()
+ : getRootLink().getTargetFileSystem();
+ resolveResult = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
+ targetFs, root.fullPath, SlashPath);
+ return resolveResult;
+ }
- /**
- * linkMergeSlash has been configured. The root of this mount table has
- * been linked to the root directory of a file system.
- * The first non-slash path component should be name of the mount table.
- */
- if (root.isLink()) {
- Path remainingPath;
- StringBuilder remainingPathStr = new StringBuilder();
- // ignore first slash
- for (int i = 1; i < path.length; i++) {
- remainingPathStr.append("/").append(path[i]);
+ /**
+ * linkMergeSlash has been configured. The root of this mount table has
+ * been linked to the root directory of a file system.
+ * The first non-slash path component should be name of the mount table.
+ */
+ if (root.isLink()) {
+ Path remainingPath;
+ StringBuilder remainingPathStr = new StringBuilder();
+ // ignore first slash
+ for (int i = 1; i < path.length; i++) {
+ remainingPathStr.append("/").append(path[i]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ getRootLink().getTargetFileSystem(), root.fullPath, remainingPath);
+ return resolveResult;
}
- remainingPath = new Path(remainingPathStr.toString());
- ResolveResult<T> res = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- getRootLink().getTargetFileSystem(), root.fullPath, remainingPath);
- return res;
- }
- Preconditions.checkState(root.isInternalDir());
- INodeDir<T> curInode = getRootDir();
+ Preconditions.checkState(root.isInternalDir());
+ INodeDir<T> curInode = getRootDir();
- int i;
- // ignore first slash
- for (i = 1; i < path.length - (resolveLastComponent ? 0 : 1); i++) {
- INode<T> nextInode = curInode.resolveInternal(path[i]);
- if (nextInode == null) {
- if (hasFallbackLink()) {
- return new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- getRootFallbackLink().getTargetFileSystem(),
- root.fullPath, new Path(p));
- } else {
- StringBuilder failedAt = new StringBuilder(path[0]);
- for (int j = 1; j <= i; ++j) {
- failedAt.append('/').append(path[j]);
+ // Try to resolve path in the regex mount point
+ resolveResult = tryResolveInRegexMountpoint(p, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
+ }
+
+ int i;
+ // ignore first slash
+ for (i = 1; i < path.length - (resolveLastComponent ? 0 : 1); i++) {
+ INode<T> nextInode = curInode.resolveInternal(path[i]);
+ if (nextInode == null) {
+ if (hasFallbackLink()) {
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ getRootFallbackLink().getTargetFileSystem(), root.fullPath,
+ new Path(p));
+ return resolveResult;
+ } else {
+ StringBuilder failedAt = new StringBuilder(path[0]);
+ for (int j = 1; j <= i; ++j) {
+ failedAt.append('/').append(path[j]);
+ }
+ throw (new FileNotFoundException(
+ "File/Directory does not exist: " + failedAt.toString()));
}
- throw (new FileNotFoundException(
- "File/Directory does not exist: " + failedAt.toString()));
}
- }
- if (nextInode.isLink()) {
- final INodeLink<T> link = (INodeLink<T>) nextInode;
- final Path remainingPath;
- if (i >= path.length - 1) {
- remainingPath = SlashPath;
- } else {
- StringBuilder remainingPathStr = new StringBuilder("/" + path[i + 1]);
- for (int j = i + 2; j < path.length; ++j) {
- remainingPathStr.append('/').append(path[j]);
+ if (nextInode.isLink()) {
+ final INodeLink<T> link = (INodeLink<T>) nextInode;
+ final Path remainingPath;
+ if (i >= path.length - 1) {
+ remainingPath = SlashPath;
+ } else {
+ StringBuilder remainingPathStr =
+ new StringBuilder("/" + path[i + 1]);
+ for (int j = i + 2; j < path.length; ++j) {
+ remainingPathStr.append('/').append(path[j]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
}
- remainingPath = new Path(remainingPathStr.toString());
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ link.getTargetFileSystem(), nextInode.fullPath, remainingPath);
+ return resolveResult;
+ } else if (nextInode.isInternalDir()) {
+ curInode = (INodeDir<T>) nextInode;
}
- final ResolveResult<T> res =
- new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- link.getTargetFileSystem(), nextInode.fullPath, remainingPath);
- return res;
- } else if (nextInode.isInternalDir()) {
- curInode = (INodeDir<T>) nextInode;
+ }
+
+ // We have resolved to an internal dir in mount table.
+ Path remainingPath;
+ if (resolveLastComponent) {
+ remainingPath = SlashPath;
+ } else {
+ // note we have taken care of when path is "/" above
+ // for internal dirs rem-path does not start with / since the lookup
+ // that follows will do a children.get(remaningPath) and will have to
+ // strip-out the initial /
+ StringBuilder remainingPathStr = new StringBuilder("/" + path[i]);
+ for (int j = i + 1; j < path.length; ++j) {
+ remainingPathStr.append('/').append(path[j]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
+ }
+ resolveResult = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
+ curInode.getInternalDirFs(), curInode.fullPath, remainingPath);
+ return resolveResult;
+ } finally {
+ if (pathResolutionCacheCapacity > 0 && resolveResult != null) {
+ addResolveResultToCache(p, resolveLastComponent, resolveResult);
}
}
+ }
- // We have resolved to an internal dir in mount table.
- Path remainingPath;
- if (resolveLastComponent) {
- remainingPath = SlashPath;
- } else {
- // note we have taken care of when path is "/" above
- // for internal dirs rem-path does not start with / since the lookup
- // that follows will do a children.get(remaningPath) and will have to
- // strip-out the initial /
- StringBuilder remainingPathStr = new StringBuilder("/" + path[i]);
- for (int j = i + 1; j < path.length; ++j) {
- remainingPathStr.append('/').append(path[j]);
+ /**
+ * Walk through all regex mount points to see
+ * whether the path match any regex expressions.
+ *
+ * @param srcPath
+ * @param resolveLastComponent
+ * @return
+ */
+ protected ResolveResult<T> tryResolveInRegexMountpoint(final String srcPath,
+ final boolean resolveLastComponent) {
+ for (RegexMountPoint regexMountPoint : regexMountPointList) {
+ ResolveResult resolveResult =
+ regexMountPoint.resolve(srcPath, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
}
- remainingPath = new Path(remainingPathStr.toString());
}
- final ResolveResult<T> res =
- new ResolveResult<T>(ResultKind.INTERNAL_DIR,
- curInode.getInternalDirFs(), curInode.fullPath, remainingPath);
- return res;
+ return null;
+ }
+
+ /**
+ * Build resolve result return to caller.
+ *
+ * @param resultKind
+ * @param resolvedPathStr
+ * @param targetOfResolvedPathStr
+ * @param remainingPath
+ * @return
+ */
+ protected ResolveResult<T> buildResolveResultForRegexMountPoint(
+ ResultKind resultKind, String resolvedPathStr,
+ String targetOfResolvedPathStr, Path remainingPath) {
+ try {
+ T targetFs = getTargetFileSystem(new URI(targetOfResolvedPathStr));
+ return new ResolveResult<T>(resultKind, targetFs, resolvedPathStr,
+ remainingPath);
+ } catch (IOException ex) {
+ LOGGER.error(String.format(
+ "Got Exception while build resolve result."
+ + " ResultKind:%s, resolvedPathStr:%s,"
+ + " targetOfResolvedPathStr:%s, remainingPath:%s,"
+ + " will return null.",
+ resultKind, resolvedPathStr, targetOfResolvedPathStr, remainingPath),
+ ex);
+ return null;
+ } catch (URISyntaxException uex) {
+ LOGGER.error(String.format(
+ "Got Exception while build resolve result."
+ + " ResultKind:%s, resolvedPathStr:%s,"
+ + " targetOfResolvedPathStr:%s, remainingPath:%s,"
+ + " will return null.",
+ resultKind, resolvedPathStr, targetOfResolvedPathStr, remainingPath),
+ uex);
+ return null;
+ }
+ }
+
+ /**
+ * Return resolution cache capacity.
+ *
+ * @return
+ */
+ public int getPathResolutionCacheCapacity() {
+ return pathResolutionCacheCapacity;
+ }
+
+ private void addResolveResultToCache(final String pathStr,
+ final Boolean resolveLastComponent,
+ final ResolveResult<T> resolveResult) {
+ try {
+ cacheRWLock.writeLock().lock();
+ String key = getResolveCacheKeyStr(pathStr, resolveLastComponent);
+ pathResolutionCache.put(key, resolveResult);
+ } finally {
+ cacheRWLock.writeLock().unlock();
+ }
+ }
+
+ private ResolveResult<T> getResolveResultFromCache(final String pathStr,
+ final Boolean resolveLastComponent) {
+ if (pathResolutionCacheCapacity <= 0) {
+ return null;
+ }
+ try {
+ cacheRWLock.readLock().lock();
Review comment:
same as above.
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##########
@@ -226,7 +239,14 @@ void addLink(final String pathComponent, final INodeLink<T> link)
* Config prefix: fs.viewfs.mounttable.<mnt_tbl_name>.linkNfly
* Refer: {@link Constants#CONFIG_VIEWFS_LINK_NFLY}
*/
- NFLY;
+ NFLY,
+ /**
+ * Link entry which source are regex exrepssions and target refer matched
+ * group from source
+ * Config prefix: fs.viewfs.mounttable.<mnt_tbl_name>.linkMerge
Review comment:
you mean fs.viewfs.mounttable.<mnt_tbl_name>.linkRegex ?
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkRegex.java
##########
@@ -0,0 +1,320 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.FsConstants;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSNNTopology;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE;
+import static org.apache.hadoop.fs.viewfs.RegexMountPoint.INTERCEPTOR_INTERNAL_SEP;
+
+/**
+ * Test linkRegex node type for view file system.
+ */
+public class TestViewFileSystemLinkRegex extends ViewFileSystemBaseTest {
+ public static final Logger LOGGER =
+ LoggerFactory.getLogger(TestViewFileSystemLinkRegex.class);
+
+ private static FileSystem fsDefault;
+ private static MiniDFSCluster cluster;
+ private static Configuration clusterConfig;
+ private static final int NAME_SPACES_COUNT = 3;
+ private static final int DATA_NODES_COUNT = 3;
+ private static final int FS_INDEX_DEFAULT = 0;
+ private static final FileSystem[] FS_HDFS = new FileSystem[NAME_SPACES_COUNT];
+ private static final String CLUSTER_NAME =
+ "TestViewFileSystemLinkRegexCluster";
+ private static final File TEST_DIR = GenericTestUtils
+ .getTestDir(TestViewFileSystemLinkRegex.class.getSimpleName());
+ private static final String TEST_BASE_PATH =
+ "/tmp/TestViewFileSystemLinkRegex";
+
+ @Override
+ protected FileSystemTestHelper createFileSystemHelper() {
+ return new FileSystemTestHelper(TEST_BASE_PATH);
+ }
+
+ @BeforeClass
+ public static void clusterSetupAtBeginning() throws IOException {
+ SupportsBlocks = true;
+ clusterConfig = ViewFileSystemTestSetup.createConfig();
+ clusterConfig.setBoolean(
+ DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY,
+ true);
+ cluster = new MiniDFSCluster.Builder(clusterConfig).nnTopology(
+ MiniDFSNNTopology.simpleFederatedTopology(NAME_SPACES_COUNT))
+ .numDataNodes(DATA_NODES_COUNT).build();
+ cluster.waitClusterUp();
+
+ for (int i = 0; i < NAME_SPACES_COUNT; i++) {
+ FS_HDFS[i] = cluster.getFileSystem(i);
+ }
+ fsDefault = FS_HDFS[FS_INDEX_DEFAULT];
+ }
+
+ @AfterClass
+ public static void clusterShutdownAtEnd() throws Exception {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+
+ @Override
+ @Before
+ public void setUp() throws Exception {
+ fsTarget = fsDefault;
+ super.setUp();
+ }
+
+ /**
+ * Override this so that we don't set the targetTestRoot to any path under the
+ * root of the FS, and so that we don't try to delete the test dir, but rather
+ * only its contents.
+ */
+ @Override
+ void initializeTargetTestRoot() throws IOException {
+ targetTestRoot = fsDefault.makeQualified(new Path("/"));
+ for (FileStatus status : fsDefault.listStatus(targetTestRoot)) {
+ fsDefault.delete(status.getPath(), true);
+ }
+ }
+
+ @Override
+ void setupMountPoints() {
+ super.setupMountPoints();
+ }
+
+ @Override
+ int getExpectedDelegationTokenCount() {
+ return 1; // all point to the same fs so 1 unique token
+ }
+
+ @Override
+ int getExpectedDelegationTokenCountWithCredentials() {
+ return 1;
+ }
+
+ public String buildReplaceInterceptorSettingString(String srcRegex,
+ String replaceString) {
+ return
+ RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + INTERCEPTOR_INTERNAL_SEP + srcRegex + INTERCEPTOR_INTERNAL_SEP
+ + replaceString;
+ }
+
+ public String linkInterceptorSettings(
+ List<String> interceptorSettingStrList) {
+ StringBuilder stringBuilder = new StringBuilder();
+ int listSize = interceptorSettingStrList.size();
+ for (int i = 0; i < listSize; ++i) {
+ stringBuilder.append(interceptorSettingStrList.get(i));
+ if (i < listSize - 1) {
+ stringBuilder.append(RegexMountPoint.INTERCEPTOR_SEP);
+ }
+ }
+ return stringBuilder.toString();
+ }
+
+ @Test
+ public void testConfLinkRegexIndexMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ // (^/(\w+),/targetTestRoot/$1)
+ // => /targetTestRoot/testConfLinkRegexIndexMapping1
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/(\\w+)";
+ String dstPathStr = targetTestRoot + "$1";
+ Path srcPath = new Path("/testConfLinkRegexIndexMapping1");
+ Path expectedResolveResult =
+ new Path(dstPathStr.replace("$1", "testConfLinkRegexIndexMapping1"));
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ // Test ${1} format
+ // ^/(\w+, /targetTestRoot/${1})
+ // => /targetTestRoot/testConfLinkRegexIndexMapping2
+ dstPathStr = targetTestRoot + "${1}";
+ srcPath = new Path("/testConfLinkRegexIndexMapping2");
+ expectedResolveResult =
+ new Path(dstPathStr.replace("${1}", "testConfLinkRegexIndexMapping2"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping3/file1
+ dstPathStr = targetTestRoot + "$1";
+ srcPath = new Path("/testConfLinkRegexIndexMapping3/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping3/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1/)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping4/file1
+ dstPathStr = targetTestRoot + "$1/";
+ srcPath = new Path("/testConfLinkRegexIndexMapping4/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping4/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+ }
+
+ @Test
+ public void testConfLinkRegexNamedGroupMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ // ^/(?<firstDir>\\w+) = > /targetTestRoot/$firstDir
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/(?<firstDir>\\w+)";
+ String dstPathStr = targetTestRoot + "$firstDir";
+ Path srcPath = new Path("/testConfLinkRegexNamedGroupMapping1");
+ Path expectedResolveResult = new Path(
+ dstPathStr.replace("$firstDir", "testConfLinkRegexNamedGroupMapping1"));
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
Review comment:
same as above...please take care of other similar places.
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
##########
@@ -1430,4 +1432,49 @@ public void testGetContentSummaryWithFileInLocalFS() throws Exception {
summaryAfter.getLength());
}
}
+
+ @Test
+ public void testMountPointCache() throws Exception {
+ conf.setInt(Constants.CONFIG_VIEWFS_PATH_RESOLUTION_CACHE_CAPACITY, 1);
+ conf.setBoolean("fs.viewfs.impl.disable.cache", true);
Review comment:
Not about this test, In general on inner cache disabling: There was a suggestion for avoiding explicitly user disabling it. If that not possible, you may want to have check while analyzing mount points itself that, if cache is enabled and using regex mounts, then you may want to fail fs initialization itself?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##########
@@ -646,102 +714,222 @@ boolean isInternalDir() {
}
/**
- * Resolve the pathname p relative to root InodeDir
+ * Resolve the pathname p relative to root InodeDir.
* @param p - input path
* @param resolveLastComponent
* @return ResolveResult which allows further resolution of the remaining path
* @throws FileNotFoundException
*/
ResolveResult<T> resolve(final String p, final boolean resolveLastComponent)
throws FileNotFoundException {
- String[] path = breakIntoPathComponents(p);
- if (path.length <= 1) { // special case for when path is "/"
- T targetFs = root.isInternalDir() ?
- getRootDir().getInternalDirFs() : getRootLink().getTargetFileSystem();
- ResolveResult<T> res = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
- targetFs, root.fullPath, SlashPath);
- return res;
- }
+ ResolveResult<T> resolveResult = null;
+ resolveResult = getResolveResultFromCache(p, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
+ }
+
+ try {
+ String[] path = breakIntoPathComponents(p);
+ if (path.length <= 1) { // special case for when path is "/"
+ T targetFs = root.isInternalDir() ?
+ getRootDir().getInternalDirFs()
+ : getRootLink().getTargetFileSystem();
+ resolveResult = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
+ targetFs, root.fullPath, SlashPath);
+ return resolveResult;
+ }
- /**
- * linkMergeSlash has been configured. The root of this mount table has
- * been linked to the root directory of a file system.
- * The first non-slash path component should be name of the mount table.
- */
- if (root.isLink()) {
- Path remainingPath;
- StringBuilder remainingPathStr = new StringBuilder();
- // ignore first slash
- for (int i = 1; i < path.length; i++) {
- remainingPathStr.append("/").append(path[i]);
+ /**
+ * linkMergeSlash has been configured. The root of this mount table has
+ * been linked to the root directory of a file system.
+ * The first non-slash path component should be name of the mount table.
+ */
+ if (root.isLink()) {
+ Path remainingPath;
+ StringBuilder remainingPathStr = new StringBuilder();
+ // ignore first slash
+ for (int i = 1; i < path.length; i++) {
+ remainingPathStr.append("/").append(path[i]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ getRootLink().getTargetFileSystem(), root.fullPath, remainingPath);
+ return resolveResult;
}
- remainingPath = new Path(remainingPathStr.toString());
- ResolveResult<T> res = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- getRootLink().getTargetFileSystem(), root.fullPath, remainingPath);
- return res;
- }
- Preconditions.checkState(root.isInternalDir());
- INodeDir<T> curInode = getRootDir();
+ Preconditions.checkState(root.isInternalDir());
+ INodeDir<T> curInode = getRootDir();
- int i;
- // ignore first slash
- for (i = 1; i < path.length - (resolveLastComponent ? 0 : 1); i++) {
- INode<T> nextInode = curInode.resolveInternal(path[i]);
- if (nextInode == null) {
- if (hasFallbackLink()) {
- return new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- getRootFallbackLink().getTargetFileSystem(),
- root.fullPath, new Path(p));
- } else {
- StringBuilder failedAt = new StringBuilder(path[0]);
- for (int j = 1; j <= i; ++j) {
- failedAt.append('/').append(path[j]);
+ // Try to resolve path in the regex mount point
+ resolveResult = tryResolveInRegexMountpoint(p, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
+ }
+
+ int i;
+ // ignore first slash
+ for (i = 1; i < path.length - (resolveLastComponent ? 0 : 1); i++) {
+ INode<T> nextInode = curInode.resolveInternal(path[i]);
+ if (nextInode == null) {
+ if (hasFallbackLink()) {
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ getRootFallbackLink().getTargetFileSystem(), root.fullPath,
+ new Path(p));
+ return resolveResult;
+ } else {
+ StringBuilder failedAt = new StringBuilder(path[0]);
+ for (int j = 1; j <= i; ++j) {
+ failedAt.append('/').append(path[j]);
+ }
+ throw (new FileNotFoundException(
+ "File/Directory does not exist: " + failedAt.toString()));
}
- throw (new FileNotFoundException(
- "File/Directory does not exist: " + failedAt.toString()));
}
- }
- if (nextInode.isLink()) {
- final INodeLink<T> link = (INodeLink<T>) nextInode;
- final Path remainingPath;
- if (i >= path.length - 1) {
- remainingPath = SlashPath;
- } else {
- StringBuilder remainingPathStr = new StringBuilder("/" + path[i + 1]);
- for (int j = i + 2; j < path.length; ++j) {
- remainingPathStr.append('/').append(path[j]);
+ if (nextInode.isLink()) {
+ final INodeLink<T> link = (INodeLink<T>) nextInode;
+ final Path remainingPath;
+ if (i >= path.length - 1) {
+ remainingPath = SlashPath;
+ } else {
+ StringBuilder remainingPathStr =
+ new StringBuilder("/" + path[i + 1]);
+ for (int j = i + 2; j < path.length; ++j) {
+ remainingPathStr.append('/').append(path[j]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
}
- remainingPath = new Path(remainingPathStr.toString());
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ link.getTargetFileSystem(), nextInode.fullPath, remainingPath);
+ return resolveResult;
+ } else if (nextInode.isInternalDir()) {
+ curInode = (INodeDir<T>) nextInode;
}
- final ResolveResult<T> res =
- new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- link.getTargetFileSystem(), nextInode.fullPath, remainingPath);
- return res;
- } else if (nextInode.isInternalDir()) {
- curInode = (INodeDir<T>) nextInode;
+ }
+
+ // We have resolved to an internal dir in mount table.
+ Path remainingPath;
+ if (resolveLastComponent) {
+ remainingPath = SlashPath;
+ } else {
+ // note we have taken care of when path is "/" above
+ // for internal dirs rem-path does not start with / since the lookup
+ // that follows will do a children.get(remaningPath) and will have to
+ // strip-out the initial /
+ StringBuilder remainingPathStr = new StringBuilder("/" + path[i]);
+ for (int j = i + 1; j < path.length; ++j) {
+ remainingPathStr.append('/').append(path[j]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
+ }
+ resolveResult = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
+ curInode.getInternalDirFs(), curInode.fullPath, remainingPath);
+ return resolveResult;
+ } finally {
+ if (pathResolutionCacheCapacity > 0 && resolveResult != null) {
+ addResolveResultToCache(p, resolveLastComponent, resolveResult);
}
}
+ }
- // We have resolved to an internal dir in mount table.
- Path remainingPath;
- if (resolveLastComponent) {
- remainingPath = SlashPath;
- } else {
- // note we have taken care of when path is "/" above
- // for internal dirs rem-path does not start with / since the lookup
- // that follows will do a children.get(remaningPath) and will have to
- // strip-out the initial /
- StringBuilder remainingPathStr = new StringBuilder("/" + path[i]);
- for (int j = i + 1; j < path.length; ++j) {
- remainingPathStr.append('/').append(path[j]);
+ /**
+ * Walk through all regex mount points to see
+ * whether the path match any regex expressions.
+ *
+ * @param srcPath
+ * @param resolveLastComponent
+ * @return
+ */
+ protected ResolveResult<T> tryResolveInRegexMountpoint(final String srcPath,
+ final boolean resolveLastComponent) {
+ for (RegexMountPoint regexMountPoint : regexMountPointList) {
+ ResolveResult resolveResult =
+ regexMountPoint.resolve(srcPath, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
}
- remainingPath = new Path(remainingPathStr.toString());
}
- final ResolveResult<T> res =
- new ResolveResult<T>(ResultKind.INTERNAL_DIR,
- curInode.getInternalDirFs(), curInode.fullPath, remainingPath);
- return res;
+ return null;
+ }
+
+ /**
+ * Build resolve result return to caller.
+ *
+ * @param resultKind
+ * @param resolvedPathStr
+ * @param targetOfResolvedPathStr
+ * @param remainingPath
+ * @return
+ */
+ protected ResolveResult<T> buildResolveResultForRegexMountPoint(
+ ResultKind resultKind, String resolvedPathStr,
+ String targetOfResolvedPathStr, Path remainingPath) {
+ try {
+ T targetFs = getTargetFileSystem(new URI(targetOfResolvedPathStr));
+ return new ResolveResult<T>(resultKind, targetFs, resolvedPathStr,
+ remainingPath);
+ } catch (IOException ex) {
+ LOGGER.error(String.format(
+ "Got Exception while build resolve result."
+ + " ResultKind:%s, resolvedPathStr:%s,"
+ + " targetOfResolvedPathStr:%s, remainingPath:%s,"
+ + " will return null.",
+ resultKind, resolvedPathStr, targetOfResolvedPathStr, remainingPath),
+ ex);
+ return null;
+ } catch (URISyntaxException uex) {
+ LOGGER.error(String.format(
+ "Got Exception while build resolve result."
+ + " ResultKind:%s, resolvedPathStr:%s,"
+ + " targetOfResolvedPathStr:%s, remainingPath:%s,"
+ + " will return null.",
+ resultKind, resolvedPathStr, targetOfResolvedPathStr, remainingPath),
+ uex);
+ return null;
+ }
+ }
+
+ /**
+ * Return resolution cache capacity.
+ *
+ * @return
+ */
+ public int getPathResolutionCacheCapacity() {
+ return pathResolutionCacheCapacity;
+ }
+
Review comment:
Since this patch already big and this cache improvement seems to be generic for all cases, can we file separate Sub JIRA for this cache part?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/RegexMountPoint.java
##########
@@ -0,0 +1,244 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.regex.PatternSyntaxException;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.util.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.viewfs.InodeTree.SlashPath;
+
+/**
+ * Regex mount point is build to implement regex based mount point.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+class RegexMountPoint<T> {
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(RegexMountPoint.class.getName());
+
+ private InodeTree inodeTree;
+ private String srcPathRegex;
+ private Pattern srcPattern;
+ private String dstPath;
+ private String interceptorSettingsString;
+ private List<RegexMountPointInterceptor> interceptorList;
+
+ public static final String SETTING_SRCREGEX_SEP = "#.";
+ public static final char INTERCEPTOR_SEP = ';';
+ public static final char INTERCEPTOR_INTERNAL_SEP = ':';
+ // ${var},$var
+ public static final Pattern VAR_PATTERN_IN_DEST =
+ Pattern.compile("\\$((\\{\\w+\\})|(\\w+))");
+
+ // key => $key or key = > ${key}
+ private Map<String, Set<String>> varInDestPathMap;
+
+ public Map<String, Set<String>> getVarInDestPathMap() {
+ return varInDestPathMap;
+ }
+
+ RegexMountPoint(InodeTree inodeTree, String sourcePathRegex,
+ String destPath, String settingsStr) {
+ this.inodeTree = inodeTree;
+ this.srcPathRegex = sourcePathRegex;
+ this.dstPath = destPath;
+ this.interceptorSettingsString = settingsStr;
+ this.interceptorList = new ArrayList<>();
+ }
+
+ /**
+ * Initialize regex mount point.
+ *
+ * @throws IOException
+ */
+ public void initialize() throws IOException {
+ try {
+ srcPattern = Pattern.compile(srcPathRegex);
+ } catch (PatternSyntaxException ex) {
+ throw new IOException(
+ "Failed to initialized mount point due to bad src path regex:"
+ + srcPathRegex + ", dstPath:" + dstPath, ex);
+ }
+ varInDestPathMap = getVarListInString(dstPath);
+ initializeInterceptors();
+ }
+
+ private void initializeInterceptors() throws IOException {
+ if (interceptorSettingsString == null
+ || interceptorSettingsString.isEmpty()) {
+ return;
+ }
+ String[] interceptorStrArray =
+ StringUtils.split(interceptorSettingsString, INTERCEPTOR_SEP);
+ for (String interceptorStr : interceptorStrArray) {
+ RegexMountPointInterceptor interceptor =
+ RegexMountPointInterceptorFactory.create(interceptorStr);
+ if (interceptor == null) {
+ throw new IOException(
+ "Illegal settings String " + interceptorSettingsString);
+ }
+ interceptor.initialize();
+ interceptorList.add(interceptor);
+ }
+ }
+
+ /**
+ * Get $var1 and $var2 style variables in string.
+ *
+ * @param input
+ * @return
+ */
+ public static Map<String, Set<String>> getVarListInString(String input) {
+ Map<String, Set<String>> varMap = new HashMap<>();
+ Matcher matcher = VAR_PATTERN_IN_DEST.matcher(input);
+ while (matcher.find()) {
+ // $var or ${var}
+ String varName = matcher.group(0);
+ // var or {var}
+ String strippedVarName = matcher.group(1);
+ if (strippedVarName.startsWith("{")) {
+ // {varName} = > varName
+ strippedVarName =
+ strippedVarName.substring(1, strippedVarName.length() - 1);
+ }
+ varMap.putIfAbsent(strippedVarName, new HashSet<>());
+ varMap.get(strippedVarName).add(varName);
+ }
+ return varMap;
+ }
+
+ public String getSrcPathRegex() {
+ return srcPathRegex;
+ }
+
+ public Pattern getSrcPattern() {
+ return srcPattern;
+ }
+
+ public String getDstPath() {
+ return dstPath;
+ }
+
+ public static Pattern getVarPatternInDest() {
+ return VAR_PATTERN_IN_DEST;
+ }
+
+ /**
+ * Get resolved path from regex mount points.
+ * @param srcPath
+ * @param resolveLastComponent
+ * @return
+ */
+ public InodeTree.ResolveResult<T> resolve(final String srcPath,
+ final boolean resolveLastComponent) {
+ String pathStrToResolve = srcPath;
+ if (!resolveLastComponent) {
+ int lastSlashIndex = srcPath.lastIndexOf(SlashPath.toString());
+ if (lastSlashIndex == -1) {
+ return null;
+ }
+ pathStrToResolve = srcPath.substring(0, lastSlashIndex);
+ }
+ for (RegexMountPointInterceptor interceptor : interceptorList) {
+ pathStrToResolve = interceptor.interceptSource(pathStrToResolve);
+ }
Review comment:
nit: typo: give space after comma
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##########
@@ -646,102 +714,222 @@ boolean isInternalDir() {
}
/**
- * Resolve the pathname p relative to root InodeDir
+ * Resolve the pathname p relative to root InodeDir.
* @param p - input path
* @param resolveLastComponent
* @return ResolveResult which allows further resolution of the remaining path
* @throws FileNotFoundException
*/
ResolveResult<T> resolve(final String p, final boolean resolveLastComponent)
throws FileNotFoundException {
- String[] path = breakIntoPathComponents(p);
- if (path.length <= 1) { // special case for when path is "/"
- T targetFs = root.isInternalDir() ?
- getRootDir().getInternalDirFs() : getRootLink().getTargetFileSystem();
- ResolveResult<T> res = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
- targetFs, root.fullPath, SlashPath);
- return res;
- }
+ ResolveResult<T> resolveResult = null;
+ resolveResult = getResolveResultFromCache(p, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
+ }
+
+ try {
+ String[] path = breakIntoPathComponents(p);
+ if (path.length <= 1) { // special case for when path is "/"
+ T targetFs = root.isInternalDir() ?
+ getRootDir().getInternalDirFs()
+ : getRootLink().getTargetFileSystem();
+ resolveResult = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
+ targetFs, root.fullPath, SlashPath);
+ return resolveResult;
+ }
- /**
- * linkMergeSlash has been configured. The root of this mount table has
- * been linked to the root directory of a file system.
- * The first non-slash path component should be name of the mount table.
- */
- if (root.isLink()) {
- Path remainingPath;
- StringBuilder remainingPathStr = new StringBuilder();
- // ignore first slash
- for (int i = 1; i < path.length; i++) {
- remainingPathStr.append("/").append(path[i]);
+ /**
+ * linkMergeSlash has been configured. The root of this mount table has
+ * been linked to the root directory of a file system.
+ * The first non-slash path component should be name of the mount table.
+ */
+ if (root.isLink()) {
+ Path remainingPath;
+ StringBuilder remainingPathStr = new StringBuilder();
+ // ignore first slash
+ for (int i = 1; i < path.length; i++) {
+ remainingPathStr.append("/").append(path[i]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ getRootLink().getTargetFileSystem(), root.fullPath, remainingPath);
+ return resolveResult;
}
- remainingPath = new Path(remainingPathStr.toString());
- ResolveResult<T> res = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- getRootLink().getTargetFileSystem(), root.fullPath, remainingPath);
- return res;
- }
- Preconditions.checkState(root.isInternalDir());
- INodeDir<T> curInode = getRootDir();
+ Preconditions.checkState(root.isInternalDir());
+ INodeDir<T> curInode = getRootDir();
- int i;
- // ignore first slash
- for (i = 1; i < path.length - (resolveLastComponent ? 0 : 1); i++) {
- INode<T> nextInode = curInode.resolveInternal(path[i]);
- if (nextInode == null) {
- if (hasFallbackLink()) {
- return new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- getRootFallbackLink().getTargetFileSystem(),
- root.fullPath, new Path(p));
- } else {
- StringBuilder failedAt = new StringBuilder(path[0]);
- for (int j = 1; j <= i; ++j) {
- failedAt.append('/').append(path[j]);
+ // Try to resolve path in the regex mount point
+ resolveResult = tryResolveInRegexMountpoint(p, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
+ }
+
+ int i;
+ // ignore first slash
+ for (i = 1; i < path.length - (resolveLastComponent ? 0 : 1); i++) {
+ INode<T> nextInode = curInode.resolveInternal(path[i]);
+ if (nextInode == null) {
+ if (hasFallbackLink()) {
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ getRootFallbackLink().getTargetFileSystem(), root.fullPath,
+ new Path(p));
+ return resolveResult;
+ } else {
+ StringBuilder failedAt = new StringBuilder(path[0]);
+ for (int j = 1; j <= i; ++j) {
+ failedAt.append('/').append(path[j]);
+ }
+ throw (new FileNotFoundException(
+ "File/Directory does not exist: " + failedAt.toString()));
}
- throw (new FileNotFoundException(
- "File/Directory does not exist: " + failedAt.toString()));
}
- }
- if (nextInode.isLink()) {
- final INodeLink<T> link = (INodeLink<T>) nextInode;
- final Path remainingPath;
- if (i >= path.length - 1) {
- remainingPath = SlashPath;
- } else {
- StringBuilder remainingPathStr = new StringBuilder("/" + path[i + 1]);
- for (int j = i + 2; j < path.length; ++j) {
- remainingPathStr.append('/').append(path[j]);
+ if (nextInode.isLink()) {
+ final INodeLink<T> link = (INodeLink<T>) nextInode;
+ final Path remainingPath;
+ if (i >= path.length - 1) {
+ remainingPath = SlashPath;
+ } else {
+ StringBuilder remainingPathStr =
+ new StringBuilder("/" + path[i + 1]);
+ for (int j = i + 2; j < path.length; ++j) {
+ remainingPathStr.append('/').append(path[j]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
}
- remainingPath = new Path(remainingPathStr.toString());
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ link.getTargetFileSystem(), nextInode.fullPath, remainingPath);
+ return resolveResult;
+ } else if (nextInode.isInternalDir()) {
+ curInode = (INodeDir<T>) nextInode;
}
- final ResolveResult<T> res =
- new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- link.getTargetFileSystem(), nextInode.fullPath, remainingPath);
- return res;
- } else if (nextInode.isInternalDir()) {
- curInode = (INodeDir<T>) nextInode;
+ }
+
+ // We have resolved to an internal dir in mount table.
+ Path remainingPath;
+ if (resolveLastComponent) {
+ remainingPath = SlashPath;
+ } else {
+ // note we have taken care of when path is "/" above
+ // for internal dirs rem-path does not start with / since the lookup
+ // that follows will do a children.get(remaningPath) and will have to
+ // strip-out the initial /
+ StringBuilder remainingPathStr = new StringBuilder("/" + path[i]);
+ for (int j = i + 1; j < path.length; ++j) {
+ remainingPathStr.append('/').append(path[j]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
+ }
+ resolveResult = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
+ curInode.getInternalDirFs(), curInode.fullPath, remainingPath);
+ return resolveResult;
+ } finally {
+ if (pathResolutionCacheCapacity > 0 && resolveResult != null) {
+ addResolveResultToCache(p, resolveLastComponent, resolveResult);
}
}
+ }
- // We have resolved to an internal dir in mount table.
- Path remainingPath;
- if (resolveLastComponent) {
- remainingPath = SlashPath;
- } else {
- // note we have taken care of when path is "/" above
- // for internal dirs rem-path does not start with / since the lookup
- // that follows will do a children.get(remaningPath) and will have to
- // strip-out the initial /
- StringBuilder remainingPathStr = new StringBuilder("/" + path[i]);
- for (int j = i + 1; j < path.length; ++j) {
- remainingPathStr.append('/').append(path[j]);
+ /**
+ * Walk through all regex mount points to see
+ * whether the path match any regex expressions.
+ *
+ * @param srcPath
+ * @param resolveLastComponent
+ * @return
+ */
+ protected ResolveResult<T> tryResolveInRegexMountpoint(final String srcPath,
+ final boolean resolveLastComponent) {
+ for (RegexMountPoint regexMountPoint : regexMountPointList) {
+ ResolveResult resolveResult =
+ regexMountPoint.resolve(srcPath, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
}
- remainingPath = new Path(remainingPathStr.toString());
}
- final ResolveResult<T> res =
- new ResolveResult<T>(ResultKind.INTERNAL_DIR,
- curInode.getInternalDirFs(), curInode.fullPath, remainingPath);
- return res;
+ return null;
+ }
+
+ /**
+ * Build resolve result return to caller.
+ *
+ * @param resultKind
+ * @param resolvedPathStr
+ * @param targetOfResolvedPathStr
+ * @param remainingPath
+ * @return
+ */
+ protected ResolveResult<T> buildResolveResultForRegexMountPoint(
+ ResultKind resultKind, String resolvedPathStr,
+ String targetOfResolvedPathStr, Path remainingPath) {
+ try {
+ T targetFs = getTargetFileSystem(new URI(targetOfResolvedPathStr));
+ return new ResolveResult<T>(resultKind, targetFs, resolvedPathStr,
+ remainingPath);
+ } catch (IOException ex) {
+ LOGGER.error(String.format(
+ "Got Exception while build resolve result."
+ + " ResultKind:%s, resolvedPathStr:%s,"
+ + " targetOfResolvedPathStr:%s, remainingPath:%s,"
+ + " will return null.",
+ resultKind, resolvedPathStr, targetOfResolvedPathStr, remainingPath),
+ ex);
+ return null;
+ } catch (URISyntaxException uex) {
+ LOGGER.error(String.format(
+ "Got Exception while build resolve result."
+ + " ResultKind:%s, resolvedPathStr:%s,"
+ + " targetOfResolvedPathStr:%s, remainingPath:%s,"
+ + " will return null.",
+ resultKind, resolvedPathStr, targetOfResolvedPathStr, remainingPath),
+ uex);
+ return null;
+ }
+ }
+
+ /**
+ * Return resolution cache capacity.
+ *
+ * @return
+ */
+ public int getPathResolutionCacheCapacity() {
+ return pathResolutionCacheCapacity;
+ }
+
+ private void addResolveResultToCache(final String pathStr,
+ final Boolean resolveLastComponent,
+ final ResolveResult<T> resolveResult) {
+ try {
+ cacheRWLock.writeLock().lock();
+ String key = getResolveCacheKeyStr(pathStr, resolveLastComponent);
+ pathResolutionCache.put(key, resolveResult);
+ } finally {
+ cacheRWLock.writeLock().unlock();
+ }
+ }
+
+ private ResolveResult<T> getResolveResultFromCache(final String pathStr,
+ final Boolean resolveLastComponent) {
+ if (pathResolutionCacheCapacity <= 0) {
+ return null;
+ }
+ try {
+ cacheRWLock.readLock().lock();
+ String key = getResolveCacheKeyStr(pathStr, resolveLastComponent);
+ return (ResolveResult<T>) pathResolutionCache.get(key);
+ } finally {
+ cacheRWLock.readLock().unlock();
+ }
+ }
+
+ public static String getResolveCacheKeyStr(final String path,
+ Boolean resolveLastComp) {
+ return path + ",resolveLastComp" + resolveLastComp;
+ }
+
Review comment:
Please move annotation to separate line below.
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
##########
@@ -1417,7 +1418,8 @@ public void testGetContentSummaryWithFileInLocalFS() throws Exception {
fos.write(expected.getBytes());
}
ConfigUtil.addLink(conf,
- "/internalDir/internalDir2/linkToLocalFile", localFile.toURI());
Review comment:
seems like no change here. Can we avoid it?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##########
@@ -646,102 +714,222 @@ boolean isInternalDir() {
}
/**
- * Resolve the pathname p relative to root InodeDir
+ * Resolve the pathname p relative to root InodeDir.
* @param p - input path
* @param resolveLastComponent
* @return ResolveResult which allows further resolution of the remaining path
* @throws FileNotFoundException
*/
ResolveResult<T> resolve(final String p, final boolean resolveLastComponent)
throws FileNotFoundException {
- String[] path = breakIntoPathComponents(p);
- if (path.length <= 1) { // special case for when path is "/"
- T targetFs = root.isInternalDir() ?
- getRootDir().getInternalDirFs() : getRootLink().getTargetFileSystem();
- ResolveResult<T> res = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
- targetFs, root.fullPath, SlashPath);
- return res;
- }
+ ResolveResult<T> resolveResult = null;
+ resolveResult = getResolveResultFromCache(p, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
+ }
+
+ try {
+ String[] path = breakIntoPathComponents(p);
+ if (path.length <= 1) { // special case for when path is "/"
+ T targetFs = root.isInternalDir() ?
+ getRootDir().getInternalDirFs()
+ : getRootLink().getTargetFileSystem();
+ resolveResult = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
+ targetFs, root.fullPath, SlashPath);
+ return resolveResult;
+ }
- /**
- * linkMergeSlash has been configured. The root of this mount table has
- * been linked to the root directory of a file system.
- * The first non-slash path component should be name of the mount table.
- */
- if (root.isLink()) {
- Path remainingPath;
- StringBuilder remainingPathStr = new StringBuilder();
- // ignore first slash
- for (int i = 1; i < path.length; i++) {
- remainingPathStr.append("/").append(path[i]);
+ /**
+ * linkMergeSlash has been configured. The root of this mount table has
+ * been linked to the root directory of a file system.
+ * The first non-slash path component should be name of the mount table.
+ */
+ if (root.isLink()) {
+ Path remainingPath;
+ StringBuilder remainingPathStr = new StringBuilder();
+ // ignore first slash
+ for (int i = 1; i < path.length; i++) {
+ remainingPathStr.append("/").append(path[i]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ getRootLink().getTargetFileSystem(), root.fullPath, remainingPath);
+ return resolveResult;
}
- remainingPath = new Path(remainingPathStr.toString());
- ResolveResult<T> res = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- getRootLink().getTargetFileSystem(), root.fullPath, remainingPath);
- return res;
- }
- Preconditions.checkState(root.isInternalDir());
- INodeDir<T> curInode = getRootDir();
+ Preconditions.checkState(root.isInternalDir());
+ INodeDir<T> curInode = getRootDir();
- int i;
- // ignore first slash
- for (i = 1; i < path.length - (resolveLastComponent ? 0 : 1); i++) {
- INode<T> nextInode = curInode.resolveInternal(path[i]);
- if (nextInode == null) {
- if (hasFallbackLink()) {
- return new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- getRootFallbackLink().getTargetFileSystem(),
- root.fullPath, new Path(p));
- } else {
- StringBuilder failedAt = new StringBuilder(path[0]);
- for (int j = 1; j <= i; ++j) {
- failedAt.append('/').append(path[j]);
+ // Try to resolve path in the regex mount point
+ resolveResult = tryResolveInRegexMountpoint(p, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
+ }
+
+ int i;
+ // ignore first slash
+ for (i = 1; i < path.length - (resolveLastComponent ? 0 : 1); i++) {
+ INode<T> nextInode = curInode.resolveInternal(path[i]);
+ if (nextInode == null) {
+ if (hasFallbackLink()) {
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ getRootFallbackLink().getTargetFileSystem(), root.fullPath,
+ new Path(p));
+ return resolveResult;
+ } else {
+ StringBuilder failedAt = new StringBuilder(path[0]);
+ for (int j = 1; j <= i; ++j) {
+ failedAt.append('/').append(path[j]);
+ }
+ throw (new FileNotFoundException(
+ "File/Directory does not exist: " + failedAt.toString()));
}
- throw (new FileNotFoundException(
- "File/Directory does not exist: " + failedAt.toString()));
}
- }
- if (nextInode.isLink()) {
- final INodeLink<T> link = (INodeLink<T>) nextInode;
- final Path remainingPath;
- if (i >= path.length - 1) {
- remainingPath = SlashPath;
- } else {
- StringBuilder remainingPathStr = new StringBuilder("/" + path[i + 1]);
- for (int j = i + 2; j < path.length; ++j) {
- remainingPathStr.append('/').append(path[j]);
+ if (nextInode.isLink()) {
+ final INodeLink<T> link = (INodeLink<T>) nextInode;
+ final Path remainingPath;
+ if (i >= path.length - 1) {
+ remainingPath = SlashPath;
+ } else {
+ StringBuilder remainingPathStr =
+ new StringBuilder("/" + path[i + 1]);
+ for (int j = i + 2; j < path.length; ++j) {
+ remainingPathStr.append('/').append(path[j]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
}
- remainingPath = new Path(remainingPathStr.toString());
+ resolveResult = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
+ link.getTargetFileSystem(), nextInode.fullPath, remainingPath);
+ return resolveResult;
+ } else if (nextInode.isInternalDir()) {
+ curInode = (INodeDir<T>) nextInode;
}
- final ResolveResult<T> res =
- new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
- link.getTargetFileSystem(), nextInode.fullPath, remainingPath);
- return res;
- } else if (nextInode.isInternalDir()) {
- curInode = (INodeDir<T>) nextInode;
+ }
+
+ // We have resolved to an internal dir in mount table.
+ Path remainingPath;
+ if (resolveLastComponent) {
+ remainingPath = SlashPath;
+ } else {
+ // note we have taken care of when path is "/" above
+ // for internal dirs rem-path does not start with / since the lookup
+ // that follows will do a children.get(remaningPath) and will have to
+ // strip-out the initial /
+ StringBuilder remainingPathStr = new StringBuilder("/" + path[i]);
+ for (int j = i + 1; j < path.length; ++j) {
+ remainingPathStr.append('/').append(path[j]);
+ }
+ remainingPath = new Path(remainingPathStr.toString());
+ }
+ resolveResult = new ResolveResult<T>(ResultKind.INTERNAL_DIR,
+ curInode.getInternalDirFs(), curInode.fullPath, remainingPath);
+ return resolveResult;
+ } finally {
+ if (pathResolutionCacheCapacity > 0 && resolveResult != null) {
+ addResolveResultToCache(p, resolveLastComponent, resolveResult);
}
}
+ }
- // We have resolved to an internal dir in mount table.
- Path remainingPath;
- if (resolveLastComponent) {
- remainingPath = SlashPath;
- } else {
- // note we have taken care of when path is "/" above
- // for internal dirs rem-path does not start with / since the lookup
- // that follows will do a children.get(remaningPath) and will have to
- // strip-out the initial /
- StringBuilder remainingPathStr = new StringBuilder("/" + path[i]);
- for (int j = i + 1; j < path.length; ++j) {
- remainingPathStr.append('/').append(path[j]);
+ /**
+ * Walk through all regex mount points to see
+ * whether the path match any regex expressions.
+ *
+ * @param srcPath
+ * @param resolveLastComponent
+ * @return
+ */
+ protected ResolveResult<T> tryResolveInRegexMountpoint(final String srcPath,
+ final boolean resolveLastComponent) {
+ for (RegexMountPoint regexMountPoint : regexMountPointList) {
+ ResolveResult resolveResult =
+ regexMountPoint.resolve(srcPath, resolveLastComponent);
+ if (resolveResult != null) {
+ return resolveResult;
}
- remainingPath = new Path(remainingPathStr.toString());
}
- final ResolveResult<T> res =
- new ResolveResult<T>(ResultKind.INTERNAL_DIR,
- curInode.getInternalDirFs(), curInode.fullPath, remainingPath);
- return res;
+ return null;
+ }
+
+ /**
+ * Build resolve result return to caller.
+ *
+ * @param resultKind
+ * @param resolvedPathStr
+ * @param targetOfResolvedPathStr
+ * @param remainingPath
+ * @return
+ */
+ protected ResolveResult<T> buildResolveResultForRegexMountPoint(
+ ResultKind resultKind, String resolvedPathStr,
+ String targetOfResolvedPathStr, Path remainingPath) {
+ try {
+ T targetFs = getTargetFileSystem(new URI(targetOfResolvedPathStr));
+ return new ResolveResult<T>(resultKind, targetFs, resolvedPathStr,
+ remainingPath);
+ } catch (IOException ex) {
+ LOGGER.error(String.format(
+ "Got Exception while build resolve result."
+ + " ResultKind:%s, resolvedPathStr:%s,"
+ + " targetOfResolvedPathStr:%s, remainingPath:%s,"
+ + " will return null.",
+ resultKind, resolvedPathStr, targetOfResolvedPathStr, remainingPath),
+ ex);
+ return null;
+ } catch (URISyntaxException uex) {
+ LOGGER.error(String.format(
+ "Got Exception while build resolve result."
+ + " ResultKind:%s, resolvedPathStr:%s,"
+ + " targetOfResolvedPathStr:%s, remainingPath:%s,"
+ + " will return null.",
+ resultKind, resolvedPathStr, targetOfResolvedPathStr, remainingPath),
+ uex);
+ return null;
+ }
+ }
+
+ /**
+ * Return resolution cache capacity.
+ *
+ * @return
+ */
+ public int getPathResolutionCacheCapacity() {
+ return pathResolutionCacheCapacity;
+ }
+
+ private void addResolveResultToCache(final String pathStr,
+ final Boolean resolveLastComponent,
+ final ResolveResult<T> resolveResult) {
+ try {
+ cacheRWLock.writeLock().lock();
Review comment:
below key construction can be outside of lock?
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
##########
@@ -366,6 +366,82 @@ Don't want to change scheme or difficult to copy mount-table configurations to a
Please refer to the [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html)
+Regex Pattern Based Mount Points
+--------------------------------
+
+The view file system mount points were a Key-Value based mapping system. It is not friendly for user cases which mapping config could be abstracted to rules. E.g. Users want to provide a GCS bucket per user and there might be thousands of users in total. The old key-value based approach won't work well for several reasons:
+
+1. The mount table is used by FileSystem clients. There's a cost to spread the config to all clients and we should avoid it if possible. The [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html) could help the distribution by central mount table management. But the mount table still have to be updated on every change. The change could be greatly avoided if provide a rule-based mount table..
+
+2. The client have to understand all the KVs in the mount table. This is not ideal when the mountable grows to thousands of items. E.g. thousands of file systems might be initialized even users only need one. And the config itself will become bloated at scale.
Review comment:
I agree with this issue and filed a JIRA for it. https://issues.apache.org/jira/browse/HADOOP-17028
This issue should be solved once implement that.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkRegex.java
##########
@@ -0,0 +1,320 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.FsConstants;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSNNTopology;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE;
+import static org.apache.hadoop.fs.viewfs.RegexMountPoint.INTERCEPTOR_INTERNAL_SEP;
+
+/**
+ * Test linkRegex node type for view file system.
+ */
+public class TestViewFileSystemLinkRegex extends ViewFileSystemBaseTest {
+ public static final Logger LOGGER =
+ LoggerFactory.getLogger(TestViewFileSystemLinkRegex.class);
+
+ private static FileSystem fsDefault;
+ private static MiniDFSCluster cluster;
+ private static Configuration clusterConfig;
+ private static final int NAME_SPACES_COUNT = 3;
+ private static final int DATA_NODES_COUNT = 3;
+ private static final int FS_INDEX_DEFAULT = 0;
+ private static final FileSystem[] FS_HDFS = new FileSystem[NAME_SPACES_COUNT];
+ private static final String CLUSTER_NAME =
+ "TestViewFileSystemLinkRegexCluster";
+ private static final File TEST_DIR = GenericTestUtils
+ .getTestDir(TestViewFileSystemLinkRegex.class.getSimpleName());
+ private static final String TEST_BASE_PATH =
+ "/tmp/TestViewFileSystemLinkRegex";
+
+ @Override
+ protected FileSystemTestHelper createFileSystemHelper() {
+ return new FileSystemTestHelper(TEST_BASE_PATH);
+ }
+
+ @BeforeClass
+ public static void clusterSetupAtBeginning() throws IOException {
+ SupportsBlocks = true;
+ clusterConfig = ViewFileSystemTestSetup.createConfig();
+ clusterConfig.setBoolean(
+ DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY,
+ true);
+ cluster = new MiniDFSCluster.Builder(clusterConfig).nnTopology(
+ MiniDFSNNTopology.simpleFederatedTopology(NAME_SPACES_COUNT))
+ .numDataNodes(DATA_NODES_COUNT).build();
+ cluster.waitClusterUp();
+
+ for (int i = 0; i < NAME_SPACES_COUNT; i++) {
+ FS_HDFS[i] = cluster.getFileSystem(i);
+ }
+ fsDefault = FS_HDFS[FS_INDEX_DEFAULT];
+ }
+
+ @AfterClass
+ public static void clusterShutdownAtEnd() throws Exception {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+
+ @Override
+ @Before
+ public void setUp() throws Exception {
+ fsTarget = fsDefault;
+ super.setUp();
+ }
+
+ /**
+ * Override this so that we don't set the targetTestRoot to any path under the
+ * root of the FS, and so that we don't try to delete the test dir, but rather
+ * only its contents.
+ */
+ @Override
+ void initializeTargetTestRoot() throws IOException {
+ targetTestRoot = fsDefault.makeQualified(new Path("/"));
+ for (FileStatus status : fsDefault.listStatus(targetTestRoot)) {
+ fsDefault.delete(status.getPath(), true);
+ }
+ }
+
+ @Override
+ void setupMountPoints() {
+ super.setupMountPoints();
+ }
+
+ @Override
+ int getExpectedDelegationTokenCount() {
+ return 1; // all point to the same fs so 1 unique token
+ }
+
+ @Override
+ int getExpectedDelegationTokenCountWithCredentials() {
+ return 1;
+ }
+
+ public String buildReplaceInterceptorSettingString(String srcRegex,
+ String replaceString) {
+ return
+ RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + INTERCEPTOR_INTERNAL_SEP + srcRegex + INTERCEPTOR_INTERNAL_SEP
+ + replaceString;
+ }
+
+ public String linkInterceptorSettings(
+ List<String> interceptorSettingStrList) {
+ StringBuilder stringBuilder = new StringBuilder();
+ int listSize = interceptorSettingStrList.size();
+ for (int i = 0; i < listSize; ++i) {
+ stringBuilder.append(interceptorSettingStrList.get(i));
+ if (i < listSize - 1) {
+ stringBuilder.append(RegexMountPoint.INTERCEPTOR_SEP);
+ }
+ }
+ return stringBuilder.toString();
+ }
+
+ @Test
+ public void testConfLinkRegexIndexMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ // (^/(\w+),/targetTestRoot/$1)
+ // => /targetTestRoot/testConfLinkRegexIndexMapping1
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/(\\w+)";
+ String dstPathStr = targetTestRoot + "$1";
+ Path srcPath = new Path("/testConfLinkRegexIndexMapping1");
+ Path expectedResolveResult =
+ new Path(dstPathStr.replace("$1", "testConfLinkRegexIndexMapping1"));
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ // Test ${1} format
+ // ^/(\w+, /targetTestRoot/${1})
+ // => /targetTestRoot/testConfLinkRegexIndexMapping2
+ dstPathStr = targetTestRoot + "${1}";
+ srcPath = new Path("/testConfLinkRegexIndexMapping2");
+ expectedResolveResult =
+ new Path(dstPathStr.replace("${1}", "testConfLinkRegexIndexMapping2"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping3/file1
+ dstPathStr = targetTestRoot + "$1";
+ srcPath = new Path("/testConfLinkRegexIndexMapping3/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping3/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
Review comment:
same as above comment
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestRegexMountPointResolvedDstPathReplaceInterceptor.java
##########
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.IOException;
+
+import org.apache.hadoop.fs.Path;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.hadoop.fs.viewfs.RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH;
+
+/**
+ * Test RegexMountPointResolvedDstPathReplaceInterceptor.
+ */
+public class TestRegexMountPointResolvedDstPathReplaceInterceptor {
+
+ public String createSerializedString(String regex, String replaceString) {
+ return REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + regex
+ + RegexMountPoint.INTERCEPTOR_INTERNAL_SEP + replaceString;
+ }
+
+ @Test public void testDeserializeFromStringNormalCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertTrue(interceptor.getSrcRegexString().equals(srcRegex));
+ Assert.assertTrue(interceptor.getReplaceString().equals(replaceString));
+ Assert.assertTrue(interceptor.getSrcRegexPattern() == null);
+ interceptor.initialize();
+ Assert.assertTrue(
+ interceptor.getSrcRegexPattern().toString().equals(srcRegex));
+ }
+
+ @Test public void testDeserializeFromStringBadCase() throws IOException {
+ String srcRegex = "-";
+ String replaceString = "_";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ serializedString = serializedString + ":ddd";
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ RegexMountPointResolvedDstPathReplaceInterceptor
+ .deserializeFromString(serializedString);
+ Assert.assertEquals(interceptor, null);
+ }
+
+ @Test public void testSerialization() {
+ String srcRegex = "word1";
+ String replaceString = "word2";
+ String serializedString = createSerializedString(srcRegex, replaceString);
+ RegexMountPointResolvedDstPathReplaceInterceptor interceptor =
+ new RegexMountPointResolvedDstPathReplaceInterceptor(srcRegex,
+ replaceString);
+ Assert.assertEquals(interceptor.serializeToString(), serializedString);
+ }
+
+ @Test public void testInterceptSource() {
Review comment:
Nits: please add annotation in separate line above. You may want to change for all such lines below and above.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
##########
@@ -366,6 +366,82 @@ Don't want to change scheme or difficult to copy mount-table configurations to a
Please refer to the [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html)
+Regex Pattern Based Mount Points
+--------------------------------
+
+The view file system mount points were a Key-Value based mapping system. It is not friendly for user cases which mapping config could be abstracted to rules. E.g. Users want to provide a GCS bucket per user and there might be thousands of users in total. The old key-value based approach won't work well for several reasons:
+
+1. The mount table is used by FileSystem clients. There's a cost to spread the config to all clients and we should avoid it if possible. The [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html) could help the distribution by central mount table management. But the mount table still have to be updated on every change. The change could be greatly avoided if provide a rule-based mount table..
+
+2. The client have to understand all the KVs in the mount table. This is not ideal when the mountable grows to thousands of items. E.g. thousands of file systems might be initialized even users only need one. And the config itself will become bloated at scale.
+
+### Understand the Difference
+
+In the key-value based mount table, view file system treats every mount point as a partition. There's several file system APIs which will lead to operation on all partitions. E.g. there's an HDFS cluster with multiple mount. Users want to run “hadoop fs -put file viewfs://hdfs.namenode.apache.org/tmp/” cmd to copy data from local disk to our HDFS cluster. The cmd will trigger ViewFileSystem to call setVerifyChecksum() method which will initialize the file system for every mount point.
+For a regex-base rule mount table entry, we couldn't know what's corresponding path until parsing. So the regex based mount table entry will be ignored on such cases and the file system will be created upon accessing. The inner cache of ViewFs is also not available for regex-base mount points now as it assumes target file system doesn't change after viewfs initialization. Please disable it if you want to use regex-base mount table. We also need to change the rename strategy to SAME_FILESYSTEM_ACROSS_MOUNTPOINT for the same reason.
+```xml
+<property>
+ <name>fs.viewfs.enable.inner.cache</name>
+ <value>false</value>
Review comment:
Instead of asking users to disable, isn't it the better way is to add getTargetFileSystem(uri, userCache=true/false) method?
So, that the users already using other mount points need not disable for the sake if regex based mount points right?
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkRegex.java
##########
@@ -0,0 +1,320 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.FsConstants;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSNNTopology;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE;
+import static org.apache.hadoop.fs.viewfs.RegexMountPoint.INTERCEPTOR_INTERNAL_SEP;
+
+/**
+ * Test linkRegex node type for view file system.
+ */
+public class TestViewFileSystemLinkRegex extends ViewFileSystemBaseTest {
+ public static final Logger LOGGER =
+ LoggerFactory.getLogger(TestViewFileSystemLinkRegex.class);
+
+ private static FileSystem fsDefault;
+ private static MiniDFSCluster cluster;
+ private static Configuration clusterConfig;
+ private static final int NAME_SPACES_COUNT = 3;
+ private static final int DATA_NODES_COUNT = 3;
+ private static final int FS_INDEX_DEFAULT = 0;
+ private static final FileSystem[] FS_HDFS = new FileSystem[NAME_SPACES_COUNT];
+ private static final String CLUSTER_NAME =
+ "TestViewFileSystemLinkRegexCluster";
+ private static final File TEST_DIR = GenericTestUtils
+ .getTestDir(TestViewFileSystemLinkRegex.class.getSimpleName());
+ private static final String TEST_BASE_PATH =
+ "/tmp/TestViewFileSystemLinkRegex";
+
+ @Override
+ protected FileSystemTestHelper createFileSystemHelper() {
+ return new FileSystemTestHelper(TEST_BASE_PATH);
+ }
+
+ @BeforeClass
+ public static void clusterSetupAtBeginning() throws IOException {
+ SupportsBlocks = true;
+ clusterConfig = ViewFileSystemTestSetup.createConfig();
+ clusterConfig.setBoolean(
+ DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY,
+ true);
+ cluster = new MiniDFSCluster.Builder(clusterConfig).nnTopology(
+ MiniDFSNNTopology.simpleFederatedTopology(NAME_SPACES_COUNT))
+ .numDataNodes(DATA_NODES_COUNT).build();
+ cluster.waitClusterUp();
+
+ for (int i = 0; i < NAME_SPACES_COUNT; i++) {
+ FS_HDFS[i] = cluster.getFileSystem(i);
+ }
+ fsDefault = FS_HDFS[FS_INDEX_DEFAULT];
+ }
+
+ @AfterClass
+ public static void clusterShutdownAtEnd() throws Exception {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+
+ @Override
+ @Before
+ public void setUp() throws Exception {
+ fsTarget = fsDefault;
+ super.setUp();
+ }
+
+ /**
+ * Override this so that we don't set the targetTestRoot to any path under the
+ * root of the FS, and so that we don't try to delete the test dir, but rather
+ * only its contents.
+ */
+ @Override
+ void initializeTargetTestRoot() throws IOException {
+ targetTestRoot = fsDefault.makeQualified(new Path("/"));
+ for (FileStatus status : fsDefault.listStatus(targetTestRoot)) {
+ fsDefault.delete(status.getPath(), true);
+ }
+ }
+
+ @Override
+ void setupMountPoints() {
+ super.setupMountPoints();
+ }
+
+ @Override
+ int getExpectedDelegationTokenCount() {
+ return 1; // all point to the same fs so 1 unique token
+ }
+
+ @Override
+ int getExpectedDelegationTokenCountWithCredentials() {
+ return 1;
+ }
+
+ public String buildReplaceInterceptorSettingString(String srcRegex,
+ String replaceString) {
+ return
+ RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + INTERCEPTOR_INTERNAL_SEP + srcRegex + INTERCEPTOR_INTERNAL_SEP
+ + replaceString;
+ }
+
+ public String linkInterceptorSettings(
+ List<String> interceptorSettingStrList) {
+ StringBuilder stringBuilder = new StringBuilder();
+ int listSize = interceptorSettingStrList.size();
+ for (int i = 0; i < listSize; ++i) {
+ stringBuilder.append(interceptorSettingStrList.get(i));
+ if (i < listSize - 1) {
+ stringBuilder.append(RegexMountPoint.INTERCEPTOR_SEP);
+ }
+ }
+ return stringBuilder.toString();
+ }
+
+ @Test
+ public void testConfLinkRegexIndexMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ // (^/(\w+),/targetTestRoot/$1)
+ // => /targetTestRoot/testConfLinkRegexIndexMapping1
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/(\\w+)";
+ String dstPathStr = targetTestRoot + "$1";
+ Path srcPath = new Path("/testConfLinkRegexIndexMapping1");
+ Path expectedResolveResult =
+ new Path(dstPathStr.replace("$1", "testConfLinkRegexIndexMapping1"));
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ // Test ${1} format
+ // ^/(\w+, /targetTestRoot/${1})
+ // => /targetTestRoot/testConfLinkRegexIndexMapping2
+ dstPathStr = targetTestRoot + "${1}";
+ srcPath = new Path("/testConfLinkRegexIndexMapping2");
+ expectedResolveResult =
+ new Path(dstPathStr.replace("${1}", "testConfLinkRegexIndexMapping2"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping3/file1
+ dstPathStr = targetTestRoot + "$1";
+ srcPath = new Path("/testConfLinkRegexIndexMapping3/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping3/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1/)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping4/file1
+ dstPathStr = targetTestRoot + "$1/";
+ srcPath = new Path("/testConfLinkRegexIndexMapping4/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping4/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
Review comment:
you may want to close vfs as well?
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkRegex.java
##########
@@ -0,0 +1,320 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.FsConstants;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSNNTopology;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE;
+import static org.apache.hadoop.fs.viewfs.RegexMountPoint.INTERCEPTOR_INTERNAL_SEP;
+
+/**
+ * Test linkRegex node type for view file system.
+ */
+public class TestViewFileSystemLinkRegex extends ViewFileSystemBaseTest {
+ public static final Logger LOGGER =
+ LoggerFactory.getLogger(TestViewFileSystemLinkRegex.class);
+
+ private static FileSystem fsDefault;
+ private static MiniDFSCluster cluster;
+ private static Configuration clusterConfig;
+ private static final int NAME_SPACES_COUNT = 3;
+ private static final int DATA_NODES_COUNT = 3;
+ private static final int FS_INDEX_DEFAULT = 0;
+ private static final FileSystem[] FS_HDFS = new FileSystem[NAME_SPACES_COUNT];
+ private static final String CLUSTER_NAME =
+ "TestViewFileSystemLinkRegexCluster";
+ private static final File TEST_DIR = GenericTestUtils
+ .getTestDir(TestViewFileSystemLinkRegex.class.getSimpleName());
+ private static final String TEST_BASE_PATH =
+ "/tmp/TestViewFileSystemLinkRegex";
+
+ @Override
+ protected FileSystemTestHelper createFileSystemHelper() {
+ return new FileSystemTestHelper(TEST_BASE_PATH);
+ }
+
+ @BeforeClass
+ public static void clusterSetupAtBeginning() throws IOException {
+ SupportsBlocks = true;
+ clusterConfig = ViewFileSystemTestSetup.createConfig();
+ clusterConfig.setBoolean(
+ DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY,
+ true);
+ cluster = new MiniDFSCluster.Builder(clusterConfig).nnTopology(
+ MiniDFSNNTopology.simpleFederatedTopology(NAME_SPACES_COUNT))
+ .numDataNodes(DATA_NODES_COUNT).build();
+ cluster.waitClusterUp();
+
+ for (int i = 0; i < NAME_SPACES_COUNT; i++) {
+ FS_HDFS[i] = cluster.getFileSystem(i);
+ }
+ fsDefault = FS_HDFS[FS_INDEX_DEFAULT];
+ }
+
+ @AfterClass
+ public static void clusterShutdownAtEnd() throws Exception {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+
+ @Override
+ @Before
+ public void setUp() throws Exception {
+ fsTarget = fsDefault;
+ super.setUp();
+ }
+
+ /**
+ * Override this so that we don't set the targetTestRoot to any path under the
+ * root of the FS, and so that we don't try to delete the test dir, but rather
+ * only its contents.
+ */
+ @Override
+ void initializeTargetTestRoot() throws IOException {
+ targetTestRoot = fsDefault.makeQualified(new Path("/"));
+ for (FileStatus status : fsDefault.listStatus(targetTestRoot)) {
+ fsDefault.delete(status.getPath(), true);
+ }
+ }
+
+ @Override
+ void setupMountPoints() {
+ super.setupMountPoints();
+ }
+
+ @Override
+ int getExpectedDelegationTokenCount() {
+ return 1; // all point to the same fs so 1 unique token
+ }
+
+ @Override
+ int getExpectedDelegationTokenCountWithCredentials() {
+ return 1;
+ }
+
+ public String buildReplaceInterceptorSettingString(String srcRegex,
+ String replaceString) {
+ return
+ RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + INTERCEPTOR_INTERNAL_SEP + srcRegex + INTERCEPTOR_INTERNAL_SEP
+ + replaceString;
+ }
+
+ public String linkInterceptorSettings(
+ List<String> interceptorSettingStrList) {
+ StringBuilder stringBuilder = new StringBuilder();
+ int listSize = interceptorSettingStrList.size();
+ for (int i = 0; i < listSize; ++i) {
+ stringBuilder.append(interceptorSettingStrList.get(i));
+ if (i < listSize - 1) {
+ stringBuilder.append(RegexMountPoint.INTERCEPTOR_SEP);
+ }
+ }
+ return stringBuilder.toString();
+ }
+
+ @Test
+ public void testConfLinkRegexIndexMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ // (^/(\w+),/targetTestRoot/$1)
+ // => /targetTestRoot/testConfLinkRegexIndexMapping1
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/(\\w+)";
+ String dstPathStr = targetTestRoot + "$1";
+ Path srcPath = new Path("/testConfLinkRegexIndexMapping1");
+ Path expectedResolveResult =
+ new Path(dstPathStr.replace("$1", "testConfLinkRegexIndexMapping1"));
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ // Test ${1} format
+ // ^/(\w+, /targetTestRoot/${1})
+ // => /targetTestRoot/testConfLinkRegexIndexMapping2
+ dstPathStr = targetTestRoot + "${1}";
+ srcPath = new Path("/testConfLinkRegexIndexMapping2");
+ expectedResolveResult =
+ new Path(dstPathStr.replace("${1}", "testConfLinkRegexIndexMapping2"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping3/file1
+ dstPathStr = targetTestRoot + "$1";
+ srcPath = new Path("/testConfLinkRegexIndexMapping3/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping3/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1/)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping4/file1
+ dstPathStr = targetTestRoot + "$1/";
+ srcPath = new Path("/testConfLinkRegexIndexMapping4/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping4/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+ }
+
+ @Test
+ public void testConfLinkRegexNamedGroupMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ // ^/(?<firstDir>\\w+) = > /targetTestRoot/$firstDir
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/(?<firstDir>\\w+)";
+ String dstPathStr = targetTestRoot + "$firstDir";
+ Path srcPath = new Path("/testConfLinkRegexNamedGroupMapping1");
+ Path expectedResolveResult = new Path(
+ dstPathStr.replace("$firstDir", "testConfLinkRegexNamedGroupMapping1"));
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ // Test ${1} format
+ dstPathStr = targetTestRoot + "${firstDir}";
+ srcPath = new Path("/testConfLinkRegexNamedGroupMapping2");
+ expectedResolveResult = new Path(dstPathStr
+ .replace("${firstDir}", "testConfLinkRegexNamedGroupMapping2"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
Review comment:
Could we assert the results of listStatus. Probably create few files under that expected target path. So, the ls will get some file and assert to make sure it's getting from target.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLinkRegex.java
##########
@@ -0,0 +1,320 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.fs.viewfs;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileSystemTestHelper;
+import org.apache.hadoop.fs.FsConstants;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.MiniDFSNNTopology;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.fs.viewfs.Constants.CONFIG_VIEWFS_ENABLE_INNER_CACHE;
+import static org.apache.hadoop.fs.viewfs.RegexMountPoint.INTERCEPTOR_INTERNAL_SEP;
+
+/**
+ * Test linkRegex node type for view file system.
+ */
+public class TestViewFileSystemLinkRegex extends ViewFileSystemBaseTest {
+ public static final Logger LOGGER =
+ LoggerFactory.getLogger(TestViewFileSystemLinkRegex.class);
+
+ private static FileSystem fsDefault;
+ private static MiniDFSCluster cluster;
+ private static Configuration clusterConfig;
+ private static final int NAME_SPACES_COUNT = 3;
+ private static final int DATA_NODES_COUNT = 3;
+ private static final int FS_INDEX_DEFAULT = 0;
+ private static final FileSystem[] FS_HDFS = new FileSystem[NAME_SPACES_COUNT];
+ private static final String CLUSTER_NAME =
+ "TestViewFileSystemLinkRegexCluster";
+ private static final File TEST_DIR = GenericTestUtils
+ .getTestDir(TestViewFileSystemLinkRegex.class.getSimpleName());
+ private static final String TEST_BASE_PATH =
+ "/tmp/TestViewFileSystemLinkRegex";
+
+ @Override
+ protected FileSystemTestHelper createFileSystemHelper() {
+ return new FileSystemTestHelper(TEST_BASE_PATH);
+ }
+
+ @BeforeClass
+ public static void clusterSetupAtBeginning() throws IOException {
+ SupportsBlocks = true;
+ clusterConfig = ViewFileSystemTestSetup.createConfig();
+ clusterConfig.setBoolean(
+ DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY,
+ true);
+ cluster = new MiniDFSCluster.Builder(clusterConfig).nnTopology(
+ MiniDFSNNTopology.simpleFederatedTopology(NAME_SPACES_COUNT))
+ .numDataNodes(DATA_NODES_COUNT).build();
+ cluster.waitClusterUp();
+
+ for (int i = 0; i < NAME_SPACES_COUNT; i++) {
+ FS_HDFS[i] = cluster.getFileSystem(i);
+ }
+ fsDefault = FS_HDFS[FS_INDEX_DEFAULT];
+ }
+
+ @AfterClass
+ public static void clusterShutdownAtEnd() throws Exception {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+
+ @Override
+ @Before
+ public void setUp() throws Exception {
+ fsTarget = fsDefault;
+ super.setUp();
+ }
+
+ /**
+ * Override this so that we don't set the targetTestRoot to any path under the
+ * root of the FS, and so that we don't try to delete the test dir, but rather
+ * only its contents.
+ */
+ @Override
+ void initializeTargetTestRoot() throws IOException {
+ targetTestRoot = fsDefault.makeQualified(new Path("/"));
+ for (FileStatus status : fsDefault.listStatus(targetTestRoot)) {
+ fsDefault.delete(status.getPath(), true);
+ }
+ }
+
+ @Override
+ void setupMountPoints() {
+ super.setupMountPoints();
+ }
+
+ @Override
+ int getExpectedDelegationTokenCount() {
+ return 1; // all point to the same fs so 1 unique token
+ }
+
+ @Override
+ int getExpectedDelegationTokenCountWithCredentials() {
+ return 1;
+ }
+
+ public String buildReplaceInterceptorSettingString(String srcRegex,
+ String replaceString) {
+ return
+ RegexMountPointInterceptorType.REPLACE_RESOLVED_DST_PATH.getConfigName()
+ + INTERCEPTOR_INTERNAL_SEP + srcRegex + INTERCEPTOR_INTERNAL_SEP
+ + replaceString;
+ }
+
+ public String linkInterceptorSettings(
+ List<String> interceptorSettingStrList) {
+ StringBuilder stringBuilder = new StringBuilder();
+ int listSize = interceptorSettingStrList.size();
+ for (int i = 0; i < listSize; ++i) {
+ stringBuilder.append(interceptorSettingStrList.get(i));
+ if (i < listSize - 1) {
+ stringBuilder.append(RegexMountPoint.INTERCEPTOR_SEP);
+ }
+ }
+ return stringBuilder.toString();
+ }
+
+ @Test
+ public void testConfLinkRegexIndexMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ // (^/(\w+),/targetTestRoot/$1)
+ // => /targetTestRoot/testConfLinkRegexIndexMapping1
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/(\\w+)";
+ String dstPathStr = targetTestRoot + "$1";
+ Path srcPath = new Path("/testConfLinkRegexIndexMapping1");
+ Path expectedResolveResult =
+ new Path(dstPathStr.replace("$1", "testConfLinkRegexIndexMapping1"));
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ // Test ${1} format
+ // ^/(\w+, /targetTestRoot/${1})
+ // => /targetTestRoot/testConfLinkRegexIndexMapping2
+ dstPathStr = targetTestRoot + "${1}";
+ srcPath = new Path("/testConfLinkRegexIndexMapping2");
+ expectedResolveResult =
+ new Path(dstPathStr.replace("${1}", "testConfLinkRegexIndexMapping2"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping3/file1
+ dstPathStr = targetTestRoot + "$1";
+ srcPath = new Path("/testConfLinkRegexIndexMapping3/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping3/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ //(^/(\w+)/file1, /targetTestRoot/$1/)
+ // = > /targetTestRoot/testConfLinkRegexIndexMapping4/file1
+ dstPathStr = targetTestRoot + "$1/";
+ srcPath = new Path("/testConfLinkRegexIndexMapping4/file1");
+ expectedResolveResult = new Path(
+ dstPathStr.replace("$1", "testConfLinkRegexIndexMapping4/file1"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+ }
+
+ @Test
+ public void testConfLinkRegexNamedGroupMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ // ^/(?<firstDir>\\w+) = > /targetTestRoot/$firstDir
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/(?<firstDir>\\w+)";
+ String dstPathStr = targetTestRoot + "$firstDir";
+ Path srcPath = new Path("/testConfLinkRegexNamedGroupMapping1");
+ Path expectedResolveResult = new Path(
+ dstPathStr.replace("$firstDir", "testConfLinkRegexNamedGroupMapping1"));
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+
+ // Test ${1} format
+ dstPathStr = targetTestRoot + "${firstDir}";
+ srcPath = new Path("/testConfLinkRegexNamedGroupMapping2");
+ expectedResolveResult = new Path(dstPathStr
+ .replace("${firstDir}", "testConfLinkRegexNamedGroupMapping2"));
+ outputStream = fsTarget.create(expectedResolveResult);
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+ }
+
+ @Test
+ public void testConfLinkRegexFixedDestMapping() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/\\w+";
+ String dstPathStr =
+ targetTestRoot + "testConfLinkRegexFixedDestMappingFile";
+ Path expectedResolveResult = new Path(dstPathStr);
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil.addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, null);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(
+ expectedResolveResult.equals(vfs.resolvePath(new Path("/misc1"))));
+ Assert.assertTrue(
+ expectedResolveResult.equals(vfs.resolvePath(new Path("/misc2"))));
+ }
+
+ @Test
+ public void testConfLinkRegexWithSingleInterceptor() throws Exception {
+ conf.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+ URI viewFsUri =
+ new URI(FsConstants.VIEWFS_SCHEME, CLUSTER_NAME, "/", null, null);
+ String regexStr = "^/user/(?<username>\\w+)";
+ String dstPathStr = targetTestRoot + "$username";
+ // Replace "_" with "-"
+ String settingString = buildReplaceInterceptorSettingString("_", "-");
+ Path srcPath = new Path("/user/hadoop_user1/hadoop_file1");
+ Path expectedResolveResult =
+ new Path(targetTestRoot, "hadoop-user1/hadoop_file1");
+ FSDataOutputStream outputStream = fsTarget.create((expectedResolveResult));
+ fsTarget.listStatus(expectedResolveResult);
+ outputStream.close();
+ ConfigUtil
+ .addLinkRegex(conf, CLUSTER_NAME, regexStr, dstPathStr, settingString);
+ FileSystem vfs = FileSystem.get(viewFsUri, conf);
+ Assert.assertTrue(expectedResolveResult.equals(vfs.resolvePath(srcPath)));
+ Assert.assertEquals(0L, vfs.getFileStatus(srcPath).getLen());
+ }
+
Review comment:
Do you mind writing some description test steps in javadoc. It will be helpful to better understand about scenarios.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] JohnZZGithub commented on pull request #2185: HADOOP-15891. provide Regex Based Mount Point In Inode Tree
Posted by GitBox <gi...@apache.org>.
JohnZZGithub commented on pull request #2185:
URL: https://github.com/apache/hadoop/pull/2185#issuecomment-672293070
Thank you! let me check ViewFs.java. Function wise, this patch worked for MR jobs and HDFS use cases in our internal clusters.
> I will review it in a day or two, Thanks
> BTW, you may need the similar changes in ViewFs.java as well, I think nfly also missed there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org