You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@twill.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/01/04 22:40:39 UTC
[jira] [Commented] (TWILL-158) Provides an implementation of
Location and LocationFactory using FileContext instead of FileSystem
[ https://issues.apache.org/jira/browse/TWILL-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15081865#comment-15081865 ]
ASF GitHub Bot commented on TWILL-158:
--------------------------------------
Github user poornachandra commented on a diff in the pull request:
https://github.com/apache/incubator-twill/pull/73#discussion_r48784930
--- Diff: twill-yarn/src/main/java/org/apache/twill/filesystem/FileContextLocationFactory.java ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.twill.filesystem;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Throwables;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileContext;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.UnsupportedFileSystemException;
+
+import java.net.URI;
+import java.util.Objects;
+
+/**
+ * A {@link LocationFactory} implementation that uses {@link FileContext} to create {@link Location}.
+ */
+public class FileContextLocationFactory implements LocationFactory {
+
+ private final Configuration configuration;
+ private final FileContext fc;
+ private final Path pathBase;
+
+ /**
+ * Same as {@link #FileContextLocationFactory(Configuration, String) FileContextLocationFactory(configuration, "/")}.
+ */
+ public FileContextLocationFactory(Configuration configuration) {
+ this(configuration, "/");
+ }
+
+ /**
+ * Creates a new instance.
+ *
+ * @param configuration the hadoop configuration
+ * @param pathBase base path for all non-absolute location created through this {@link LocationFactory}.
+ */
+ public FileContextLocationFactory(Configuration configuration, String pathBase) {
+ this.configuration = configuration;
+ this.fc = createFileContext(configuration);
+ this.pathBase = new Path(pathBase.startsWith("/") ? pathBase : "/" + pathBase);
+ }
+
+ @Override
+ public Location create(String path) {
+ if (path.startsWith("/")) {
+ path = path.substring(1);
+ }
+ Path locationPath;
+ if (path.isEmpty()) {
+ locationPath = pathBase;
+ } else {
+ locationPath = new Path(path);
+ }
+ locationPath = locationPath.makeQualified(fc.getDefaultFileSystem().getUri(), pathBase);
+ return new FileContextLocation(this, fc, locationPath);
+ }
+
+ @Override
+ public Location create(URI uri) {
+ URI contextURI = fc.getWorkingDirectory().toUri();
+ if (Objects.equals(contextURI.getScheme(), uri.getScheme())
+ && Objects.equals(contextURI.getAuthority(), uri.getAuthority())) {
+ // A full URI
+ return new FileContextLocation(this, fc, new Path(uri));
+ }
+
+ if (uri.isAbsolute()) {
--- End diff --
It will be good to have a comment on what a full URI and an absolute URI mean.
> Provides an implementation of Location and LocationFactory using FileContext instead of FileSystem
> --------------------------------------------------------------------------------------------------
>
> Key: TWILL-158
> URL: https://issues.apache.org/jira/browse/TWILL-158
> Project: Apache Twill
> Issue Type: New Feature
> Reporter: Terence Yim
> Assignee: Terence Yim
> Fix For: 0.7.0-incubating
>
>
> FileContext is more intended for application developer. It provides better support for the rename operation, while the current FileSystem one uses a hacky way to implement rename().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)