[jira] [Commented] (HADOOP-18684) Fix S3A filesystem such that the scheme matches the URI scheme

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17707861#comment-17707861 ] 

ASF GitHub Bot commented on HADOOP-18684:
-----------------------------------------

steveloughran commented on code in PR #5521:
URL: https://github.com/apache/hadoop/pull/5521#discussion_r1155685349


##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##########
@@ -1196,7 +1196,11 @@ public void abortOutstandingMultipartUploads(long seconds)
    */
   @Override
   public String getScheme() {
-    return "s3a";
+    if (this.uri != null && this.uri.getScheme() != null){

Review Comment:
   just add a field "scheme" with default value FS_S3A; in initialize() set it to the URI scheme



##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/ITestS3AFileContext.java:
##########
@@ -13,20 +13,22 @@
  */
 package org.apache.hadoop.fs.s3a.fileContext;
 
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.*;
-import org.junit.Test;
-
-import java.io.IOException;
 import java.net.URI;
 import java.net.URISyntaxException;
 
+import org.junit.Test;
 import static org.junit.Assert.assertEquals;

Review Comment:
   move to bottom





> Fix S3A filesystem such that the scheme matches the URI scheme
> --------------------------------------------------------------
>
>                 Key: HADOOP-18684
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18684
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 3.3.5
>            Reporter: Harshit Gupta
>            Priority: Major
>              Labels: pull-request-available
>
> Certain codepaths use the FileContext API's to perform FS based operations such as yarn log aggregations. While trying to reuse the S3A connector for GCS based workloads the yarn log aggregation was not happening. Upon further investigation it was observed that FileContext API have hardcoded URI scheme checks that need to disabled/updated to make S3A compatible with non AWS stores.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org