You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2020/12/01 21:42:00 UTC

[jira] [Commented] (HADOOP-17402) Add GCS FS impl reference to core-default.xml

    [ https://issues.apache.org/jira/browse/HADOOP-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241893#comment-17241893 ] 

Steve Loughran commented on HADOOP-17402:
-----------------------------------------

> GCS connector has hadoop.fs.Filesystem service declared like here, is this the right way?

yes. Does force a load of the class, which may have side effects or even fail to load. it can also be very slow. Hence our interest in having some separate declaration

the S3A error isn't as useful as you think as if the hadoop-aws JAR is present but the aws-sdk JAR isn't you end up with different linkage problems. If you look at HADOOP-14132 we've discussed actually including in the service declaration a list of mandatory dependencies which we could look for and fail with meaningful messages.

Take a look at my cloudstore JAR to see what I envisage. 
https://github.com/steveloughran/cloudstore

> Add GCS FS impl reference to core-default.xml
> ---------------------------------------------
>
>                 Key: HADOOP-17402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17402
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Rafal Wojdyla
>            Priority: Major
>
> Akin to current S3 default configuration add GCS configuration, specifically to declare the GCS implementation. [GCS connector|https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage]. Has this not been done since the GCS connector is not part of the hadoop/ASF codebase, or is there any other blocker?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org