You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2017/03/01 10:52:45 UTC
[jira] [Commented] (HADOOP-14132) Filesystem discovery to stop
loading implementation classes
[ https://issues.apache.org/jira/browse/HADOOP-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889964#comment-15889964 ]
Steve Loughran commented on HADOOP-14132:
-----------------------------------------
I was thinking of three things actually
# for the in-hadoop classes, we just move the declarations back into core-defaults. Easy & doable today. We already have the AbstractFileSystem declarations there.
# do what you say, offer a new mech, encourage external filesystems to move to it; don't bother with our own classes
# leave the old loader alone, deprecated.
We should reach out to the people doing other filesystems: redhat, google, EMR s3, to get them on board, as the lightweight mechanism is the one they should move to.
The gotcha I Can think of is if someone external adds the new mech and also retains the old one for x-version support. We might want to backport this to 2.7.x
> Filesystem discovery to stop loading implementation classes
> -----------------------------------------------------------
>
> Key: HADOOP-14132
> URL: https://issues.apache.org/jira/browse/HADOOP-14132
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/adl, fs/azure, fs/oss, fs/s3, fs/swift
> Affects Versions: 2.7.3
> Reporter: Steve Loughran
> Assignee: Steve Loughran
>
> Integration testing of Hadoop with the HADOOP-14040 has shown up that the move to a shaded AWS JAR is slowing all hadoop client code down.
> I believe this is due to how we use service discovery to identify FS implementations: the implementation classes themselves are instantiated.
> This has known problems today with classloading, but clearly impacts performance too, especially with complex transitive dependencies unique to the loaded class.
> Proposed: have lightweight service declaration classes which implement an interface declaring
> # schema
> # classname of FileSystem impl
> # classname of AbstractFS impl
> # homepage (for third party code, support, etc)
> These are what we register and scan in the FS to look for services.
> This will leave the question about what to do for existing filesystems? I think we'll need to retain the old code for external ones, while moving the hadoop modules to the new ones
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org