You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Esfandiar Manii (JIRA)" <ji...@apache.org> on 2018/07/19 17:43:00 UTC
[jira] [Commented] (HADOOP-15547) WASB: listStatus performance
[ https://issues.apache.org/jira/browse/HADOOP-15547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549616#comment-16549616 ]
Esfandiar Manii commented on HADOOP-15547:
------------------------------------------
Thanks [~stevel@apache.org] Could you please submit this to the branch?
> WASB: listStatus performance
> ----------------------------
>
> Key: HADOOP-15547
> URL: https://issues.apache.org/jira/browse/HADOOP-15547
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
> Affects Versions: 2.9.1, 3.0.2
> Reporter: Thomas Marquardt
> Assignee: Thomas Marquardt
> Priority: Major
> Attachments: HADOOP-15547-004.patch, HADOOP-15547-004.patch, HADOOP-15547.001.patch, HADOOP-15547.002.patch, HADOOP-15547.003.patch
>
>
> The WASB implementation of Filesystem.listStatus is very slow due to O(n!) algorithm to remove duplicates and uses too much memory due to the extra conversion from BlobListItem to FileMetadata to FileStatus. It takes over 30 minutes to list 700,000 files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org