You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2022/10/03 09:44:00 UTC
[jira] [Commented] (HDFS-16789) Update FileSystem class to read disk usage from actual usage value instead of file's length
[ https://issues.apache.org/jira/browse/HDFS-16789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17612228#comment-17612228 ]
Steve Loughran commented on HDFS-16789:
---------------------------------------
bq. retrieve the disk usage value directly from file's length instead of actual usage value.
bq. We need to update FileSystem & related classes to read disk usage from actual usage value.
what do you mean here? number of blocks consumed on hdd? as that info isn't available to most filesystems.
GetContentSummary is used in places in hive and spark to estimate workload; for it the summary is exactly what's needed. its unfortunate there that it is so inefficient on cloud storage (single thread treewalk). HADOOP-13704 did some speedup for s3a, but it is still an expensive operation. and the other clouds are still doing that treewalk.
# hive/spark should stop using it
# a multithreaded implementation would be nice
> Update FileSystem class to read disk usage from actual usage value instead of file's length
> -------------------------------------------------------------------------------------------
>
> Key: HDFS-16789
> URL: https://issues.apache.org/jira/browse/HDFS-16789
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: dfs
> Reporter: Dave Teng
> Assignee: Dave Teng
> Priority: Major
>
> Currently FileSystem class retrieve the disk usage value directly from [file's length|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L1895] instead of actual usage value.
> We need to update FileSystem & related classes to read disk usage from actual usage value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org