You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Konstantin Shvachko (Jira)" <ji...@apache.org> on 2021/11/11 18:30:00 UTC
[jira] [Commented] (HADOOP-17999) ViewFileSystem.setVerifyChecksum
should not initialize all target filesystems
[ https://issues.apache.org/jira/browse/HADOOP-17999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442389#comment-17442389 ]
Konstantin Shvachko commented on HADOOP-17999:
----------------------------------------------
Was looking at the patch. Why do we need setting checksum in ViewFileSystem globally for all mount points?
Seems it should be a per-target-file-system parameter, which is set when the mount point is initialized from its own config.
> ViewFileSystem.setVerifyChecksum should not initialize all target filesystems
> -----------------------------------------------------------------------------
>
> Key: HADOOP-17999
> URL: https://issues.apache.org/jira/browse/HADOOP-17999
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Abhishek Das
> Assignee: Abhishek Das
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently setVerifyChecksum and setWriteChecksum initializes all target file systems which causes delay in hadoop shell copy commands such as get, put, copyFromLocal etc.
> This also eventually causes OOM.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org