You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Yuanhao Lu (Jira)" <ji...@apache.org> on 2020/08/08 06:18:00 UTC
[jira] [Commented] (OOZIE-3605) ShareLib installation does not
honor dfs.replication in HDFS configuration
[ https://issues.apache.org/jira/browse/OOZIE-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173589#comment-17173589 ]
Yuanhao Lu commented on OOZIE-3605:
-----------------------------------
The "[15] new bugs found" seems not relevant to the change. Could anyone please take a look?
> ShareLib installation does not honor dfs.replication in HDFS configuration
> --------------------------------------------------------------------------
>
> Key: OOZIE-3605
> URL: https://issues.apache.org/jira/browse/OOZIE-3605
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 5.1.0, 5.2.0
> Reporter: Yuanhao Lu
> Priority: Major
> Attachments: OOZIE-3605-001.patch
>
>
> The change in https://issues.apache.org/jira/browse/OOZIE-2791 hardcoded replication factor to be 3 when using multi-threaded copying while single-threaded copy will follow the replication factor in hdfs-site.xml.
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L391]
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L306]
> This could be problematic when a cluster has less than 3 data nodes. Since the replication can never be 3 in this case, hdfs report will show a lot of files are under replicated.
> {code:java}
> $ hdfs dfsadmin -report | head
> Configured Capacity: 148067303424 (137.90 GB)
> Present Capacity: 147612037120 (137.47 GB)
> DFS Remaining: 145914187776 (135.89 GB)
> DFS Used: 1697849344 (1.58 GB)
> DFS Used%: 1.15%
> Under replicated blocks: 1003
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with replication factor 1): 0
> Pending deletion blocks: 0
> {code}
> And the message from hdfs fsck will be like
> {code:java}
> /user/oozie/share/lib/lib_20200707223334/git/commons-codec-1.10.jar: Under replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742826_2002. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/commons-lang3-3.3.2.jar: Under replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742810_1986. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/httpclient-4.5.9.jar: Under replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742815_1991. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)