[jira] [Created] (OOZIE-3605) ShareLib installation does not honor dfs.replication in HDFS configuration

["Yuanhao Lu (Jira)" <ji...@apache.org>]

Yuanhao Lu created OOZIE-3605:
---------------------------------

             Summary: ShareLib installation does not honor dfs.replication in HDFS configuration
                 Key: OOZIE-3605
                 URL: https://issues.apache.org/jira/browse/OOZIE-3605
             Project: Oozie
          Issue Type: Bug
    Affects Versions: 5.2.0, 5.1.0
            Reporter: Yuanhao Lu


The change in https://issues.apache.org/jira/browse/OOZIE-2791 hardcoded replication factor to be 3 when using multi-threaded copying while single-threaded copy will follow the replication factor in hdfs-site.xml.

[https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L391]

[https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L306]

This could be problematic when a cluster has less than 3 data nodes. Since the replication can never be 3 in this case, hdfs report will show a lot of files are under replicated.
{code:java}
$  hdfs dfsadmin -report | head
Configured Capacity: 148067303424 (137.90 GB)
Present Capacity: 147612037120 (137.47 GB)
DFS Remaining: 145914187776 (135.89 GB)
DFS Used: 1697849344 (1.58 GB)
DFS Used%: 1.15%
Under replicated blocks: 1003
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Pending deletion blocks: 0
{code}
And the message from hdfs fsck will be like
{code:java}
/user/oozie/share/lib/lib_20200707223334/git/commons-codec-1.10.jar:  Under replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742826_2002. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
/user/oozie/share/lib/lib_20200707223334/git/commons-lang3-3.3.2.jar:  Under replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742810_1986. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
/user/oozie/share/lib/lib_20200707223334/git/httpclient-4.5.9.jar:  Under replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742815_1991. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)