[jira] [Created] (OOZIE-2821) Using Hadoop Archives for Oozie ShareLib

["Attila Sasvari (JIRA)" <ji...@apache.org>]

Attila Sasvari created OOZIE-2821:
-------------------------------------

             Summary: Using Hadoop Archives for Oozie ShareLib
                 Key: OOZIE-2821
                 URL: https://issues.apache.org/jira/browse/OOZIE-2821
             Project: Oozie
          Issue Type: New Feature
            Reporter: Attila Sasvari


Oozie ShareLib is a collection of lots of jar files that are required by Oozie actions. Right now, these jars are uploaded one by one with Oozie ShareLib installation. There can more hundreds of such jars, and many of them are pretty small, significantly smaller than a HDFS block size. Storing a large number of small files in HDFS is inefficient (for example due to the fact that there is an object maintained for each file in the NameNode's memory and blocks containing the small files might be much bigger then the actual files). When an action is executed, these jar files are copied to the distributed cache.

It  would worth to investigate the possibility of using [Hadoop archives|http://hadoop.apache.org/docs/r2.6.5/hadoop-mapreduce-client/hadoop-mapreduce-client-core/HadoopArchives.html] for handling  Oozie ShareLib files, because it could result in better utilisation of HDFS. 





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)