[jira] [Updated] (PIG-4246) HBaseStorage should implement getShipFiles

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rohini Palaniswamy updated PIG-4246:
------------------------------------
    Attachment: PIG-4246-1.patch

> HBaseStorage should implement getShipFiles
> ------------------------------------------
>
>                 Key: PIG-4246
>                 URL: https://issues.apache.org/jira/browse/PIG-4246
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Rohini Palaniswamy
>             Fix For: 0.14.0
>
>         Attachments: PIG-4246-1.patch
>
>
> HBaseStorage.initializeHBaseClassLoaderResources() uses TableMapReduceUtil APIs to add dependency jars. That sets the tmpjars setting which makes JobClient ship the jars to hdfs and use that path in distributed cache. That bypasses the optimizations in PIG-2672 and PIG-3861 which avoid shipping the jars to hdfs. Instead it should implement the getShipFiles() API introduced in PIG-4141 so that PIG-2672 or PIG-3861 avoid shipping the same jar multiple times to hdfs for a job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)