[jira] [Work logged] (CRUNCH-683) Avoid unnecessary listStatus calls from getSize computation

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CRUNCH-683?focusedWorklogId=246241&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246241 ]

ASF GitHub Bot logged work on CRUNCH-683:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 21/May/19 17:31
            Start Date: 21/May/19 17:31
    Worklog Time Spent: 10m 
      Work Description: ben-roling commented on pull request #26: CRUNCH-683 avoid unnecessary listStatus() calls from getPathSize()
URL: https://github.com/apache/crunch/pull/26
 
 
   Since @jonhemphill has been tied up in other things and hasn't been able to complete #23, I'm throwing this out to try to wrap it up.
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 246241)
    Time Spent: 40m  (was: 0.5h)

> Avoid unnecessary listStatus calls from getSize computation
> -----------------------------------------------------------
>
>                 Key: CRUNCH-683
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-683
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.14.0
>            Reporter: Jon Hemphill
>            Assignee: Josh Wills
>            Priority: Major
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> The getPathSize computation in SourceTargetHelper currently makes unnecessary listStatus calls when recursing over a directory, which can cause performance issues when the filesystem is an object store such as S3. The performance can be improved with the addition of a private method to use for the getPathSize recursion that takes a known FIleStatus object as a parameter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)