[jira] [Updated] (CRUNCH-463) Copying the Configuration object in every CrunchInputSplit causes OOM errors for jobs with lots of splits

["Josh Wills (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CRUNCH-463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Wills updated CRUNCH-463:
------------------------------

    Attachment: CRUNCH-463.patch

Yeah, you're right-- we don't need to copy the Configuration object during initialization, it will already be configured correctly using the FormatBundle for the split.

> Copying the Configuration object in every CrunchInputSplit causes OOM errors for jobs with lots of splits
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: CRUNCH-463
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-463
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.10.0
>            Reporter: Hector Izquierdo Seliva
>            Assignee: Josh Wills
>         Attachments: CRUNCH-463.patch
>
>
> Trying to run a job with 11k input files and that yields about 25k splits results in OOM errors due to too many copies of the Configuration object being created when the CrunchInputSplit is initialised. I know that that's the result of CRUNCH-313, but perhaps a better way to deal with that problem should be found.



--
This message was sent by Atlassian JIRA
(v6.2#6252)