You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Chirag Todarka (Jira)" <ji...@apache.org> on 2019/11/15 22:11:00 UTC

[jira] [Created] (GOBBLIN-968) Honor file split size for HadoopFileInputSource

Chirag Todarka created GOBBLIN-968:
--------------------------------------

             Summary: Honor file split size for HadoopFileInputSource
                 Key: GOBBLIN-968
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-968
             Project: Apache Gobblin
          Issue Type: Bug
          Components: gobblin-core
    Affects Versions: 0.14.0
            Reporter: Chirag Todarka
            Assignee: Abhishek Tiwari


For HadoopFileInputSource it is splitting the source file only in default chunk size (which is 128MB) and not honoring the min and mix split size mentioned in pull file.

For example even if below entires are present in pull file it will still chunk the source file in smaller 134217728 bytes files and not in 268435456 bytes files.

{{source.hadoop.file.input.split.minsize=268435456}}
{{source.hadoop.file.input.split.maxsize=268435456}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)