You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Chirag Todarka (Jira)" <ji...@apache.org> on 2019/11/15 22:11:00 UTC
[jira] [Created] (GOBBLIN-968) Honor file split size for
HadoopFileInputSource
Chirag Todarka created GOBBLIN-968:
--------------------------------------
Summary: Honor file split size for HadoopFileInputSource
Key: GOBBLIN-968
URL: https://issues.apache.org/jira/browse/GOBBLIN-968
Project: Apache Gobblin
Issue Type: Bug
Components: gobblin-core
Affects Versions: 0.14.0
Reporter: Chirag Todarka
Assignee: Abhishek Tiwari
For HadoopFileInputSource it is splitting the source file only in default chunk size (which is 128MB) and not honoring the min and mix split size mentioned in pull file.
For example even if below entires are present in pull file it will still chunk the source file in smaller 134217728 bytes files and not in 268435456 bytes files.
{{source.hadoop.file.input.split.minsize=268435456}}
{{source.hadoop.file.input.split.maxsize=268435456}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)