You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/03/19 21:48:00 UTC
[jira] [Work logged] (GOBBLIN-968) Honor file split size for
HadoopFileInputSource
[ https://issues.apache.org/jira/browse/GOBBLIN-968?focusedWorklogId=406613&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406613 ]
ASF GitHub Bot logged work on GOBBLIN-968:
------------------------------------------
Author: ASF GitHub Bot
Created on: 19/Mar/20 21:47
Start Date: 19/Mar/20 21:47
Worklog Time Spent: 10m
Work Description: jhsenjaliya commented on issue #2816: [GOBBLIN-968] Honor file split size for HadoopFileInputSource
URL: https://github.com/apache/incubator-gobblin/pull/2816#issuecomment-601432412
LGTM, @sv2000 can u pls take a look?
Unless there is a common framework to pass on such MR parameters ( or rather execution engine parameters) , we have to keep taking these configs as part of the gobblin config.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 406613)
Remaining Estimate: 23.5h (was: 23h 40m)
Time Spent: 0.5h (was: 20m)
> Honor file split size for HadoopFileInputSource
> -----------------------------------------------
>
> Key: GOBBLIN-968
> URL: https://issues.apache.org/jira/browse/GOBBLIN-968
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-core
> Affects Versions: 0.14.0
> Reporter: Chirag Todarka
> Assignee: Abhishek Tiwari
> Priority: Major
> Labels: easyfix
> Original Estimate: 24h
> Time Spent: 0.5h
> Remaining Estimate: 23.5h
>
> For HadoopFileInputSource it is splitting the source file only in default chunk size (which is 128MB) and not honoring the min and mix split size mentioned in pull file.
> For example even if below entires are present in pull file it will still chunk the source file in smaller 134217728 bytes files and not in 268435456 bytes files.
> {{source.hadoop.file.input.split.minsize=268435456}}
> {{source.hadoop.file.input.split.maxsize=268435456}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)