You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/10/26 16:35:00 UTC
[jira] [Work logged] (GOBBLIN-968) Honor file split size for
HadoopFileInputSource
[ https://issues.apache.org/jira/browse/GOBBLIN-968?focusedWorklogId=504796&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-504796 ]
ASF GitHub Bot logged work on GOBBLIN-968:
------------------------------------------
Author: ASF GitHub Bot
Created on: 26/Oct/20 16:34
Start Date: 26/Oct/20 16:34
Worklog Time Spent: 10m
Work Description: codecov-io edited a comment on pull request #2816:
URL: https://github.com/apache/incubator-gobblin/pull/2816#issuecomment-555263057
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2816?src=pr&el=h1) Report
> Merging [#2816](https://codecov.io/gh/apache/incubator-gobblin/pull/2816?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/a714c943bddfe49c19830444434aa64501294b2d?el=desc) will **increase** coverage by `0.07%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/2816?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2816 +/- ##
============================================
+ Coverage 45.47% 45.55% +0.07%
- Complexity 8908 8950 +42
============================================
Files 1899 1900 +1
Lines 71096 71217 +121
Branches 7833 7858 +25
============================================
+ Hits 32332 32440 +108
- Misses 35784 35790 +6
- Partials 2980 2987 +7
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2816?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...source/extractor/hadoop/HadoopFileInputSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2816/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvaGFkb29wL0hhZG9vcEZpbGVJbnB1dFNvdXJjZS5qYXZh) | `59.32% <0.00%> (-4.32%)` | `6.00 <0.00> (ø)` | |
| [...ava/org/apache/gobblin/fsm/FiniteStateMachine.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2816/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ZzbS9GaW5pdGVTdGF0ZU1hY2hpbmUuamF2YQ==) | `73.48% <0.00%> (-3.04%)` | `18.00% <0.00%> (-3.00%)` | |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2816/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `32.71% <0.00%> (-2.81%)` | `11.00% <0.00%> (-1.00%)` | |
| [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2816/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `70.00% <0.00%> (-2.23%)` | `13.00% <0.00%> (ø%)` | |
| [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2816/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `63.42% <0.00%> (-1.86%)` | `27.00% <0.00%> (-1.00%)` | |
| [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2816/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0.00%> (-0.94%)` | `24.00% <0.00%> (ø%)` | |
| [...ement/copy/replication/SourceHadoopFsEndPoint.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2816/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcmVwbGljYXRpb24vU291cmNlSGFkb29wRnNFbmRQb2ludC5qYXZh) | `19.04% <0.00%> (-0.63%)` | `6.00% <0.00%> (ø%)` | |
| [...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2816/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==) | `30.20% <0.00%> (-0.34%)` | `24.00% <0.00%> (ø%)` | |
| [...n/java/org/apache/gobblin/runtime/api/JobSpec.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2816/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYlNwZWMuamF2YQ==) | `80.13% <0.00%> (-0.01%)` | `17.00% <0.00%> (ø%)` | |
| [...verter/avro/AvroRecursionEliminatingConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2816/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9hdnJvL0F2cm9SZWN1cnNpb25FbGltaW5hdGluZ0NvbnZlcnRlci5qYXZh) | `75.00% <0.00%> (ø)` | `5.00% <0.00%> (?%)` | |
| ... and [9 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2816/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2816?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2816?src=pr&el=footer). Last update [a714c94...993efd9](https://codecov.io/gh/apache/incubator-gobblin/pull/2816?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 504796)
Remaining Estimate: 23h 20m (was: 23.5h)
Time Spent: 40m (was: 0.5h)
> Honor file split size for HadoopFileInputSource
> -----------------------------------------------
>
> Key: GOBBLIN-968
> URL: https://issues.apache.org/jira/browse/GOBBLIN-968
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-core
> Affects Versions: 0.14.0
> Reporter: Chirag Todarka
> Assignee: Abhishek Tiwari
> Priority: Major
> Labels: easyfix
> Original Estimate: 24h
> Time Spent: 40m
> Remaining Estimate: 23h 20m
>
> For HadoopFileInputSource it is splitting the source file only in default chunk size (which is 128MB) and not honoring the min and mix split size mentioned in pull file.
> For example even if below entires are present in pull file it will still chunk the source file in smaller 134217728 bytes files and not in 268435456 bytes files.
> {{source.hadoop.file.input.split.minsize=268435456}}
> {{source.hadoop.file.input.split.maxsize=268435456}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)