You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2020/06/01 21:50:44 UTC
[GitHub] [incubator-gobblin] zxcware opened a new pull request #3019: [GOBBLIN-1174] Fail job on FileBasedSource ls invalid source directory
zxcware opened a new pull request #3019:
URL: https://github.com/apache/incubator-gobblin/pull/3019
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title.
- https://issues.apache.org/jira/browse/GOBBLIN-1174
### Description
- [x] Here are some details about my PR:
- Today, a gobblin job that uses a FileBasedSource will succeed on an invalid `ConfigurationKeys.SOURCE_FILEBASED_DATA_DIRECTORY`, for example the path doesn't exist. In this case, the job should fail to signal wrong configuration.
### Tests
- [x] My PR adds the following unit tests:
- `FileBasedSourceTest.testFailOnInvalidSourceDirectory` verifies that a `RuntimeException` was thrown with an invalid path
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] codecov-commenter edited a comment on pull request #3019: [GOBBLIN-1174] Fail job on FileBasedSource ls invalid source directory
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3019:
URL: https://github.com/apache/incubator-gobblin/pull/3019#issuecomment-637825508
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@3f84a01`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `55.55%`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #3019 +/- ##
=========================================
Coverage ? 45.74%
Complexity ? 9297
=========================================
Files ? 1951
Lines ? 74246
Branches ? 8215
=========================================
Hits ? 33961
Misses ? 37114
Partials ? 3171
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...in/source/extractor/filebased/FileBasedSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZmlsZWJhc2VkL0ZpbGVCYXNlZFNvdXJjZS5qYXZh) | `54.13% <0.00%> (ø)` | `13.00 <0.00> (?)` | |
| [...obblin/source/extractor/hadoop/AvroFileSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvaGFkb29wL0F2cm9GaWxlU291cmNlLmphdmE=) | `94.11% <100.00%> (ø)` | `4.00 <1.00> (?)` | |
| [.../util/recordcount/LateFileRecordCountProvider.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvcmVjb3JkY291bnQvTGF0ZUZpbGVSZWNvcmRDb3VudFByb3ZpZGVyLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...config/store/api/ConfigStoreCreationException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb25maWctbWFuYWdlbWVudC9nb2JibGluLWNvbmZpZy1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbmZpZy9zdG9yZS9hcGkvQ29uZmlnU3RvcmVDcmVhdGlvbkV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [.../org/apache/gobblin/service/ServiceConfigKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9TZXJ2aWNlQ29uZmlnS2V5cy5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...agement/copy/SubsetFilesCopyableDatasetFinder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvU3Vic2V0RmlsZXNDb3B5YWJsZURhdGFzZXRGaW5kZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...a/org/apache/gobblin/cluster/HelixTaskFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhUYXNrRmFjdG9yeS5qYXZh) | `58.82% <0.00%> (ø)` | `3.00% <0.00%> (?%)` | |
| [...rg/apache/gobblin/dataset/NoopDatasetResolver.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YXNldC9Ob29wRGF0YXNldFJlc29sdmVyLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...blin/converter/filter/AvroProjectionConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29udmVydGVyL2ZpbHRlci9BdnJvUHJvamVjdGlvbkNvbnZlcnRlci5qYXZh) | `77.77% <0.00%> (ø)` | `6.00% <0.00%> (?%)` | |
| [...obblin/metastore/JobHistoryDataSourceProvider.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL0pvYkhpc3RvcnlEYXRhU291cmNlUHJvdmlkZXIuamF2YQ==) | `44.44% <0.00%> (ø)` | `1.00% <0.00%> (?%)` | |
| ... and [1943 more](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=footer). Last update [3f84a01...7a828c3](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] codecov-commenter edited a comment on pull request #3019: [GOBBLIN-1174] Fail job on FileBasedSource ls invalid source directory
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3019:
URL: https://github.com/apache/incubator-gobblin/pull/3019#issuecomment-637825508
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@3f84a01`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `60.00%`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #3019 +/- ##
=========================================
Coverage ? 45.67%
Complexity ? 9295
=========================================
Files ? 1956
Lines ? 74394
Branches ? 8248
=========================================
Hits ? 33976
Misses ? 37242
Partials ? 3176
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...in/source/extractor/filebased/FileBasedSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZmlsZWJhc2VkL0ZpbGVCYXNlZFNvdXJjZS5qYXZh) | `54.13% <0.00%> (ø)` | `13.00 <0.00> (?)` | |
| [...obblin/source/extractor/hadoop/AvroFileSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvaGFkb29wL0F2cm9GaWxlU291cmNlLmphdmE=) | `93.75% <100.00%> (ø)` | `4.00 <1.00> (?)` | |
| [...g/apache/gobblin/cluster/SingleTaskRunnerMain.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFza1J1bm5lck1haW4uamF2YQ==) | `63.63% <0.00%> (ø)` | `3.00% <0.00%> (?%)` | |
| [...va/org/apache/gobblin/publisher/JdbcPublisher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tc3FsL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3B1Ymxpc2hlci9KZGJjUHVibGlzaGVyLmphdmE=) | `71.79% <0.00%> (ø)` | `12.00% <0.00%> (?%)` | |
| [...extractor/extract/google/GoogleDriveExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvb2dsZS1pbmdlc3Rpb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc291cmNlL2V4dHJhY3Rvci9leHRyYWN0L2dvb2dsZS9Hb29nbGVEcml2ZUV4dHJhY3Rvci5qYXZh) | `100.00% <0.00%> (ø)` | `2.00% <0.00%> (?%)` | |
| [...in/instrumented/fork/InstrumentedForkOperator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vaW5zdHJ1bWVudGVkL2ZvcmsvSW5zdHJ1bWVudGVkRm9ya09wZXJhdG9yLmphdmE=) | `100.00% <0.00%> (ø)` | `2.00% <0.00%> (?%)` | |
| [...obblin/metrics/influxdb/InfluxDBEventReporter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tbWV0cmljcy1pbmZsdXhkYi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9tZXRyaWNzL2luZmx1eGRiL0luZmx1eERCRXZlbnRSZXBvcnRlci5qYXZh) | `66.66% <0.00%> (ø)` | `10.00% <0.00%> (?%)` | |
| [...obblin/compaction/parser/CompactionPathParser.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vcGFyc2VyL0NvbXBhY3Rpb25QYXRoUGFyc2VyLmphdmE=) | `92.30% <0.00%> (ø)` | `9.00% <0.00%> (?%)` | |
| [...earch/typemapping/AvroGenericRecordTypeMapper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3R5cGVtYXBwaW5nL0F2cm9HZW5lcmljUmVjb3JkVHlwZU1hcHBlci5qYXZh) | `73.33% <0.00%> (ø)` | `5.00% <0.00%> (?%)` | |
| [...n/runtime/scheduler/JobSpecSchedulerListeners.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc2NoZWR1bGVyL0pvYlNwZWNTY2hlZHVsZXJMaXN0ZW5lcnMuamF2YQ==) | `54.54% <0.00%> (ø)` | `7.00% <0.00%> (?%)` | |
| ... and [1948 more](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=footer). Last update [3f84a01...c8abdc9](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] codecov-commenter commented on pull request #3019: [GOBBLIN-1174] Fail job on FileBasedSource ls invalid source directory
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #3019:
URL: https://github.com/apache/incubator-gobblin/pull/3019#issuecomment-637825508
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@3f84a01`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #3019 +/- ##
========================================
Coverage ? 9.28%
Complexity ? 1697
========================================
Files ? 1951
Lines ? 74246
Branches ? 8215
========================================
Hits ? 6896
Misses ? 66677
Partials ? 673
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...in/source/extractor/filebased/FileBasedSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZmlsZWJhc2VkL0ZpbGVCYXNlZFNvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...obblin/source/extractor/hadoop/AvroFileSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvaGFkb29wL0F2cm9GaWxlU291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...pache/gobblin/runtime/mapreduce/MRTaskFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWFwcmVkdWNlL01SVGFza0ZhY3RvcnkuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...n/java/org/apache/gobblin/stream/StreamEntity.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL1N0cmVhbUVudGl0eS5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...n/runtime/util/MultiWorkUnitUnpackingIterator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdXRpbC9NdWx0aVdvcmtVbml0VW5wYWNraW5nSXRlcmF0b3IuamF2YQ==) | `61.53% <0.00%> (ø)` | `8.00% <0.00%> (?%)` | |
| [...management/copy/TimestampBasedCopyableDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvVGltZXN0YW1wQmFzZWRDb3B5YWJsZURhdGFzZXQuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...iter/partitioner/SchemaBasedWriterPartitioner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3dyaXRlci9wYXJ0aXRpb25lci9TY2hlbWFCYXNlZFdyaXRlclBhcnRpdGlvbmVyLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...lin/kafka/writer/KafkaWriterConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2thZmthL3dyaXRlci9LYWZrYVdyaXRlckNvbmZpZ3VyYXRpb25LZXlzLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...n/version/finder/DateTimeDatasetVersionFinder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L3JldGVudGlvbi92ZXJzaW9uL2ZpbmRlci9EYXRlVGltZURhdGFzZXRWZXJzaW9uRmluZGVyLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| ... and [1943 more](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=footer). Last update [3f84a01...7a828c3](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] ZihanLi58 removed a comment on pull request #3019: [GOBBLIN-1174] Fail job on FileBasedSource ls invalid source directory
Posted by GitBox <gi...@apache.org>.
ZihanLi58 removed a comment on pull request #3019:
URL: https://github.com/apache/incubator-gobblin/pull/3019#issuecomment-637878683
+1 LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] ZihanLi58 commented on pull request #3019: [GOBBLIN-1174] Fail job on FileBasedSource ls invalid source directory
Posted by GitBox <gi...@apache.org>.
ZihanLi58 commented on pull request #3019:
URL: https://github.com/apache/incubator-gobblin/pull/3019#issuecomment-637878683
+1 LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] yukuai518 commented on a change in pull request #3019: [GOBBLIN-1174] Fail job on FileBasedSource ls invalid source directory
Posted by GitBox <gi...@apache.org>.
yukuai518 commented on a change in pull request #3019:
URL: https://github.com/apache/incubator-gobblin/pull/3019#discussion_r434785957
##########
File path: gobblin-core/src/main/java/org/apache/gobblin/source/extractor/hadoop/AvroFileSource.java
##########
@@ -28,13 +28,16 @@
import com.google.common.collect.Lists;
+import lombok.extern.slf4j.Slf4j;
+
import org.apache.gobblin.configuration.ConfigurationKeys;
import org.apache.gobblin.configuration.State;
import org.apache.gobblin.configuration.WorkUnitState;
import org.apache.gobblin.source.extractor.Extractor;
import org.apache.gobblin.source.extractor.filebased.FileBasedSource;
+@Slf4j
public class AvroFileSource extends FileBasedSource<Schema, GenericRecord> {
private static final Logger LOGGER = LoggerFactory.getLogger(AvroFileSource.class);
Review comment:
Can we clean up the LOGGER here if we are using the Slf4j wrapper?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] asfgit closed pull request #3019: [GOBBLIN-1174] Fail job on FileBasedSource ls invalid source directory
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3019:
URL: https://github.com/apache/incubator-gobblin/pull/3019
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] codecov-commenter edited a comment on pull request #3019: [GOBBLIN-1174] Fail job on FileBasedSource ls invalid source directory
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3019:
URL: https://github.com/apache/incubator-gobblin/pull/3019#issuecomment-637825508
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@3f84a01`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `60.00%`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #3019 +/- ##
=========================================
Coverage ? 45.67%
Complexity ? 9295
=========================================
Files ? 1956
Lines ? 74394
Branches ? 8248
=========================================
Hits ? 33976
Misses ? 37242
Partials ? 3176
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...in/source/extractor/filebased/FileBasedSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZmlsZWJhc2VkL0ZpbGVCYXNlZFNvdXJjZS5qYXZh) | `54.13% <0.00%> (ø)` | `13.00 <0.00> (?)` | |
| [...obblin/source/extractor/hadoop/AvroFileSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvaGFkb29wL0F2cm9GaWxlU291cmNlLmphdmE=) | `93.75% <100.00%> (ø)` | `4.00 <1.00> (?)` | |
| [.../org/apache/gobblin/runtime/CountBasedLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQ291bnRCYXNlZExpbWl0ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [.../modules/flowgraph/FlowGraphConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvRmxvd0dyYXBoQ29uZmlndXJhdGlvbktleXMuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...he/gobblin/kafka/client/Kafka08ConsumerClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMDgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4va2Fma2EvY2xpZW50L0thZmthMDhDb25zdW1lckNsaWVudC5qYXZh) | `55.69% <0.00%> (ø)` | `22.00% <0.00%> (?%)` | |
| [...ervice/modules/orchestration/AzkabanJobHelper.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9zZXJ2aWNlL21vZHVsZXMvb3JjaGVzdHJhdGlvbi9BemthYmFuSm9iSGVscGVyLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...rc/main/java/gobblin/source/workunit/WorkUnit.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9nb2JibGluL3NvdXJjZS93b3JrdW5pdC9Xb3JrVW5pdC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...g/apache/gobblin/runtime/JobShutdownException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vcnVudGltZS9Kb2JTaHV0ZG93bkV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| [...alitychecker/row/RowLevelPolicyCheckerBuilder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3F1YWxpdHljaGVja2VyL3Jvdy9Sb3dMZXZlbFBvbGljeUNoZWNrZXJCdWlsZGVyLmphdmE=) | `80.00% <0.00%> (ø)` | `8.00% <0.00%> (?%)` | |
| [...ractor/extract/sftp/SftpLightWeightFileSystem.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9zZnRwL1NmdHBMaWdodFdlaWdodEZpbGVTeXN0ZW0uamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | |
| ... and [1948 more](https://codecov.io/gh/apache/incubator-gobblin/pull/3019/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=footer). Last update [3f84a01...c8abdc9](https://codecov.io/gh/apache/incubator-gobblin/pull/3019?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] yukuai518 commented on pull request #3019: [GOBBLIN-1174] Fail job on FileBasedSource ls invalid source directory
Posted by GitBox <gi...@apache.org>.
yukuai518 commented on pull request #3019:
URL: https://github.com/apache/incubator-gobblin/pull/3019#issuecomment-639120136
+1 LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #3019: [GOBBLIN-1174] Fail job on FileBasedSource ls invalid source directory
Posted by GitBox <gi...@apache.org>.
zxcware commented on a change in pull request #3019:
URL: https://github.com/apache/incubator-gobblin/pull/3019#discussion_r435453152
##########
File path: gobblin-core/src/main/java/org/apache/gobblin/source/extractor/hadoop/AvroFileSource.java
##########
@@ -28,13 +28,16 @@
import com.google.common.collect.Lists;
+import lombok.extern.slf4j.Slf4j;
+
import org.apache.gobblin.configuration.ConfigurationKeys;
import org.apache.gobblin.configuration.State;
import org.apache.gobblin.configuration.WorkUnitState;
import org.apache.gobblin.source.extractor.Extractor;
import org.apache.gobblin.source.extractor.filebased.FileBasedSource;
+@Slf4j
public class AvroFileSource extends FileBasedSource<Schema, GenericRecord> {
private static final Logger LOGGER = LoggerFactory.getLogger(AvroFileSource.class);
Review comment:
nice catch! Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org