You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2021/08/20 00:54:31 UTC
[GitHub] [gobblin] ZihanLi58 opened a new pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
ZihanLi58 opened a new pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1521
### Description
- [ ] Here are some details about my PR, including screenshots (if applicable):
Using embedded gobblin to run streaming job, and create a guid to help user quickly onboard
* Change kafka09ConsumerClient to make get earliest and latest offset won't change the current position of the consumer, and support assignAndSeek() method
* Change FsStateStore rename method to enable overwrite so that we can write to existing state store
* Change reference in kafka-hdfs-streaming-avro.job to OSS class to give user better idea about how the config looks like if they want to ingest as avro, although we don't use that config in this local job since we don't have schema registry setup.
* change KafkaStreamingExtractor to support the scenario where schemaRegistry is not setup
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
Test we can run the job locally
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#issuecomment-902354878
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3372](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (58f3989) into [master](https://codecov.io/gh/apache/gobblin/commit/6691a37d863d35cb5d217355ff8667f29cdb03e8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6691a37) will **increase** coverage by `6.33%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3372 +/- ##
============================================
+ Coverage 42.63% 48.96% +6.33%
- Complexity 4575 8698 +4123
============================================
Files 1028 1676 +648
Lines 40878 64569 +23691
Branches 4549 7416 +2867
============================================
+ Hits 17427 31616 +14189
- Misses 21730 29958 +8228
- Partials 1721 2995 +1274
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/gobblin/metastore/FsStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL0ZzU3RhdGVTdG9yZS5qYXZh) | `36.05% <100.00%> (ø)` | |
| [.../source/extractor/extract/google/GoogleCommon.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvb2dsZS1pbmdlc3Rpb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc291cmNlL2V4dHJhY3Rvci9leHRyYWN0L2dvb2dsZS9Hb29nbGVDb21tb24uamF2YQ==) | | |
| [...pache/gobblin/service/AvroJobSpecDeserializer.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMDgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9BdnJvSm9iU3BlY0Rlc2VyaWFsaXplci5qYXZh) | | |
| [...che/gobblin/kafka/schemareg/HttpClientFactory.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2thZmthL3NjaGVtYXJlZy9IdHRwQ2xpZW50RmFjdG9yeS5qYXZh) | | |
| [...gobblin/compliance/purger/HivePurgerPublisher.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY29tcGxpYW5jZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9jb21wbGlhbmNlL3B1cmdlci9IaXZlUHVyZ2VyUHVibGlzaGVyLmphdmE=) | | |
| [...pache/gobblin/metrics/kafka/ProducerCloseable.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMDgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9rYWZrYS9Qcm9kdWNlckNsb3NlYWJsZS5qYXZh) | | |
| [...pache/gobblin/converter/AvroHttpJoinConverter.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taHR0cC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9jb252ZXJ0ZXIvQXZyb0h0dHBKb2luQ29udmVydGVyLmphdmE=) | | |
| [...blin/metrics/kafka/KafkaMetricReporterFactory.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvS2Fma2FNZXRyaWNSZXBvcnRlckZhY3RvcnkuamF2YQ==) | | |
| [...zedRecordToEncryptedSerializedRecordConverter.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY3J5cHRvLXByb3ZpZGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9TZXJpYWxpemVkUmVjb3JkVG9FbmNyeXB0ZWRTZXJpYWxpemVkUmVjb3JkQ29udmVydGVyLmphdmE=) | | |
| [.../converter/EnvelopePayloadExtractingConverter.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9FbnZlbG9wZVBheWxvYWRFeHRyYWN0aW5nQ29udmVydGVyLmphdmE=) | | |
| ... and [1398 more](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6691a37...58f3989](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#issuecomment-902354878
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3372](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (58f3989) into [master](https://codecov.io/gh/apache/gobblin/commit/6691a37d863d35cb5d217355ff8667f29cdb03e8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6691a37) will **increase** coverage by `4.12%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3372 +/- ##
============================================
+ Coverage 42.63% 46.75% +4.12%
+ Complexity 4575 3117 -1458
============================================
Files 1028 648 -380
Lines 40878 25368 -15510
Branches 4549 3018 -1531
============================================
- Hits 17427 11861 -5566
+ Misses 21730 12240 -9490
+ Partials 1721 1267 -454
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/gobblin/metastore/FsStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL0ZzU3RhdGVTdG9yZS5qYXZh) | `36.05% <100.00%> (ø)` | |
| [...e/gobblin/crypto/JCEKSKeystoreCredentialStore.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY3J5cHRvL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NyeXB0by9KQ0VLU0tleXN0b3JlQ3JlZGVudGlhbFN0b3JlLmphdmE=) | | |
| [.../extract/kafka/StartOffsetOutOfRangeException.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9TdGFydE9mZnNldE91dE9mUmFuZ2VFeGNlcHRpb24uamF2YQ==) | | |
| [...g/apache/gobblin/azkaban/AzkabanGobblinDaemon.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Hb2JibGluRGFlbW9uLmphdmE=) | | |
| [...ource/extractor/extract/jdbc/PostgresqlSource.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tc3FsL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9qZGJjL1Bvc3RncmVzcWxTb3VyY2UuamF2YQ==) | | |
| [...org/apache/gobblin/writer/R2RestWriterBuilder.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taHR0cC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi93cml0ZXIvUjJSZXN0V3JpdGVyQnVpbGRlci5qYXZh) | | |
| [...kunit/packer/KafkaTopicGroupingWorkUnitPacker.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS93b3JrdW5pdC9wYWNrZXIvS2Fma2FUb3BpY0dyb3VwaW5nV29ya1VuaXRQYWNrZXIuamF2YQ==) | | |
| [...a/org/apache/gobblin/writer/JdbcWriterBuilder.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tc3FsL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3dyaXRlci9KZGJjV3JpdGVyQnVpbGRlci5qYXZh) | | |
| [...rg/apache/gobblin/compliance/utils/ProxyUtils.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY29tcGxpYW5jZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9jb21wbGlhbmNlL3V0aWxzL1Byb3h5VXRpbHMuamF2YQ==) | | |
| [...hbase/writer/CouchbaseWriterConfigurationKeys.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyQ29uZmlndXJhdGlvbktleXMuamF2YQ==) | | |
| ... and [370 more](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6691a37...58f3989](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] ZihanLi58 commented on pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
ZihanLi58 commented on pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#issuecomment-902350732
@sv2000 @autumnust can you help take a look at the change when you have time? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#issuecomment-902354878
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3372](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (18951a8) into [master](https://codecov.io/gh/apache/gobblin/commit/6691a37d863d35cb5d217355ff8667f29cdb03e8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6691a37) will **increase** coverage by `0.41%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3372 +/- ##
============================================
+ Coverage 42.63% 43.05% +0.41%
+ Complexity 4575 1959 -2616
============================================
Files 1028 395 -633
Lines 40878 16952 -23926
Branches 4549 2090 -2459
============================================
- Hits 17427 7298 -10129
+ Misses 21730 8843 -12887
+ Partials 1721 811 -910
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `92.85% <0.00%> (-7.15%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `34.71% <0.00%> (-3.31%)` | :arrow_down: |
| [...ache/gobblin/cluster/GobblinHelixMultiManager.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TXVsdGlNYW5hZ2VyLmphdmE=) | `52.33% <0.00%> (-1.84%)` | :arrow_down: |
| [...gobblin/compliance/HivePartitionVersionPolicy.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY29tcGxpYW5jZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9jb21wbGlhbmNlL0hpdmVQYXJ0aXRpb25WZXJzaW9uUG9saWN5LmphdmE=) | | |
| [.../main/java/org/apache/gobblin/http/HttpClient.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taHR0cC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9odHRwL0h0dHBDbGllbnQuamF2YQ==) | | |
| [...pache/gobblin/metrics/graphite/GraphitePusher.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tbWV0cmljcy1ncmFwaGl0ZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9tZXRyaWNzL2dyYXBoaXRlL0dyYXBoaXRlUHVzaGVyLmphdmE=) | | |
| [...l/limiter/stressTest/RandomDelayStartStressor.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UtY2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9zdHJlc3NUZXN0L1JhbmRvbURlbGF5U3RhcnRTdHJlc3Nvci5qYXZh) | | |
| [...org/apache/gobblin/writer/R2RestWriterBuilder.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taHR0cC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi93cml0ZXIvUjJSZXN0V3JpdGVyQnVpbGRlci5qYXZh) | | |
| [...obblin/util/limiter/RestliServiceBasedLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UtY2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9SZXN0bGlTZXJ2aWNlQmFzZWRMaW1pdGVyLmphdmE=) | | |
| ... and [627 more](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6691a37...18951a8](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] ZihanLi58 commented on a change in pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
ZihanLi58 commented on a change in pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#discussion_r695313863
##########
File path: gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaStreamingExtractor.java
##########
@@ -323,8 +323,10 @@ public KafkaStreamingExtractor(WorkUnitState state) {
@Override
public S getSchema() {
try {
- Schema schema = (Schema) this._schemaRegistry.get().getLatestSchemaByTopic(this.topicPartitions.get(0).getTopicName());
- return (S) schema;
+ if(this._schemaRegistry.isPresent()) {
+ return (S)(Schema) this._schemaRegistry.get().getLatestSchemaByTopic(this.topicPartitions.get(0).getTopicName());
+ }
+ return (S) this.topicPartitions.iterator().next().getTopicName();
Review comment:
It's because many converter does not accept null in convertSchema() method. And simply return null will cause issue there. I just use the same behavior as KafkaSimpleStreamingExtractor where if the schema registry is null, we just return topic name as schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#issuecomment-902354878
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3372](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5f6fe3f) into [master](https://codecov.io/gh/apache/gobblin/commit/6691a37d863d35cb5d217355ff8667f29cdb03e8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6691a37) will **increase** coverage by `2.88%`.
> The diff coverage is `25.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3372 +/- ##
============================================
+ Coverage 42.63% 45.52% +2.88%
- Complexity 4575 9000 +4425
============================================
Files 1028 1803 +775
Lines 40878 71668 +30790
Branches 4549 8019 +3470
============================================
+ Hits 17427 32624 +15197
- Misses 21730 36052 +14322
- Partials 1721 2992 +1271
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | `58.93% <0.00%> (-0.23%)` | :arrow_down: |
| [...ava/org/apache/gobblin/metastore/FsStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL0ZzU3RhdGVTdG9yZS5qYXZh) | `36.05% <100.00%> (ø)` | |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0.00%> (-14.29%)` | :arrow_down: |
| [...ache/gobblin/cluster/GobblinHelixMultiManager.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TXVsdGlNYW5hZ2VyLmphdmE=) | `52.33% <0.00%> (-1.84%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/writer/Destination.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vd3JpdGVyL0Rlc3RpbmF0aW9uLmphdmE=) | | |
| [...rc/main/java/org/apache/gobblin/source/Source.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc291cmNlL1NvdXJjZS5qYXZh) | | |
| [...ava/org/apache/gobblin/broker/SimpleScopeType.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYnJva2VyL1NpbXBsZVNjb3BlVHlwZS5qYXZh) | | |
| [...a/org/apache/gobblin/runtime/api/SpecExecutor.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vcnVudGltZS9hcGkvU3BlY0V4ZWN1dG9yLmphdmE=) | | |
| [...va/org/apache/gobblin/dataset/DatasetResolver.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YXNldC9EYXRhc2V0UmVzb2x2ZXIuamF2YQ==) | | |
| ... and [1277 more](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6691a37...5f6fe3f](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter commented on pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#issuecomment-902354878
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3372](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (58f3989) into [master](https://codecov.io/gh/apache/gobblin/commit/6691a37d863d35cb5d217355ff8667f29cdb03e8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6691a37) will **increase** coverage by `0.46%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3372 +/- ##
============================================
+ Coverage 42.63% 43.09% +0.46%
+ Complexity 4575 1961 -2614
============================================
Files 1028 395 -633
Lines 40878 16951 -23927
Branches 4549 2089 -2460
============================================
- Hits 17427 7305 -10122
+ Misses 21730 8837 -12893
+ Partials 1721 809 -912
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/gobblin/metastore/FsStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL0ZzU3RhdGVTdG9yZS5qYXZh) | `36.05% <100.00%> (ø)` | |
| [...obblin/service/modules/flowgraph/BaseFlowEdge.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvQmFzZUZsb3dFZGdlLmphdmE=) | | |
| [...apache/gobblin/writer/FluentDataWriterBuilder.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vd3JpdGVyL0ZsdWVudERhdGFXcml0ZXJCdWlsZGVyLmphdmE=) | | |
| [.../src/main/java/org/apache/gobblin/ack/Ackable.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYWNrL0Fja2FibGUuamF2YQ==) | | |
| [.../src/main/java/org/apache/gobblin/fork/Forker.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZm9yay9Gb3JrZXIuamF2YQ==) | | |
| [...asticsearch/typemapping/FieldMappingException.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3R5cGVtYXBwaW5nL0ZpZWxkTWFwcGluZ0V4Y2VwdGlvbi5qYXZh) | | |
| [...ava/org/apache/gobblin/http/ApacheHttpRequest.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taHR0cC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9odHRwL0FwYWNoZUh0dHBSZXF1ZXN0LmphdmE=) | | |
| [...ache/gobblin/kafka/serialize/LiAvroSerializer.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9rYWZrYS9zZXJpYWxpemUvTGlBdnJvU2VyaWFsaXplci5qYXZh) | | |
| [.../gobblin/dataset/test/SimpleDatasetForTesting.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YXNldC90ZXN0L1NpbXBsZURhdGFzZXRGb3JUZXN0aW5nLmphdmE=) | | |
| [...obblin/service/validator/TemplateUriValidator.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS92YWxpZGF0b3IvVGVtcGxhdGVVcmlWYWxpZGF0b3IuamF2YQ==) | | |
| ... and [623 more](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6691a37...58f3989](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#issuecomment-902354878
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3372](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5f6fe3f) into [master](https://codecov.io/gh/apache/gobblin/commit/6691a37d863d35cb5d217355ff8667f29cdb03e8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6691a37) will **increase** coverage by `3.79%`.
> The diff coverage is `25.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3372 +/- ##
============================================
+ Coverage 42.63% 46.42% +3.79%
- Complexity 4575 10155 +5580
============================================
Files 1028 2056 +1028
Lines 40878 80085 +39207
Branches 4549 8948 +4399
============================================
+ Hits 17427 37183 +19756
- Misses 21730 39452 +17722
- Partials 1721 3450 +1729
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | `58.93% <0.00%> (-0.23%)` | :arrow_down: |
| [...ava/org/apache/gobblin/metastore/FsStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL0ZzU3RhdGVTdG9yZS5qYXZh) | `36.05% <100.00%> (ø)` | |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0.00%> (-14.29%)` | :arrow_down: |
| [...ache/gobblin/cluster/GobblinHelixMultiManager.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TXVsdGlNYW5hZ2VyLmphdmE=) | `52.33% <0.00%> (-1.84%)` | :arrow_down: |
| [...pache/gobblin/writer/DataWriterWrapperBuilder.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3dyaXRlci9EYXRhV3JpdGVyV3JhcHBlckJ1aWxkZXIuamF2YQ==) | `83.33% <0.00%> (ø)` | |
| [...management/partition/CopyableDatasetRequestor.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L3BhcnRpdGlvbi9Db3B5YWJsZURhdGFzZXRSZXF1ZXN0b3IuamF2YQ==) | `45.16% <0.00%> (ø)` | |
| [...version/hive/provider/HdfsBasedUpdateProvider.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9wcm92aWRlci9IZGZzQmFzZWRVcGRhdGVQcm92aWRlci5qYXZh) | `42.85% <0.00%> (ø)` | |
| [...nt/copy/writer/FileAwareInputStreamDataWriter.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvd3JpdGVyL0ZpbGVBd2FyZUlucHV0U3RyZWFtRGF0YVdyaXRlci5qYXZh) | `73.86% <0.00%> (ø)` | |
| [...bblin/metrics/event/EntityMissingEventBuilder.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9ldmVudC9FbnRpdHlNaXNzaW5nRXZlbnRCdWlsZGVyLmphdmE=) | `79.16% <0.00%> (ø)` | |
| ... and [1025 more](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6691a37...5f6fe3f](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#issuecomment-902354878
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3372](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (18951a8) into [master](https://codecov.io/gh/apache/gobblin/commit/6691a37d863d35cb5d217355ff8667f29cdb03e8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6691a37) will **increase** coverage by `3.79%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3372 +/- ##
============================================
+ Coverage 42.63% 46.43% +3.79%
- Complexity 4575 10156 +5581
============================================
Files 1028 2056 +1028
Lines 40878 80085 +39207
Branches 4549 8948 +4399
============================================
+ Hits 17427 37184 +19757
- Misses 21730 39452 +17722
- Partials 1721 3449 +1728
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | `58.93% <0.00%> (-0.23%)` | :arrow_down: |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `92.85% <0.00%> (-7.15%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `34.71% <0.00%> (-3.31%)` | :arrow_down: |
| [...ache/gobblin/cluster/GobblinHelixMultiManager.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TXVsdGlNYW5hZ2VyLmphdmE=) | `52.33% <0.00%> (-1.84%)` | :arrow_down: |
| [...pache/gobblin/converter/serde/OrcSerDeWrapper.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9zZXJkZS9PcmNTZXJEZVdyYXBwZXIuamF2YQ==) | `100.00% <0.00%> (ø)` | |
| [...a/org/apache/gobblin/runtime/util/MetricGroup.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdXRpbC9NZXRyaWNHcm91cC5qYXZh) | `100.00% <0.00%> (ø)` | |
| [...e/extractor/exception/ExtractPrepareException.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXhjZXB0aW9uL0V4dHJhY3RQcmVwYXJlRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...src/main/java/org/apache/gobblin/runtime/Task.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFzay5qYXZh) | `69.12% <0.00%> (ø)` | |
| [...ce/extractor/extract/restapi/RestApiExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9yZXN0YXBpL1Jlc3RBcGlFeHRyYWN0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| ... and [1026 more](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6691a37...18951a8](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] sv2000 commented on a change in pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#discussion_r695069713
##########
File path: gobblin-docs/user-guide/Run-Gobblin-Streaming-kafka-hdfs-Locally.md
##########
@@ -0,0 +1,30 @@
+# Table of Contents
+
+[TOC]
+
+# Introduction
+
+Gobblin streaming enable user to ingest data from Kafka to HDFS in streaming manner. Normally it's running on yarn, to get the full function,
+you need have helix cluster, yarn cluster, kafka server and schema registry set up. But we also have one local mode of streaming task using EmbeddedGobblin
+for user to easily start and test out the job. Here is the steps:
+
+# Setup local kafka cluster
+
+Just follow [kafka quick start](https://kafka.apache.org/quickstart) to set up your kafka cluster, and create test topic "testEvents"
+
+# Run EmbeddedGobblin to start the job
+
+We are using configuration: /gobblin-modules/gobblin-kafka-09/src/test/resources/kafkaHDFSStreaming.conf to execute the job.
Review comment:
"We are using" -> "We use the"
##########
File path: gobblin-metastore/src/main/java/org/apache/gobblin/metastore/FsStateStore.java
##########
@@ -217,7 +217,7 @@ public void putAll(String storeName, String tableName, Collection<T> states) thr
}
protected void renamePath(Path tmpTablePath, Path tablePath) throws IOException {
- HadoopUtils.renamePath(this.fs, tmpTablePath, tablePath);
+ HadoopUtils.renamePath(this.fs, tmpTablePath, tablePath, true);
Review comment:
We are changing the behavior of renamePath(). I am assuming you need this to overwrite the watermarks.
##########
File path: gobblin-docs/user-guide/Run-Gobblin-Streaming-kafka-hdfs-Locally.md
##########
@@ -0,0 +1,30 @@
+# Table of Contents
+
+[TOC]
+
+# Introduction
+
+Gobblin streaming enable user to ingest data from Kafka to HDFS in streaming manner. Normally it's running on yarn, to get the full function,
+you need have helix cluster, yarn cluster, kafka server and schema registry set up. But we also have one local mode of streaming task using EmbeddedGobblin
+for user to easily start and test out the job. Here is the steps:
+
+# Setup local kafka cluster
+
+Just follow [kafka quick start](https://kafka.apache.org/quickstart) to set up your kafka cluster, and create test topic "testEvents"
Review comment:
Drop "Just".
##########
File path: gobblin-docs/user-guide/Run-Gobblin-Streaming-kafka-hdfs-Locally.md
##########
@@ -0,0 +1,30 @@
+# Table of Contents
+
+[TOC]
+
+# Introduction
+
+Gobblin streaming enable user to ingest data from Kafka to HDFS in streaming manner. Normally it's running on yarn, to get the full function,
+you need have helix cluster, yarn cluster, kafka server and schema registry set up. But we also have one local mode of streaming task using EmbeddedGobblin
+for user to easily start and test out the job. Here is the steps:
+
+# Setup local kafka cluster
+
+Just follow [kafka quick start](https://kafka.apache.org/quickstart) to set up your kafka cluster, and create test topic "testEvents"
+
+# Run EmbeddedGobblin to start the job
+
+We are using configuration: /gobblin-modules/gobblin-kafka-09/src/test/resources/kafkaHDFSStreaming.conf to execute the job.
+
+To run the job, in your intellij, you can run the test in /gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/kafka/KafkaStreamingLocalTest
+by remove the line '(enabled=false)'. In order to run the test in IDE, you may need to manually delete log4j-over-slf4j jar in IDE
Review comment:
remove -> removing.
##########
File path: gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaStreamingExtractor.java
##########
@@ -323,8 +323,10 @@ public KafkaStreamingExtractor(WorkUnitState state) {
@Override
public S getSchema() {
try {
- Schema schema = (Schema) this._schemaRegistry.get().getLatestSchemaByTopic(this.topicPartitions.get(0).getTopicName());
- return (S) schema;
+ if(this._schemaRegistry.isPresent()) {
+ return (S)(Schema) this._schemaRegistry.get().getLatestSchemaByTopic(this.topicPartitions.get(0).getTopicName());
+ }
+ return (S) this.topicPartitions.iterator().next().getTopicName();
Review comment:
return null? Don't understand why we are returning topic name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] ZihanLi58 merged pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
ZihanLi58 merged pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#issuecomment-902354878
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3372](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (58f3989) into [master](https://codecov.io/gh/apache/gobblin/commit/6691a37d863d35cb5d217355ff8667f29cdb03e8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6691a37) will **increase** coverage by `3.80%`.
> The diff coverage is `25.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3372 +/- ##
============================================
+ Coverage 42.63% 46.43% +3.80%
- Complexity 4575 10157 +5582
============================================
Files 1028 2056 +1028
Lines 40878 80080 +39202
Branches 4549 8947 +4398
============================================
+ Hits 17427 37183 +19756
- Misses 21730 39449 +17719
- Partials 1721 3448 +1727
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...tractor/extract/kafka/KafkaStreamingExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVN0cmVhbWluZ0V4dHJhY3Rvci5qYXZh) | `58.93% <0.00%> (-0.23%)` | :arrow_down: |
| [...ava/org/apache/gobblin/metastore/FsStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL0ZzU3RhdGVTdG9yZS5qYXZh) | `36.05% <100.00%> (ø)` | |
| [...pache/gobblin/compaction/hive/SerialCompactor.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vaGl2ZS9TZXJpYWxDb21wYWN0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...blin/yarn/event/ApplicationReportArrivalEvent.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vZXZlbnQvQXBwbGljYXRpb25SZXBvcnRBcnJpdmFsRXZlbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...onverter/avro/JsonIntermediateToAvroConverter.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9hdnJvL0pzb25JbnRlcm1lZGlhdGVUb0F2cm9Db252ZXJ0ZXIuamF2YQ==) | `34.28% <0.00%> (ø)` | |
| [...lin/compaction/mapreduce/orc/OrcKeyComparator.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL29yYy9PcmNLZXlDb21wYXJhdG9yLmphdmE=) | `73.07% <0.00%> (ø)` | |
| [...org/apache/gobblin/fork/CopyableGenericRecord.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ZvcmsvQ29weWFibGVHZW5lcmljUmVjb3JkLmphdmE=) | `57.14% <0.00%> (ø)` | |
| [...gobblin/source/DatePartitionedDailyAvroSource.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9EYXRlUGFydGl0aW9uZWREYWlseUF2cm9Tb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...n/filesystem/MetricsFileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZmlsZXN5c3RlbS9NZXRyaWNzRmlsZVN5c3RlbUluc3RydW1lbnRhdGlvbi5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...management/copy/TimestampBasedCopyableDataset.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvVGltZXN0YW1wQmFzZWRDb3B5YWJsZURhdGFzZXQuamF2YQ==) | `79.48% <0.00%> (ø)` | |
| ... and [1021 more](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6691a37...58f3989](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#issuecomment-902354878
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3372](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5f6fe3f) into [master](https://codecov.io/gh/apache/gobblin/commit/6691a37d863d35cb5d217355ff8667f29cdb03e8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6691a37) will **increase** coverage by `5.54%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3372 +/- ##
============================================
+ Coverage 42.63% 48.18% +5.54%
- Complexity 4575 7541 +2966
============================================
Files 1028 1423 +395
Lines 40878 56157 +15279
Branches 4549 6488 +1939
============================================
+ Hits 17427 27057 +9630
- Misses 21730 26561 +4831
- Partials 1721 2539 +818
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/gobblin/metastore/FsStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL0ZzU3RhdGVTdG9yZS5qYXZh) | `36.05% <100.00%> (ø)` | |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0.00%> (-14.29%)` | :arrow_down: |
| [...ache/gobblin/cluster/GobblinHelixMultiManager.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TXVsdGlNYW5hZ2VyLmphdmE=) | `52.33% <0.00%> (-1.84%)` | :arrow_down: |
| [...lin/service/modules/flow/MultiHopFlowCompiler.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93L011bHRpSG9wRmxvd0NvbXBpbGVyLmphdmE=) | | |
| [...rg/apache/gobblin/metrics/kafka/LoggingPusher.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvTG9nZ2luZ1B1c2hlci5qYXZh) | | |
| [...n/service/modules/dataset/FSDatasetDescriptor.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9kYXRhc2V0L0ZTRGF0YXNldERlc2NyaXB0b3IuamF2YQ==) | | |
| [.../gobblin/dataset/test/SimpleDatasetForTesting.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YXNldC90ZXN0L1NpbXBsZURhdGFzZXRGb3JUZXN0aW5nLmphdmE=) | | |
| [...bblin/broker/gobblin\_scopes/TaskScopeInstance.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYnJva2VyL2dvYmJsaW5fc2NvcGVzL1Rhc2tTY29wZUluc3RhbmNlLmphdmE=) | | |
| [...bblin/metastore/ZkStateStoreConfigurationKeys.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL1prU3RhdGVTdG9yZUNvbmZpZ3VyYXRpb25LZXlzLmphdmE=) | | |
| ... and [1654 more](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6691a37...5f6fe3f](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#issuecomment-902354878
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3372](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (18951a8) into [master](https://codecov.io/gh/apache/gobblin/commit/6691a37d863d35cb5d217355ff8667f29cdb03e8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6691a37) will **increase** coverage by `4.10%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3372 +/- ##
============================================
+ Coverage 42.63% 46.73% +4.10%
+ Complexity 4575 3115 -1460
============================================
Files 1028 648 -380
Lines 40878 25369 -15509
Branches 4549 3019 -1530
============================================
- Hits 17427 11857 -5570
+ Misses 21730 12243 -9487
+ Partials 1721 1269 -452
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `92.85% <0.00%> (-7.15%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `34.71% <0.00%> (-3.31%)` | :arrow_down: |
| [...ache/gobblin/cluster/GobblinHelixMultiManager.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TXVsdGlNYW5hZ2VyLmphdmE=) | `52.33% <0.00%> (-1.84%)` | :arrow_down: |
| [...odules/orchestration/ServiceAzkabanConfigKeys.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9zZXJ2aWNlL21vZHVsZXMvb3JjaGVzdHJhdGlvbi9TZXJ2aWNlQXprYWJhbkNvbmZpZ0tleXMuamF2YQ==) | | |
| [.../main/java/org/apache/gobblin/http/HttpClient.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taHR0cC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9odHRwL0h0dHBDbGllbnQuamF2YQ==) | | |
| [...gestion/google/webmaster/UrlTriePrefixGrouper.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvb2dsZS1pbmdlc3Rpb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vaW5nZXN0aW9uL2dvb2dsZS93ZWJtYXN0ZXIvVXJsVHJpZVByZWZpeEdyb3VwZXIuamF2YQ==) | | |
| [...che/gobblin/type/SerializedRecordWithMetadata.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tbWV0YWRhdGEvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdHlwZS9TZXJpYWxpemVkUmVjb3JkV2l0aE1ldGFkYXRhLmphdmE=) | | |
| [...ogle/webmaster/GoogleWebmasterDataFetcherImpl.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvb2dsZS1pbmdlc3Rpb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vaW5nZXN0aW9uL2dvb2dsZS93ZWJtYXN0ZXIvR29vZ2xlV2VibWFzdGVyRGF0YUZldGNoZXJJbXBsLmphdmE=) | | |
| [...apache/gobblin/couchbase/common/TupleDocument.java](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS9jb21tb24vVHVwbGVEb2N1bWVudC5qYXZh) | | |
| ... and [375 more](https://codecov.io/gh/apache/gobblin/pull/3372/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6691a37...18951a8](https://codecov.io/gh/apache/gobblin/pull/3372?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] sv2000 commented on a change in pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#discussion_r695069170
##########
File path: gobblin-docs/user-guide/Run-Gobblin-Streaming-kafka-hdfs-Locally.md
##########
@@ -0,0 +1,30 @@
+# Table of Contents
+
+[TOC]
+
+# Introduction
+
+Gobblin streaming enable user to ingest data from Kafka to HDFS in streaming manner. Normally it's running on yarn, to get the full function,
Review comment:
Reword to: "Gobblin supports streaming mode that allows continuous ingestion of data from Kafka to HDFS. The streaming mode has been deployed in production at LinkedIn as a Gobblin cluster that uses Yarn for container allocation and Helix for task coordination. Here, we describe how to set up a Kafka -> HDFS pipeline in local mode for users to easily start and test out a streaming ingestion pipeline. "
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] ZihanLi58 commented on a change in pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
ZihanLi58 commented on a change in pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#discussion_r695318503
##########
File path: gobblin-metastore/src/main/java/org/apache/gobblin/metastore/FsStateStore.java
##########
@@ -217,7 +217,7 @@ public void putAll(String storeName, String tableName, Collection<T> states) thr
}
protected void renamePath(Path tmpTablePath, Path tablePath) throws IOException {
- HadoopUtils.renamePath(this.fs, tmpTablePath, tablePath);
+ HadoopUtils.renamePath(this.fs, tmpTablePath, tablePath, true);
Review comment:
Yes, Fs state store does not accept add/update watermark into an existing store now, but in our case, we need to continually publish new file and update watermark so I change the behavior to make it able to overwrite existing state file. Do you have any concern for this change? i.e we have pipeline that rely on this feature? If so, I can also make this behavior configurable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] sv2000 commented on a change in pull request #3372: [GOBBLIN-1521] Create local mode of streaming kafka job to help user quickly onboard
Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#discussion_r696176561
##########
File path: gobblin-metastore/src/main/java/org/apache/gobblin/metastore/FsStateStore.java
##########
@@ -217,7 +217,7 @@ public void putAll(String storeName, String tableName, Collection<T> states) thr
}
protected void renamePath(Path tmpTablePath, Path tablePath) throws IOException {
- HadoopUtils.renamePath(this.fs, tmpTablePath, tablePath);
+ HadoopUtils.renamePath(this.fs, tmpTablePath, tablePath, true);
Review comment:
Can you try using FileContextBasedFsStateStore that allows overwriting on rename?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org