You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2021/01/23 07:32:48 UTC
[GitHub] [incubator-gobblin] shirshanka opened a new pull request #3212: [GOBBLIN-1371] Improve README to reflect current capabilities
shirshanka opened a new pull request #3212:
URL: https://github.com/apache/incubator-gobblin/pull/3212
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1371
### Description
- [x] Here are some details about my PR, including screenshots (if applicable):
Added text reflecting data management focus (versus just ingestion)
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
docs only
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] shirshanka commented on pull request #3212: [GOBBLIN-1371] Improve README to reflect current capabilities
Posted by GitBox <gi...@apache.org>.
shirshanka commented on pull request #3212:
URL: https://github.com/apache/incubator-gobblin/pull/3212#issuecomment-767754173
Rendered version: https://github.com/shirshanka/gobblin/blob/better-readme/README.md
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] asfgit closed pull request #3212: [GOBBLIN-1371] Improve README to reflect current capabilities
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3212:
URL: https://github.com/apache/incubator-gobblin/pull/3212
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #3212: [GOBBLIN-1371] Improve README to reflect current capabilities
Posted by GitBox <gi...@apache.org>.
sv2000 commented on a change in pull request #3212:
URL: https://github.com/apache/incubator-gobblin/pull/3212#discussion_r564710086
##########
File path: README.md
##########
@@ -6,11 +6,32 @@
[](https://communityinviter.com/apps/apache-gobblin/apache-gobblin)
[](https://codecov.io/github/apache/incubator-gobblin)
-Apache Gobblin is a universal data ingestion framework for extracting, transforming, and loading large volume of data from a variety of data sources: databases, rest APIs, FTP/SFTP servers, filers, etc., onto Hadoop.
+Apache Gobblin is a highly scalable data management solution for structured and byte-oriented data in heterogeneous data ecosystems.
-Apache Gobblin handles the common routine tasks required for all data ingestion ETLs, including job/task scheduling, task partitioning, error handling, state management, data quality checking, data publishing, etc.
+It offers the following capabilities:
+- Ingestion and export of data from a variety of sources and sinks into and out of the data lake. Gobblin is optimized and designed for ELT patterns with inline transformations on ingest (small t).
Review comment:
Do you think we can move this sentence to the top para after "Apace Gobblin is a highly scalable..."?
##########
File path: README.md
##########
@@ -6,11 +6,32 @@
[](https://communityinviter.com/apps/apache-gobblin/apache-gobblin)
[](https://codecov.io/github/apache/incubator-gobblin)
-Apache Gobblin is a universal data ingestion framework for extracting, transforming, and loading large volume of data from a variety of data sources: databases, rest APIs, FTP/SFTP servers, filers, etc., onto Hadoop.
+Apache Gobblin is a highly scalable data management solution for structured and byte-oriented data in heterogeneous data ecosystems.
-Apache Gobblin handles the common routine tasks required for all data ingestion ETLs, including job/task scheduling, task partitioning, error handling, state management, data quality checking, data publishing, etc.
+It offers the following capabilities:
+- Ingestion and export of data from a variety of sources and sinks into and out of the data lake. Gobblin is optimized and designed for ELT patterns with inline transformations on ingest (small t).
+- Data Organization within the lake (e.g. compaction, partitioning, deduplication)
+- Lifecycle Management of data within the lake (e.g. data retention)
+- Compliance Management of data across the ecosystem (e.g. fine-grain data deletions)
+
+Common Patterns used in production
+- Stream / Batch ingestion of Kafka to Data Lake (HDFS, S3, ADLS)
+- Bulk-loading serving stores from the Data Lake (e.g. HDFS -> Couchbase)
+- Support for data sync across Federated Data Lake (HDFS <-> HDFS, HDFS <-> S3, S3 <-> ADLS)
+- Integrate external vendor API-s (e.g. Salesforce, Dynamics etc.) with data store (HDFS, Couchbase etc)
+- Enforcing Data retention policies and GDPR deletion on HDFS / ADLS
+
+Highlights
+- Battle tested at scale: Runs in production at petabyte-scale at companies like LinkedIn, PayPal, Verizon etc.
+- Feature rich: Supports job/task scheduling, task partitioning, fault tolerance, error handling, state management for incremental processing, data quality checking, atomic data publishing etc.
+- Supports stream and batch execution modes
+- Control Plane (Gobblin-as-a-service) supports programmatic triggering and orchestration of data plane operations.
+
+Apache Gobblin is NOT
+- A general purpose data transformation engine like Spark or Flink. Gobblin can delegate complex-data processing tasks to Spark, Hive etc.
+- A data storage system like Apache Kafka or HDFS. Gobblin integrates with these systems as sources or sinks.
+- A general-purpose workflow execution system like Airflow, Azkaban, Dagster, Luigi. You can use these systems to kick-off Gobblin jobs or use Gobblin’s in-built Quartz based scheduler.
Review comment:
We can mention here that Gobblin-as-a-Service has a workflow execution system for scheduling and orchestrating Gobblin jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] codecov-io commented on pull request #3212: [GOBBLIN-1371] Improve README to reflect current capabilities
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #3212:
URL: https://github.com/apache/incubator-gobblin/pull/3212#issuecomment-765885243
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=h1) Report
> Merging [#3212](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=desc) (b460a21) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/5ecf9a175d745e1b2b46bea5de4ba89512866e5e?el=desc) (5ecf9a1) will **decrease** coverage by `37.09%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #3212 +/- ##
============================================
- Coverage 46.26% 9.17% -37.10%
+ Complexity 9796 1738 -8058
============================================
Files 2019 2019
Lines 77506 77506
Branches 8607 8607
============================================
- Hits 35861 7109 -28752
- Misses 38310 69701 +31391
+ Partials 3335 696 -2639
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...c/main/java/org/apache/gobblin/util/FileUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvRmlsZVV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...n/java/org/apache/gobblin/fork/CopyableSchema.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ZvcmsvQ29weWFibGVTY2hlbWEuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...java/org/apache/gobblin/stream/ControlMessage.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc3RyZWFtL0NvbnRyb2xNZXNzYWdlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...va/org/apache/gobblin/dataset/DatasetResolver.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YXNldC9EYXRhc2V0UmVzb2x2ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...va/org/apache/gobblin/converter/EmptyIterable.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9FbXB0eUl0ZXJhYmxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...org/apache/gobblin/ack/BasicAckableForTesting.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYWNrL0Jhc2ljQWNrYWJsZUZvclRlc3RpbmcuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...n/java/org/apache/gobblin/salesforce/SfConfig.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2ZDb25maWcuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/gobblin/yarn/HelixMessageSubTypes.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vSGVsaXhNZXNzYWdlU3ViVHlwZXMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...va/org/apache/gobblin/cluster/SingleHelixTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlSGVsaXhUYXNrLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| [.../apache/gobblin/records/ControlMessageHandler.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vcmVjb3Jkcy9Db250cm9sTWVzc2FnZUhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| ... and [1062 more](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=footer). Last update [5ecf9a1...b460a21](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] codecov-io edited a comment on pull request #3212: [GOBBLIN-1371] Improve README to reflect current capabilities
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #3212:
URL: https://github.com/apache/incubator-gobblin/pull/3212#issuecomment-765885243
# [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=h1) Report
> Merging [#3212](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=desc) (afecc3d) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/5ecf9a175d745e1b2b46bea5de4ba89512866e5e?el=desc) (5ecf9a1) will **decrease** coverage by `0.10%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #3212 +/- ##
============================================
- Coverage 46.26% 46.16% -0.11%
+ Complexity 9796 9766 -30
============================================
Files 2019 2019
Lines 77506 77506
Branches 8607 8607
============================================
- Hits 35861 35781 -80
- Misses 38310 38393 +83
+ Partials 3335 3332 -3
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...lin/metrics/kafka/KafkaKeyValueProducerPusher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9tZXRyaWNzL2thZmthL0thZmthS2V5VmFsdWVQcm9kdWNlclB1c2hlci5qYXZh) | `0.00% <0.00%> (-82.76%)` | `0.00% <0.00%> (-7.00%)` | |
| [...che/gobblin/metrics/kafka/KafkaProducerPusher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9tZXRyaWNzL2thZmthL0thZmthUHJvZHVjZXJQdXNoZXIuamF2YQ==) | `0.00% <0.00%> (-82.15%)` | `0.00% <0.00%> (-7.00%)` | |
| [...he/gobblin/kafka/serialize/LiAvroDeserializer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9rYWZrYS9zZXJpYWxpemUvTGlBdnJvRGVzZXJpYWxpemVyLmphdmE=) | `0.00% <0.00%> (-66.67%)` | `0.00% <0.00%> (-3.00%)` | |
| [...ache/gobblin/kafka/serialize/LiAvroSerializer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9rYWZrYS9zZXJpYWxpemUvTGlBdnJvU2VyaWFsaXplci5qYXZh) | `0.00% <0.00%> (-50.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [.../apache/gobblin/kafka/writer/Kafka1DataWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9rYWZrYS93cml0ZXIvS2Fma2ExRGF0YVdyaXRlci5qYXZh) | `0.00% <0.00%> (-46.16%)` | `0.00% <0.00%> (-9.00%)` | |
| [...ava/org/apache/gobblin/fsm/FiniteStateMachine.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ZzbS9GaW5pdGVTdGF0ZU1hY2hpbmUuamF2YQ==) | `73.48% <0.00%> (-3.04%)` | `18.00% <0.00%> (-3.00%)` | |
| [...a/management/copy/publisher/CopyDataPublisher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcHVibGlzaGVyL0NvcHlEYXRhUHVibGlzaGVyLmphdmE=) | `74.00% <0.00%> (-1.34%)` | `31.00% <0.00%> (-1.00%)` | |
| [...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==) | `60.00% <0.00%> (+0.40%)` | `29.00% <0.00%> (+1.00%)` | |
| [...tractor/extract/kafka/KafkaProduceRateTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVByb2R1Y2VSYXRlVHJhY2tlci5qYXZh) | `87.82% <0.00%> (+0.86%)` | `27.00% <0.00%> (ø%)` | |
| [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `72.22% <0.00%> (+2.22%)` | `13.00% <0.00%> (ø%)` | |
| ... and [2 more](https://codecov.io/gh/apache/incubator-gobblin/pull/3212/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=footer). Last update [5ecf9a1...40f4fba](https://codecov.io/gh/apache/incubator-gobblin/pull/3212?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-gobblin] shirshanka commented on pull request #3212: [GOBBLIN-1371] Improve README to reflect current capabilities
Posted by GitBox <gi...@apache.org>.
shirshanka commented on pull request #3212:
URL: https://github.com/apache/incubator-gobblin/pull/3212#issuecomment-765883095
@abti @sv2000 : PTAL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org