You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gobblin.apache.org by hu...@apache.org on 2019/04/10 17:29:35 UTC
[incubator-gobblin] branch master updated: [GOBBLIN-719] fix
invalid git links for classes in docs
This is an automated email from the ASF dual-hosted git repository.
hutran pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-gobblin.git
The following commit(s) were added to refs/heads/master by this push:
new a6c5134 [GOBBLIN-719] fix invalid git links for classes in docs
a6c5134 is described below
commit a6c51342afb40f3a9859ec98814f426b74fcff87
Author: Jay Sen <js...@paypal.com>
AuthorDate: Wed Apr 10 10:29:28 2019 -0700
[GOBBLIN-719] fix invalid git links for classes in docs
[GOBBLIN-719] fix invalid git links for classes in
docs
fixing all file paths and linksA
Closes #2586 from jhsenjaliya/master
---
gobblin-admin/README.md | 2 +-
.../apache/gobblin/source/extractor/Extractor.java | 2 +-
.../cluster/GobblinClusterConfigurationKeys.java | 2 +-
gobblin-docs/Getting-Started.md | 20 +++++++-------
.../adaptors/Hive-Avro-To-ORC-Converter.md | 10 +++----
gobblin-docs/case-studies/Kafka-HDFS-Ingestion.md | 14 +++++-----
gobblin-docs/case-studies/Publishing-Data-to-S3.md | 4 +--
gobblin-docs/case-studies/Writing-ORC-Data.md | 4 +--
.../developer-guide/Documentation-Architecture.md | 2 +-
gobblin-docs/metrics/Existing-Reporters.md | 18 ++++++------
.../metrics/Gobblin-Metrics-Architecture.md | 14 +++++-----
...-generation-instrumentation-for-applications.md | 2 +-
gobblin-docs/metrics/Implementing-New-Reporters.md | 2 +-
gobblin-docs/metrics/Metrics-for-Gobblin-ETL.md | 6 ++--
gobblin-docs/sinks/AvroHdfsDataWriter.md | 2 +-
gobblin-docs/sinks/Couchbase-Writer.md | 6 ++--
gobblin-docs/sinks/Kafka.md | 4 +--
gobblin-docs/sources/QueryBasedSource.md | 16 +++++------
gobblin-docs/sources/RestApiSource.md | 4 +--
gobblin-docs/user-guide/Building-Gobblin.md | 2 +-
gobblin-docs/user-guide/Compaction.md | 2 +-
gobblin-docs/user-guide/Gobblin-Deployment.md | 2 +-
gobblin-docs/user-guide/Gobblin-Schedulers.md | 17 ++++++------
gobblin-docs/user-guide/Gobblin-on-Yarn.md | 32 +++++++++++-----------
gobblin-docs/user-guide/Hive-Registration.md | 18 ++++++------
.../user-guide/Job-Execution-History-Store.md | 2 +-
gobblin-docs/user-guide/Monitoring.md | 2 +-
.../user-guide/State-Management-and-Watermarks.md | 8 +++---
.../Working-with-Job-Configuration-Files.md | 2 +-
.../user-guide/Working-with-the-ForkOperator.md | 20 +++++++-------
gradle/scripts/computeVersions.gradle | 2 +-
mkdocs.yml | 2 +-
32 files changed, 123 insertions(+), 122 deletions(-)
diff --git a/gobblin-admin/README.md b/gobblin-admin/README.md
index 1a9a1f9..0d455b5 100644
--- a/gobblin-admin/README.md
+++ b/gobblin-admin/README.md
@@ -5,7 +5,7 @@ development, will likely change, and have many new features planned. The
two interfaces provided are a command line interface and a GUI,
accessible via a web server. The current state of the interfaces relies
on the [Job Execution History
-Store](https://github.com/linkedin/gobblin/wiki/Job%20Execution%20History%20Store),
+Store](https://gobblin.readthedocs.io/en/latest/user-guide/Job-Execution-History-Store/),
which must be enabled and running for the interfaces to work.
## CLI
diff --git a/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java b/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java
index 2ea900b..9eb31a7 100644
--- a/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java
+++ b/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java
@@ -87,7 +87,7 @@ public interface Extractor<S, D> extends Closeable {
*
* @return the calculated high watermark
* @deprecated there is no longer support for reporting the high watermark via this method, please see
- * <a href="https://github.com/linkedin/gobblin/wiki/Watermarks">Watermarks</a> for more information.
+ * <a href="https://gobblin.readthedocs.io/en/latest/user-guide/State-Management-and-Watermarks/">Watermarks</a> for more information.
*/
@Deprecated
long getHighWatermark();
diff --git a/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java b/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java
index a760c65..f8604fc 100644
--- a/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java
+++ b/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java
@@ -101,7 +101,7 @@ public class GobblinClusterConfigurationKeys {
* A path pointing to a directory that contains job execution files to be executed by Gobblin. This directory can
* have a nested structure.
*
- * @see <a href="https://github.com/linkedin/gobblin/wiki/Working-with-Job-Configuration-Files">Job Config Files</a>
+ * @see <a href="https://gobblin.readthedocs.io/en/latest/user-guide/Working-with-Job-Configuration-Files/">Job Config Files</a>
*/
public static final String JOB_CONF_PATH_KEY = GOBBLIN_CLUSTER_PREFIX + "job.conf.path";
public static final String INPUT_WORK_UNIT_DIR_NAME = "_workunits";
diff --git a/gobblin-docs/Getting-Started.md b/gobblin-docs/Getting-Started.md
index f8cfbad..d575997 100644
--- a/gobblin-docs/Getting-Started.md
+++ b/gobblin-docs/Getting-Started.md
@@ -19,7 +19,7 @@ There are two way to obtain a distribution tarball
## Downloading a Pre-Built Distribution
-Download the latest Gobblin release from the [Release Page](https://github.com/linkedin/gobblin/releases). You will want to download the `gobblin-distribution-[RELEASE-VERSION].tar.gz` file.
+Download the latest Gobblin release from the [Release Page](https://github.com/apache/incubator-gobblin/releases). You will want to download the `gobblin-distribution-[RELEASE-VERSION].tar.gz` file.
## Building a Distribution
@@ -84,17 +84,17 @@ For this example, we will once again run the Wikipedia example. The records will
## Preliminary
-Each Gobblin job minimally involves several constructs, e.g. [Source](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/Source.java), [Extractor](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/extractor/Extractor.java), [DataWriter](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/writer/DataWriter.java) and [DataPublisher](https://github.com/linkedin/gobblin/blob/master/gobb [...]
+Each Gobblin job minimally involves several constructs, e.g. [Source](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/Source.java), [Extractor](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java), [DataWriter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/writer/DataWriter.java) and [DataPublis [...]
-Some of the classes relevant to this example include [WikipediaSource](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/java/gobblin/example/wikipedia/WikipediaSource.java), [WikipediaExtractor](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/java/gobblin/example/wikipedia/WikipediaExtractor.java), [WikipediaConverter](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/java/gobblin/example/wikipedia/WikipediaConverter [...]
+Some of the classes relevant to this example include [WikipediaSource](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/wikipedia/WikipediaSource.java), [WikipediaExtractor](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/wikipedia/WikipediaExtractor.java), [WikipediaConverter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/jav [...]
-To run Gobblin in standalone daemon mode we need a Gobblin configuration file (such as uses [gobblin-standalone.properties](https://github.com/linkedin/gobblin/blob/master/conf/gobblin-standalone.properties)). And for each job we wish to run, we also need a job configuration file (such as [wikipedia.pull](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull)). The Gobblin configuration file, which is passed to Gobblin as a command line argumen [...]
+To run Gobblin in standalone daemon mode we need a Gobblin configuration file (such as uses [gobblin-standalone.properties](https://github.com/apache/incubator-gobblin/blob/master/conf/gobblin-standalone-v2.properties)). And for each job we wish to run, we also need a job configuration file (such as [wikipedia.pull](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull)). The Gobblin configuration file, which is passed to Gobblin as a c [...]
A list of commonly used configuration properties can be found here: [Configuration Properties Glossary](user-guide/Configuration-Properties-Glossary).
## Steps
-* Create a folder to store the job configuration file. Put [wikipedia.pull](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull) in this folder, and set environment variable `GOBBLIN_JOB_CONFIG_DIR` to point to this folder. Also, make sure that the environment variable `JAVA_HOME` is set correctly.
+* Create a folder to store the job configuration file. Put [wikipedia.pull](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull) in this folder, and set environment variable `GOBBLIN_JOB_CONFIG_DIR` to point to this folder. Also, make sure that the environment variable `JAVA_HOME` is set correctly.
* Create a folder as Gobblin's working directory. Gobblin will write job output as well as other information there, such as locks and state-store (for more information, see the [Standalone Deployment](user-guide/Gobblin-Deployment#Standalone-Deployment) page). Set environment variable `GOBBLIN_WORK_DIR` to point to that folder.
@@ -106,7 +106,7 @@ A list of commonly used configuration properties can be found here: [Configurati
bin/gobblin-standalone.sh start
```
-This script will launch Gobblin and pass the Gobblin configuration file ([gobblin-standalone.properties](https://github.com/linkedin/gobblin/blob/master/conf/gobblin-standalone.properties)) as an argument.
+This script will launch Gobblin and pass the Gobblin configuration file ([gobblin-standalone.properties](https://github.com/apache/incubator-gobblin/blob/master/conf/gobblin-standalone-v2.properties)) as an argument.
The job log, which contains the progress and status of the job, will be written into `logs/gobblin-current.log` (to change where the log is written, modify the Log4j configuration file `conf/log4j-standalone.xml`). Stdout will be written into `nohup.out`.
@@ -157,12 +157,12 @@ java -jar avro-tools-1.8.1.jar tojson --pretty [job_output].avro > output.json
`output.json` will contain all retrieved records in JSON format.
-Note that since this job configuration file we used ([wikipedia.pull](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull)) doesn't specify a job schedule, the job will run immediately and will run only once. To schedule a job to run at a certain time and/or repeatedly, set the `job.schedule` property with a cron-based syntax. For example, `job.schedule=0 0/2 * * * ?` will run the job every two minutes. See [this link](http://www.quartz-sched [...]
+Note that since this job configuration file we used ([wikipedia.pull](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull)) doesn't specify a job schedule, the job will run immediately and will run only once. To schedule a job to run at a certain time and/or repeatedly, set the `job.schedule` property with a cron-based syntax. For example, `job.schedule=0 0/2 * * * ?` will run the job every two minutes. See [this link](http://www.quar [...]
# Other Example Jobs
-Besides the Wikipedia example, we have another example job [SimpleJson](https://github.com/linkedin/gobblin/tree/master/gobblin-example/src/main/java/gobblin/example/simplejson), which extracts records from JSON files and store them in Avro files.
+Besides the Wikipedia example, we have another example job [SimpleJson](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/simplejson.pull), which extracts records from JSON files and store them in Avro files.
-To create your own jobs, simply implement the relevant interfaces such as [Source](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/Source.java), [Extractor](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/extractor/Extractor.java), [Converter](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/converter/Converter.java) and [DataWriter](https://github.com/linkedin/gobblin/blob/ [...]
+To create your own jobs, simply implement the relevant interfaces such as [Source](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/Source.java), [Extractor](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java), [Converter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/converter/Converter.java) a [...]
-On a side note: while users are free to directly implement the Extractor interface (e.g., WikipediaExtractor), Gobblin also provides several extractor implementations based on commonly used protocols, e.g., [KafkaExtractor](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/kafka/KafkaExtractor.java), [RestApiExtractor](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/resta [...]
+On a side note: while users are free to directly implement the Extractor interface (e.g., WikipediaExtractor), Gobblin also provides several extractor implementations based on commonly used protocols, e.g., [KafkaExtractor](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaExtractor.java), [RestApiExtractor](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/s [...]
diff --git a/gobblin-docs/adaptors/Hive-Avro-To-ORC-Converter.md b/gobblin-docs/adaptors/Hive-Avro-To-ORC-Converter.md
index c03c6e4..98e99bf 100644
--- a/gobblin-docs/adaptors/Hive-Avro-To-ORC-Converter.md
+++ b/gobblin-docs/adaptors/Hive-Avro-To-ORC-Converter.md
@@ -39,16 +39,16 @@ FROM db_name_avro.table_avro WHERE year='2016';
## Source and Extractor
-Gobblin provides [`HiveSource`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/source/HiveSource.java) which is a generic source that connects to the hive metastore and creates `WorkUnits` for any Hive `Partitions` and `Tables` whitelisted. The [`HiveConvertExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/extractor/HiveCon [...]
+Gobblin provides [`HiveSource`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/source/HiveSource.java) which is a generic source that connects to the hive metastore and creates `WorkUnits` for any Hive `Partitions` and `Tables` whitelisted. The [`HiveConvertExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/manage [...]
-The `HiveSource` uses the `HiveDatasetFinder` to find all hive tables and partitions that satisfy a whitelist. For each table/partition it creates a workunit is the `updateTime` is greater than the `lowWatermark`. By default a [`PartitionLevelWatermarker`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/watermarker/PartitionLevelWatermarker.java) is used. This watermarker tracks watermarks for every partition o [...]
+The `HiveSource` uses the `HiveDatasetFinder` to find all hive tables and partitions that satisfy a whitelist. For each table/partition it creates a workunit is the `updateTime` is greater than the `lowWatermark`. By default a [`PartitionLevelWatermarker`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/watermarker/PartitionLevelWatermarker.java) is used. This watermarker tracks watermarks fo [...]
The `HiveConvertExtractor` builds `QueryBasedHiveConversionEntity`s. The extractor makes necessary calls to the Hive Metastore to get table/partition metadata. The metadata is then wrapped into a `QueryBasedHiveConversionEntity`.
## Converter
The converter builds the Hive DDLs/DMLs required to perform the Avro to ORC conversion. Gobblin supports conversion of Avro to both flattened ORC and nested ORC.
-The abstract converter [`AbstractAvroToOrcConverter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/converter/AbstractAvroToOrcConverter.java) builds DDLs/DMLs for any destination ORC format. Concrete subclass [`HiveAvroToFlattenedOrcConverter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/converter/HiveAvroToFlattenedOrcConverte [...]
+The abstract converter [`AbstractAvroToOrcConverter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/converter/AbstractAvroToOrcConverter.java) builds DDLs/DMLs for any destination ORC format. Concrete subclass [`HiveAvroToFlattenedOrcConverter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/co [...]
The converter builds the following different DDLs/DMLs
@@ -63,12 +63,12 @@ The converter builds the following different DDLs/DMLs
## Writer
-The writer in this context executes the Hive DDLs/DMLs generated by the converter. [`HiveQueryExecutionWriter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/writer/HiveQueryExecutionWriter.java) uses Hive JDBC connector to execute the DDLs. The DDLs write ORC data into staging tables. After the writer has completed `HiveQueryExecutionWriter#write()`, ORC data will be available in the staging tables.
+The writer in this context executes the Hive DDLs/DMLs generated by the converter. [`HiveQueryExecutionWriter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/writer/HiveQueryExecutionWriter.java) uses Hive JDBC connector to execute the DDLs. The DDLs write ORC data into staging tables. After the writer has completed `HiveQueryExecutionWriter#write()`, ORC data will be available in the stagi [...]
## Publisher
-The publisher [`HiveConvertPublisher`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/publisher/HiveConvertPublisher.java) executes hive DDLs to publish staging ORC tables to final ORC tables. The publisher also cleans up staging tables.
+The publisher [`HiveConvertPublisher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/publisher/HiveConvertPublisher.java) executes hive DDLs to publish staging ORC tables to final ORC tables. The publisher also cleans up staging tables.
By default publishing happens per dataset (dataset = table in this context). If a dataset fails, other datasets will still be published but the job will fail. The commit policy is configurable.
diff --git a/gobblin-docs/case-studies/Kafka-HDFS-Ingestion.md b/gobblin-docs/case-studies/Kafka-HDFS-Ingestion.md
index feaca55..352b7e3 100644
--- a/gobblin-docs/case-studies/Kafka-HDFS-Ingestion.md
+++ b/gobblin-docs/case-studies/Kafka-HDFS-Ingestion.md
@@ -107,17 +107,17 @@ After the job finishes, the job output file will be in `/gobblintest/job-output/
## Source and Extractor
-Gobblin provides two abstract classes, [`KafkaSource`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/kafka/KafkaSource.java) and [`KafkaExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/kafka/KafkaExtractor.java). `KafkaSource` creates a workunit for each Kafka topic partition to be pulled, then merges and groups the workunits based on the desired number of [...]
+Gobblin provides two abstract classes, [`KafkaSource`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaSource.java) and [`KafkaExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaExtractor.java). `KafkaSource` creates a workunit for each Kafka topic partit [...]
To use them in a Kafka-HDFS ingestion job, one should subclass `KafkaExtractor` and implement method `decodeRecord(MessageAndOffset)`, which takes a `MessageAndOffset` object pulled from the Kafka broker and decodes it into a desired object. One should also subclass `KafkaSource` and implement `getExtractor(WorkUnitState)` which should return an instance of the Extractor class.
-As examples, take a look at [`KafkaSimpleSource`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/kafka/KafkaSimpleSource.java), [`KafkaSimpleExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/kafka/KafkaSimpleExtractor.java), and [`KafkaAvroExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/k [...]
+As examples, take a look at [`KafkaSimpleSource`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaSimpleSource.java), [`KafkaSimpleExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaSimpleExtractor.java), and [`KafkaAvroExtractor`](https://github.com/apac [...]
`KafkaSimpleExtractor` simply returns the payload of the `MessageAndOffset` object as a byte array. A job that uses `KafkaSimpleExtractor` may use a `Converter` to convert the byte array to whatever format desired. For example, if the desired output format is JSON, one may implement an `ByteArrayToJsonConverter` to convert the byte array to JSON. Alternatively one may implement a `KafkaJsonExtractor`, which extends `KafkaExtractor` and convert the `MessageAndOffset` object into a JSON ob [...]
## Writer and Publisher
-Any desired writer and publisher can be used, e.g., one may use the [`AvroHdfsDataWriter`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/writer/AvroHdfsDataWriter.java) and the [`BaseDataPublisher`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/publisher/BaseDataPublisher.java), similar as the [Wikipedia example job](../Getting-Started). If plain text output file is desired, one may use [`SimpleDataWriter`](https:/ [...]
+Any desired writer and publisher can be used, e.g., one may use the [`AvroHdfsDataWriter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/AvroHdfsDataWriter.java) and the [`BaseDataPublisher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/BaseDataPublisher.java), similar as the [Wikipedia example job](https://github.com/apache/incubator-gobblin/blob/master/gobbl [...]
# Job Config Properties
@@ -145,7 +145,7 @@ extract.limit.timeLimitTimeunit=minutes
## Task Level Metrics
-Task level metrics can be created in `Extractor`, `Converter` and `Writer` by extending [`InstrumentedExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/instrumented/extractor/InstrumentedExtractor.java), [`InstrumentedConverter`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/instrumented/converter/InstrumentedConverter.java) and [`InstrumentedDataWriter`](https://github.com/linkedin/gobblin/blob/master/gobb [...]
+Task level metrics can be created in `Extractor`, `Converter` and `Writer` by extending [`InstrumentedExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/extractor/InstrumentedExtractor.java), [`InstrumentedConverter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/converter/InstrumentedConverter.java) and [`InstrumentedDataWriter`](http [...]
For example, `KafkaExtractor` extends `InstrumentedExtractor`. So you can do the following in subclasses of `KafkaExtractor`:
@@ -158,11 +158,11 @@ Besides Counter, Meter and Histogram are also supported.
## Task Level Events
-Task level events can be submitted by creating an [`EventSubmitter`](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/event/EventSubmitter.java) instance and using `EventSubmitter.submit()` or `EventSubmitter.getTimingEvent()`.
+Task level events can be submitted by creating an [`EventSubmitter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/event/EventSubmitter.java) instance and using `EventSubmitter.submit()` or `EventSubmitter.getTimingEvent()`.
## Job Level Metrics
-To create job level metrics, one may extend [`AbstractJobLauncher`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/AbstractJobLauncher.java) and create metrics there. For example:
+To create job level metrics, one may extend [`AbstractJobLauncher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/AbstractJobLauncher.java) and create metrics there. For example:
```
Optional<JobMetrics> jobMetrics = this.jobContext.getJobMetricsOptional();
@@ -186,7 +186,7 @@ For more details about metrics, events and reporting them, please see Gobblin Me
For each topic partition that should be ingested, `KafkaSource` first retrieves the last offset pulled by the previous run, which should be the first offset of the current run. It also retrieves the earliest and latest offsets currently available from the Kafka cluster and verifies that the first offset is between the earliest and the latest offsets. The latest offset is the last offset to be pulled by the current workunit. Since new records may be constantly published to Kafka and old r [...]
-For each partition, after the first and last offsets are determined, a workunit is created. If the number of Kafka partitions exceeds the desired number of workunits specified by property `mr.job.max.mappers`, `KafkaSource` will merge and group them into `n` [`MultiWorkUnit`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/workunit/MultiWorkUnit.java)s where `n=mr.job.max.mappers`. This is done using [`KafkaWorkUnitPacker`](https://github.com/link [...]
+For each partition, after the first and last offsets are determined, a workunit is created. If the number of Kafka partitions exceeds the desired number of workunits specified by property `mr.job.max.mappers`, `KafkaSource` will merge and group them into `n` [`MultiWorkUnit`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/workunit/MultiWorkUnit.java)s where `n=mr.job.max.mappers`. This is done using [`KafkaWorkUnitPacker`](http [...]
## Single-Level Packing
diff --git a/gobblin-docs/case-studies/Publishing-Data-to-S3.md b/gobblin-docs/case-studies/Publishing-Data-to-S3.md
index 8d1e89a..9866e85 100644
--- a/gobblin-docs/case-studies/Publishing-Data-to-S3.md
+++ b/gobblin-docs/case-studies/Publishing-Data-to-S3.md
@@ -95,7 +95,7 @@ Go to the [S3 dashboard](https://console.aws.amazon.com/s3)
## Setting Up Gobblin on EC2
* Download and Build Gobblin Locally
- * On your local machine, clone the [Gobblin repository](https://github.com/linkedin/gobblin): `git clone git@github.com:linkedin/gobblin.git` (this assumes you have [Git](https://en.wikipedia.org/wiki/Git_(software)) installed locally)
+ * On your local machine, clone the [Gobblin repository](https://github.com/apache/incubator-gobblin): `git clone git@github.com:linkedin/gobblin.git` (this assumes you have [Git](https://en.wikipedia.org/wiki/Git_(software)) installed locally)
* Build Gobblin using the following commands (it is important to use Hadoop version 2.6.0 as it includes the `s3a` file system implementation):
```
cd gobblin
@@ -132,7 +132,7 @@ Assuming we are running Gobblin in [standalone mode](../user-guide/Gobblin-Deplo
Assuming we want Gobblin to run in standalone mode, follow the usual steps for [standalone deployment](../user-guide/Gobblin-Deployment#Standalone-Deployment).
-For the sake of this walkthrough, we will launch the Gobblin [wikipedia example](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull). Directions on how to run this example can be found [here](../Getting-Started). The command to launch Gobblin should look similar to:
+For the sake of this walkthrough, we will launch the Gobblin [wikipedia example](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull). Directions on how to run this example can be found [here](../Getting-Started). The command to launch Gobblin should look similar to:
```
sh bin/gobblin-standalone.sh start --workdir /home/ec2-user/gobblin-dist/work --logdir /home/ec2-user/gobblin-dist/logs --conf /home/ec2-user/gobblin-dist/config
```
diff --git a/gobblin-docs/case-studies/Writing-ORC-Data.md b/gobblin-docs/case-studies/Writing-ORC-Data.md
index 26da218..411409a 100644
--- a/gobblin-docs/case-studies/Writing-ORC-Data.md
+++ b/gobblin-docs/case-studies/Writing-ORC-Data.md
@@ -4,7 +4,7 @@
# Introduction
-Gobblin is capable of writing data to ORC files by leveraging Hive's SerDe library. Gobblin has native integration with Hive SerDe's library via the [HiveSerDeWrapper](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/HiveSerDeManager.java) class.
+Gobblin is capable of writing data to ORC files by leveraging Hive's SerDe library. Gobblin has native integration with Hive SerDe's library via the [HiveSerDeWrapper](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveSerDeManager.java) class.
This document will briefly explain how Gobblin integrates with Hive's SerDe library, and show an example of writing ORC files.
@@ -20,7 +20,7 @@ Gobblin integrates with the Hive SerDe's in a few different places. Here is a li
# Writing to an ORC File
-An end-to-end example of writing to an ORC file is provided in the configuration found [here](https://github.com/linkedin/gobblin/tree/master/gobblin-example/src/main/resources/wikipedia-orc.pull). This `.pull` file is almost identical to the Wikipedia example discussed in the [Getting Started Guide](../Getting-Started.md). The only difference is that the output is written in ORC instead of Avro. The configuration file mentioned above can be directly used as a template for writing data t [...]
+An end-to-end example of writing to an ORC file is provided in the configuration found [here](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia-orc.pull). This `.pull` file is almost identical to the Wikipedia example discussed in the [Getting Started Guide](../Getting-Started.md). The only difference is that the output is written in ORC instead of Avro. The configuration file mentioned above can be directly used as a template for writin [...]
* `converter.classes` requires two additional converters: `gobblin.converter.avro.AvroRecordToAvroWritableConverter` and `gobblin.converter.serde.HiveSerDeConverter`
* The output of the first converter (the `WikipediaConverter`) returns Avro `GenericRecord`s
diff --git a/gobblin-docs/developer-guide/Documentation-Architecture.md b/gobblin-docs/developer-guide/Documentation-Architecture.md
index 80de729..15a04bc 100644
--- a/gobblin-docs/developer-guide/Documentation-Architecture.md
+++ b/gobblin-docs/developer-guide/Documentation-Architecture.md
@@ -21,4 +21,4 @@ ReadTheDocs is an open source, free tool that can build documentation in a GitHu
# Additional Information
-For more information on how this architecture was decided and the different tradeoffs between other documentation services, check out the original PR: https://github.com/linkedin/gobblin/pull/788 and GitHub Issue: https://github.com/linkedin/gobblin/issues/724
+For more information on how this architecture was decided and the different tradeoffs between other documentation services, check out the original PR: https://github.com/apache/incubator-gobblin/pull/788
diff --git a/gobblin-docs/metrics/Existing-Reporters.md b/gobblin-docs/metrics/Existing-Reporters.md
index e573941..909f285 100644
--- a/gobblin-docs/metrics/Existing-Reporters.md
+++ b/gobblin-docs/metrics/Existing-Reporters.md
@@ -6,15 +6,15 @@ Table of Contents
Metric Reporters
================
-* [Output Stream Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/reporter/OutputStreamReporter.java): allows printing metrics to any OutputStream, including STDOUT and files.
-* [Kafka Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/kafka/KafkaReporter.java): emits metrics to Kafka topic as Json messages.
-* [Kafka Avro Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/kafka/KafkaAvroReporter.java): emits metrics to Kafka topic as Avro messages with schema [MetricReport](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/avro/MetricReport.avsc).
-* [Graphite Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/graphite/GraphiteReporter.java): emits metrics to Graphite. This reporter has a different, deprecated construction API included in its javadoc.
-* [Influx DB Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/influxdb/InfluxDBReporter.java): emits metrics to Influx DB. This reporter has a different, deprecated construction API included in its javadoc.
-* [Hadoop Counter Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/hadoop/HadoopCounterReporter.java): emits metrics as Hadoop counters at the end of the execution. Available for old and new Hadoop API. This reporter has a different, deprecated construction API included in its javadoc. Due to limits on the number of Hadoop counters that can be created, this reporter is not recommended except for applications with very few metrics.
+* [Output Stream Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/OutputStreamReporter.java): allows printing metrics to any OutputStream, including STDOUT and files.
+* [Kafka Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaReporter.java): emits metrics to Kafka topic as Json messages.
+* [Kafka Avro Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaAvroReporter.java): emits metrics to Kafka topic as Avro messages with schema [MetricReport](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/MetricReport.avsc).
+* [Graphite Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-metrics-graphite/src/main/java/org/apache/gobblin/metrics/graphite/GraphiteReporter.java): emits metrics to Graphite. This reporter has a different, deprecated construction API included in its javadoc.
+* [Influx DB Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-metrics-influxdb/src/main/java/org/apache/gobblin/metrics/influxdb/InfluxDBReporter.java): emits metrics to Influx DB. This reporter has a different, deprecated construction API included in its javadoc.
+* [Hadoop Counter Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-metrics-hadoop/src/main/java/org/apache/gobblin/metrics/hadoop/HadoopCounterReporter.java): emits metrics as Hadoop counters at the end of the execution. Available for old and new Hadoop API. This reporter has a different, deprecated construction API included in its javadoc. Due to limits on the number of Hadoop counters that can be created, this reporter is not recommended except [...]
Event Reporters
===============
-* [Output Stream Event Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/reporter/OutputStreamEventReporter.java): Emits events to any output stream, including STDOUT and files.
-* [Kafka Event Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/kafka/KafkaEventReporter.java): Emits events to Kafka topic as Json messages.
-* [Kafka Avro Event Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/kafka/KafkaAvroEventReporter.java): Emits events to Kafka topic as Avro messages using the schema [GobblinTrackingEvent](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/avro/GobblinTrackingEvent.avsc).
+* [Output Stream Event Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/OutputStreamEventReporter.java): Emits events to any output stream, including STDOUT and files.
+* [Kafka Event Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaEventReporter.java): Emits events to Kafka topic as Json messages.
+* [Kafka Avro Event Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaAvroEventReporter.java): Emits events to Kafka topic as Avro messages using the schema [GobblinTrackingEvent](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GobblinTrackingEvent.avsc).
diff --git a/gobblin-docs/metrics/Gobblin-Metrics-Architecture.md b/gobblin-docs/metrics/Gobblin-Metrics-Architecture.md
index 4e15523..f479ac4 100644
--- a/gobblin-docs/metrics/Gobblin-Metrics-Architecture.md
+++ b/gobblin-docs/metrics/Gobblin-Metrics-Architecture.md
@@ -19,15 +19,15 @@ Each metric context contains the following instance variables:
* A `String` `name`. The name is not used by the core metrics engine, but can be accessed by users to identify the context.
* A reference to the parent metric context, or null if it has no parent.
* A list of children metric context references, stored as soft references.
-* An object of type [Tagged](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/Tagged.java) containing the tags for this metric context.
-* A `Set` of notification targets. Notification targets are objects of type [Function](http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/base/Function.html)<[Notification](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/notification/Notification.java), Void> which are all called every time there is a new notification. Notifications can be submitted to the Metric Context using the method `sendNotification(Notification)`. [...]
+* An object of type [Tagged](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/Tagged.java) containing the tags for this metric context.
+* A `Set` of notification targets. Notification targets are objects of type [Function](https://google.github.io/guava/releases/15.0/api/docs/com/google/common/base/Function.html)<[Notification](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/notification/Notification.java), Void> which are all called every time there is a new notification. Notifications can be submitted to the Metric Context using [...]
* A lazily instantiated `ExecutorService` used for asynchronously executing the notification targets. The executor service will only be started the first time there is a notification and the number of notification targets is positive.
* A `ConcurrentMap` from metric names to `Metric` for all metrics registered in this Metric Context. Metrics can be added to this map using the `register(Metric)`, `register(String, Metric)`, or `registerAll(MetricSet)`, although it is recommended to instead use the methods to create and register the metrics. Metric Context implements getter methods for all metrics, as well as for each type of metric individually (`getMetrics`, `getGauges`, `getCounters`, `getHistograms`, `getMeters`, `g [...]
Metrics
=======
-All metrics extend the interface [ContextAwareMetric](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/ContextAwareMetric.java). Each metric type in Dropwizard Metrics is extended to a Context Aware type: `ContextAwareCounter`, `ContextAwareGauge`, `ContextAwareHistogram`, `ContextAwareMeter`, `ContextAwareTimer`.
+All metrics extend the interface [ContextAwareMetric](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/ContextAwareMetric.java). Each metric type in Dropwizard Metrics is extended to a Context Aware type: `ContextAwareCounter`, `ContextAwareGauge`, `ContextAwareHistogram`, `ContextAwareMeter`, `ContextAwareTimer`.
Context Aware metrics all always created from the Metric Context where they will be registered. For example, to get a counter under Metric Context `context`, the user would call `context.counter("counter.name")`. This method first checks all registered metrics in the Metric Context to find a counter with that name, if it succeeds, it simply returns that counter. If a counter with that name has not been registered in `context`, then a new `ContextAwareCounter` is created and registered in [...]
@@ -38,16 +38,16 @@ Users can also register objects of type `com.codahale.metrics.Metric` with any M
Events
======
-Events are objects of type [GobblinTrackingEvent](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/avro/GobblinTrackingEvent.avsc), which is a type generated from an Avro schema. Events have:
+Events are objects of type [GobblinTrackingEvent](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GobblinTrackingEvent.avsc), which is a type generated from an Avro schema. Events have:
* A `namespace`.
* A `name`.
* A `timestamp`.
* A `Map<String,String>` of `metadata`.
-Events are submitted using the `MetricContext#submitEvent(GobblinTrackingEvent)` method. When called, this method packages the event into an [EventNotification](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/notification/EventNotification.java) and submits it to the metric context using the method `MetricContext#sendNotification(Notification)`. This notification is passed to all metrics context ancestors. Each notification target of each anc [...]
+Events are submitted using the `MetricContext#submitEvent(GobblinTrackingEvent)` method. When called, this method packages the event into an [EventNotification](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/notification/EventNotification.java) and submits it to the metric context using the method `MetricContext#sendNotification(Notification)`. This notification is passed to all metrics context an [...]
-Events can be created manually using Avro constructors, and using the method `context.submitEvent(GobblinTrackinEvent)`, but this is unfriendly when trying to build events incrementally, especially when using metadata. To address this, users can instead use [EventSubmitter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/event/EventSubmitter.java) which is an abstraction around the Avro constructor for GobblinTrackingEvent.
+Events can be created manually using Avro constructors, and using the method `context.submitEvent(GobblinTrackinEvent)`, but this is unfriendly when trying to build events incrementally, especially when using metadata. To address this, users can instead use [EventSubmitter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/event/EventSubmitter.java) which is an abstraction around the Avro constructor [...]
Event Submitter
---------------
@@ -59,7 +59,7 @@ Reporters
Reporters export the metrics and/or events of a metric context to a sink. Reporters extend the interface `com.codahale.metrics.Reporter`. Most reporters will attach themselves to a Metric Context. The reporter can then navigate the Metric Context tree where the Metric Context belongs, get tags and metrics, get notified of events, and export them to the sink.
-The two best entry points for developing reporters are [RecursiveScheduledMetricReporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/reporter/RecursiveScheduledMetricReporter.java) and [EventReporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/reporter/EventReporter.java). These classes do most of the heavy lifting for reporting metrics and events respectively. They are both scheduled re [...]
+The two best entry points for developing reporters are [RecursiveScheduledMetricReporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/RecursiveScheduledMetricReporter.java) and [EventReporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/EventReporter.java). These classes do most of t [...]
RecursiveScheduleMetricReporter
-------------------------------
diff --git a/gobblin-docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications.md b/gobblin-docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications.md
index d26eb2b..9289de2 100644
--- a/gobblin-docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications.md
+++ b/gobblin-docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications.md
@@ -34,5 +34,5 @@ Each metric context manages a set of metrics (like counters, timers, meters, and
</p>
<p>
- To learn more about Gobblin Metrics, check out the <a href="https://github.com/linkedin/gobblin/wiki/Gobblin%20Metrics">Wiki</a> and the <a href="https://github.com/linkedin/gobblin">Gobblin project</a> in Github.
+ To learn more about Gobblin Metrics, check out the <a href="https://gobblin.readthedocs.io/en/latest/metrics/Gobblin-Metrics/">Wiki</a> and the <a href="https://github.com/apache/incubator-gobblin">Gobblin project</a> in Github.
</p>
diff --git a/gobblin-docs/metrics/Implementing-New-Reporters.md b/gobblin-docs/metrics/Implementing-New-Reporters.md
index 42be354..1c79ea5 100644
--- a/gobblin-docs/metrics/Implementing-New-Reporters.md
+++ b/gobblin-docs/metrics/Implementing-New-Reporters.md
@@ -3,7 +3,7 @@ Table of Contents
[TOC]
-The two best entry points for implementing custom reporters are [RecursiveScheduledMetricReporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/reporter/RecursiveScheduledMetricReporter.java) and [EventReporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/reporter/EventReporter.java). Each of these classes automatically schedules reporting, extracts the correct metrics, and calls a single m [...]
+The two best entry points for implementing custom reporters are [RecursiveScheduledMetricReporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/RecursiveScheduledMetricReporter.java) and [EventReporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/EventReporter.java). Each of these cla [...]
In the interest of giving more control to the users, metric and event reporters are kept separate, allowing users to more easily specify separate sinks for events and metrics. However, it is possible to implement a single report that handles both events and metrics.
diff --git a/gobblin-docs/metrics/Metrics-for-Gobblin-ETL.md b/gobblin-docs/metrics/Metrics-for-Gobblin-ETL.md
index c0321be..3ece6c7 100644
--- a/gobblin-docs/metrics/Metrics-for-Gobblin-ETL.md
+++ b/gobblin-docs/metrics/Metrics-for-Gobblin-ETL.md
@@ -126,13 +126,13 @@ Customizing Instrumentation
Custom constructs
-----------------
-When using a custom construct (for example a custom extractor for your data source), you will get the above mentioned instrumentation for free. However, you may want to implement additional metrics. To aid with this, instead of extending the usual class Extractor, you can extend the class `gobblin.instrumented.extractor.InstrumentedExtractor`. Similarly, for each construct there is an instrumented version that allows extension of the default metrics ([InstrumentedExtractor](https://githu [...]
+When using a custom construct (for example a custom extractor for your data source), you will get the above mentioned instrumentation for free. However, you may want to implement additional metrics. To aid with this, instead of extending the usual class Extractor, you can extend the class `gobblin.instrumented.extractor.InstrumentedExtractor`. Similarly, for each construct there is an instrumented version that allows extension of the default metrics ([InstrumentedExtractor](https://githu [...]
All of the instrumented constructs have Javadoc providing with additional information. In general, when extending an instrumented construct, you will have to implement a different method. For example, when extending an InstrumentedExtractor, instead of implementing `readRecord`, you will implement `readRecordImpl`. To make this clearer for the user, implementing `readRecord` will throw a compilation error, and the javadoc of each method specifies the method that should be implemented.
### Instrumentable Interface
-Instrumented constructs extend the interface [Instrumentable](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/instrumented/Instrumentable.java). It contains the following methods:
+Instrumented constructs extend the interface [Instrumentable](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/Instrumentable.java). It contains the following methods:
* `getMetricContext()`: get the default metric context generated for that instance of the construct, with all the appropriate tags. Use this metric context to create any additional metrics.
* `isInstrumentationEnabled()`: returns true if instrumentation is enabled.
@@ -149,6 +149,6 @@ Instrumented constructs have a set of callback methods that are called at differ
Custom Reporters
----------------
-Besides the reporters implemented by default (file, Kafka, and JMX), users can add custom reporters to the classpath and instruct Gobblin to use these reporters. To do this, users should extend the interface [CustomReporterFactory](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/metrics/CustomReporterFactory.java), and specify a comma-separated list of CustomReporterFactory classes in the configuration key `metrics.reporting.custom.builders`.
+Besides the reporters implemented by default (file, Kafka, and JMX), users can add custom reporters to the classpath and instruct Gobblin to use these reporters. To do this, users should extend the interface [CustomReporterFactory](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/CustomReporterFactory.java), and specify a comma-separated list of CustomReporterFactory classes in the configuration key [...]
Gobblin will automatically search for these CustomReporterFactory implementations, instantiate each one with a parameter-less constructor, and then call the method `newScheduledReporter(MetricContext, Properties)`, where the properties contain all of the input configurations supplied to Gobblin. Gobblin will then manage this `ScheduledReporter`.
diff --git a/gobblin-docs/sinks/AvroHdfsDataWriter.md b/gobblin-docs/sinks/AvroHdfsDataWriter.md
index f74d6a4..8ac89fb 100644
--- a/gobblin-docs/sinks/AvroHdfsDataWriter.md
+++ b/gobblin-docs/sinks/AvroHdfsDataWriter.md
@@ -9,7 +9,7 @@ Writes Avro records to Avro data files on Hadoop file systems.
writer.builder.class=org.apache.gobblin.writer.AvroDataWriterBuilder
writer.destination.type=HDFS
-For more info, see [`AvroHdfsDataWriter`](https://github.com/linkedin/gobblin/search?utf8=%E2%9C%93&q=AvroHdfsDataWriter)
+For more info, see [`AvroHdfsDataWriter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/AvroHdfsDataWriter.java)
# Configuration
diff --git a/gobblin-docs/sinks/Couchbase-Writer.md b/gobblin-docs/sinks/Couchbase-Writer.md
index 8b96b30..6527008 100644
--- a/gobblin-docs/sinks/Couchbase-Writer.md
+++ b/gobblin-docs/sinks/Couchbase-Writer.md
@@ -1,4 +1,4 @@
-TODO [Issue #1544](https://github.com/linkedin/gobblin/issues/1544)
+TODO [GOBBLIN-59](https://issues.apache.org/jira/browse/GOBBLIN-59)
# Description
@@ -8,7 +8,7 @@ Writes records to Couchbase.
# Usage
-This writer is part of the gobblin-couchbase [Gobblin Module](developer-guide/GobblinModules). See
- [`CouchbaseWriter`](https://github.com/linkedin/gobblin/search?utf8=%E2%9C%93&q=CouchbaseWriter) for more info.
+This writer is part of the gobblin-couchbase [Gobblin Module](https://github.com/apache/incubator-gobblin/tree/master/gobblin-modules/gobblin-couchbase). See
+ [`CouchbaseWriter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-couchbase/src/main/java/org/apache/gobblin/couchbase/writer/CouchbaseWriter.java) for more info.
# Configuration
\ No newline at end of file
diff --git a/gobblin-docs/sinks/Kafka.md b/gobblin-docs/sinks/Kafka.md
index af47f4b..10c243f 100644
--- a/gobblin-docs/sinks/Kafka.md
+++ b/gobblin-docs/sinks/Kafka.md
@@ -15,7 +15,7 @@ The Kafka writer allows users to create pipelines that ingest data from Gobblin
#Steps
-* Edit the [wikipedia-kafka.pull](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia-kafka.pull) example to get started with setting up ingestion into Kafka. This is a very similar pipeline to the [wikipedia.pull](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull) example which pulls pages from 5 titles from Wikipedia to HDFS. The main differences to note are:
+* Edit the [wikipedia-kafka.pull](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia-kafka.pull) example to get started with setting up ingestion into Kafka. This is a very similar pipeline to the [wikipedia.pull](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull) example which pulls pages from 5 titles from Wikipedia to HDFS. The main differences to note are:
* The `writer.builder.class` is set to `gobblin.kafka.writer.KafkaDataWriterBuilder`. This is the class that creates a Kafka writer.
* The `writer.kafka.topic` is set to `WikipediaExample`. This is the topic that the writer will write the records to.
* The `writer.kafka.producerConfig.bootstrap.servers` is set to `localhost:9092`. This is the address of the kafka broker(s) that the writer must write to.
@@ -53,7 +53,7 @@ INFO [TaskExecutor-0] gobblin.runtime.Fork 261 - Committing data for fork 0 of
INFO [TaskExecutor-0] gobblin.kafka.writer.KafkaDataWriter 211 - Successfully committed 20 records.
```
-* To verify that the records have indeed been ingested into Kafka, you can run a kafka console consumer or run Gobblin's [kafka-console pull file](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia-kafka.pull) which prints the events from Kafka onto the console.
+* To verify that the records have indeed been ingested into Kafka, you can run a kafka console consumer or run Gobblin's [kafka-console pull file](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/kafka-console.pull) which prints the events from Kafka onto the console.
#Configuration Details
diff --git a/gobblin-docs/sources/QueryBasedSource.md b/gobblin-docs/sources/QueryBasedSource.md
index 217f389..baf6b83 100644
--- a/gobblin-docs/sources/QueryBasedSource.md
+++ b/gobblin-docs/sources/QueryBasedSource.md
@@ -1,11 +1,11 @@
[TOC]
# Introduction
-[`QueryBasedSource`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/QueryBasedSource.java)
+[`QueryBasedSource`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedSource.java)
represents a category of sources whose data is pulled by sending queries. A dataset of a source is identified as a
-[`SourceEntity`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/QueryBasedSource.java#L96).
+[`SourceEntity`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedSource.java#L96).
Query can be done by sending HTTP requests or SQL commands. A source often, but not always, has a corresponding
-[`QueryBasedExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/QueryBasedExtractor.java),
+[`QueryBasedExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedExtractor.java),
which defines the way and implements common routines to extract data from the source.
# Constructs
@@ -25,7 +25,7 @@ Like other categories of sources, a `QueryBasedSource` focuses on creating work
- pick up previously failed work units.
At last, it will group several work units as
-[`MultiWorkUnit`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/workunit/MultiWorkUnit.java)
+[`MultiWorkUnit`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/workunit/MultiWorkUnit.java)
according to the `mr.job.max.mappers` configuration (Note: other categories of source might have a different approach to group work units into `MultiWorkUnit`).
## `QueryBasedExtractor`
@@ -37,13 +37,13 @@ according to the `mr.job.max.mappers` configuration (Note: other categories of s
</p>
Currently in Gobblin, depending on how an extractor communicates with a source
-(or [different communication protocols](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/ProtocolSpecificLayer.java)),
+(or [different communication protocols](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/ProtocolSpecificLayer.java)),
a `QueryBasedExtractor` falls into 2 categories:
-[`RestApiExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/restapi/RestApiExtractor.java)
+[`RestApiExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/restapi/RestApiExtractor.java)
and
-[`JdbcExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/jdbc/JdbcExtractor.java).
+[`JdbcExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-sql/src/main/java/org/apache/gobblin/source/jdbc/JdbcExtractor.java).
A specific extractor has to provide some
-[source specific logic](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/SourceSpecificLayer.java)
+[source specific logic](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/SourceSpecificLayer.java)
in order to successfully extract information from the source.
### `build`
diff --git a/gobblin-docs/sources/RestApiSource.md b/gobblin-docs/sources/RestApiSource.md
index 1f25096..ab9dde9 100644
--- a/gobblin-docs/sources/RestApiSource.md
+++ b/gobblin-docs/sources/RestApiSource.md
@@ -3,7 +3,7 @@
# Introduction
A RestApiSource is a [QueryBasedSource](../sources/QueryBasedSource.md) which uses [RESTful](https://en.wikipedia.org/wiki/Representational_state_transfer)
Api for query. `RestApiExtractor` is a `QueryBasedExtractor` that uses REST to communicate with the source. To establish the communication,
-a [`RestApiConnector`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/restapi/RestApiConnector.java) is
+a [`RestApiConnector`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/restapi/RestApiConnector.java) is
required.
# Constructs
@@ -36,7 +36,7 @@ Depends on the routines, [getX], [constructGetXQuery], [extractXFromResponse] ar
There are other interactions between the `RestApiExtractor` layer and `SourceSpecificLayer`. The key points are:
-- A [`ProtocolSpecificLayer`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/ProtocolSpecificLayer.java), such as
+- A [`ProtocolSpecificLayer`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/ProtocolSpecificLayer.java), such as
`RestApiExtractor`, understands the protocol and sets up a routine to communicate with the source
- A `SourceSpecificLayer`, such as `SalesforceExtractor`, knows the source and fits into the routine by providing and analyzing source specific information
diff --git a/gobblin-docs/user-guide/Building-Gobblin.md b/gobblin-docs/user-guide/Building-Gobblin.md
index 1e51cc4..f29e85c 100644
--- a/gobblin-docs/user-guide/Building-Gobblin.md
+++ b/gobblin-docs/user-guide/Building-Gobblin.md
@@ -7,7 +7,7 @@
This page outlines all the options that can be specified when building Gobblin using Gradle. The typical way of building Gobblin is to first checkout the code-base from GitHub and then build the code-base using Gradle.
```
-git clone https://github.com/linkedin/gobblin.git
+git clone https://github.com/apache/incubator-gobblin.git
cd gobblin
./gradlew assemble
```
diff --git a/gobblin-docs/user-guide/Compaction.md b/gobblin-docs/user-guide/Compaction.md
index 5784d39..74bf985 100644
--- a/gobblin-docs/user-guide/Compaction.md
+++ b/gobblin-docs/user-guide/Compaction.md
@@ -32,7 +32,7 @@ Suppose we ingest data from a Kafka broker, and we would like to publish the dat
## Basic Usage
-`MRCompactor.compact()` is the entry point for MapReduce-based compaction. The compaction unit is [`Dataset`](https://github.com/linkedin/gobblin/blob/master/gobblin-compaction/src/main/java/gobblin/compaction/dataset/Dataset.java). `MRCompactor` uses a [`DatasetsFinder`](https://github.com/linkedin/gobblin/blob/master/gobblin-compaction/src/main/java/gobblin/compaction/dataset/DatasetsFinder.java) to find all datasets eligible for compaction. Implementations of `DatasetsFinder` include [...]
+`MRCompactor.compact()` is the entry point for MapReduce-based compaction. The compaction unit is [`Dataset`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/dataset/Dataset.java). `MRCompactor` uses a [`DatasetsFinder`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/dataset/DatasetsFinder.java) to find all datasets eligible for compaction. Implem [...]
In the above example use case, for hourly compaction, each dataset contains an hour's data in the `hourly_staging` folder, e.g., `/data/kafka_topics/PageViewEvent/hourly_staging/2015/10/29/08`; for daily compaction, each dataset contains 24 hourly folder of a day, e.g., `/data/kafka_topics/PageViewEvent/hourly/2015/10/29`. In hourly compaction, you may use the following config properties:
diff --git a/gobblin-docs/user-guide/Gobblin-Deployment.md b/gobblin-docs/user-guide/Gobblin-Deployment.md
index 4b78387..00b8aef 100644
--- a/gobblin-docs/user-guide/Gobblin-Deployment.md
+++ b/gobblin-docs/user-guide/Gobblin-Deployment.md
@@ -7,7 +7,7 @@ Deployment Overview <a name="Standalone-Overview"></a>
--------------------
One important feature of Gobblin is that it can be run on different platforms. Currently, Gobblin can run in standalone mode (which runs on a single machine), and on Hadoop MapReduce mode (which runs on a Hadoop cluster). This page summarizes the different deployment modes of Gobblin. It is important to understand the architecture of Gobblin in a specific deployment mode, so this page also describes the architecture of each deployment mode.
-Gobblin supports Java 7 and up, but can only run on Hadoop 2.x. By default, Gobblin will build against Hadoop 2.x, run `./gradlew clean build`. More information on how to build Gobblin can be found [here](https://github.com/linkedin/gobblin/blob/master/README.md). All directories/paths referred below are relative to `gobblin-dist`.
+Gobblin supports Java 7 and up, but can only run on Hadoop 2.x. By default, Gobblin will build against Hadoop 2.x, run `./gradlew clean build`. More information on how to build Gobblin can be found [here](https://github.com/apache/incubator-gobblin/blob/master/README.md). All directories/paths referred below are relative to `gobblin-dist`.
Standalone Architecture <a name="Standalone-Architecture"></a>
--------------------
diff --git a/gobblin-docs/user-guide/Gobblin-Schedulers.md b/gobblin-docs/user-guide/Gobblin-Schedulers.md
index b0c1022..4166164 100644
--- a/gobblin-docs/user-guide/Gobblin-Schedulers.md
+++ b/gobblin-docs/user-guide/Gobblin-Schedulers.md
@@ -8,7 +8,7 @@ Gobblin jobs can be scheduled on a recurring basis using a few different tools.
# Quartz
-Gobblin has a built in Quartz scheduler as part of the [`JobScheduler`](https://github.com/linkedin/gobblin/blob/master/gobblin-scheduler/src/main/java/gobblin/scheduler/JobScheduler.java) class. This class integrates with the Gobblin [`SchedulerDaemon`](https://github.com/linkedin/gobblin/blob/master/gobblin-scheduler/src/main/java/gobblin/scheduler/SchedulerDaemon.java), which can be run using the Gobblin [`bin/gobblin-standalone.sh](https://github.com/linkedin/gobblin/blob/master/bin/ [...]
+Gobblin has a built in Quartz scheduler as part of the [`JobScheduler`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/scheduler/JobScheduler.java) class. This class integrates with the Gobblin [`SchedulerDaemon`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/scheduler/SchedulerDaemon.java), which can be run using the Gobblin [`bin/gobblin-standalone.sh](https://github.com [...]
So in order to take advantage of the Quartz scheduler two steps need to be taken:
@@ -18,9 +18,9 @@ So in order to take advantage of the Quartz scheduler two steps need to be taken
# Azkaban
-Gobblin can be launched via [Azkaban](https://azkaban.github.io/), and open-source Workflow Manager for scheduling and launching Hadoop jobs. Gobblin's [`AzkabanJobLauncher`](https://github.com/linkedin/gobblin/blob/master/gobblin-azkaban/src/main/java/gobblin/azkaban/AzkabanJobLauncher.java) can be used to launch a Gobblin job through Azkaban.
+Gobblin can be launched via [Azkaban](https://azkaban.github.io/), and open-source Workflow Manager for scheduling and launching Hadoop jobs. Gobblin's [`AzkabanJobLauncher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/azkaban/AzkabanJobLauncher.java) can be used to launch a Gobblin job through Azkaban.
-One has to follow the typical setup to create a zip file that can be uploaded to Azkaban (it should include all dependent jars, which can be found in `gobblin-dist.tar.gz`). The `.job` file for the Azkaban Job should contain all configuration properties that would be put in a `.pull` file (for example, the [Wikipedia Example](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull) `.pull` file). All Gobblin system dependent properties (e.g. [`co [...]
+One has to follow the typical setup to create a zip file that can be uploaded to Azkaban (it should include all dependent jars, which can be found in `gobblin-dist.tar.gz`). The `.job` file for the Azkaban Job should contain all configuration properties that would be put in a `.pull` file (for example, the [Wikipedia Example](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull) `.pull` file). All Gobblin system dependent properties (e [...]
In the Azkaban `.job` file, the `type` parameter should be set to `hadoopJava` (see [here](http://azkaban.github.io/azkaban/docs/latest/#hadoopjava-type) for more information about the `hadoopJava` Job Type). The `job.class` parameter should be set to `gobblin.azkaban.AzkabanJobLauncher`.
@@ -38,13 +38,14 @@ This guide focuses on getting Gobblin to run in as a stand alone Java Process. T
#### Example Config Files
-[`gobblin-oozie/src/main/resources/local`](https://github.com/linkedin/gobblin/tree/master/gobblin-oozie/src/main/resources/local) contains sample configuration files for launching Gobblin Oozie. There are a number of important files in this directory:
+[`gobblin-oozie/src/main/resources/local`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local) contains sample configuration files for launching Gobblin Oozie. There are a number of important files in this directory:
-`gobblin-oozie-example-system.properties` contains default system level properties for Gobblin. When launched with Oozie, Gobblin will run inside a map task; it is thus recommended to configure Gobblin to write directly to HDFS rather than the local file system. The property `fs.uri` in this file should be changed to point to the NameNode of the Hadoop File System the job should write to. By default, all data is written under a folder called `gobblin-out`; to change this modify the `gobb [...]
+[`gobblin-oozie-example-system.properties`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-system.properties) contains default system level properties for Gobblin. When launched with Oozie, Gobblin will run inside a map task; it is thus recommended to configure Gobblin to write directly to HDFS rather than the local file system. The property `fs.uri` in this file should be changed to point to the NameNode of the Hadoop [...]
-`gobblin-oozie-example-workflow.properties` contains default Oozie properties for any job launched. It is also the entry point for launching an Oozie job (e.g. to launch an Oozie job from the command line you execute `oozie job -config gobblin-oozie-example-workflow.properties -run`). In this file one needs to update the `name.node` and `resource.manager` to the values specific to their environment. Another important property in this file is `oozie.wf.application.path`; it points to a fo [...]
+[`gobblin-oozie-example-workflow.properties`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-workflow.properties) contains default Oozie properties for any job launched. It is also the entry point for launching an Oozie job (e.g. to launch an Oozie job from the command line you execute `oozie job -config gobblin-oozie-example-workflow.properties -run`). In this file one needs to update the `name.node` and `resource.man [...]
-`gobblin-oozie-example-workflow.xml` contains an example Oozie workflow. This example simply launches a Java process that invokes the main method of the [`CliLocalJobLauncher`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/local/CliLocalJobLauncher.java). The main method of this class expects two file paths to be passed to it (once again these files need to be on HDFS). The `jobconfig` arg should point to a file on HDFS containing all job c [...]
+[`gobblin-oozie-example-workflow.xml`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-workflow.xml) contains an example Oozie workflow. This example simply launches a Java process that invokes the main method of the [`CliLocalJobLauncher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/local/CliLocalJobLauncher.java). The main method of this class expect [...]
+).
<!---Ying Do you think we can add some descriptions about launching through MR mode? The simplest way is to use the <shell> tag and invoke `gobblin-mapreduce.sh`. I've tested it before.-->
@@ -66,7 +67,7 @@ Launching Gobblin in mapreduce Mode works quite similar to the local mode. In th
#### Example Config Files
-[`gobblin-oozie/src/main/resources/mapreduce`](https://github.com/linkedin/gobblin/tree/master/gobblin-oozie/src/main/resources/mapreduce) contains sample configuration files for launching Gobblin Oozie in Mapreduce mode. The main difference to launching Gobblin Oozie in Local mode are a view extra MapReduce related configuration variables in the sysconfig.properties file and launching CliMRJobLauncher instead CliLocalJobLauncher.
+[`gobblin-oozie/src/main/resources/mapreduce`](https://github.com/apache/incubator-gobblin/tree/master/gobblin-oozie/src/test/resources/mapreduce) contains sample configuration files for launching Gobblin Oozie in Mapreduce mode. The main difference to launching Gobblin Oozie in Local mode are a view extra MapReduce related configuration variables in the sysconfig.properties file and launching CliMRJobLauncher instead CliLocalJobLauncher.
#### Further steps
diff --git a/gobblin-docs/user-guide/Gobblin-on-Yarn.md b/gobblin-docs/user-guide/Gobblin-on-Yarn.md
index 194cf4a..548f0f4 100644
--- a/gobblin-docs/user-guide/Gobblin-on-Yarn.md
+++ b/gobblin-docs/user-guide/Gobblin-on-Yarn.md
@@ -41,7 +41,7 @@ Heiix relies on ZooKeeper for its operations, and particularly for maintaining t
## Gobblin Yarn Application Launcher
-The Gobblin Yarn Application Launcher (implemented by class [`GobblinYarnAppLauncher`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/GobblinYarnAppLauncher.java)) is the client/driver of a Gobblin Yarn application. The first thing the `GobblinYarnAppLauncher` does when it starts is to register itself with Helix as a _spectator_ and creates a new Helix cluster with name specified through the configuration property `gobblin.yarn.helix.cluster.name` [...]
+The Gobblin Yarn Application Launcher (implemented by class [`GobblinYarnAppLauncher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java)) is the client/driver of a Gobblin Yarn application. The first thing the `GobblinYarnAppLauncher` does when it starts is to register itself with Helix as a _spectator_ and creates a new Helix cluster with name specified through the configuration property `gobblin.yarn. [...]
The `GobblinYarnAppLauncher` then sets up the Gobblin Yarn application and submits it to run on Yarn. Once the Yarn application successfully starts running, it starts an application state monitor that periodically checks the state of the Gobblin Yarn application. If the state is one of the exit states (`FINISHED`, `FAILED`, or `KILLED`), the `GobblinYarnAppLauncher` shuts down itself.
@@ -49,19 +49,19 @@ Upon successfully submitting the application to run on Yarn, the `GobblinYarnApp
### YarnAppSecurityManager
-The [`YarnAppSecurityManager`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/YarnAppSecurityManager.java) works with the [`YarnContainerSecurityManager`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/YarnContainerSecurityManager.java) running in the ApplicationMaster and the WorkUnitRunner for a complete solution for security and delegation token management. The `YarnAppSecurityManager` is responsible for [...]
+The [`YarnAppSecurityManager`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnAppSecurityManager.java) works with the [`YarnContainerSecurityManager`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnContainerSecurityManager.java) running in the ApplicationMaster and the WorkUnitRunner for a complete solution for security and delegation token management. The `Yarn [...]
### LogCopier
-The service [`LogCopier`](https://github.com/linkedin/gobblin/blob/master/gobblin-utility/src/main/java/gobblin/util/logs/LogCopier.java) in `GobblinYarnAppLauncher` streams the ApplicationMaster and WorkUnitRunner logs in near real-time from the central location on HDFS where the logs are streamed to from the ApplicationMaster and WorkUnitRunner containers, to the local directory specified through the configuration property `gobblin.yarn.logs.sink.root.dir` on the machine where the `Gob [...]
+The service [`LogCopier`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/logs/LogCopier.java) in `GobblinYarnAppLauncher` streams the ApplicationMaster and WorkUnitRunner logs in near real-time from the central location on HDFS where the logs are streamed to from the ApplicationMaster and WorkUnitRunner containers, to the local directory specified through the configuration property `gobblin.yarn.logs.sink.root.dir` on the mac [...]
## Gobblin ApplicationMaster
-The ApplicationMaster process runs the [`GobblinApplicationMaster`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/GobblinApplicationMaster.java), which uses a `ServiceManager` to manage the services supporting the operation of the ApplicationMaster process. The services running in `GobblinApplicationMaster` will be discussed later. When it starts, the first thing `GobblinApplicationMaster` does is to connect to ZooKeeper and register itself as a [...]
+The ApplicationMaster process runs the [`GobblinApplicationMaster`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinApplicationMaster.java), which uses a `ServiceManager` to manage the services supporting the operation of the ApplicationMaster process. The services running in `GobblinApplicationMaster` will be discussed later. When it starts, the first thing `GobblinApplicationMaster` does is to connect to ZooKeeper and re [...]
### YarnService
-The service [`YarnService`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/YarnService.java) handles all Yarn-related task including the following:
+The service [`YarnService`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnService.java) handles all Yarn-related task including the following:
* Registering and un-registering the ApplicationMaster with the Yarn ResourceManager.
* Requesting the initial set of containers from the Yarn ResourceManager.
@@ -71,37 +71,37 @@ This design makes it switch to a different resource manager, e.g., Mesos, by rep
### GobblinHelixJobScheduler
-[`GobblinApplicationMaster`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/GobblinApplicationMaster.java) runs the [`GobblinHelixJobScheduler`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/GobblinHelixJobScheduler.java) that schedules jobs to run through the Helix [Distributed Task Execution Framework](http://helix.apache.org/0.7.1-docs/recipes/task_dag_execution.html). For each Gobblin job run, the `Gob [...]
+[`GobblinApplicationMaster`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinApplicationMaster.java) runs the [`GobblinHelixJobScheduler`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobScheduler.java) that schedules jobs to run through the Helix [Distributed Task Execution Framework](http://helix.apache.org/0.7.1-docs/recipes/task_dag_executi [...]
-Like the [`LocalJobLauncher`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/local/LocalJobLauncher.java) and [`MRJobLauncher`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/mapreduce/MRJobLauncher.java), the `GobblinHelixJobLauncher` handles output data commit and job state persistence.
+Like the [`LocalJobLauncher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/local/LocalJobLauncher.java) and [`MRJobLauncher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java), the `GobblinHelixJobLauncher` handles output data commit and job state persistence.
### LogCopier
-The service [`LogCopier`](https://github.com/linkedin/gobblin/blob/master/gobblin-utility/src/main/java/gobblin/util/logs/LogCopier.java) in `GobblinApplicationMaster` streams the ApplicationMaster logs in near real-time from the machine running the ApplicationMaster container to a central location on HDFS so the logs can be accessed at runtime. More details on this can be found in [Log Aggregation](#log-aggregation).
+The service [`LogCopier`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/logs/LogCopier.java) in `GobblinApplicationMaster` streams the ApplicationMaster logs in near real-time from the machine running the ApplicationMaster container to a central location on HDFS so the logs can be accessed at runtime. More details on this can be found in [Log Aggregation](#log-aggregation).
### YarnContainerSecurityManager
-The [`YarnContainerSecurityManager`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/YarnContainerSecurityManager.java) runs in both the ApplicationMaster and the WorkUnitRunner. When it starts, it registers a message handler with the `HelixManager` for handling messages on refreshes of the delegation token. Once such a message is received, the `YarnContainerSecurityManager` gets the path to the token file on HDFS from the message, and updated the [...]
+The [`YarnContainerSecurityManager`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnContainerSecurityManager.java) runs in both the ApplicationMaster and the WorkUnitRunner. When it starts, it registers a message handler with the `HelixManager` for handling messages on refreshes of the delegation token. Once such a message is received, the `YarnContainerSecurityManager` gets the path to the token file on HDFS from the messag [...]
## Gobblin WorkUnitRunner
-The WorkUnitRunner process runs the [`GobblinWorkUnitRunner`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/GobblinWorkUnitRunner.java), which uses a `ServiceManager` to manage the services supporting the operation of the WorkUnitRunner process. The services running in `GobblinWorkUnitRunner` will be discussed later. When it starts, the first thing `GobblinWorkUnitRunner` does is to connect to ZooKeeper and register itself as a Helix _participant [...]
+The WorkUnitRunner process runs as part of [`GobblinTaskRunner`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java), which uses a `ServiceManager` to manage the services supporting the operation of the WorkUnitRunner process. The services running in `GobblinWorkUnitRunner` will be discussed later. When it starts, the first thing `GobblinWorkUnitRunner` does is to connect to ZooKeeper and register itself [...]
### TaskExecutor
-The [`TaskExecutor`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/TaskExecutor.java) remains the same as in the standalone and MR modes, and is purely responsible for running tasks assigned to a WorkUnitRunner.
+The [`TaskExecutor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/TaskExecutor.java) remains the same as in the standalone and MR modes, and is purely responsible for running tasks assigned to a WorkUnitRunner.
### GobblinHelixTaskStateTracker
-The [`GobblinHelixTaskStateTracker`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/GobblinHelixTaskStateTracker.java) has a similar responsibility as the `LocalTaskStateTracker` and `MRTaskStateTracker`: keeping track of the state of running tasks including operational metrics, e.g., total records pulled, records pulled per second, total bytes pulled, bytes pulled per second, etc.
+The [`GobblinHelixTaskStateTracker`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixTaskStateTracker.java) has a similar responsibility as the `LocalTaskStateTracker` and `MRTaskStateTracker`: keeping track of the state of running tasks including operational metrics, e.g., total records pulled, records pulled per second, total bytes pulled, bytes pulled per second, etc.
### LogCopier
-The service [`LogCopier`](https://github.com/linkedin/gobblin/blob/master/gobblin-utility/src/main/java/gobblin/util/logs/LogCopier.java) in `GobblinWorkUnitRunner` streams the WorkUnitRunner logs in near real-time from the machine running the WorkUnitRunner container to a central location on HDFS so the logs can be accessed at runtime. More details on this can be found in [Log Aggregation](#log-aggregation).
+The service [`LogCopier`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/logs/LogCopier.java) in `GobblinWorkUnitRunner` streams the WorkUnitRunner logs in near real-time from the machine running the WorkUnitRunner container to a central location on HDFS so the logs can be accessed at runtime. More details on this can be found in [Log Aggregation](#log-aggregation).
### YarnContainerSecurityManager
-The [`YarnContainerSecurityManager`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/YarnContainerSecurityManager.java) in `GobblinWorkUnitRunner` works in the same way as it in `GobblinApplicationMaster`.
+The [`YarnContainerSecurityManager`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnContainerSecurityManager.java) in `GobblinWorkUnitRunner` works in the same way as it in `GobblinApplicationMaster`.
## Failure Handling
@@ -305,8 +305,8 @@ Gobblin on Yarn is backward compatible and supports existing Gobblin jobs runnin
# Monitoring
-Gobblin Yarn uses the [Gobblin Metrics](../metrics/Gobblin-Metrics) library for collecting and reporting metrics at the container, job, and task levels. Each `GobblinWorkUnitRunner` maintains a [`ContainerMetrics`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/ContainerMetrics.java) that is the parent of the [`JobMetrics`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/util/JobMetrics.java) of each j [...]
+Gobblin Yarn uses the [Gobblin Metrics](../metrics/Gobblin-Metrics) library for collecting and reporting metrics at the container, job, and task levels. Each `GobblinWorkUnitRunner` maintains a [`ContainerMetrics`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/ContainerMetrics.java) that is the parent of the [`JobMetrics`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gob [...]
Collected metrics can be reported to various sinks such as Kafka, files, and JMX, depending on the configuration. Specifically, `metrics.enabled` controls whether metrics collecting and reporting are enabled or not. `metrics.reporting.kafka.enabled`, `metrics.reporting.file.enabled`, and `metrics.reporting.jmx.enabled` control whether collected metrics should be reported or not to Kafka, files, and JMX, respectively. Please refer to [Metrics Properties](Configuration-Properties-Glossary# [...]
-In addition to metric collecting and reporting, Gobblin Yarn also supports writing job execution information to a MySQL-backed job execution history store, which keeps track of job execution information. Please refer to the [DDL](https://github.com/linkedin/gobblin/blob/master/gobblin-metastore/src/main/resources/gobblin_job_history_store.sql) for the relevant MySQL tables. Detailed information on the job execution history store including how to configure it can be found [here](Job-Execu [...]
+In addition to metric collecting and reporting, Gobblin Yarn also supports writing job execution information to a MySQL-backed job execution history store, which keeps track of job execution information. Please refer to the [DDL](https://github.com/apache/incubator-gobblin/tree/master/gobblin-metastore/src/main/resources/db/migration) for the relevant MySQL tables. Detailed information on the job execution history store including how to configure it can be found [here](Job-Execution-Hist [...]
diff --git a/gobblin-docs/user-guide/Hive-Registration.md b/gobblin-docs/user-guide/Hive-Registration.md
index c9cb6bb..5751e50 100644
--- a/gobblin-docs/user-guide/Hive-Registration.md
+++ b/gobblin-docs/user-guide/Hive-Registration.md
@@ -6,17 +6,17 @@ Gobblin has the ability to register the ingested/compacted data in Hive. This al
## How Hive Registration Works in Gobblin
-Hive registration is done in [`HiveRegister`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/HiveRegister.java). After the data is published, the publisher or compaction runner will create an instance of `HiveRegister`, and for each path that should be registered in Hive, the publisher or compaction runner will use a specific [`HiveRegistrationPolicy`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/ [...]
+Hive registration is done in [`HiveRegister`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveRegister.java). After the data is published, the publisher or compaction runner will create an instance of `HiveRegister`, and for each path that should be registered in Hive, the publisher or compaction runner will use a specific [`HiveRegistrationPolicy`](https://github.com/apache/incubator-gobblin/blob/master/gobblin- [...]
### `HiveSpec`
-A `HiveSpec` specifies how a path should be registered in Hive, i.e., which database, which table, which partition should the path be registered. An example is [`SimpleHiveSpec`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/spec/SimpleHiveSpec.java).
+A `HiveSpec` specifies how a path should be registered in Hive, i.e., which database, which table, which partition should the path be registered. An example is [`SimpleHiveSpec`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/SimpleHiveSpec.java).
### `HiveRegistrationPolicy`
`HiveRegistrationPolicy` is responsible for generating `HiveSpec`s given a path. For example, if you want paths ending with a date (e.g., `/(something)/2016/05/22`) to be registered in the corresponding daily partition (e.g., `daily-2016-05-22`), you can create an implementation of `HiveRegistrationPolicy` that contains the logic of converting such a path into a Hive partition.
-An example is [`HiveRegistrationPolicyBase`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/policy/HiveRegistrationPolicyBase.java), which provides base implementation for getting database names and table names for a path:
+An example is [`HiveRegistrationPolicyBase`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/policy/HiveRegistrationPolicyBase.java), which provides base implementation for getting database names and table names for a path:
* A database/table name can be specified explicitly in `hive.database.name` or `hive.table.name`.
* Alternatively, a database/table regex can be provided in `hive.database.regex` or `hive.table.regex`. The regex will be matched against the path to be registered, and if they match, the first group is considered the database/table name.
@@ -27,13 +27,13 @@ One should in general extend `HiveRegistrationPolicyBase` when implementing a ne
### `HiveSerDeManager`
-If the data to be registered is in a format other than plain text (CSV, TSV, etc.), you often need to use a SerDe and specify some SerDe properties including the type of SerDe, input format, output format, schema, etc. This is done in [`HiveSerDeManager`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/HiveSerDeManager.java), which is part of a [`HiveRegistrationUnit`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-regist [...]
+If the data to be registered is in a format other than plain text (CSV, TSV, etc.), you often need to use a SerDe and specify some SerDe properties including the type of SerDe, input format, output format, schema, etc. This is done in [`HiveSerDeManager`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveSerDeManager.java), which is part of a [`HiveRegistrationUnit`](https://github.com/apache/incubator-gobblin/blob [...]
### Predicate and Activity
-One or more `Predicate`s can be attached to a `HiveSpec`. If a `HiveSpec` contains `Predicate`s, unless `Predicate`s return `true`, the `HiveSpec` will not be registered. This is useful in cases where, for example, one only wants to register a path in Hive if a particular Hive table or partition doesn't already exist. An example is [`TableNotExistPredicate`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/spec/predicate/TableNotExistPr [...]
+One or more `Predicate`s can be attached to a `HiveSpec`. If a `HiveSpec` contains `Predicate`s, unless `Predicate`s return `true`, the `HiveSpec` will not be registered. This is useful in cases where, for example, one only wants to register a path in Hive if a particular Hive table or partition doesn't already exist. An example is [`TableNotExistPredicate`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/predic [...]
-One or more [`Activity`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/spec/activity/Activity.java)s can be attached to a `HiveSpec`. There are two types of activities: pre-activities and post-activities, which will be executed before and after a `HiveSpec` is registered, respectively. This is useful, for example, when you need to drop/alter a table/partition before or after a path is registered. An example is [`DropTableActivity`](h [...]
+One or more [`Activity`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/activity/Activity.java)s can be attached to a `HiveSpec`. There are two types of activities: pre-activities and post-activities, which will be executed before and after a `HiveSpec` is registered, respectively. This is useful, for example, when you need to drop/alter a table/partition before or after a path is registered. An example is [`Dr [...]
## How to Use Hive Registration in Your Gobblin Job
@@ -46,10 +46,10 @@ Example table/partition properties are "owner" and "retention", example storage
If you are running a Gobblin ingestion job:
-* If data is published in the job (which is the default case), use a job-level data publisher that can perform Hive registration, such as [`BaseDataPublisherWithHiveRegistration`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/publisher/BaseDataPublisherWithHiveRegistration.java). If you need to do Hive registration with a different publisher than `BaseDataPublisher`, you will need to extend that publisher to do Hive registration, which will be similar [...]
-* If data is published in the tasks, use [`HiveRegistrationPublisher`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/publisher/HiveRegistrationPublisher.java) as the job-level data publisher. This publisher does not publish any data; it only does Hive registration.
+* If data is published in the job (which is the default case), use a job-level data publisher that can perform Hive registration, such as [`BaseDataPublisherWithHiveRegistration`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/BaseDataPublisherWithHiveRegistration.java). If you need to do Hive registration with a different publisher than `BaseDataPublisher`, you will need to extend that publisher to do Hive registration, wh [...]
+* If data is published in the tasks, use [`HiveRegistrationPublisher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/HiveRegistrationPublisher.java) as the job-level data publisher. This publisher does not publish any data; it only does Hive registration.
-If you are running a Gobblin compaction job: add [`HiveRegistrationCompactorListener`](https://github.com/linkedin/gobblin/blob/master/gobblin-compaction/src/main/java/gobblin/compaction/hive/registration/HiveRegistrationCompactorListener.java) to the list of compaction listeners by adding the class name to property `compaction.listeners`.
+If you are running a Gobblin compaction job: add [`HiveRegistrationCompactorListener`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/hive/registration/HiveRegistrationCompactorListener.java) to the list of compaction listeners by adding the class name to property `compaction.listeners`.
## Hive Registration Config Properties
diff --git a/gobblin-docs/user-guide/Job-Execution-History-Store.md b/gobblin-docs/user-guide/Job-Execution-History-Store.md
index 04357a0..cbf3d01 100644
--- a/gobblin-docs/user-guide/Job-Execution-History-Store.md
+++ b/gobblin-docs/user-guide/Job-Execution-History-Store.md
@@ -54,7 +54,7 @@ The following table summarizes task-level execution information the Job Executio
Default Implementation
--------------------------------
-The default implementation of the Job Execution History Store stores job execution information into a MySQL database in a few different tables. Specifically, the following tables are used and should be created before writing to the store is enabled. Checkout the MySQL [DDLs](https://github.com/linkedin/gobblin/blob/master/gobblin-metastore/src/main/resources/gobblin_job_history_store.sql) of the tables for detailed columns of each table.
+The default implementation of the Job Execution History Store stores job execution information into a MySQL database in a few different tables. Specifically, the following tables are used and should be created before writing to the store is enabled. Checkout the MySQL [DDLs](https://github.com/apache/incubator-gobblin/tree/master/gobblin-metastore/src/main/resources/db/migration) of the tables for detailed columns of each table.
* Table `gobblin_job_executions` stores basic information about a job execution including the start and end times, job running state, number of launched and completed tasks, etc.
* Table `gobblin_task_executions` stores basic information on task executions of a job, including the start and end times, task running state, task failure message if any, etc, of each task.
diff --git a/gobblin-docs/user-guide/Monitoring.md b/gobblin-docs/user-guide/Monitoring.md
index 7db9fc0..c70b103 100644
--- a/gobblin-docs/user-guide/Monitoring.md
+++ b/gobblin-docs/user-guide/Monitoring.md
@@ -42,7 +42,7 @@ Among the above metrics, `${metric_group}.${id}.records` and `${metric_group}.${
Job Execution History Store
--------------------
-Gobblin also supports writing job execution information to a job execution history store backed by a database of choice. Gobblin uses MySQL by default and it ships with the SQL [DDLs](https://github.com/linkedin/gobblin/wiki/files/gobblin_job_history_store_ddlwq.sql) of the relevant MySQL tables, although it still allows users to choose which database to use as long as the schema of the tables is compatible. Users can use the properties `job.history.store.url` and `job.history.store.jdb [...]
+Gobblin also supports writing job execution information to a job execution history store backed by a database of choice. Gobblin uses MySQL by default and it ships with the SQL [DDLs](https://github.com/apache/incubator-gobblin/tree/master/gobblin-metastore/src/main/resources/db/migration) of the relevant MySQL tables, although it still allows users to choose which database to use as long as the schema of the tables is compatible. Users can use the properties `job.history.store.url` and [...]
```
job.history.store.url=jdbc:mysql://localhost/gobblin
diff --git a/gobblin-docs/user-guide/State-Management-and-Watermarks.md b/gobblin-docs/user-guide/State-Management-and-Watermarks.md
index 2e53918..da5b5ea 100644
--- a/gobblin-docs/user-guide/State-Management-and-Watermarks.md
+++ b/gobblin-docs/user-guide/State-Management-and-Watermarks.md
@@ -23,7 +23,7 @@ In the next run, the `Source` will call `SourceState.getPreviousWorkUnitStates()
**watermark type**
-A watermark can be of any custom type by implementing the [`Watermark`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/extractor/Watermark.java) interface. For example, for Kafka-HDFS ingestion, if each `WorkUnit` is responsible for pulling a single Kafka topic partition, a watermark is a single `long` value representing a Kafka offset. If each `WorkUnit` is responsible for pulling multiple Kafka topic partitions, a watermark can be a list of `lo [...]
+A watermark can be of any custom type by implementing the [`Watermark`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Watermark.java) interface. For example, for Kafka-HDFS ingestion, if each `WorkUnit` is responsible for pulling a single Kafka topic partition, a watermark is a single `long` value representing a Kafka offset. If each `WorkUnit` is responsible for pulling multiple Kafka topic partitions, a watermark c [...]
### Task Failures
@@ -31,7 +31,7 @@ A task may pull some data and then fail. If a task fails and job commit policy s
### Multi-Dataset Jobs
-Currently the only state store implementation Gobblin provides is [`FsStateStore`](https://github.com/linkedin/gobblin/blob/master/gobblin-metastore/src/main/java/gobblin/metastore/FsStateStore.java) which uses Hadoop SequenceFiles to store the states. By default, each job run reads the SequenceFile created by the previous run, and generates a new SequenceFile. This creates a pitfall when a job pulls data from multiple datasets: if a data set is skipped in a job run for whatever reason ( [...]
+Currently the only state store implementation Gobblin provides is [`FsStateStore`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metastore/src/main/java/org/apache/gobblin/metastore/FsStateStore.java) which uses Hadoop SequenceFiles to store the states. By default, each job run reads the SequenceFile created by the previous run, and generates a new SequenceFile. This creates a pitfall when a job pulls data from multiple datasets: if a data set is skipped in a job run fo [...]
**Example**: suppose we schedule a Gobblin job to pull a Kafka topic from a Kafka broker, which has 10 partitions. In this case each partition is a dataset. In one of the job runs, a partition is skipped due to either being blacklisted or some failure. If no `WorkUnit` is created for this partition, this partition's watermark will not be checked in to the state store, and will not be available for the next run.
@@ -47,7 +47,7 @@ Note that when using Dataset URNs, **each `WorkUnit` can only have one `dataset.
## Gobblin State Deep Dive
-Gobblin involves several types of states during a job run, such as `JobState`, `TaskState`, `WorkUnit`, etc. They all extend the [`State`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/configuration/State.java) class, which is a wrapper around [`Properties`](https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html) and provides some useful utility functions.
+Gobblin involves several types of states during a job run, such as `JobState`, `TaskState`, `WorkUnit`, etc. They all extend the [`State`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/configuration/State.java) class, which is a wrapper around [`Properties`](https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html) and provides some useful utility functions.
### `State` class hierarchy
@@ -78,7 +78,7 @@ The `MultiWorkUnit` is useful for finer-grained control and load balancing. With
* The job launcher then passes the `JobState` (as a `SourceState` object) to the `Source`, based on which the `Source` will create a set of `WorkUnit`s. Note that when creating `WorkUnit`s, the `Source` should not add properties in `SourceState` into the `WorkUnit`s, which will be done when each `WorkUnit` is executed in a `Task`. The reason is that since the job launcher runs in a single JVM, creating a large number of `WorkUnit`s, each containing a copy of the `SourceState`, may cause OOM.
* The job launcher prepares to run the `WorkUnit`s.
- * In standalone mode, the job launcher will add properties in the `JobState` into each `WorkUnit` (if a property in `JobState` already exists in the `WorkUnit`, it will NOT be overwritten, i.e., the value in the `WorkUnit` takes precedence). Then for each `WorkUnit` it creates a `Task` to run the `WorkUnit`, and submits all these Tasks to a [`TaskExecutor`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/TaskExecutor.java) which will run the [...]
+ * In standalone mode, the job launcher will add properties in the `JobState` into each `WorkUnit` (if a property in `JobState` already exists in the `WorkUnit`, it will NOT be overwritten, i.e., the value in the `WorkUnit` takes precedence). Then for each `WorkUnit` it creates a `Task` to run the `WorkUnit`, and submits all these Tasks to a [`TaskExecutor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/TaskExecutor.java) [...]
* In MR mode, the job launcher will serialize the `JobState` and each `WorkUnit` into a file, which will be picked up by the mappers. It then creates, configures and submits a Hadoop job.
After this step, the job launcher will be waiting till all tasks finish.
diff --git a/gobblin-docs/user-guide/Working-with-Job-Configuration-Files.md b/gobblin-docs/user-guide/Working-with-Job-Configuration-Files.md
index 87aaa19..b4f3d42 100644
--- a/gobblin-docs/user-guide/Working-with-Job-Configuration-Files.md
+++ b/gobblin-docs/user-guide/Working-with-Job-Configuration-Files.md
@@ -5,7 +5,7 @@ Table of Contents
Job Configuration Basics
--------------------
-A Job configuration file is a text file with extension `.pull` or `.job` that defines the job properties that can be loaded into a Java [Properties](http://docs.oracle.com/javase/7/docs/api/java/util/Properties.html) object. Gobblin uses [commons-configuration](http://commons.apache.org/proper/commons-configuration/) to allow variable substitutions in job configuration files. You can find some example Gobblin job configuration files [here](https://github.com/linkedin/gobblin/tree/master/ [...]
+A Job configuration file is a text file with extension `.pull` or `.job` that defines the job properties that can be loaded into a Java [Properties](http://docs.oracle.com/javase/7/docs/api/java/util/Properties.html) object. Gobblin uses [commons-configuration](http://commons.apache.org/proper/commons-configuration/) to allow variable substitutions in job configuration files. You can find some example Gobblin job configuration files [here](https://github.com/apache/incubator-gobblin/tree [...]
A Job configuration file typically includes the following properties, in additional to any mandatory configuration properties required by the custom [Gobblin Constructs](Gobblin-Architecture#gobblin-constructs) classes. For a complete reference of all configuration properties supported by Gobblin, please refer to [Configuration Properties Glossary](Configuration-Properties-Glossary).
diff --git a/gobblin-docs/user-guide/Working-with-the-ForkOperator.md b/gobblin-docs/user-guide/Working-with-the-ForkOperator.md
index 40bf854..2416c9e 100644
--- a/gobblin-docs/user-guide/Working-with-the-ForkOperator.md
+++ b/gobblin-docs/user-guide/Working-with-the-ForkOperator.md
@@ -6,7 +6,7 @@ Table of Contents
Overview of the ForkOperator
--------------------
-The [`ForkOperator`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/fork/ForkOperator.java) is a type of control operators that allow a task flow to branch into multiple streams (or forked branches) as represented by a [`Fork`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/Fork.java), each of which goes to a separately configured sink with its own data writer. The `ForkOperator` gives users more flexibilit [...]
+The [`ForkOperator`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/ForkOperator.java) is a type of control operators that allow a task flow to branch into multiple streams (or forked branches) as represented by a [`Fork`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java), each of which goes to a separately configured sink with its own data writer. The [...]
<p align="center">
<figure>
@@ -20,21 +20,21 @@ Using the ForkOperator
### Basics of Usage
-The [`ForkOperator`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/fork/ForkOperator.java), like most other operators in a Gobblin task flow, is pluggable through the configuration, or more specifically , the configuration property `fork.operator.class` that points to a class that implements the `ForkOperator` interface. For instance:
+The [`ForkOperator`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/ForkOperator.java), like most other operators in a Gobblin task flow, is pluggable through the configuration, or more specifically , the configuration property `fork.operator.class` that points to a class that implements the `ForkOperator` interface. For instance:
```
fork.operator.class=org.apache.gobblin.fork.IdentityForkOperator
```
-By default, if no `ForkOperator` class is specified, internally Gobblin uses the default implementation [`IdentityForkOperator`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/fork/IdentityForkOperator.java) with a single forked branch (although it does supports multiple forked branches). The `IdentityForkOperator` simply unconditionally forwards the schema and ingested data records to all the forked branches, the number of which is specified through t [...]
+By default, if no `ForkOperator` class is specified, internally Gobblin uses the default implementation [`IdentityForkOperator`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/IdentityForkOperator.java) with a single forked branch (although it does supports multiple forked branches). The `IdentityForkOperator` simply unconditionally forwards the schema and ingested data records to all the forked branches, the number of which is [...]
The _expected_ number of forked branches is given by the method `getBranches` of the `ForkOperator`. This number must match the size of the list of `Boolean`s returned by `forkSchema` as well as the size of the list of `Boolean`s returned by `forkDataRecords`. Otherwise, `ForkBranchMismatchException` will be thrown. Note that the `ForkOperator` itself _is not making and returning a copy_ for the input schema and data records, but rather just providing a `Boolean` for each forked branch t [...]
The use of the `ForkOperator` with _the possibility that the schema and/or data records may be forwarded to more than one forked branches_ has some special requirement on the input schema and data records to the `ForkOperator`. Specifically, because the same schema or data records may be forwarded to more than branches that may alter the schema or data records in place, it is necessary for the Gobblin task flow to make a copy of the input schema or data records for each forked branch so [...]
-To guarantee that it is always able to make a copy in such a case, Gobblin requires the input schema and data records to be of type `Copyable` when there are more than one forked branch. [`Copyable`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/fork/Copyable.java) is an interface that defines a method `copy` for making a copy of an instance of a given type. The Gobblin task flow will check if the input schema and data records are instances of `Copyabl [...]
+To guarantee that it is always able to make a copy in such a case, Gobblin requires the input schema and data records to be of type `Copyable` when there are more than one forked branch. [`Copyable`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/Copyable.java) is an interface that defines a method `copy` for making a copy of an instance of a given type. The Gobblin task flow will check if the input schema and data records are in [...]
-Gobblin ships with some built-in `Copyable` implementations, e.g., [`CopyableSchema`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/fork/CopyableSchema.java) and [`CopyableGenericRecord`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/fork/CopyableGenericRecord.java) for Avro's `Schema` and `GenericRecord`.
+Gobblin ships with some built-in `Copyable` implementations, e.g., [`CopyableSchema`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/CopyableSchema.java) and [`CopyableGenericRecord`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/CopyableGenericRecord.java) for Avro's `Schema` and `GenericRecord`.
### Per-Fork Configuration
@@ -51,15 +51,15 @@ In a normal task flow where the default `IdentityForkOperator` with a single bra
* The failure of any forked branch means the failure of the whole task flow, i.e., the task succeeds if and only if all the forked branches succeed.
* A forked branch stops processing any outstanding incoming data records in the queue if it fails in the middle of processing the data.
* The failure and subsequent stop/completion of any forked branch does not prevent other branches from processing their copies of the ingested data records. The task will wait until all the branches to finish, regardless if they succeed or fail.
-* The commit of output data of forks is determined by the job commit policy (see [`JobCommitPolicy`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/JobCommitPolicy.java)) specified. If `JobCommitPolicy.COMMIT_ON_FULL_SUCCESS` (or `full` in short) is used, the output data of the entire job will be discarded if any forked branch fails, which will fail the task and consequently the job. If instead `JobCommitPolicy.COMMIT_SUCCESSFUL_TASKS` [...]
+* The commit of output data of forks is determined by the job commit policy (see [`JobCommitPolicy`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/source/extractor/JobCommitPolicy.java)) specified. If `JobCommitPolicy.COMMIT_ON_FULL_SUCCESS` (or `full` in short) is used, the output data of the entire job will be discarded if any forked branch fails, which will fail the task and consequently the job. If instead `JobCommitPolicy. [...]
### Performance Tuning
-Internally, each forked branch as represented by a [`Fork`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/Fork.java) maintains a bounded record queue (implemented by [`BoundedBlockingRecordQueue`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/BoundedBlockingRecordQueue.java)), which serves as a buffer between the pre-fork stream and the forked stream of the particular branch. The size if this [...]
+Internally, each forked branch as represented by a [`Fork`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java) maintains a bounded record queue (implemented by [`BoundedBlockingRecordQueue`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/BoundedBlockingRecordQueue.java)), which serves as a buffer between the pre-fork stream and the forked stream [...]
In terms of the number of forked branches, we have seen use cases with a half dozen forked branches, and we are anticipating uses cases with much larger numbers. Again, when using a large number of forked branches, the size of the record queues and the timeout time need to be carefully tuned.
-The [`BoundedBlockingRecordQueue`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/BoundedBlockingRecordQueue.java) in each [`Fork`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/Fork.java) keeps trach of the following queue statistics that can be output to the logs if the `DEBUG` logging level is turned on. Those statistics provide good indications on the performance of the forks.
+The [`BoundedBlockingRecordQueue`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/BoundedBlockingRecordQueue.java) in each [`Fork`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java) keeps trach of the following queue statistics that can be output to the logs if the `DEBUG` logging level is turned on. Those statistics provide good indications on [...]
* Queue size, i.e., the number of records in queue.
* Queue fill ratio, i.e., a ratio of the number of records in queue over the queue capacity.
@@ -70,7 +70,7 @@ The [`BoundedBlockingRecordQueue`](https://github.com/linkedin/gobblin/blob/mast
### Comparison with PartitionedDataWriter
-Gobblin ships with a special type of `DataWriter`s called [`PartitionedDataWriter`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/writer/PartitionedDataWriter.java) that allow ingested records to be written in a partitioned fashion using a `WriterPartitioner` into different locations in the same sink. The `WriterPartitioner` determines the specific partition for each data record. So there's certain overlap in terms of functionality between the `ForkOp [...]
+Gobblin ships with a special type of `DataWriter`s called [`PartitionedDataWriter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/PartitionedDataWriter.java) that allow ingested records to be written in a partitioned fashion using a `WriterPartitioner` into different locations in the same sink. The `WriterPartitioner` determines the specific partition for each data record. So there's certain overlap in terms of functionality [...]
* The `ForkOperator` requires the number of forked branches to be known and returned through `getBranches` before the task starts, whereas the `PartitionedDataWriter` does not have this requirement.
* The `PartitionedDataWriter` writes each data record to a single partition, whereas the `ForkOperator` allows data records to be forwarded to any number of forked branches.
@@ -81,7 +81,7 @@ Gobblin ships with a special type of `DataWriter`s called [`PartitionedDataWrite
Writing your Own ForkOperator
--------------------
-Since the built-in default implementation [`IdentityForkOperator`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/fork/IdentityForkOperator.java) simply blindly forks the input schema and data records to every branches, it's often necessary to have a custom implementation of the `ForkOperator` interface for more fine-grained control on the actual branching. Checkout the interface [`ForkOperator`](https://github.com/linkedin/gobblin/blob/master/gobblin- [...]
+Since the built-in default implementation [`IdentityForkOperator`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/IdentityForkOperator.java) simply blindly forks the input schema and data records to every branches, it's often necessary to have a custom implementation of the `ForkOperator` interface for more fine-grained control on the actual branching. Checkout the interface [`ForkOperator`](https://github.com/apache/incubator-g [...]
Best Practices
--------------------
diff --git a/gradle/scripts/computeVersions.gradle b/gradle/scripts/computeVersions.gradle
index cf7baaa..10fdf25 100644
--- a/gradle/scripts/computeVersions.gradle
+++ b/gradle/scripts/computeVersions.gradle
@@ -18,7 +18,7 @@
if (!project.hasProperty('version') || project.version == 'unspecified') {
try {
exec {
- commandLine 'git', 'fetch', '-t', 'https://github.com/linkedin/gobblin.git', 'master'
+ commandLine 'git', 'fetch', '-t', 'https://github.com/apache/incubator-gobblin.git', 'master'
}
def versionOut = new ByteArrayOutputStream()
exec {
diff --git a/mkdocs.yml b/mkdocs.yml
index b6a77f9..5c4a345 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -2,7 +2,7 @@ site_name: Gobblin Documentation
site_description: Documentation for Gobblin, a universal data ingestion framework for Hadoop
site_author: Abhishek Tiwari, Aditya Sharma, Chavdar Botev, Issac Buenrostro, Min Tu, Narasimha Veeramreedy, Pradhan Cadabam, Raul Agepati, Sahil Takiar, Shirshanka Das, Yinan Li, Ying Dai, Ziyang Liu
-repo_url: https://github.com/linkedin/gobblin
+repo_url: https://github.com/apache/incubator-gobblin
repo_name: Gobblin
docs_dir: gobblin-docs