You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hop.apache.org by ha...@apache.org on 2021/09/16 06:35:43 UTC
[incubator-hop] branch master updated: HOP-3267 descriptions for
remaining user manual pages, tech manual
This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-hop.git
The following commit(s) were added to refs/heads/master by this push:
new 0012816 HOP-3267 descriptions for remaining user manual pages, tech manual
new 2bb7517 Merge pull request #1064 from bamaer/HOP-3267
0012816 is described below
commit 0012816354b985c38552460dd0b2981c5c21b000
Author: Bart Maertens <ba...@know.bi>
AuthorDate: Wed Sep 15 20:50:21 2021 +0200
HOP-3267 descriptions for remaining user manual pages, tech manual
---
.../modules/ROOT/pages/docker-container.adoc | 2 ++
.../ROOT/pages/hop-vs-kettle/hop-vs-kettle.adoc | 2 ++
.../ROOT/pages/hop-vs-kettle/if-you-know-kettle.adoc | 7 ++++---
.../pages/hop-vs-kettle/import-kettle-projects.adoc | 3 ++-
.../modules/ROOT/pages/hop-vs-kettle/index.adoc | 2 ++
.../images/hop-gui/pipeline/failed-pipeline.png | Bin 6513 -> 48382 bytes
.../hop-user-manual/modules/ROOT/pages/concepts.adoc | 1 +
.../hop-user-manual/modules/ROOT/pages/hop-usps.adoc | 2 ++
docs/hop-user-manual/modules/ROOT/pages/index.adoc | 1 +
.../modules/ROOT/pages/pipeline/create-pipeline.adoc | 18 +++++++++++-------
.../modules/ROOT/pages/pipeline/errorhandling.adoc | 1 +
.../ROOT/pages/pipeline/hop-pipeline-editor.adoc | 5 +++++
.../ROOT/pages/pipeline/metadata-injection.adoc | 3 ++-
.../modules/ROOT/pages/pipeline/partitioning.adoc | 1 +
.../ROOT/pages/pipeline/pipeline-unit-testing.adoc | 1 +
.../modules/ROOT/pages/pipeline/pipelines.adoc | 5 +++++
.../pages/pipeline/run-preview-debug-pipeline.adoc | 4 +++-
.../ROOT/pages/plugins/import/kettle-import.adoc | 3 ++-
.../modules/ROOT/pages/protips/index.adoc | 1 +
.../modules/ROOT/pages/samples/samples.adoc | 1 +
.../modules/ROOT/pages/technology/azure/index.adoc | 3 ++-
.../ROOT/pages/technology/cassandra/index.adoc | 2 ++
.../modules/ROOT/pages/technology/google/index.adoc | 10 ++++++++++
.../modules/ROOT/pages/technology/neo4j/index.adoc | 1 +
.../technology/neo4j/working-with-neo4j-data.adoc | 2 ++
.../modules/ROOT/pages/technology/technology.adoc | 6 ++++--
.../modules/ROOT/pages/variables.adoc | 1 +
docs/hop-user-manual/modules/ROOT/pages/vfs.adoc | 1 +
.../modules/ROOT/pages/vfs/aws-s3-vfs.adoc | 1 +
.../ROOT/pages/vfs/azure-blob-storage-vfs.adoc | 1 +
.../modules/ROOT/pages/vfs/dropbox-vfs.adoc | 1 +
.../ROOT/pages/vfs/google-cloud-storage-vfs.adoc | 1 +
.../modules/ROOT/pages/vfs/google-drive-vfs.adoc | 1 +
.../modules/ROOT/pages/workflow/actions.adoc | 6 +++++-
.../modules/ROOT/pages/workflow/create-workflow.adoc | 5 +++--
.../ROOT/pages/workflow/run-debug-workflow.adoc | 1 +
.../native-local-workflow-engine.adoc | 4 +++-
.../native-remote-workflow-engine.adoc | 2 ++
.../workflow-run-configurations.adoc | 2 ++
.../modules/ROOT/pages/workflow/workflows.adoc | 3 +++
40 files changed, 96 insertions(+), 21 deletions(-)
diff --git a/docs/hop-tech-manual/modules/ROOT/pages/docker-container.adoc b/docs/hop-tech-manual/modules/ROOT/pages/docker-container.adoc
index ad46fa2..7e169a8 100644
--- a/docs/hop-tech-manual/modules/ROOT/pages/docker-container.adoc
+++ b/docs/hop-tech-manual/modules/ROOT/pages/docker-container.adoc
@@ -14,6 +14,8 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
+:description: Apache Hop provides a Docker image for long (Hop Server) and short-lived (hop-run) containers. An additional image is available for Hop Web. Both images are available on Docker Hub.
+
[[DockerContainer-DockerContainer]]
= Docker container
diff --git a/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/hop-vs-kettle.adoc b/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/hop-vs-kettle.adoc
index 7d91261..486a554 100644
--- a/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/hop-vs-kettle.adoc
+++ b/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/hop-vs-kettle.adoc
@@ -14,6 +14,7 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
+:description: Apache Hop is an independent platform that originated from the same code base as Kettle (Pentaho Data Integration). This page provides an overview of the differences in concepts, configuration, engines and features between Hop and Kettle/PDI
[[HopvsKettle]]
:imagesdir: ../assets/images
:openvar: ${
@@ -22,6 +23,7 @@ under the License.
= Hop vs Kettle - A Quick Comparison
+
== Concepts
[width=90%",cols="6,2,2",options="header"]
diff --git a/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/if-you-know-kettle.adoc b/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/if-you-know-kettle.adoc
index 9bea13e..2760552 100644
--- a/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/if-you-know-kettle.adoc
+++ b/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/if-you-know-kettle.adoc
@@ -17,16 +17,17 @@ under the License.
[[IfYouKnowKettle]]
:imagesdir: ../assets/images
:page-pagination:
+:description: Hop wants to allow data engineers to be able to deliver high quality work, deliver that work fast and integrated with bleeding edge technology.
= If You Know Kettle (Pentaho Data Integration)
== Why Hop?
-With Hop, we want to allow data engineers to be able to deliver high quality work, deliver that work fast and integrated with bleeding edge technology.
+Hop wants to allow data engineers to be able to deliver high quality work, deliver that work fast and integrated with bleeding edge technology.
-We want Hop to be completely open source, and are eager to hear your feedback on our https://chat.project-hop.org[chat] and just as eager to see your bug tickets and feature request in https://jira.project-hop.org[our JIRA].
+We want Hop to be completely open source, and are eager to hear your feedback on our https://chat.project-hop.org[chat] and just as eager to see your bug tickets and feature request in https://issues.apache.org/jira/projects/HOP/issues[JIRA].
-As an open source first project, we'll start the https://www.apache.org/[Apache Software Foundation] https://incubator.apache.org/[incubation] process soon, and aim to become and ASF https://projects.apache.org/[top level project] as soon as possible.
+As an open source first project, we started the https://www.apache.org/[Apache Software Foundation] https://incubator.apache.org/[incubation] process in September 2020 and aim to become and ASF https://projects.apache.org/[top level project] as soon as possible.
Check our https://hop.apache.org/docs/qa/[Q&A] for more information on why Hop was created and what the project is all about.
diff --git a/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/import-kettle-projects.adoc b/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/import-kettle-projects.adoc
index f8ce305..f1ebbd3 100644
--- a/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/import-kettle-projects.adoc
+++ b/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/import-kettle-projects.adoc
@@ -18,12 +18,13 @@ under the License.
:imagesdir: ../assets/images
:page-pagination:
:page-pagination-no-next:
+:description: Compatibility with Kettle/PDI was never a goal for Apache Hop (Incubating), but since a lot of organizations have invested vast amounts of resources in Kettle/PDI project development, the Apache Hop (Incubating) community provides a way to import Kettle/PDI code into Hop and convert the imported code the the Hop ways of working.
= Import Kettle (PDI) Projects in Apache Hop (Incubating)
As stated in the https://hop.apache.org/docs/qa/[Q&A], Apache Hop (Incubating) used Kettle (aka Pentaho Data Integration or PDI) as a starting point in late 2019. A lot has happened in the meantime on both Apache Hop (Incubating) and Pentaho Data Integration.
-Compatibility with Kettle/PDI never was a goal for Apache Hop (Incubating), but since a lot of organizations have invested vast amounts of resources in Kettle/PDI project development, the Apache Hop (Incubating) community provides a way to import Kettle/PDI code into Hop and convert the imported code the the Hop ways of working.
+Compatibility with Kettle/PDI was never a goal for Apache Hop (Incubating), but since a lot of organizations have invested vast amounts of resources in Kettle/PDI project development, the Apache Hop (Incubating) community provides a way to import Kettle/PDI code into Hop and convert the imported code the the Hop ways of working.
== Imported Items
diff --git a/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/index.adoc b/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/index.adoc
index 59c0d6a..0612c55 100644
--- a/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/index.adoc
+++ b/docs/hop-tech-manual/modules/ROOT/pages/hop-vs-kettle/index.adoc
@@ -14,6 +14,8 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
+:description: Hop initially (late 2019) started as a fork of the Kettle (Pentaho Data Integration). Hop and Kettle/PDI are independent projects, each with their own roadmap and priorities. Given these different roadmaps, architectural vision and development track, Hop and Kettle/PDI are incompatible.
+
[[HopAndKettle]]
:imagesdir: ../assets/images
:page-pagination:
diff --git a/docs/hop-user-manual/modules/ROOT/assets/images/hop-gui/pipeline/failed-pipeline.png b/docs/hop-user-manual/modules/ROOT/assets/images/hop-gui/pipeline/failed-pipeline.png
index f2de0bb..15245ca 100644
Binary files a/docs/hop-user-manual/modules/ROOT/assets/images/hop-gui/pipeline/failed-pipeline.png and b/docs/hop-user-manual/modules/ROOT/assets/images/hop-gui/pipeline/failed-pipeline.png differ
diff --git a/docs/hop-user-manual/modules/ROOT/pages/concepts.adoc b/docs/hop-user-manual/modules/ROOT/pages/concepts.adoc
index dfe8f20..27ec033 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/concepts.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/concepts.adoc
@@ -16,6 +16,7 @@ under the License.
////
[[Concepts]]
:imagesdir: ../assets/images
+:description: Hop comes with a number of concepts: a variety of tools, a large number of metadata types, projects and enviromments. At the core of literally everything in Hop is metadata.
= Concepts
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-usps.adoc b/docs/hop-user-manual/modules/ROOT/pages/hop-usps.adoc
index 2b03b91..9f5368d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-usps.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-usps.adoc
@@ -14,6 +14,8 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
+:description: Hop differentiates itself from other platforms through an absolute focus on metadata, a visual code editor, a kernel + plugins architecture, portable run configurations, unit and integration testing and life cycle management.
+
[[USPs]]
= Hop Unique Selling Propositions
diff --git a/docs/hop-user-manual/modules/ROOT/pages/index.adoc b/docs/hop-user-manual/modules/ROOT/pages/index.adoc
index 6dbd9f3..85cf94a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/index.adoc
@@ -16,6 +16,7 @@ under the License.
////
[[Index]]
:imagesdir: ../assets/images
+:description: As a relatively young platform, Hop covers a area of functionalities that is both wide and deep. This page provides an overview of topic that may be of interest to you.
= Hop User Manual
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/create-pipeline.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/create-pipeline.adoc
index dfbdcaa..8bca2c9 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/create-pipeline.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/create-pipeline.adoc
@@ -16,15 +16,19 @@ under the License.
////
[[CreatePipeline]]
:imagesdir: ../assets/images
+:description: Pipelines are the fundamental building blocks of your Hop projects. Pipelines read from data sources and perform operations (joins, lookups, filter and lots more) on the data before it is written out to a target platform.
+
= Create a Pipeline
-TODO: this content is based on the xref:getting-started/index.adoc[Getting Started Guide].
-Elaborate more.
+== How pipelines work
+
+Pipelines are the fundamental building blocks of your Hop projects.
+
+Pipelines perform the heavy lifting: they read data from a variety of sources, perform a number of operations (combine, clean, enrich, transform etc) and write the data out to some target platform. A pipeline executes all of these operations in a _predefined order_ and in _parallel_.
-Pipelines are the fundamentals of your project.
-A pipeline contains operations executed in a predefined order.
-Pipelines consist of a chain of transforms that read, process or write data.
-The image below shows a simple pipeline that reads data from a table, adds a message to the data and sends out an email.
+In the image below, a very simple pipeline reads data from a database, adds a message to the data and sends out an email.
+All of these operations are executed in a predefined order (read from the database, add the message, send the mail) and in parallel.
+The pipelines executes these transforms, Imagine our database table or query contains thousands of rows. The pipeline will start reading results from the query, pass those on to the 'Add message' transform. Once the message has been added, we'll send a mail from the Mail transform. All of these will operate in parallel, so the Mail transform will already be sending mails while the table input is still reading records from the table or query.
image::hop-gui/pipeline/simple-pipeline.png[Hop - Simple Pipeline,75%,align="left"]
@@ -46,7 +50,7 @@ Some transforms can read from or write to other transforms conditionally to or f
The hop is unaware of it.
Hops can be disabled by clicking on them, or through right-click -> disable.
-== Creation
+== Create a pipeline
Create a new pipeline through the work item dialog.
You'll be presented with the dialog shown below.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/errorhandling.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/errorhandling.adoc
index 607f063..b821ca9 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/errorhandling.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/errorhandling.adoc
@@ -16,6 +16,7 @@ under the License.
////
[[PipelineErrorHandling]]
:imagesdir: ../assets/images
+:description: When a major failure occurs in one of the transforms, the pipeline is notified and halts all active operations. This is fine in most cases, pipeline failures are typically handled in the parent workflow. In some cases however, there are errors you'll want to handle gracefully without halting the entire pipeline.
= Pipeline Error Handling
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/hop-pipeline-editor.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/hop-pipeline-editor.adoc
index 12c785c..63070b8 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/hop-pipeline-editor.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/hop-pipeline-editor.adoc
@@ -16,9 +16,14 @@ under the License.
////
[[HopPipelineEditor]]
:imagesdir: ../assets/images
+:description: You'll perform a lot of operations on the pipeline canvas, but a lot of additional functionality is available from the pipeline editor's main toolbar
= Toolbar
+You learned to create a pipeline in xref:pipeline/create-pipeline.adoc[Create a Pipeline]. You'll perform a lot of operations on the pipeline canvas, but a lot of additional functionality is available from the pipeline editor's main toolbar.
+
+There are two other important toolbars to work with xref:pipeline/pipeline-unit-testing.adoc[unit tests] and xref:projects/index.adoc[projects and environments]. Check the respective pages to learn more about managing your projects and environments and to learn how to write unit tests for your pipelines.
+
Let's walk through the top toolbar:
image::getting-started/getting-started-pipeline-toolbar.png[Hop - Pipeline Toolbar,align="left"]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/metadata-injection.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/metadata-injection.adoc
index c42494d..6b84010 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/metadata-injection.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/metadata-injection.adoc
@@ -16,10 +16,11 @@ under the License.
////
[[MetadataInjection]]
:imagesdir: ../assets/images
+:description: Metadata injection inserts data from various sources into a template pipeline at runtime to reduce repetitive tasks.
= Metadata Injection
-Metadata injection inserts data from various sources into a pipeline at runtime to reduce repetitive tasks.
+Metadata injection inserts data from various sources into a template pipeline at runtime to reduce repetitive tasks.
For example, you might have a simple pipeline to load transaction data values from a supplier, filter specific values, and output them to a file.
If you have more than one supplier, you would need to run this simple pipeline for each supplier.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/partitioning.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/partitioning.adoc
index eddc807..fb88b64 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/partitioning.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/partitioning.adoc
@@ -15,6 +15,7 @@ specific language governing permissions and limitations
under the License.
////
:imagesdir: ../assets/images
+:description: Partitioning allows you to distribute all the data from a set into distinct subsets according to the rule applied on a table or row, where these subsets form a partition of the original set with no item replicated into multiple groups.
= Partitioning
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-unit-testing.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-unit-testing.adoc
index 69647e9..3605c21 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-unit-testing.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-unit-testing.adoc
@@ -18,6 +18,7 @@ under the License.
:imagesdir: ../../assets/images
:openvar: ${
:closevar: }
+:description: Hop unit tests simulate inputs in the form of _Input data sets_ and validates output against _Golden data sets_. A unit test is a combination of zero or more input sets and golden data sets along with a bunch of tweaks you can do to the pipelines prior to testing.
= Pipeline Unit Testing
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipelines.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipelines.adoc
index 7ae704e..b8eca4f 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipelines.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipelines.adoc
@@ -16,9 +16,14 @@ under the License.
////
[[Pipelines]]
:imagesdir: ../assets/images
+:description: Pipelines, together with workflows, are the main building blocks in Hop. Pipelines perform the heavy data lifting: in a pipeline, you read data from one or more sources, perform a number of operations (joins, lookups, filters and lots more) and finally write the processed data to one or more target platforms.
= Pipelines
+Pipelines, together with workflows, are the main building blocks in Hop. Pipelines perform the heavy data lifting: in a pipeline, you read data from one or more sources, perform a number of operations (joins, lookups, filters and lots more) and finally write the processed data to one or more target platforms.
+
+Pipelines are an extensive topic. Check the pages below to learn more about working with pipelines:
+
* xref:pipeline/hop-pipeline-editor.adoc[Pipeline Editor]
* xref:pipeline/create-pipeline.adoc[Create a Pipeline]
* xref:pipeline/run-preview-debug-pipeline.adoc[Run, Preview and Debug a Pipeline]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/run-preview-debug-pipeline.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/run-preview-debug-pipeline.adoc
index 977cf7c..fb4ad48 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/run-preview-debug-pipeline.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/run-preview-debug-pipeline.adoc
@@ -16,6 +16,7 @@ under the License.
////
[[RunPreviewDebugPipeline]]
:imagesdir: ../assets/images
+:description: Hop Gui offers a lot of functionality to let data developers run, preview and debug pipelines. Feedback is provided through the UI, logging and run metrics.
= Run, Preview and Debug a Pipeline
@@ -61,7 +62,8 @@ The transforms also show a little table icon, giving you access to a preview of
image::hop-gui/pipeline/preview-pipeline.png[Preview pipeline,75%,align="left"]
-When a pipeline has failed a red failed icon is displayed on the transform containing an error.
+When a pipeline fails, a red triangle is shown in the upper right corner of the failed transform.
+Hover your mouse over the red error triangle to take a quick peek at the error message. The full stacktrace is available in the logs. Check xref:pipeline/errorhandling.adoc[pipeline error handling] to learn about how you can handle errors in your pipeline gracefully (which is not necessarily what you want).
image::hop-gui/pipeline/failed-pipeline.png[Failed pipeline,75%,align="left"]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/plugins/import/kettle-import.adoc b/docs/hop-user-manual/modules/ROOT/pages/plugins/import/kettle-import.adoc
index daafe8c..63a5142 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/plugins/import/kettle-import.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/plugins/import/kettle-import.adoc
@@ -19,10 +19,11 @@ under the License.
:openvar: ${
:closevar: }
+:description: Hop is an independent platform that originates from the same code base as Kettle (Pentaho Data Integration). Even though Hop and Kettle are not compatible, you can upgrade your existing Kettle projects by simply importing them into Hop.
= Kettle Import
-Whenever you want to import metadata from Kettle you can use this plugin.
+Whenever you want to import metadata from Kettle (Pentaho Data Integration) you can use this plugin.
== Usage
diff --git a/docs/hop-user-manual/modules/ROOT/pages/protips/index.adoc b/docs/hop-user-manual/modules/ROOT/pages/protips/index.adoc
index 5e9ed0d..9b33106 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/protips/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/protips/index.adoc
@@ -16,6 +16,7 @@ under the License.
////
[[ProTips]]
:imagesdir: ../assets/images
+:description: Hop is a large platform with a lot of options. These pro tips consist of useful variables, keyboard shortcuts and mouse actions etc
= Pro Tips
diff --git a/docs/hop-user-manual/modules/ROOT/pages/samples/samples.adoc b/docs/hop-user-manual/modules/ROOT/pages/samples/samples.adoc
index 04e03ef..53caf68 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/samples/samples.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/samples/samples.adoc
@@ -14,6 +14,7 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
+:description: Hop comes with a set of samples for workflows, pipelines, actions, transforms and other metadata objects. These samples are included in your default Hop installation as the Samples project.
[[Samples]]
= Samples
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/azure/index.adoc b/docs/hop-user-manual/modules/ROOT/pages/technology/azure/index.adoc
index d2187aa..097af33 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/azure/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/azure/index.adoc
@@ -16,6 +16,7 @@ under the License.
////
:documentationPath: /technology/azure/
:language: en_US
+:description: Hop supports Azure in Apache VFS (Blob Storage) and Azure Event Hubs listeners and writers.
= Azure
@@ -35,4 +36,4 @@ Hop supports Azure through a number of metadata types:
Apache xref:vfs.adoc[VFS Support] in Hop allows you to directly read from a multitude of file systems and protocols, including Azure:
-* xref:vfs/azure-blob-storage-vfs.adoc[Azure Blog Storage]: read from and write data directly in Azure Blob Storage
\ No newline at end of file
+* xref:vfs/azure-blob-storage-vfs.adoc[Azure Blob Storage]: read from and write data directly in Azure Blob Storage
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/cassandra/index.adoc b/docs/hop-user-manual/modules/ROOT/pages/technology/cassandra/index.adoc
index 2f903d3..47d9da2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/cassandra/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/cassandra/index.adoc
@@ -16,6 +16,7 @@ under the License.
////
:documentationPath: /technology/cassandra/
:language: en_US
+:description: Apache Hop supports Apache Cassandra through a Cassandra connection metadata type, an Exec CQL action and transforms for Cassandra input and output, and an SSTable output.
= Cassandra Technology
@@ -46,3 +47,4 @@ Hop supports Cassandra in the following metadata objects:
* xref:pipeline/transforms/cassandra-input.adoc[Cassandra Input]: Reads from a Cassandra cluster through a CQL query.
* xref:pipeline/transforms/cassandra-output.adoc[Cassandra Output]: Write data to a table in a Cassandra cluster.
+* xref:pipeline/transforms/sstable-output.adoc[SSTable Output]: Write data to a filesystem directory as a Cassandra SSTable.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/google/index.adoc b/docs/hop-user-manual/modules/ROOT/pages/technology/google/index.adoc
index 29f075b..7193c23 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/google/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/google/index.adoc
@@ -16,6 +16,7 @@ under the License.
////
:documentationPath: /technology/google/
:language: en_US
+:description: The Google platform is supported in Hop through a number of (Beam) transforms and Apache VFS support for Google Drive and Google Cloud Storage. Additionally, a number of external transform plugins are available.
= Google Technology
@@ -26,6 +27,15 @@ We briefly touch upon them below.
== Pipeline Transforms
+* xref:pipeline/transforms/beambigqueryinput.adoc[BigQuery Input (Beam)]
+* xref:pipeline/transforms/beambigqueryoutput.adoc[BigQuery Output (Beam)]
+* xref:pipeline/transforms/beambigtableinput.adoc[Bigtable Input (Beam)]
+* xref:pipeline/transforms/beambigtableinput.adoc[Bigtable Output (Beam)]
+* xref:pipeline/transforms/beamgcppublisher.adoc[GCP Pub/Sub Publisher (Beam)]
+* xref:pipeline/transforms/beamgcpsubscriber.adoc[GCP Pub/Sub Subscriber (Beam)]
+
+Check the xref:plugins/external-plugins.adoc[external plugins] for more information on a number of additional Google plugins that can't be or won't be included with Apache Hop (Google Sheets Input and Output, Google Analytics)
+
== VFS
Apache xref:vfs.adoc[VFS Support] in Hop allows you to directly read from a multitude of file systems and protocols, including Google:
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/index.adoc b/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/index.adoc
index 1d7766c..961047e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/index.adoc
@@ -16,6 +16,7 @@ under the License.
////
:documentationPath: /technology/neo4j/
:language: en_US
+:description: Apache Hop has extensive support for the Neo4j graph database through a dedicated connection metadata type, a graph model type and tens of transforms and actions. Additionally, Hop logging data can be written to a Neo4j database and can be consulted directly from the Neo4j logging perspective in Hop Gui.
= Neo4j
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/working-with-neo4j-data.adoc b/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/working-with-neo4j-data.adoc
index c6e0a09..09dd9b9 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/working-with-neo4j-data.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/working-with-neo4j-data.adoc
@@ -14,6 +14,8 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
+:description: Apache Hop has extensive support for the Neo4j graph database, including a library of samples that explain how to load data to and read from your Neo4j graphs through Apache Hop.
+
= Working with Neo4j data
:toc:
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc b/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
index 7da841f..3109c6d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
@@ -15,9 +15,11 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
+:description: Hop integrates with tens if not hundreds of different technologies. The Azure, Apache Cassandra, Google and Neo4j platforms are supported at various points in Hop and are covered in their own section.
+
= Technology
-This section of the documentation contains technology-stack specific information.
+This section contains technology-stack specific information.
The goal is to inform you about how Apache Hop offers support for the technology in question.
It also provides starting points for you if you want to learn more about the technology.
@@ -25,5 +27,5 @@ The technologies supported by Hop currently include:
* xref:technology/azure/index.adoc[Azure]
* xref:technology/cassandra/index.adoc[Apache Cassandra]
+* xref:technology/google/index.adoc[Google]
* xref:technology/neo4j/index.adoc[Neo4j]
-* xref:technology/google/index.adoc[Google]
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/variables.adoc b/docs/hop-user-manual/modules/ROOT/pages/variables.adoc
index cbbda20..6ce72d3 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/variables.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/variables.adoc
@@ -18,6 +18,7 @@ under the License.
:imagesdir: ../assets/images
:openvar: ${
:closevar: }
+:description: Well-designed Hop solutions never use hard coded values. Hop offers variables on the global, project and environment levels, and allows workflows and parameters to provide parameters, to set and read variables in runtime etc.
= Variables
diff --git a/docs/hop-user-manual/modules/ROOT/pages/vfs.adoc b/docs/hop-user-manual/modules/ROOT/pages/vfs.adoc
index ace1882..a91fca6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs.adoc
@@ -16,6 +16,7 @@ under the License.
////
[[VFS]]
:imagesdir: ../assets/images
+:description: Apache VFS, part of the Apache Commons project, allows Hop data developers to access a large variety of file and data formats over a convenient url syntax.
= Virtual File Systems
diff --git a/docs/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc b/docs/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc
index b1a0c86..2c17e55 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc
@@ -17,6 +17,7 @@ under the License.
:documentationPath: /vfs/
:language: en_US
+:description: Apache Hop supports reading from and writing to AWS S3 buckets from almost anywhere in the platform through Apache VFS
= AWS S3
diff --git a/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc b/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
index b42f836..fee6cb6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
@@ -17,6 +17,7 @@ under the License.
:documentationPath: /vfs/
:language: en_US
+:description: Apache Hop supports reading from and writing to Azure Blob Storage from almost anywhere in the platform through Apache VFS
= Azure Storage
diff --git a/docs/hop-user-manual/modules/ROOT/pages/vfs/dropbox-vfs.adoc b/docs/hop-user-manual/modules/ROOT/pages/vfs/dropbox-vfs.adoc
index beb706e..1f10795 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/dropbox-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/dropbox-vfs.adoc
@@ -17,6 +17,7 @@ under the License.
:documentationPath: /vfs/
:language: en_US
+:description: Apache Hop supports reading from and writing to Dropbox from almost anywhere in the platform through Apache VFS
= Dropbox VFS
diff --git a/docs/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc b/docs/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc
index 1b08720..8e91533 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc
@@ -17,6 +17,7 @@ under the License.
:documentationPath: /vfs/
:language: en_US
+:description: Apache Hop supports reading from and writing to Google Cloud Storage from almost anywhere in the platform through Apache VFS
= Google Cloud Storage VFS
diff --git a/docs/hop-user-manual/modules/ROOT/pages/vfs/google-drive-vfs.adoc b/docs/hop-user-manual/modules/ROOT/pages/vfs/google-drive-vfs.adoc
index 3bc9cc6..fbaf27e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/google-drive-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/google-drive-vfs.adoc
@@ -17,6 +17,7 @@ under the License.
:documentationPath: /vfs/
:language: en_US
+:description: Apache Hop supports reading from and writing to Google Drive from almost anywhere in the platform through Apache VFS
= Google Drive VFS
diff --git a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions.adoc b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions.adoc
index 83237e7..911f4f0 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions.adoc
@@ -14,10 +14,14 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
+:description: Actions are used the objects that are executed in a workflow. They are logical operators that do not manipulate your data but create actions that have to be performed around your pipelines.
+
[[actions-plugins]]
= Action Plugins
-Actions are used to create a workflow, they are logical operators that do not manipulate your data but create actions that have to be performed around your pipelines.
+Actions are used the objects that are executed in a workflow. They are logical operators that do not manipulate your data but create actions that have to be performed around your pipelines.
+
+Actions in a workflow typically do not process your data, nor do they pass on data to the next action. When an action finishes running, it returns a true or false error code that can be used to drive the workflows behavior through true/false or unconditional (ignore exit code) workflows hops.
// tag::website-links[]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/workflow/create-workflow.adoc b/docs/hop-user-manual/modules/ROOT/pages/workflow/create-workflow.adoc
index 398979c..cef658a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/workflow/create-workflow.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/create-workflow.adoc
@@ -16,10 +16,11 @@ under the License.
////
[[CreateWorkflow]]
:imagesdir: ../assets/images
+:description: Workflows are process that orchestrate a variety of actions defined between a start and an end point.
= Create a Workflow
-Workflows are process that orchestrate a variety of actions defined between a starting and endpoint.
+Workflows are processes that orchestrate a variety of actions defined between a start and an end point.
These actions can be, but are not limited to:
* Executing other workflows
@@ -28,7 +29,7 @@ These actions can be, but are not limited to:
* Error messaging handling
* Notification handling
-A simple workflow is displayed below, the flow will check if a folder is empty, if not a pipeline is executed.
+A simple workflow is displayed below. This workflow will check if a folder is empty, if not a pipeline is executed.
image::hop-gui/workflow/simple-workflow.png[Hop-Simple Workflow,75%,align="left"]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/workflow/run-debug-workflow.adoc b/docs/hop-user-manual/modules/ROOT/pages/workflow/run-debug-workflow.adoc
index f350e5a..c4ad753 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/workflow/run-debug-workflow.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/run-debug-workflow.adoc
@@ -16,6 +16,7 @@ under the License.
////
[[RunDebugWorkflow]]
:imagesdir: ../assets/images
+:description: Running a workflow requires a workflow run configuration and (optionally) a number of parameters.
= Run and Debug a Workflow
diff --git a/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/native-local-workflow-engine.adoc b/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/native-local-workflow-engine.adoc
index f0f1113..c90d17d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/native-local-workflow-engine.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/native-local-workflow-engine.adoc
@@ -16,11 +16,13 @@ under the License.
////
[[LocalWorkflowEngine]]
:imagesdir: ../assets/images
+:description: The local run configuration runs workflows on your local environment.
+
= Local Workflow Engine
== Hop local workflow engine
-The local run configuration runs workflows locally.
+The local run configuration runs workflows on your local environment.
=== Options
diff --git a/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/native-remote-workflow-engine.adoc b/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/native-remote-workflow-engine.adoc
index bcd1f4b..fea7ce7 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/native-remote-workflow-engine.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/native-remote-workflow-engine.adoc
@@ -16,6 +16,8 @@ under the License.
////
[[RemoteWorkflowEngine]]
:imagesdir: ../assets/images
+:description: The remote run configuration runs workflows on a remote Hop Server. This run configuration requires little configuration, but requires a Hop Server and a Hop Server metadata definition.
+
= Remote Workflow Engine
== Hop Remote workflow engine
diff --git a/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/workflow-run-configurations.adoc b/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/workflow-run-configurations.adoc
index 9eaff93..d021676 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/workflow-run-configurations.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/workflow-run-configurations.adoc
@@ -16,6 +16,8 @@ under the License.
////
[[WorkflowRunConfigurations]]
:imagesdir: ../assets/images
+:description: Run configurations decouple the design and execution phases of Hop workflow development. A workflow is a definition of _how_ data is processed, a run configurations defines _where_ the workflow is executed. Hop supports a local and remote workflow runtime engines
+
= Workflow Run Configurations
== Description
diff --git a/docs/hop-user-manual/modules/ROOT/pages/workflow/workflows.adoc b/docs/hop-user-manual/modules/ROOT/pages/workflow/workflows.adoc
index eb453b7..4e8cb13 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/workflow/workflows.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/workflows.adoc
@@ -16,9 +16,12 @@ under the License.
////
[[Workflows]]
:imagesdir: ../assets/images
+:description: Workflows are one of the core building blocks in Apache Hop. Where pipelines do the heavy data lifting, workflows take care of the orchestration work: prepare the environment, fetch remote files, perform error handling and executing child workflows and pipelines.
= Workflows
+Workflows are one of the core building blocks in Apache Hop. Where pipelines do the heavy data lifting, workflows take care of the orchestration work: prepare the environment, fetch remote files, perform error handling and executing child workflows and pipelines.
+
** xref:workflow/create-workflow.adoc[Create a Workflow]
** xref:workflow/run-debug-workflow.adoc[Run and Debug a Workflow]
** xref:workflow/workflow-run-configurations/workflow-run-configurations.adoc[Workflow Run Configurations]