You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hop.apache.org by ha...@apache.org on 2021/04/24 20:54:41 UTC
[incubator-hop] branch master updated: HOP-2713, HOP-2714,
HOP-2818 technology doc updates for Azure, Cassandra, Google, Neo4j,
added missing plugin placeholders for Neo4j transforms
This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-hop.git
The following commit(s) were added to refs/heads/master by this push:
new 097b6b6 HOP-2713, HOP-2714, HOP-2818 technology doc updates for Azure, Cassandra, Google, Neo4j, added missing plugin placeholders for Neo4j transforms
new 27cf008 Merge pull request #765 from bamaer/HOP-2714
097b6b6 is described below
commit 097b6b62a78f9455df6f2d499f85fdcf0ef2ef4b
Author: Bart Maertens <ba...@know.bi>
AuthorDate: Sat Apr 24 08:17:06 2021 +0200
HOP-2713, HOP-2714, HOP-2818 technology doc updates for Azure, Cassandra, Google, Neo4j, added missing plugin placeholders for Neo4j transforms
---
docs/hop-user-manual/modules/ROOT/nav.adoc | 13 +++++--
.../ROOT/pages/best-practices/best-practices.adoc | 2 +-
.../transforms/neo4j-gencsv.adoc} | 6 +--
.../transforms/neo4j-getloginfo.adoc} | 6 +--
.../transforms/neo4j-import.adoc} | 6 +--
.../transforms/neo4j-split-graph.adoc} | 6 +--
.../modules/ROOT/pages/technology/azure/index.adoc | 37 ++++++++++++++++++
.../ROOT/pages/technology/cassandra/index.adoc | 45 ++++++++++++++++++++++
.../google/{google-info.adoc => index.adoc} | 18 ++++-----
.../neo4j/{neo4j-info.adoc => index.adoc} | 24 +++++++++++-
.../modules/ROOT/pages/technology/technology.adoc | 7 ++++
11 files changed, 141 insertions(+), 29 deletions(-)
diff --git a/docs/hop-user-manual/modules/ROOT/nav.adoc b/docs/hop-user-manual/modules/ROOT/nav.adoc
index dff29d8..391418b 100644
--- a/docs/hop-user-manual/modules/ROOT/nav.adoc
+++ b/docs/hop-user-manual/modules/ROOT/nav.adoc
@@ -160,10 +160,12 @@ under the License.
*** xref:pipeline/transforms/mqttsubscriber.adoc[MQTT Subscriber]
*** xref:pipeline/transforms/multimerge.adoc[Multiway Merge Join]
*** xref:pipeline/transforms/neo4j-cypher.adoc[Neo4j Cypher]
+*** xref:pipeline/transforms/neo4j-gencsv.adoc[Neo4j Generate CSVs]
+*** xref:pipeline/transforms/neo4j-getloginfo.adoc[Neo4j Get Logging Info]
*** xref:pipeline/transforms/neo4j-graphoutput.adoc[Neo4j Graph Output]
-*** xref:pipeline/transforms/neo4j-cypher.adoc[Neo4j Cypher]
-*** xref:pipeline/transforms/neo4j-graphoutput.adoc[Neo4j Graph Output]
+*** xref:pipeline/transforms/neo4j-import.adoc[Neo4j Import]
*** xref:pipeline/transforms/neo4j-output.adoc[Neo4j Output]
+*** xref:pipeline/transforms/neo4j-split-graph.adoc[Neo4j Split Graph]
*** xref:pipeline/transforms/nullif.adoc[Null If]
*** xref:pipeline/transforms/numberrange.adoc[Number range]
*** xref:pipeline/transforms/pgpdecryptstream.adoc[PGP decrypt stream]
@@ -234,6 +236,7 @@ under the License.
*** xref:workflow/actions/abort.adoc[Abort]
*** xref:workflow/actions/addresultfilenames.adoc[Add filenames to result]
*** xref:workflow/actions/as400command.adoc[AS/400 Command]
+*** xref:workflow/actions/cassandra-exec-cql.adoc[Cassandra Exec Cql]
*** xref:workflow/actions/checkdbconnection.adoc[Check Db connections]
*** xref:workflow/actions/checkfilelocked.adoc[Check Files Locked]
*** xref:workflow/actions/webserviceavailable.adoc[Check if Webservice is available]
@@ -385,8 +388,10 @@ under the License.
* xref:samples/samples.adoc[Samples]
** xref:samples/dummy-sample.adoc[Dummy Sample]
* xref:technology/technology.adoc[Technology]
-** xref:technology/neo4j/neo4j-info.adoc[Neo4j]
-** xref:technology/google/google-info.adoc[Google]
+** xref:technology/azure/index.adoc[Azure]
+** xref:technology/cassandra/index.adoc[Cassandra]
+** xref:technology/neo4j/index.adoc[Neo4j]
+** xref:technology/google/index.adoc[Google]
* xref:hop-server/index.adoc[Hop Server]
** xref:hop-server/rest-api.adoc[REST api]
** xref:hop-server/web-service.adoc[Web Service]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/best-practices/best-practices.adoc b/docs/hop-user-manual/modules/ROOT/pages/best-practices/best-practices.adoc
index 8b09ecb..34c8618 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/best-practices/best-practices.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/best-practices/best-practices.adoc
@@ -77,7 +77,7 @@ xref:variables.adoc[Variables] provide an easy way to avoid hard-coding all sort
== Logging
-Take some time to capture the logging of your workflows and pipelines. Any time you run anything you want to have a trace of it. Things tend to go wrong when you least expect it and at that point you like being able to see what happened. See xref:logging/logging-basics.adoc[Logging Basics], xref:logging/logging-reflection.adoc[Logging Reflection] or consider logging to a xref:technology/neo4j/neo4j-info.adoc[Neo4j] graph database. This last one allows you to browse the logging result [...]
+Take some time to capture the logging of your workflows and pipelines. Any time you run anything you want to have a trace of it. Things tend to go wrong when you least expect it and at that point you like being able to see what happened. See xref:logging/logging-basics.adoc[Logging Basics], xref:logging/logging-reflection.adoc[Logging Reflection] or consider logging to a xref:technology/neo4j/index.adoc[Neo4j] graph database. This last one allows you to browse the logging results in [...]
== Mappings
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-gencsv.adoc
similarity index 72%
copy from docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
copy to docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-gencsv.adoc
index 66fa753..5417050 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-gencsv.adoc
@@ -1,4 +1,3 @@
-[[Technology]]
////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -15,6 +14,7 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
-= Technology
+:documentationPath: /pipeline/transforms/
+:language: en_US
-This section of the documentation contains technology-stack specific information. The goal is to inform you about how Apache Hop offers support for the technology in question. It also provides starting points for you if you want to learn more about the technology.
+= Neo4j Generate CSVs
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-getloginfo.adoc
similarity index 72%
copy from docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
copy to docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-getloginfo.adoc
index 66fa753..a063abe 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-getloginfo.adoc
@@ -1,4 +1,3 @@
-[[Technology]]
////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -15,6 +14,7 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
-= Technology
+:documentationPath: /pipeline/transforms/
+:language: en_US
-This section of the documentation contains technology-stack specific information. The goal is to inform you about how Apache Hop offers support for the technology in question. It also provides starting points for you if you want to learn more about the technology.
+= Neo4j Get Logging Info
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-import.adoc
similarity index 72%
copy from docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
copy to docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-import.adoc
index 66fa753..a01f42a 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-import.adoc
@@ -1,4 +1,3 @@
-[[Technology]]
////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -15,6 +14,7 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
-= Technology
+:documentationPath: /pipeline/transforms/
+:language: en_US
-This section of the documentation contains technology-stack specific information. The goal is to inform you about how Apache Hop offers support for the technology in question. It also provides starting points for you if you want to learn more about the technology.
+= Neo4j Import
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-split-graph.adoc
similarity index 72%
copy from docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
copy to docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-split-graph.adoc
index 66fa753..c2551e8 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-split-graph.adoc
@@ -1,4 +1,3 @@
-[[Technology]]
////
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
@@ -15,6 +14,7 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
-= Technology
+:documentationPath: /pipeline/transforms/
+:language: en_US
-This section of the documentation contains technology-stack specific information. The goal is to inform you about how Apache Hop offers support for the technology in question. It also provides starting points for you if you want to learn more about the technology.
+= Neo4j Split Graph
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/azure/index.adoc b/docs/hop-user-manual/modules/ROOT/pages/technology/azure/index.adoc
new file mode 100644
index 0000000..3a4c049
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/azure/index.adoc
@@ -0,0 +1,37 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /technology/azure/
+:language: en_US
+
+= Azure
+
+From https://en.wikipedia.org/wiki/Microsoft_Azure[Wikipedia]:
+
+Microsoft Azure, commonly referred to as Azure (/ˈæʒər/), is a cloud computing service created by Microsoft for building, testing, deploying, and managing applications and services through Microsoft-managed data centers. It provides software as a service (SaaS), platform as a service (PaaS) and infrastructure as a service (IaaS) and supports many different programming languages, tools, and frameworks, including both Microsoft-specific and third-party software and systems.
+
+Hop supports Azure through a number of metadata types:
+
+== Pipeline Transforms
+
+* xref:pipeline/transforms/azure-event-hubs-listener.adoc[Azure Event Hubs Listener]: listens indefinitely to an Event Hub on the Microsoft Azure cloud platform.
+* xref:pipeline/transforms/azure-event-hubs-writer.adoc[Azure Event Hubs Writer]: allows you to write messages (events) to a streaming service bus called Event Hubs on the Microsoft Azure cloud platform.
+
+== VFS
+
+Apache xref:vfs.adoc[VFS Support] in Hop allows you to directly read from a multitude of file systems and protocols, including Azure:
+
+* xref:vfs/azure-blob-storage-vfs.adoc[Azure Blog Storage]: read from and write data directly in Azure Blob Storage
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/cassandra/index.adoc b/docs/hop-user-manual/modules/ROOT/pages/technology/cassandra/index.adoc
new file mode 100644
index 0000000..a9db7e8
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/cassandra/index.adoc
@@ -0,0 +1,45 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /technology/cassandra/
+:language: en_US
+
+= Cassandra Technology
+
+From the https://cassandra.apache.org/[Apache Cassandra] website:
+
+The database is the right choice when you need scalability and high availability without compromising performance. http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html[Linear scalability] and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data. Cassandra's support for replicating across multiple datacenters is best-in-class, providing lower latency for your users and the peace of mind of kno [...]
+
+From https://en.wikipedia.org/wiki/Apache_Cassandra[Wikipedia]:
+
+Apache Cassandra is a free and open-source, distributed, wide-column store, NoSQL database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. Cassandra offers support for clusters spanning multiple datacenters,[2] with asynchronous masterless replication allowing low latency operations for all clients. Cassandra was designed to implement a combination of Amazon's Dynamo distributed storage [...]
+
+== Cassandra Support in Hop
+
+Hop supports Cassandra in the following metadata objects:
+
+=== Metadata Types
+
+* xref:metadata-types/cassandra/cassandra-connection.adoc[Cassandra Connection]: Create a connection to your Cassandra database cluster.
+
+=== Workflow Actions
+
+* xref:workflow/actions/cassandra-exec-cql.adoc[Cassandra Exec CQL]: Execute Cassandra https://cassandra.apache.org/doc/latest/cql/[CQL]
+
+=== Pipeline Transforms
+
+* xref:pipeline/transforms/cassandra-input.adoc[Cassandra Input]: Reads from a Cassandra cluster through a CQL query.
+* xref:pipeline/transforms/cassandra-output.adoc[Cassandra Output]: Write data to a table in a Cassandra cluster.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/google/google-info.adoc b/docs/hop-user-manual/modules/ROOT/pages/technology/google/index.adoc
similarity index 51%
rename from docs/hop-user-manual/modules/ROOT/pages/technology/google/google-info.adoc
rename to docs/hop-user-manual/modules/ROOT/pages/technology/google/index.adoc
index 69c28be..4b17f97 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/google/google-info.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/google/index.adoc
@@ -23,19 +23,15 @@ under the License.
The Google technology stack is supported in Hop through a number of plugins. We briefly touch upon them below.
-=== Google Analytics
+== Pipeline Transforms
-You can use the `Google Analytics` transform to get information from this service.
+* xref:pipeline/transforms/googleanalytics.adoc[Google Analytics]: You can use the `Google Analytics` transform to get information from this service.
+* xref:pipeline/transforms/googlesheetsinput.adoc[Google Sheets Input] and xref:pipeline/transforms/googlesheetsoutput.adoc[Google Sheets Output]: It can be very convenient to save data into a Google sheet or indeed to read from it. You can do this using the `Google Sheets Input` and `Google Sheets Output` transforms.
-== Google Sheets
+== VFS
-It can be very convenient to save data into a Google sheet or indeed to read from it. You can do this using the `Google Sheets Input` and `Google Sheets Output` transforms.
+Apache xref:vfs.adoc[VFS Support] in Hop allows you to directly read from a multitude of file systems and protocols, including Google:
-== Google Drive
-
-Hop has a Virtual File System (VFS) plugin for Google Drive. For more information see the VFS section of the documentation.
-
-== Google Cloud Storage
-
-Hop has a Virtual File System (VFS) plugin for Google Cloud Storage. For more information see the VFS section of the documentation.
+* xref:vfs/google-drive-vfs.adoc[Google Drive]: read and write data directly from and to Google Drive files and folders.
+* xref:vfs/google-cloud-storage-vfs.adoc[Google Cloud Storage]: read and write data directly from and to files and folders in Google Cloud Storage buckets
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/neo4j-info.adoc b/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/index.adoc
similarity index 58%
rename from docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/neo4j-info.adoc
rename to docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/index.adoc
index e3c10a0..45a7d38 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/neo4j-info.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/neo4j/index.adoc
@@ -25,10 +25,32 @@ Neo4j is an open source graph database which you can download from **https://neo
You can use it to represent information with nodes and relationships in a property graph. Neo4j doesn't use indexes which allows it to traverse large graphs really quickly with so-called graph algorithms. For more information on these unique graph algorithms see: https://neo4j.com/docs/graph-data-science/current/algorithms/#algorithms[Neo4j Graph Algorithms]
-== Execution lineage
+Hop supports Neo4j through the following metadata objects
+
+== Perspective: Execution lineage
You can use Neo4j to store logging and execution lineage of your workflows and pipelines.
The way you do this is simply by setting the variable NEO_LOGGING_CONNECTION to the name of the Neo4j Connection where you want the logging and lineage to be written to.
The Neo4j plugin offers a separate perspective to query this logging and lineage information. For example, it allows you to quickly jump to the place where an error occurred. This neat trick is performed by asking the database to find the shortest path between and execution node where an error occurred and without children and the "grand parent" node. The path you get is the exact path that was followed from for example the "grand parent" workflow to the exact transform where an error [...]
+== Metadata Types
+
+* xref:metadata-types/neo4j/neo4j-connection.adoc[Neo4j Connection]: create or manage a connection to a Neo4j database
+* xref:metadata-types/neo4j/neo4j-graphmodel.adoc[Neo4j Graph Model]: crearte or manage a Neo4j (sub)graph model
+
+== Workflow Actions
+
+* xref:workflow/actions/neo4j-checkconnections.adoc[Neo4j Check Connection]: check a connection to a Neo4j database
+* xref:workflow/actions/neo4j-cypherscript.adoc[Neo4j Cypher Script]: execute a Cypher script or query
+
+== Pipeline Transforms
+
+* xref:pipeline/transforms/neo4j-cypher.adoc[Neo4j Cypher]: execute a Cypher query on a Neo4j database using parameter info from input fields
+* xref:pipeline/transforms/neo4j-gencsv.adoc[Neo4j Generate CSVs]: Generate CSV files for nodes and relationships to be used with neo4j-import
+* xref:pipeline/transforms/neo4j-getloginfo.adoc[Neo4j Get Log Info]: Queries the Neo4j logging graph for execution info
+* xref:pipeline/transforms/neo4j-graphoutput.adoc[Neo4j Graph Output]: Writes to a Neo4j graph using an input fields mapping.
+* xref:pipeline/transforms/neo4j-import.adoc[Neo4j Import]: Runs a Neo4j import command using the provided CSV files
+* xref:pipeline/transforms/neo4j-output.adoc[Neo4j Output]: Write nodes and/or relationships to a Neo4j graph
+* xref:pipeline/transforms/neo4j-split-graph[Neo4j Split Graph]: Splits the nodes and relationships of a Neo4j graph
+
diff --git a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc b/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
index 66fa753..91ce1f0 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/technology/technology.adoc
@@ -18,3 +18,10 @@ under the License.
= Technology
This section of the documentation contains technology-stack specific information. The goal is to inform you about how Apache Hop offers support for the technology in question. It also provides starting points for you if you want to learn more about the technology.
+
+The technologies supported by Hop currently include:
+
+* xref:technology/azure/index.adoc[Azure]
+* xref:technology/cassandra/index.adoc[Apache Cassandra]
+* xref:technology/neo4j/index.adoc[Neo4j]
+* xref:technology/google/index.adoc[Google]
\ No newline at end of file