You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hop.apache.org by gi...@apache.org on 2021/03/13 16:18:06 UTC
[incubator-hop-docs] branch asf-site updated: Documentation updated
to 7bc39f9
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-hop-docs.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a61e5d3 Documentation updated to 7bc39f9
a61e5d3 is described below
commit a61e5d31e2e452e323e8a9e794cc58b781a65b31
Author: jenkins <bu...@apache.org>
AuthorDate: Sat Mar 13 16:18:02 2021 +0000
Documentation updated to 7bc39f9
---
.../pages/metadata-types/neo4j-graphmodel.adoc | 26 ++++
.../metadata-types/neo4j/neo4j-connection.adoc | 152 +++++++++++++++++++++
.../pages/pipeline/transforms/neo4j-cypher.adoc | 54 ++++++++
.../pipeline/transforms/neo4j-graphoutput.adoc | 45 ++++++
.../pages/pipeline/transforms/neo4j-output.adoc | 50 +++++++
.../ROOT/pages/technology/neo4j/neo4j-info.adoc | 35 +++++
.../workflow/actions/deleteresultfilenames.adoc | 6 +-
...tfilenames.adoc => neo4j-checkconnections.adoc} | 18 +--
.../pages/workflow/actions/neo4j-cypherscript.adoc | 56 ++++++++
9 files changed, 427 insertions(+), 15 deletions(-)
diff --git a/hop-user-manual/modules/ROOT/pages/metadata-types/neo4j-graphmodel.adoc b/hop-user-manual/modules/ROOT/pages/metadata-types/neo4j-graphmodel.adoc
new file mode 100644
index 0000000..b58a678
--- /dev/null
+++ b/hop-user-manual/modules/ROOT/pages/metadata-types/neo4j-graphmodel.adoc
@@ -0,0 +1,26 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /metadata-types/
+:language: en_US
+
+= Neo4j Graph Model
+
+== Description
+
+A graph model describes (part of) a graph by allowing you to define nodes with their properties, and the relationships that connect these nodes.
+You can then use such a Graph Model to map input fields to properties in the link:neo4j-graphoutput.html[Neo4j Graph Output] transform.
+
diff --git a/hop-user-manual/modules/ROOT/pages/metadata-types/neo4j/neo4j-connection.adoc b/hop-user-manual/modules/ROOT/pages/metadata-types/neo4j/neo4j-connection.adoc
new file mode 100644
index 0000000..e3b42a0
--- /dev/null
+++ b/hop-user-manual/modules/ROOT/pages/metadata-types/neo4j/neo4j-connection.adoc
@@ -0,0 +1,152 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /metadata-types/neo4j/
+:language: en_US
+:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/misc/neo4j/src/main/doc/neo4j-connection.adoc
+= Neo4j Connection
+
+== Description
+
+Connections can be created and edited using the logo (click on it to edit), and the drop down menu (arrow down) which you can find on the right of every connection line on the various Neo4j dialogs.
+
+They can also be managed with the metadata perspective in the Hop GUI. Finally, please note that Neo4j Connections are serialized as JSON in the Hop metadata folder under
+
+`metadata/neo4j-connection/`
+
+== The dialog
+
+**DON'T PANIC**: most of the options in the interface you don't need (...and most likely will never will need).
+
+== The Options
+
+|===
+|Option |Description
+
+|Connection name
+|The name of a connection uniquely identifying it
+
+|Server or IP Address
+|The IP address or hostname used to reach the Neo4j server. If you have more than one server address to specify, use a comma separated list of hosts.
+
+|Database Name (4.0)
+|The database to use (defaults to neo4j). A feature available on plugins version 5.0 or higher against Neo4j server 4.0 or higher.
+
+|Version 4 database?
+|Enable this option to help us generate the most optimized Cypher for you.
+
+|Bolt port
+|the Neo4j Bolt protocol port, default is 7687
+
+|Browser port
+|information only. The port on which the Neo4j browser runs
+
+|Use routing
+|enable this option to use the bolt+routing protocol
+
+|Routing policy
+|The bolt+routing policy to use
+
+|Username
+|the Neo4j database user to log in with
+
+|Password
+|the password to use
+
+|Use encryption?
+|Unless you generated and configured the correct SSL keys, disable this. Make sure to check our v4 upgrade notes
+
+|===
+
+== Aura
+
+Aura
+To connect to an Aura version 4 database we would set the following options:
+
+
+|===
+|Option |Value
+
+|Server or IP address
+|the provided ....neo4j.io hostname
+
+|Database name
+|neo4j
+
+|Bolt port
+|7687
+
+|Use routing
+|Enabled
+
+|neo4j:// protocol
+|Enabled
+
+|Routing policy
+|neo4j+s
+
+|Username
+|neo4j
+
+|Password
+|the provided password
+|===
+
+
+== The advanced options
+
+=== Connection Liveliness Check Timeout (ms)
+
+Pooled connections that have been idle in the pool for longer than this timeout will be tested before they are used again, to ensure they are still live. If this option is set too low, an additional network call will be incurred when acquiring a connection, which causes a performance hit.
+
+If this is set high, you may receive sessions that are backed by no longer live connections, which will lead to exceptions in your application. Assuming the database is running, these exceptions will go away if you retry acquiring sessions.
+
+Hence, this parameter tunes a balance between the likelihood of your application seeing connection problems, and performance.
+
+You normally should not need to tune this parameter. No connection liveliness check is done by default. Value 0 means connections will always be tested for validity and negative values mean connections will never be tested.
+
+=== Maximum Connection Lifetime (ms)
+
+Pooled connections older than this threshold will be closed and removed from the pool. Such discarding happens during connection acquisition so that new session is never backed by an old connection. Setting this option to a low value will cause a high connection churn and might result in a performance hit.
+
+It is recommended to set maximum lifetime to a slightly smaller value than the one configured in network equipment (load balancer, proxy, firewall, etc. can also limit maximum connection lifetime).
+
+Setting can also be used in combination with the Connection Liveness Check Timeout (ms) value. In this case, it is recommended to set liveness check to a value smaller than network equipment has and maximum lifetime to a reasonably large value to "renew" connections once in a while.
+
+Default maximum connection lifetime is 1 hour. Zero and negative values result in lifetime not being checked.
+
+=== Maximum Pool Size
+
+Configure the maximum amount of connections in the connection pool towards a single database. This setting limits total amount of connections in the pool when used in direct driver, created for URI with 'bolt' scheme. It will limit amount of connections per cluster member when used with routing driver, created for URI with 'neo4j' scheme.
+
+Acquisition will be attempted for at most configured timeout Connection Acquisition Timeout(ms) when limit is reached.
+
+Default value is 100. Negative values are allowed and result in unlimited pool. Value of 0 is not allowed.
+
+=== Connection Acquisition Time (ms)
+
+Configure the maximum amount of time the connection acquisition will attempt to acquire a connection from the connection pool. This timeout only kicks in when all existing connections are being used and no new connections can be created because maximum connection pool size has been reached. An error is thrown when a connection can't be acquired within the configured time.
+
+The default value is 60 seconds. Negative values are allowed and result in unlimited acquisition timeout. Value of 0 is allowed and results in no timeout and immediate failure when connection is unavailable.
+
+=== Maximum Transaction Retry Time (ms)
+
+Specify the maximum time transactions are allowed to retry via session read and write transactions. These methods will retry the given unit of work on ServiceUnavailableException, SessionExpiredException and TransientException with an exponential backoff using initial delay of 1 second. Default value is 30 seconds.
+
+=== Manual URLs
+
+You can specify a list of manual connection URLs to allow you to work with any advanced, legacy, bleeding edge, ... feature you can possibly want. See here for more information on Bolt connection URIs: https://neo4j.com/docs/driver-manual/current/client-applications/
+
diff --git a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-cypher.adoc b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-cypher.adoc
new file mode 100644
index 0000000..5e36dfa
--- /dev/null
+++ b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-cypher.adoc
@@ -0,0 +1,54 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/misc/neo4j/src/main/doc/neo4j-cypher.adoc
+= Neo4j Cypher
+
+== Description
+
+This transform allows you to execute any Cypher through the Neo4j Bolt protocol using the Java driver. Both reading, writing or both are supported. You can also call procedures and get the output.
+
+== Notable options
+
+
+|===
+|Option |Description
+
+|Get Cypher from input field
+|instead of hard coding statements you can actually have them created in for example a bit of JavaScript. Then each row can contain a different statement to be executed by this transform.
+
+|Cypher
+|the cypher statement to execute. You can use variables substitution (for example `${MY_VARIABLE}`) or parameter substitution (for example `{MY_PARAMETER}`). Note the missing `$`. Parameters can be specified in the next section of the options.
+
+|Use UNWIND to collect parameter values?
+|This option collects all the parameters in a map, adds all the row maps together in a list and passes this to the UNWIND statement. Consider this example:
+
+`UNWIND $customers AS customer CREATE (c:Customer { id : customer.id, name : customer.name })`
+
+we pick up `customer.id` and `customer.name` as parameters. The `customers` parameter comes from the name of the unwind map.
+
+|Name of UNWIND values map
+|the name of the map containing all the rows and values for the UNWIND statement.
+
+|Parameters
+|You can pick any Neo4j approved parameter name (without spaces and so on), the field to use and the Neo4j data type to convert to.
+
+|Returns
+|this transform can return one or more rows of data. You can specify the return values here. For every input row you can then have 0, 1 or more output rows. If you do not specify any return values the output will simply be the same as the input.
+
+|===
diff --git a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-graphoutput.adoc b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-graphoutput.adoc
new file mode 100644
index 0000000..bc831cb
--- /dev/null
+++ b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-graphoutput.adoc
@@ -0,0 +1,45 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+
+= Neo4j Graph Output
+
+== Description
+
+The Graph Output transform aims to allow input fields to be mapped automatically to a graph model. The model is for now defined locally but the goal is to have this done elsewhere in the near future. The algorithm used takes null values for fields mapped to primary properties of nodes into account and will not merge on these nodes and will not create or update relationships to it.
+
+== Limitations
+
+* Updates across unknown nodes `()--()` are not supported
+* Circular relationships to the same nodes: please create a new node with a different name and the same label to make this work. It allows you to uniquely map input fields to these node properties and get the correct results.
+
+
+== Notable options
+
+
+|===
+|Option |Description
+
+|Model
+|the model to map fields to
+
+
+|Mapping
+|Here you define the mappings
+
+|===
diff --git a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-output.adoc b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-output.adoc
new file mode 100644
index 0000000..64ef509
--- /dev/null
+++ b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-output.adoc
@@ -0,0 +1,50 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /pipeline/transforms/
+:language: en_US
+:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/misc/neo4j/src/main/doc/neo4j-output.adoc
+= Neo4j Output
+
+== Description
+
+This transform allows you to do high performance updates in one node, two nodes or two nodes and a relationship. The transform generates the required Cypher statements with accompanying parameters.
+
+== Usage
+
+You can update one node or two nodes and a relationship. Nodes and relationships can have properties and the appropriate MERGE statements will be generated based on the information you provided. Make use of the "Get fields" buttons on the right hand side of the dialog to prevent you from having to type too much.
+
+== Labels
+
+You can get one node labels from a list of fields or you can use static values (with optionally variables). If you specify both the static value will be used if the field value is null. A label will not be used for the node if the label is null or empty.
+
+
+
+== Notable options
+
+|===
+|Option |Description
+
+|Batch size
+|for better performance the transform will group records together in a single transaction
+
+|Use CREATE instead of MERGE
+|this bypasses any lookup and is faster.
+
+|Create indexes
+|creates unique field constraints for all Primary properties in the nodes to make sure performance stays good when dealing with merges and lookup of nodes for relationships. **NOTE:** Please consider only using this when doing prototyping and testing. It's better to put the index creation statements explicitly in a Cypher Script action to make sure you never forget it even when moving to a new database.
+
+|===
diff --git a/hop-user-manual/modules/ROOT/pages/technology/neo4j/neo4j-info.adoc b/hop-user-manual/modules/ROOT/pages/technology/neo4j/neo4j-info.adoc
new file mode 100644
index 0000000..5684e20
--- /dev/null
+++ b/hop-user-manual/modules/ROOT/pages/technology/neo4j/neo4j-info.adoc
@@ -0,0 +1,35 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /technology/neo4j/
+:language: en_US
+:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/misc/neo4j/src/main/doc/neo4j-info.adoc
+
+= Neo4j
+
+== Description
+
+Neo4j is an open source Graph database which you can download from **https://neo4j.com/download-center/[www.neo4j.com/download-center]**
+
+You can use it to represent information in way that allows for really fast graph queries to be executed. For more information on these unique graph algorithms see: https://neo4j.com/docs/graph-data-science/current/algorithms/#algorithms[Neo4j Graph Algorithms]
+
+== Execution lineage
+
+You can use Neo4j to store logging and execution lineage of your workflows and pipelines.
+The way you do this is simply by setting the variable NEO_LOGGING_CONNECTION to the name of the Neo4j Connection where you want the logging and lineage to be written to.
+
+The Neo4j plugin offers a separate perspective to query this logging and lineage information. For example, it allows you to quickly jump to the place where an error occurred.
+
diff --git a/hop-user-manual/modules/ROOT/pages/workflow/actions/deleteresultfilenames.adoc b/hop-user-manual/modules/ROOT/pages/workflow/actions/deleteresultfilenames.adoc
index 748ab56..1998da3 100644
--- a/hop-user-manual/modules/ROOT/pages/workflow/actions/deleteresultfilenames.adoc
+++ b/hop-user-manual/modules/ROOT/pages/workflow/actions/deleteresultfilenames.adoc
@@ -28,8 +28,8 @@ Use this action to delete all the filenames that are in the result files list of
[width="90%", options="header"]
|===
|Option|Description
-|Workflow action name|The name of the workflow action. *Note*: This name has to be unique in a single workflow. A workflow action can be placed several times on the canvas, however it will be the same workflow action.
-|Limit action to| Enable this feature if you want to limit the deletion to certain filenames in the result file list.
+|Workflow action name|The name of the workflow action.
+|Limit action to|Enable this feature if you want to limit the deletion to certain filenames in the result file list.
|Wildcard|The regular expression to limit the files to delete
-|Exclude wildcard|The regular expression to exclude certain files from being deleted.
+|Exclude wildcard|The regular expression to exclude certain files from being deleted.
|===
\ No newline at end of file
diff --git a/hop-user-manual/modules/ROOT/pages/workflow/actions/deleteresultfilenames.adoc b/hop-user-manual/modules/ROOT/pages/workflow/actions/neo4j-checkconnections.adoc
similarity index 53%
copy from hop-user-manual/modules/ROOT/pages/workflow/actions/deleteresultfilenames.adoc
copy to hop-user-manual/modules/ROOT/pages/workflow/actions/neo4j-checkconnections.adoc
index 748ab56..e1fb06b 100644
--- a/hop-user-manual/modules/ROOT/pages/workflow/actions/deleteresultfilenames.adoc
+++ b/hop-user-manual/modules/ROOT/pages/workflow/actions/neo4j-checkconnections.adoc
@@ -16,20 +16,14 @@ under the License.
////
:documentationPath: /workflow/actions/
:language: en_US
-:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/workflow/actions/deleteresultfilenames/src/main/doc/deleteresultfilenames.adoc
-= Delete filenames from result
+:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/misc/neo4j/src/main/doc/neo4j-checkconnections.adoc
+= Check Neo4j Connections
== Description
-Use this action to delete all the filenames that are in the result files list of the last workflow action.
+This action allows you to line up a list of Neo4j connections that you want to check for availability. A small static value will be retrieved from the server to see if it's properly up-and-running.
-== Options
+== Configuration
+
+Enter the names of the connections you want to check in the dialog.
-[width="90%", options="header"]
-|===
-|Option|Description
-|Workflow action name|The name of the workflow action. *Note*: This name has to be unique in a single workflow. A workflow action can be placed several times on the canvas, however it will be the same workflow action.
-|Limit action to| Enable this feature if you want to limit the deletion to certain filenames in the result file list.
-|Wildcard|The regular expression to limit the files to delete
-|Exclude wildcard|The regular expression to exclude certain files from being deleted.
-|===
\ No newline at end of file
diff --git a/hop-user-manual/modules/ROOT/pages/workflow/actions/neo4j-cypherscript.adoc b/hop-user-manual/modules/ROOT/pages/workflow/actions/neo4j-cypherscript.adoc
new file mode 100644
index 0000000..d1008f2
--- /dev/null
+++ b/hop-user-manual/modules/ROOT/pages/workflow/actions/neo4j-cypherscript.adoc
@@ -0,0 +1,56 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+:documentationPath: /workflow/actions/
+:language: en_US
+:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/misc/neo4j/src/main/doc/neo4j-cypherscript.adoc
+= Neo4j Cypher Script
+
+== Description
+
+With this action you can execute Cypher statements in a workflow.
+The goal is not to retrieve data but to execute various statements like creating or dropping indexes, calling APOC procedures, add or delete some static data and so on.
+
+== Important
+
+**To allow you to enter semi-colons (;) in commands, strings and so on we require you to separate commands with a semi-colon on a new line**
+
+== Configuration options
+
+
+|===
+|Option |Description
+
+|Connection name
+|The name of the connection on which you want to execute the cypher statements.
+
+|Script
+|The script with cypher commands seperated by a semi-colon (;) on a new line. For examples see below.
+
+|Replace variables
+|Check this option if you want to replace variables in the format `${VARIABLE_NAME}` in the script prior to execution.
+|===
+
+== Example
+
+[source,CREATE INDEX idx_person_name on :Person(name)]
+----
+CREATE INDEX idx_company_id on :Company(id)
+;
+CREATE INDEX idx_company_id on :Company(id)
+;
+----
+