You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hop.apache.org by gi...@apache.org on 2021/03/22 21:49:55 UTC
[incubator-hop-docs] branch asf-site updated: Documentation updated
to 92ccbe9
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-hop-docs.git
The following commit(s) were added to refs/heads/asf-site by this push:
new ae0b081 Documentation updated to 92ccbe9
ae0b081 is described below
commit ae0b08182baa1165e9d69263cc041e7fd1e28b0c
Author: jenkins <bu...@apache.org>
AuthorDate: Mon Mar 22 21:49:52 2021 +0000
Documentation updated to 92ccbe9
---
.../modules/ROOT/pages/pipeline/transforms/dynamicsqlrow.adoc | 4 +++-
.../modules/ROOT/pages/pipeline/transforms/filterrows.adoc | 2 +-
.../modules/ROOT/pages/pipeline/transforms/metainject.adoc | 8 +++++---
.../modules/ROOT/pages/pipeline/transforms/neo4j-cypher.adoc | 5 ++++-
.../modules/ROOT/pages/pipeline/transforms/replacestring.adoc | 4 +++-
hop-user-manual/modules/ROOT/pages/pipeline/transforms/sql.adoc | 4 +++-
.../modules/ROOT/pages/pipeline/transforms/terafast.adoc | 4 +++-
hop-user-manual/modules/ROOT/pages/plugins/projects/projects.adoc | 5 +----
.../modules/ROOT/pages/plugins/transforms/dropboxinput.adoc | 2 +-
.../modules/ROOT/pages/plugins/transforms/dropboxoutput.adoc | 2 +-
.../ROOT/pages/workflow/actions/deleteresultfilenames.adoc | 6 +++---
.../modules/ROOT/pages/workflow/actions/neo4j-cypherscript.adoc | 4 +++-
hop-user-manual/modules/ROOT/pages/workflow/actions/pipeline.adoc | 8 +++++---
.../modules/ROOT/pages/workflow/actions/simpleeval.adoc | 4 +++-
hop-user-manual/modules/ROOT/pages/workflow/actions/workflow.adoc | 6 ++++--
15 files changed, 43 insertions(+), 25 deletions(-)
diff --git a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dynamicsqlrow.adoc b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dynamicsqlrow.adoc
index ece8b69..39efe14 100644
--- a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dynamicsqlrow.adoc
+++ b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/dynamicsqlrow.adoc
@@ -17,6 +17,8 @@ under the License.
:documentationPath: /pipeline/transforms/
:language: en_US
:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/transforms/dynamicsqlrow/src/main/doc/dynamicsqlrow.adoc
+:openvar: ${
+:closevar: }
= Dynamic SQL row
== Description
@@ -33,7 +35,7 @@ The Dynamic SQL row transform allows you to execute a SQL statement that is defi
|SQL field name|Specify the field that contains the SQL to execute
|Number of rows to return|Specify the number of records to return. 0 means, return all rows
|Outer Join|- false: don't return rows where nothing is found - true: at least return one source row, the rest is NULL
-|Replace variables|In case you want to use variables in the SQL, e.g. ${table_name}, this option needs to be checked.
+|Replace variables|In case you want to use variables in the SQL, e.g. {openvart}able_name{closevar}, this option needs to be checked.
|Query only on parameters change|If your SQL statements do not change a lot, check this option to reduce the number of physical database queries.
|Template SQL|In Hop meta and data are separate so you have to specify the meta part in template SQL (field name and type).
I mean any statement that returns the same row structure.
diff --git a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filterrows.adoc b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filterrows.adoc
index c8dc3a8..24cb7cb 100644
--- a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filterrows.adoc
+++ b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/filterrows.adoc
@@ -54,7 +54,7 @@ A query:
----
SELECT field1,
field2,
-${myvar} AS field3
+${myvar} AS field3
FROM table
WHERE field1=xxxx
----
diff --git a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metainject.adoc b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metainject.adoc
index 86ccec6..3146a7f 100644
--- a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metainject.adoc
+++ b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metainject.adoc
@@ -17,6 +17,8 @@ under the License.
:documentationPath: /pipeline/transforms/
:language: en_US
:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/transforms/metainject/src/main/doc/metainject.adoc
+:openvar: ${
+:closevar: }
= Metadata Injection
== Description
@@ -46,7 +48,7 @@ The metadata is injected into the template pipeline through any transform that s
|Transform name|Name of the transform.
|Pipeline|Specify your template pipeline by entering in its path. Click Browse to display and enter the path details using the Virtual File System Browser.
-If you select a pipeline that has the same root path as the current pipeline, the variable ${Internal.Transform.Current.Directory} will automatically be inserted in place of the common root path. For example, if the current pipeline's path is /home/admin/pipeline.hpl and you select a pipeline in the folder /home/admin/path/sub.hpl then the path will automatically be converted to ${Internal.Transform.Current.Directory}/path/sub.hpl.
+If you select a pipeline that has the same root path as the current pipeline, the variable {openvar}Internal.Transform.Current.Directory{closevar} will automatically be inserted in place of the common root path. For example, if the current pipeline's path is /home/admin/pipeline.hpl and you select a pipeline in the folder /home/admin/path/sub.hpl then the path will automatically be converted to {openvar}Internal.Transform.Current.Directory{closevar}/path/sub.hpl.
|===
The ETL Metadata Injection transform features the two tabs with fields. Each tab is described below.
@@ -70,8 +72,8 @@ To specify the source field as metadata to be injected, perform the following tr
3. Optionally, select Use constant value to specify a constant value for the injected metadata through one of the following actions:
- Manually entering a value.
- - Using an internal variable to set the value (${Internal.transform.Unique.Count} for example).
- - Using a combination of manually specified values and parameter values (${FILE_PREFIX}_${FILE_DATE}.txt for example).
+ - Using an internal variable to set the value ({openvar}Internal.transform.Unique.Count{closevar} for example).
+ - Using a combination of manually specified values and parameter values ({openvarF}ILE_PREFIX{closevar}_{openvarF}ILE_DATE{closevar}.txt for example).
==== Injecting Metadata into the ETL Metadata Injection transform
diff --git a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-cypher.adoc b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-cypher.adoc
index 5e36dfa..ab433c4 100644
--- a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-cypher.adoc
+++ b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/neo4j-cypher.adoc
@@ -17,6 +17,9 @@ under the License.
:documentationPath: /pipeline/transforms/
:language: en_US
:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/misc/neo4j/src/main/doc/neo4j-cypher.adoc
+:openvar: ${
+:openvar2: {
+:closevar: }
= Neo4j Cypher
== Description
@@ -33,7 +36,7 @@ This transform allows you to execute any Cypher through the Neo4j Bolt protocol
|instead of hard coding statements you can actually have them created in for example a bit of JavaScript. Then each row can contain a different statement to be executed by this transform.
|Cypher
-|the cypher statement to execute. You can use variables substitution (for example `${MY_VARIABLE}`) or parameter substitution (for example `{MY_PARAMETER}`). Note the missing `$`. Parameters can be specified in the next section of the options.
+|the cypher statement to execute. You can use variables substitution (for example `{openvar}MY_VARIABLE{closevar}`) or parameter substitution (for example `{openvar2}MY_PARAMETER{closevar}`). Note the missing `$`. Parameters can be specified in the next section of the options.
|Use UNWIND to collect parameter values?
|This option collects all the parameters in a map, adds all the row maps together in a list and passes this to the UNWIND statement. Consider this example:
diff --git a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/replacestring.adoc b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/replacestring.adoc
index dca448e..f809338 100644
--- a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/replacestring.adoc
+++ b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/replacestring.adoc
@@ -17,6 +17,8 @@ under the License.
:documentationPath: /pipeline/transforms/
:language: en_US
:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/transforms/replacestring/src/main/doc/replacestring.adoc
+:openvar: {
+:closevar: }
= Replace in String
== Description
@@ -45,7 +47,7 @@ Replace in string is a simple search and replace. It also supports regular expre
You could parse out sections of a string as follows
-* regex = ^([0-9]{4})([0-9]{7})$
+* regex = ^([0-9]{openvar}4{closevar})([0-9]{openvar}7{closevar})$
* replace with = $2
This will replace any 11 digit numbers with the last 7 digits.
diff --git a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sql.adoc b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sql.adoc
index f264fe1..459ce2f 100644
--- a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sql.adoc
+++ b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sql.adoc
@@ -17,6 +17,8 @@ under the License.
:documentationPath: /pipeline/transforms/
:language: en_US
:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/transforms/sql/src/main/doc/sql.adoc
+:openvar: ${
+:closevar: }
= Execute SQL script
== Description
@@ -33,7 +35,7 @@ You can execute SQL scripts with this transform, either once, during the initial
|SQL script to execute|Specify the SQL to execute. Separate statements by ; and use question marks as place holders for parameter.
|Execute for each row?|Select this option to execute the SQL for each incoming row. In this case paramters can be used. When this option is unchecked, the SQL statement is executed at the transform initialization phase.
|Execute as a single statement|This option does not split the statements by ; and will send the whole SQL to the database.
-|Variable substitution|In case you want to use variables in the SQL, e.g. ${table_name}, this option needs to be checked.
+|Variable substitution|In case you want to use variables in the SQL, e.g. {openvart}able_name{closevar}, this option needs to be checked.
|Bind parameters?|Check this option to bind parameters using prepared statements, otherwise this transform will perform a literal string replacement of the parameters.
|Quote Strings?|This option adds quotes around the string according to the database dialect and also escapes special characters like CR, LF and the quote character itself.
|Parameters|The list of used parameters that will replace the question marks in the query in the given order. So the first question mark will be replaced by the first parameter, the second question mark by the second parameter etc.
diff --git a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/terafast.adoc b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/terafast.adoc
index d7c58d0..5def907 100644
--- a/hop-user-manual/modules/ROOT/pages/pipeline/transforms/terafast.adoc
+++ b/hop-user-manual/modules/ROOT/pages/pipeline/transforms/terafast.adoc
@@ -17,6 +17,8 @@ under the License.
:documentationPath: /pipeline/transforms/
:language: en_US
:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/transforms/terafast/src/main/doc/terafast.adoc
+:openvar: ${
+:closevar: }
= Teradata Bulk Loader
@@ -32,7 +34,7 @@ The Teradata Bulkloader transform supports fastloading data into a Teradata data
|Transform name|Name of the transform this name has to be unique in a single pipeline.
|Use control file| Work in control file mode.
|Control file|The path to the control file to be used.
-|Variable Substitution in control file|Make use of POI Variables (eg: ${RUN_ID}) in the control file.
+|Variable Substitution in control file|Make use of POI Variables (eg: {openvar}RUN_ID{closevar}) in the control file.
|Path to fastload|The path to the fastload command line utility.
|Error log|An optional Error log to be created by Fastload.
|Connection|A connection to the Teradata DB.
diff --git a/hop-user-manual/modules/ROOT/pages/plugins/projects/projects.adoc b/hop-user-manual/modules/ROOT/pages/plugins/projects/projects.adoc
index 4f15fc5..062d8da 100644
--- a/hop-user-manual/modules/ROOT/pages/plugins/projects/projects.adoc
+++ b/hop-user-manual/modules/ROOT/pages/plugins/projects/projects.adoc
@@ -17,9 +17,6 @@ under the License.
:documentationPath: /plugins/projects/
:language: en_US
:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/misc/projects/src/main/doc/projects.adoc
-:openvar: ${
-:closevar: }
-
= Projects & Environments
This document explains the basic structure and working of the 'projects' plugin
@@ -276,7 +273,7 @@ To execute an Apache Beam pipeline a lot of information and metadata is needed.
* By referencing the ```samples``` project Hop knows where the project is located (```config/projects/samples```)
* Since we know the location of the project, we can specify pipelines and workflows with a relative path
* The project knows where its metadata is stored (```config/projects/samples/metadata```) so it knows where to find the ```Direct``` pipeline run configuration (```config/projects/samples/metadata/pipeline-run-configuration/Direct.json```)
-* This run configuration defines its own pipeline engine specific variables, in this case the output folder : ```DATA_OUTPUT={openvar}PROJECT_HOME{closevar}/beam/output/```
+* This run configuration defines its own pipeline engine specific variables, in this case the output folder : ```DATA_OUTPUT=${PROJECT_HOME}/beam/output/```
* The output of the samples is as such written to ```config/projects/samples/beam/output```
To reference an environment you can execute using ```-e``` or ```--environment```. The only difference is that you'll have a number of extra environment variables set while executing.
diff --git a/hop-user-manual/modules/ROOT/pages/plugins/transforms/dropboxinput.adoc b/hop-user-manual/modules/ROOT/pages/plugins/transforms/dropboxinput.adoc
index 9c5e088..2d240fc 100644
--- a/hop-user-manual/modules/ROOT/pages/plugins/transforms/dropboxinput.adoc
+++ b/hop-user-manual/modules/ROOT/pages/plugins/transforms/dropboxinput.adoc
@@ -24,7 +24,7 @@ The dropbox input step reads information from a Dropbox storage account.
NOTE: create a Dropbox api key in the Dropbox https://www.dropbox.com/developers/apps/create[console].
-## Fields
+### Fields
[width="90%", options="header"]
|===
diff --git a/hop-user-manual/modules/ROOT/pages/plugins/transforms/dropboxoutput.adoc b/hop-user-manual/modules/ROOT/pages/plugins/transforms/dropboxoutput.adoc
index 847d063..a5f5b62 100644
--- a/hop-user-manual/modules/ROOT/pages/plugins/transforms/dropboxoutput.adoc
+++ b/hop-user-manual/modules/ROOT/pages/plugins/transforms/dropboxoutput.adoc
@@ -24,7 +24,7 @@ The dropbox output step writes information to a Dropbox storage account.
NOTE: create a Dropbox api key in the Dropbox https://www.dropbox.com/developers/apps/create[console].
-## Fields
+### Fields
[width="90%", options="header"]
|===
diff --git a/hop-user-manual/modules/ROOT/pages/workflow/actions/deleteresultfilenames.adoc b/hop-user-manual/modules/ROOT/pages/workflow/actions/deleteresultfilenames.adoc
index 748ab56..1998da3 100644
--- a/hop-user-manual/modules/ROOT/pages/workflow/actions/deleteresultfilenames.adoc
+++ b/hop-user-manual/modules/ROOT/pages/workflow/actions/deleteresultfilenames.adoc
@@ -28,8 +28,8 @@ Use this action to delete all the filenames that are in the result files list of
[width="90%", options="header"]
|===
|Option|Description
-|Workflow action name|The name of the workflow action. *Note*: This name has to be unique in a single workflow. A workflow action can be placed several times on the canvas, however it will be the same workflow action.
-|Limit action to| Enable this feature if you want to limit the deletion to certain filenames in the result file list.
+|Workflow action name|The name of the workflow action.
+|Limit action to|Enable this feature if you want to limit the deletion to certain filenames in the result file list.
|Wildcard|The regular expression to limit the files to delete
-|Exclude wildcard|The regular expression to exclude certain files from being deleted.
+|Exclude wildcard|The regular expression to exclude certain files from being deleted.
|===
\ No newline at end of file
diff --git a/hop-user-manual/modules/ROOT/pages/workflow/actions/neo4j-cypherscript.adoc b/hop-user-manual/modules/ROOT/pages/workflow/actions/neo4j-cypherscript.adoc
index d1008f2..728d3a8 100644
--- a/hop-user-manual/modules/ROOT/pages/workflow/actions/neo4j-cypherscript.adoc
+++ b/hop-user-manual/modules/ROOT/pages/workflow/actions/neo4j-cypherscript.adoc
@@ -17,6 +17,8 @@ under the License.
:documentationPath: /workflow/actions/
:language: en_US
:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/plugins/misc/neo4j/src/main/doc/neo4j-cypherscript.adoc
+:openvar: ${
+:closevar: }
= Neo4j Cypher Script
== Description
@@ -41,7 +43,7 @@ The goal is not to retrieve data but to execute various statements like creating
|The script with cypher commands seperated by a semi-colon (;) on a new line. For examples see below.
|Replace variables
-|Check this option if you want to replace variables in the format `${VARIABLE_NAME}` in the script prior to execution.
+|Check this option if you want to replace variables in the format `{openvar}VARIABLE_NAME{closevar}` in the script prior to execution.
|===
== Example
diff --git a/hop-user-manual/modules/ROOT/pages/workflow/actions/pipeline.adoc b/hop-user-manual/modules/ROOT/pages/workflow/actions/pipeline.adoc
index 3278f0e..7cf7bcb 100644
--- a/hop-user-manual/modules/ROOT/pages/workflow/actions/pipeline.adoc
+++ b/hop-user-manual/modules/ROOT/pages/workflow/actions/pipeline.adoc
@@ -17,6 +17,8 @@ under the License.
:documentationPath: /workflow/actions/
:language: en_US
:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/workflow/actions/pipeline/src/main/doc/pipeline.adoc
+:openvar: ${
+:closevar: }
= Pipeline
== Description
@@ -35,7 +37,7 @@ An example of a common workflow includes getting FTP files, checking existence o
|Action name|Name of the action.
|Pipeline|Specify your pipeline by entering in its path or clicking Browse.
-If you select a pipeline that has the same root path as the current pipeline, the variable ${Internal.Action.Current.Directory} will automatically be inserted in place of the common root path. For example, if the current pipeline's path is /home/admin/pipeline.hpl and you select a pipeline in the folder /home/admin/path/sub.hpl than the path will automatically be converted to ${Internal.Action.Current.Directory}/path/sub.hpl.
+If you select a pipeline that has the same root path as the current pipeline, the variable {openvar}Internal.Action.Current.Directory{closevar} will automatically be inserted in place of the common root path. For example, if the current pipeline's path is /home/admin/pipeline.hpl and you select a pipeline in the folder /home/admin/path/sub.hpl than the path will automatically be converted to {openvar}Internal.Action.Current.Directory{closevar}/path/sub.hpl.
Pipelines previously specified by reference are automatically converted to be specified by name.
|===
@@ -93,8 +95,8 @@ In most instances, it is acceptable for logging information to be available in t
|Value|Specify pipeline parameter values through one of the following actions:
- Manually entering a value (ETL workflow for example).
-- Using another parameter to set the value (${Internal.workflow.Name} for example).
-- Using a combination of manually specified values and parameter values (${FILE_PREFIX}_${FILE_DATE}.txt for example).
+- Using another parameter to set the value ({openvar}Internal.workflow.Name{closevar} for example).
+- Using a combination of manually specified values and parameter values ({openvar}FILE_PREFIX{closevar}_{openvar}FILE_DATE{closevar}.txt for example).
|Get Parameters|Get the existing parameters already associated by the pipeline.
|===
\ No newline at end of file
diff --git a/hop-user-manual/modules/ROOT/pages/workflow/actions/simpleeval.adoc b/hop-user-manual/modules/ROOT/pages/workflow/actions/simpleeval.adoc
index 8d87a19..3e8d844 100644
--- a/hop-user-manual/modules/ROOT/pages/workflow/actions/simpleeval.adoc
+++ b/hop-user-manual/modules/ROOT/pages/workflow/actions/simpleeval.adoc
@@ -17,6 +17,8 @@ under the License.
:documentationPath: /workflow/actions/
:language: en_US
:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/workflow/actions/simpleeval/src/main/doc/simpleeval.adoc
+:openvar: ${
+:closevar: }
= Simple Evaluation
== Description
@@ -57,7 +59,7 @@ Connect two actions with the output of the Simple Evaluation. The green connecti
|===
|Field|Value|Explanation
|Source: Evaluate|Variable|Select this to evaluate a variable set before
-|Source: Variable name|e. g. ${VAR1}|Enter the variable name using the usual syntax
+|Source: Variable name|e. g. {openvar}VAR1{closevar}|Enter the variable name using the usual syntax
|Source: Type|String, Number etc.|The type of your variable
|Success on: Success when variable set||When this option is selected, the true path is followed when the variable is set.
Note: The workflow action checks if the variable is defined, so an empty or null value is also true. When testing this within Hop you need to delete the line with the variable in the variables section of the Execute a workflow window. And remember: Once this variable is set, the variable keeps existing, even when deleting it from the list.
diff --git a/hop-user-manual/modules/ROOT/pages/workflow/actions/workflow.adoc b/hop-user-manual/modules/ROOT/pages/workflow/actions/workflow.adoc
index c1d7ae7..82e030f 100644
--- a/hop-user-manual/modules/ROOT/pages/workflow/actions/workflow.adoc
+++ b/hop-user-manual/modules/ROOT/pages/workflow/actions/workflow.adoc
@@ -17,6 +17,8 @@ under the License.
:documentationPath: /workflow/actions/
:language: en_US
:page-alternativeEditUrl: https://github.com/apache/incubator-hop/edit/master/workflow/actions/workflow/src/main/doc/workflow.adoc
+:openvar: ${
+:closevar: }
= Workflow
== Description
@@ -91,7 +93,7 @@ Specify which parameters will be passed to the pipeline:
|Stream column name|Allows you to capture fields of incoming records of a result set as a parameter.
|Value a| Allows you to specify the values for the pipeline's parameters. You can do this by:
- Manually typing a value (Ex: ETL workflow)
-- Use a parameter to set the value (Ex: ${Internal.workflow.Name}
-- Using a combination of manually specified values and parameter values (Ex: ${FILE_PREFIX}_${FILE_DATE}.txt)
+- Use a parameter to set the value (Ex: {openvar}Internal.workflow.Name{closevar}
+- Using a combination of manually specified values and parameter values (Ex: {openvar}FILE_PREFIX{closevar}_{openvar}FILE_DATE{closevar}.txt)
|===