You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hop.apache.org by ha...@apache.org on 2023/10/13 12:13:45 UTC
[hop] branch master updated: Update metainject.adoc
This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/master by this push:
new d5ee38d43a Update metainject.adoc
new 5c7fe508b1 Merge pull request #3249 from Mattang-Dan/patch-23
d5ee38d43a is described below
commit d5ee38d43a0036695ac85577472308151f1bda2e
Author: Mattang-Dan <10...@users.noreply.github.com>
AuthorDate: Fri Sep 22 11:03:11 2023 -0400
Update metainject.adoc
---
.../ROOT/pages/pipeline/transforms/metainject.adoc | 24 +++++++++++++++++-----
1 file changed, 19 insertions(+), 5 deletions(-)
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metainject.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metainject.adoc
index 4572ac9c2a..71545f44a3 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metainject.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/metainject.adoc
@@ -16,7 +16,7 @@ under the License.
////
:documentationPath: /pipeline/transforms/
:language: en_US
-:description: The Metadata Injection transform inserts data from various sources into a pipeline at runtime.
+:description: The ETL metadata injection transform inserts data from various sources into a pipeline at runtime.
:openvar: ${
:closevar: }
@@ -44,11 +44,19 @@ Metadata injection allows Apache Hop users to provide the required metadata for
== Usage
-The Metadata Injection transform inserts data from various sources into a pipeline at runtime. This is typically used to automate repetitive ETL tasks.
+The Metadata Injection transform (MDI transform) inserts data and transform properties into a pipeline that is dynamically configured at runtime. This is typically used to enhance reusability of a pipeline.
-A typical example is the onboarding of large amounts of text file data in a variety of formats. Even though you may have a number of different file layouts to load, the process is always the same: read the file, clean or enrich the data, and finally write the data to a target platform or some other file format. The only difference is the actual layout of the data, the steps to follow are always the same.
+E.g. Use Case: If you have 10 different CSV file definitions and don't want to create 10 pipelines for each separate file, you can use ETL metadata injection. After the MDI transform is configured (mapping data and properties to the template/target pipeline), at run-time it will dynamically populate any non configured transform properties in the target pipeline. For example, in the template file, if you leave the Table input Database fields empty, and if configured in the MDI transform, [...]
-xref:pipeline/metadata-injection.adoc[Read more]
+Basically, the ETL Metadata Injection transform overrules the default behavior for the named pipeline. Not all incoming streams need to have the same layout. It is perfectly OK to provide input to the metadata injection transform from different streams with different layouts.
+
+*Sample files:*
+
+* *Configure MDI:* read-and-inject-metadata.hpl
+
+* *Read and fill in Template:* read-file-template.hpl
+
+xref:pipeline/metadata-injection.adoc[Read more on metadata injection]
== Options
@@ -70,11 +78,12 @@ The ETL Metadata Injection transform features the two tabs with fields.
Each tab is described below.
=== Inject Metadata Tab
+This tab shows transforms and properties on the target/template file that will be modified at run-time. Map each property to a source that you would like to be populated dynamically.
[options="header"]
|===
|Option|Description
-|Target injection transform key| Lists the available fields in each transform of the template pipeline that can be injected with metadata.
+|Target injection transform key| Lists the available fields in each transform of the template pipeline that can be injected with metadata.
|Target description|Describes how the target fields relate to their target transforms.
|Source transform|Lists the transform associated with the fields to be injected into the target fields as metadata.
|Source field|Lists the fields to be injected into the target fields as metadata.
@@ -123,3 +132,8 @@ The resulting pipeline will be your template pipeline with the metadata already
If this option is not selected, metadata injection occurs, but the template pipeline does not run.
|===
+
+*Troubleshooting*
+
+* Enter a pipeline filename in "Optional target file (hpl after injection)" so that a file will be generated after injection. This pipeline will contain all the properties after injection and can be viewed and executed for testing.
+