You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by "dave-csc (via GitHub)" <gi...@apache.org> on 2024/11/05 11:25:57 UTC
[D] Streaming data in Metadata Injection (hop)
GitHub user dave-csc created a discussion: Streaming data in Metadata Injection
Hello,
I'm posting here with an actual help request (such as those filed in the Hop mailing list... BTW, is it still active?).
I set up a simple "to be injected" pipeline with a **Row Normalizer** linked to a **Dummy** transform, the latter one is needed to forward data to the "injector" (parent) pipeline.
In the parent pipeline I generate some fake data to be streamed into the Row Normalizer, and use a static **Data grid** to specify the fields to normalize them. You can see both pipelines in the attached file below.
The pipeline isn't working as expected: I got the error `Unable to find field from upstream transform: FirstName` in the log, despite `FirstName` is specified both as a field name in the streamed data, and in the Data grid as field to normalize.
Am I messing up something in the configuration, or is it a bug that needs to be reported as an issue?
Thanks in advance for your support.
_[**Next question: Why don't you use directly the Row Normalizer without injection?** In the actual scenario I'm developing, I'm not aware of the metadata to be streamed, except for some fields that need to be present. So those data would be streamed into the Row Normalizer, and the field table would be injected by generating the field list with a **Metadata structure of stream**, after removing those I know are needed for "grouping". Hence, the name "Anti-normalizer" 😃 ]_
---
Attached file:
[anti-normalizer.zip](https://github.com/user-attachments/files/17631081/anti-normalizer.zip)
GitHub link: https://github.com/apache/hop/discussions/4522
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Streaming data in Metadata Injection (hop)
Posted by "enricomariam42 (via GitHub)" <gi...@apache.org>.
GitHub user enricomariam42 added a comment to the discussion: Streaming data in Metadata Injection
Hi, put your data (Fake data transform) in the template pipeline (_inj-anti-normalizer.hpl) and use the parent ones (_test_inj-anti-normalizer.hpl) to inject the metadata only.
Take a look at the metadata-injection/read-file-template.hpl and metadata-injection/read-and-inject-metadata.hpl in the Sample project.
In the "ETL metadata injection" use "Options" "Optional target file" to easy debug.
GitHub link: https://github.com/apache/hop/discussions/4522#discussioncomment-11154993
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Streaming data in Metadata Injection (hop)
Posted by "dave-csc (via GitHub)" <gi...@apache.org>.
GitHub user dave-csc added a comment to the discussion: Streaming data in Metadata Injection
Hi @enricomariam42,
I used "Fake data" as a generic input transform, it could actually be any transform with any kind of stream traversing it (check also the "Next question" above). So probably putting it in the template pipeline is not a good solution... I'll try anyway to debug with the target file.
Thanks anyway for answering :)
GitHub link: https://github.com/apache/hop/discussions/4522#discussioncomment-11155588
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Streaming data in Metadata Injection (hop)
Posted by "enricomariam42 (via GitHub)" <gi...@apache.org>.
GitHub user enricomariam42 added a comment to the discussion: Streaming data in Metadata Injection
Hi, @dave-csc,
put all the transforms needed to play with data in the template pipeline and use the parent one to fill the fields, not to pass the data.
Check the examples carefully.
GitHub link: https://github.com/apache/hop/discussions/4522#discussioncomment-11156226
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Streaming data in Metadata Injection (hop)
Posted by "dave-csc (via GitHub)" <gi...@apache.org>.
GitHub user dave-csc added a comment to the discussion: Streaming data in Metadata Injection
Update: I managed to get the "Anti-normalizer" transform as I wished, after struggling with a possible bug in Data grid and Metadata Injection transforms (check issue #4525 for details and workaround).
You can find the result in the attached file below: you can import the files **_inj-anti-normalizer.hpl** and **anti-normalizer.hpl** in your projects, and use the latter one as a sub-pipeline with the Simple Mapping transform (check the file **_test-anti-normalizer.hpl** for the needed parameter specifications: grouping fields, key field name, value field name).
---
Attached file:
[anti-normalizer-v1.0.zip](https://github.com/user-attachments/files/17645178/anti-normalizer-v1.0.zip)
GitHub link: https://github.com/apache/hop/discussions/4522#discussioncomment-11164873
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org