You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by "dave-csc (via GitHub)" <gi...@apache.org> on 2025/04/03 14:58:30 UTC
[D] Need help for a broken pipeline... probably after a Hop update (hop)
GitHub user dave-csc created a discussion: Need help for a broken pipeline... probably after a Hop update
Hello,
I need your help to figure out what has been changed in Hop to make a pipeline of mine not working anymore.
The pipeline composition is attached at the end of the post, the archive contains 3 files:
- **anti-normalizer.hpl**, the "main" pipeline: it should provide a sort of emulation of the Row Normalizer transform, with the added value that you don't have to provide the full schema to the transform, but only the fields you would like to use for grouping
- **_inj-anti-normalizer.hpl**, the "injected" pipeline: this pipeline is injected with the data to normalize and the schema/metadata that is evaluated dinamically
- **_test-anti-normalizer.hpl**, the "testing" pipeline: it generates some fake data to demonstrate the functionality
This composition worked well in Hop 2.10: the output for the test case is correctly generated with a `RecordID` grouping column, and the pairs `elementName` and `elementValue`.
Starting from Hop 2.11 the same composition is broken. During its execution the following error, regarding the Data grid transform in the injected pipeline, is reported:
```
2025/04/03 16:28:03 - Data grid.0 - ERROR: Unexpected error
2025/04/03 16:28:03 - Data grid.0 - ERROR: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
2025/04/03 16:28:03 - Data grid.0 - at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
2025/04/03 16:28:03 - Data grid.0 - at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
2025/04/03 16:28:03 - Data grid.0 - at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)
2025/04/03 16:28:03 - Data grid.0 - at java.base/java.util.Objects.checkIndex(Objects.java:359)
2025/04/03 16:28:03 - Data grid.0 - at java.base/java.util.ArrayList.get(ArrayList.java:427)
2025/04/03 16:28:03 - Data grid.0 - at org.apache.hop.pipeline.transforms.datagrid.DataGrid.processRow(DataGrid.java:80)
2025/04/03 16:28:03 - Data grid.0 - at org.apache.hop.pipeline.transform.RunThread.run(RunThread.java:54)
2025/04/03 16:28:03 - Data grid.0 - at java.base/java.lang.Thread.run(Thread.java:833)
2025/04/03 16:28:03 - Data grid.0 - Finished processing (I=0, O=0, R=0, W=0, U=0, E=1)
```
What has been changed in Hop in order to make the pipeline fail, and what can I do to make a fix?
_**Hint for the search:** I started developing this mini-project slightly before reporting issue #4525 (you can notice that similar components are involved). The issue has been solved, but probably has broken something else..._
---
Attached files:
[anti-normalizer.zip](https://github.com/user-attachments/files/19589385/anti-normalizer.zip)
GitHub link: https://github.com/apache/hop/discussions/5128
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Need help for a broken pipeline... probably after a Hop update (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Need help for a broken pipeline... probably after a Hop update
yes, feel free to create a ticket. I can't find an easy solution at the moment
GitHub link: https://github.com/apache/hop/discussions/5128#discussioncomment-12726775
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Need help for a broken pipeline... probably after a Hop update (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Need help for a broken pipeline... probably after a Hop update
Hi @dave-csc,
I found the issue. The root cause is that the second field in the data section for the data grid has to be filled in.
Trying to find a way to resolve this. But setting it to a constant "," will solve your issue.
This was added to make it possible to inject multiple columns to a datagrid.
GitHub link: https://github.com/apache/hop/discussions/5128#discussioncomment-12725564
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Need help for a broken pipeline... probably after a Hop update (hop)
Posted by "dave-csc (via GitHub)" <gi...@apache.org>.
GitHub user dave-csc added a comment to the discussion: Need help for a broken pipeline... probably after a Hop update
> Have you tried specifying a target file name for the pipeline after injection (`Options` -> `Option target file (hpl after injection)`). Do you see any issues in the data grid if you uncheck the `Run resulting pipeline` and open the generated pipeline after the injection?
Hi @bamaer,
I'll explain the path question below, first I tried to do this test and a significant difference occurred in the resulting debug pipeline:
- with Hop 2.10 the **Data grid** transform is correctly populated, both in the "Meta" and the "Data" sections
- with Hop 2.12 only the "Meta" section is populated, the "Data" section contains 5 empty rows (as if their contents have not been injected)
> First thing I noticed is that your MDI transform points to `${HOP_CONFIG_FOLDER}/projects/_root/datanorm/_inj-anti-normalizer.hpl` as the child pipeline. As the name implies, `${HOP_CONFIG_FOLDER}` should be a folder to store configuration. Using it for project files is not what I would consider a best practice, if not just tricky.
The `${HOP_CONFIG_FOLDER}` has been moved away from its default location, so the configuration used is the same for all the installed versions of Hop. However, in the default location a `project` subfolder indeed exists, and seems to be intended to store project files (`default` and `samples` are there): so why it should be risky to keep a similar one off the default location?
Thanks for your support
GitHub link: https://github.com/apache/hop/discussions/5128#discussioncomment-12724370
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Need help for a broken pipeline... probably after a Hop update (hop)
Posted by "dave-csc (via GitHub)" <gi...@apache.org>.
GitHub user dave-csc added a comment to the discussion: Need help for a broken pipeline... probably after a Hop update
Hi @hansva,
your suggestion worked! Thank you so much :)
Should I report this in the issues?
GitHub link: https://github.com/apache/hop/discussions/5128#discussioncomment-12725697
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Need help for a broken pipeline... probably after a Hop update (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: Need help for a broken pipeline... probably after a Hop update
First thing I noticed is that your MDI transform points to `${HOP_CONFIG_FOLDER}/projects/_root/datanorm/_inj-anti-normalizer.hpl` as the child pipeline. As the name implies, `${HOP_CONFIG_FOLDER}` should be a folder to store configuration. Using it for project files is not what I would consider a best practice, if not just tricky.
Have you tried specifying a target file name for the pipeline after injection (`Options` -> `Option target file (hpl after injection)`). Do you see any issues in the data grid if you uncheck the `Run resulting pipeline` and open the generated pipeline after the injection?
GitHub link: https://github.com/apache/hop/discussions/5128#discussioncomment-12715329
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Need help for a broken pipeline... probably after a Hop update (hop)
Posted by "dave-csc (via GitHub)" <gi...@apache.org>.
GitHub user dave-csc edited a comment on the discussion: Need help for a broken pipeline... probably after a Hop update
> Have you tried specifying a target file name for the pipeline after injection (`Options` -> `Option target file (hpl after injection)`). Do you see any issues in the data grid if you uncheck the `Run resulting pipeline` and open the generated pipeline after the injection?
Hi @bamaer,
I'll explain the path question below (I exported directly my pipelines, for your tests you need to tweak the paths in the "testing" and "main" pipeline of course), first I tried to do this test and a significant difference occurred in the resulting debug pipeline:
- with Hop 2.10 the **Data grid** transform is correctly populated, both in the "Meta" and the "Data" sections
- with Hop 2.12 only the "Meta" section is populated, the "Data" section contains 5 empty rows (as if their contents have not been injected)
> First thing I noticed is that your MDI transform points to `${HOP_CONFIG_FOLDER}/projects/_root/datanorm/_inj-anti-normalizer.hpl` as the child pipeline. As the name implies, `${HOP_CONFIG_FOLDER}` should be a folder to store configuration. Using it for project files is not what I would consider a best practice, if not just tricky.
The `${HOP_CONFIG_FOLDER}` has been moved away from its default location, so the configuration used is the same for all the installed versions of Hop. However, in the default location a `project` subfolder indeed exists, and seems to be intended to store project files (`default` and `samples` are there): so why it should be risky to keep a similar one off the default location?
Thanks for your support
GitHub link: https://github.com/apache/hop/discussions/5128#discussioncomment-12724370
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org