You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by "EricTonissen (via GitHub)" <gi...@apache.org> on 2025/12/09 11:37:04 UTC
[D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
GitHub user EricTonissen created a discussion: Pipeline keeps running on simple 'Stream Lookup' action.
Anybody any idea why this Stream Lookup action doesn't continue?
<img width="306" height="236" alt="image" src="https://github.com/user-attachments/assets/f1552322-0004-4f12-8325-ae2e089920c1" />
AKKOORDJN output rows : 4990
QUERY PATARTS output rows : 12064
<img width="602" height="35" alt="image" src="https://github.com/user-attachments/assets/980c55d4-954a-4f1a-babb-5e3f5092306a" />
GitHub link: https://github.com/apache/hop/discussions/6152
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "EricTonissen (via GitHub)" <gi...@apache.org>.
GitHub user EricTonissen added a comment to the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
**ByteArrayHashIndex.put()** ends in an endless loop, if the array is full.
Better iterate the array only once. If it is full, then throw error.
//
// If we are still here, that means that we are ready to put the value down...
// Where do we need to search for an empty spot in the index?
//
int len = index.length;
while (true) {
indexPointer++;
if (indexPointer >= len) {
indexPointer = 0;
}
if (index[indexPointer] == null) {
break;
}
}
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15208326
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
Should be resolved with https://github.com/apache/hop/pull/6155. was able to create a test for the issue.
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15209160
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "dave-csc (via GitHub)" <gi...@apache.org>.
GitHub user dave-csc added a comment to the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
You most likely exceeded the buffer limit for a hop (10.000 rows by default), so your pipeline becomes deadlocked because it can't load all the necessary data for the lookup (i.e. the whole lookup query/table).
You may have better results by sorting the two streams and using a Merge join transform.
Hope this helps :)
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15207947
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
Yup. Reducing the number of row set size in your pipeline run configuration could also help.
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15207997
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
Hah, seems the previous code has the same potential issue... But because of the new "equalsByteArray" function it's just happening faster. Let me add some tests and harden that code... nice catch.
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15208477
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "EricTonissen (via GitHub)" <gi...@apache.org>.
GitHub user EricTonissen added a comment to the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
Yes, because this one has the code for correct fieldname if they are duplicate
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15208395
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
Are you running on a snapshot build?
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15208358
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "EricTonissen (via GitHub)" <gi...@apache.org>.
GitHub user EricTonissen added a comment to the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
And if I look in the StreamLookup.java, then the size of this array is 512 entries.
StreamLookup.java
if (data.hashIndex == null) {
data.hashIndex = new ByteArrayHashIndex(keyMeta);
}
Uses the contructor without size argument.
public ByteArrayHashIndex(IRowMeta keyRowMeta) {
this(keyRowMeta, 512);
}
factor2size will end with 512.
int factor2Size = 1;
while (factor2Size < size) {
factor2Size <<= 1; // Multiply by 2
}
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15208602
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "sramazzina (via GitHub)" <gi...@apache.org>.
GitHub user sramazzina added a comment to the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
We are experiencing the same issue here @hansva. Lot of our working pipelines get stuck because of this. We will pull the latest changes, make some tests during the day and let you know
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15215402
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "EricTonissen (via GitHub)" <gi...@apache.org>.
GitHub user EricTonissen added a comment to the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
Ok
<img width="1372" height="675" alt="image" src="https://github.com/user-attachments/assets/653d2f74-d9bb-4f54-96ef-23c59d5237b5" />
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15207941
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "EricTonissen (via GitHub)" <gi...@apache.org>.
GitHub user EricTonissen edited a discussion: Pipeline keeps running on simple 'Stream Lookup' action.
Anybody any idea why this Stream Lookup 'PATARTS' action doesn't continue?
<img width="306" height="236" alt="image" src="https://github.com/user-attachments/assets/f1552322-0004-4f12-8325-ae2e089920c1" />
AKKOORDJN output rows : 4990
QUERY PATARTS output rows : 12064
<img width="602" height="35" alt="image" src="https://github.com/user-attachments/assets/980c55d4-954a-4f1a-babb-5e3f5092306a" />
GitHub link: https://github.com/apache/hop/discussions/6152
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer edited a comment on the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
not with only a partial screenshot of your pipeline
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15207886
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
not with only a partial pipeline
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15207886
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "sramazzina (via GitHub)" <gi...@apache.org>.
GitHub user sramazzina added a comment to the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
@hansva we tested the change here and everything worked fine. Thank you for your fixing it
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15219129
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Pipeline keeps running on simple 'Stream Lookup' action. (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
GitHub user mattcasters added a comment to the discussion: Pipeline keeps running on simple 'Stream Lookup' action.
Eric, consider simplifying the pipeline by splitting it up into smaller parts. (De-)Serialize to disk can help or create a database table with the results in-between. See also: https://hop.apache.org//manual/latest/best-practices/index.html#_size_matters
GitHub link: https://github.com/apache/hop/discussions/6152#discussioncomment-15221331
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org