You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "Rookie625 (via GitHub)" <gi...@apache.org> on 2026/08/02 03:04:55 UTC
[I] [Bug]: [Stream Lookup] Proposal: Use threads + CountDownLatch to eliminate deadlocks (hop)
Rookie625 opened a new issue, #7752:
URL: https://github.com/apache/hop/issues/7752
### Apache Hop version?
2.18.0
### Java version?
21
### Operating system
Windows
### What happened?
### Problem Statement
The Stream Lookup transform is known to cause pipeline deadlocks when processing large datasets with split/rejoin streams [0][1]. The root cause is that when one input stream is blocked due to a full rowset buffer (e.g., after a Group By transform), Stream Lookup waits for that stream while the upstream transform cannot proceed, creating a cyclic dependency.
Current official solutions [2] require users to manually adjust pipeline design (increase rowset size, separate streams, split pipelines, or use Blocking transforms). These are workarounds, not root-cause fixes.
### Proposed Solution
Use **separate threads** (regular threads or virtual threads, depending on the runtime environment) to read each info stream independently and concurrently, with a `CountDownLatch` to coordinate the main thread. This approach eliminates deadlocks because each stream is consumed in its own thread, preventing buffer contention between upstream transforms.
**Pseudo‑code (using regular threads for compatibility with Java 11):**
```java
private boolean loadInfoStream(IStream stream, CountDownLatch latch, List<Exception> errors) {
Thread loader = new Thread(() -> {
try {
IRowSet rowSet = findInputRowSet(stream.getTransformName());
Object[] row = getRowFrom(rowSet);
while (row != null && !isStopped()) {
// Add to cache (needs thread-safe collection)
data.cache.add(row);
row = getRowFrom(rowSet);
}
} catch (Exception e) {
errors.add(e);
} finally {
latch.countDown();
}
});
loader.setName("StreamLookup-Loader-" + stream.getTransformName());
loader.setUncaughtExceptionHandler((t, e) -> errors.add(e));
loader.start();
return true;
}
```
```java
// In init() or before processing rows:
CountDownLatch latch = new CountDownLatch(infoStreams.size());
List<Exception> errors = Collections.synchronizedList(new ArrayList<>());
for (IStream stream : infoStreams) {
loadInfoStream(stream, latch, errors);
}
latch.await(); // wait for all loading threads to finish
if (!errors.isEmpty()) {
throw new HopException("Failed to load lookup data", errors.get(0));
}
```
Key benefits:
No deadlock – each stream is read independently, no circular blocking.
Transparent to users – no pipeline redesign needed.
Compatible with Java 11 – regular threads work, virtual threads can be used as an optimisation if Java 21+ is available.
References
[0] https://hop.apache.org/manual/2.18.0/how-to-guides/avoiding-deadlocks.html
[1] https://github.com/apache/hop/issues/3740
[2] https://hop.incubator.apache.org/manual/2.18.0/pipeline/transforms/streamlookup.html
### My Understanding of Apache Hop
I have been working with Apache Hop for some time and have developed a solid understanding of its pipeline execution model — particularly how transforms exchange data via rowsets, how scheduling and blocking work, and where concurrency bottlenecks can arise. For example, I analysed the Stream Lookup deadlock issue and proposed a practical solution using virtual threads and CountDownLatch to read info streams concurrently, eliminating deadlocks without forcing users to redesign their pipelines. This demonstrates my ability to dive into Hop’s internals and propose viable fixes.
### What I Can Do for Your Team
- Design and optimise Hop pipelines for large-scale data processing
- Develop custom transforms and plugins to extend Hop’s capabilities
- Debug and resolve performance issues, including deadlocks and memory bottlenecks
- Write clean, maintainable Java code that fits Hop’s architecture
- Collaborate with cross-functional teams and contribute to open-source projects
### What I Am Looking For
I am actively seeking a **paid position** (full-time or long-term contract) and I am **open to both remote and on-site opportunities**. I would love to apply my skills to real-world data engineering challenges — whether that’s building Hop-based ETL solutions, improving the Hop codebase, or integrating Hop with other data platforms. I am eager to work with an experienced team, learn from senior developers, and deliver tangible value from day one. I am flexible, open to feedback, and ready to contribute.
### A Note on My Mindset
I do not claim to be an expert yet, but I am highly motivated and have already proven that I can understand and improve complex parts of the system. I am confident that I can grow quickly and become a productive member of any Hop-focused engineering team. I would be grateful for any opportunity or referral — wherever the role is located.
### Issue Priority
Priority: 2
### Issue Component
Component: Transforms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: [Stream Lookup] Proposal: Use threads + CountDownLatch to eliminate deadlocks (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #7752:
URL: https://github.com/apache/hop/issues/7752#issuecomment-5241009789
@Rookie625 Thanks for pushing back against this long-established pattern. Fortunately we now have advanced means of fixing and testing alternative patterns. I really appreciate the issues you created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: [Stream Lookup] Proposal: Use threads + CountDownLatch to eliminate deadlocks (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters closed issue #7752: [Bug]: [Stream Lookup] Proposal: Use threads + CountDownLatch to eliminate deadlocks
URL: https://github.com/apache/hop/issues/7752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: [Stream Lookup] Proposal: Use threads + CountDownLatch to eliminate deadlocks (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #7752:
URL: https://github.com/apache/hop/issues/7752#issuecomment-5226601352
I'm also more in favour of detecting the situation and giving a clear error message (avoid the infinite lock). We have documented workarounds for this. Or we could even suggest inserting a transform to handle this (block).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: [Stream Lookup] Proposal: Use threads + CountDownLatch to eliminate deadlocks (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #7752:
URL: https://github.com/apache/hop/issues/7752#issuecomment-5227623315
I would appreciate an extra few pairs of eyes on #7854. I propose to give this some more mileage until 2.20.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: [Stream Lookup] Proposal: Use threads + CountDownLatch to eliminate deadlocks (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #7752:
URL: https://github.com/apache/hop/issues/7752#issuecomment-5226468243
It's not really a bug but a documented behavior with circular dependencies.
Loading all data into memory is perhaps less than ideal in all scenario. There's a reason why we don't have infinite buffers between transforms and that is obviously to avoid running out of memory.
That being said, if we could detect the circular dependency we could opt to serialize rows to disk to get around the limitation as a generic solution. I guess that shifts the problem to detecting that there is a circular dependency at play.
In any case, my preference would be to avoid starting extra threads and possibly causing out of memory errors and to favor a more generic solution for the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: [Stream Lookup] Proposal: Use threads + CountDownLatch to eliminate deadlocks (hop)
Posted by "Rookie625 (via GitHub)" <gi...@apache.org>.
Rookie625 commented on issue #7752:
URL: https://github.com/apache/hop/issues/7752#issuecomment-5234790367
> It's not really a bug but a documented behavior with circular dependencies. Loading all data into memory is perhaps less than ideal in all scenario. There's a reason why we don't have infinite buffers between transforms and that is obviously to avoid running out of memory.
>
> That being said, if we could detect the circular dependency we could opt to serialize rows to disk to get around the limitation as a generic solution. I guess that shifts the problem to detecting that there is a circular dependency at play.
>
> In any case, my preference would be to avoid starting extra threads and possibly causing out of memory errors and to favor a more generic solution for the problem.
Thank you for taking the time to provide such a thorough and candid explanation.
I completely see your point. Looking at this from the engine's perspective, it's absolutely not a bug—it's a conscious trade-off between performance and stability. The decision to avoid infinite buffers is precisely what prevents the engine from collapsing under memory pressure, and that foundation is something I fully respect.
My initial observation came from a different angle—more from the user experience side, where encountering a silent hang can be confusing. But I now realize that what feels like a "limitation" from the outside is actually a deliberate architectural safeguard from the inside. It's a classic case of two valid perspectives looking at the same problem.
Your idea about detecting circular dependencies and spilling to disk as a generic solution is particularly insightful. It strikes me as an elegant long-term direction that doesn't compromise the core principle of memory safety. I also appreciate your preference for avoiding extra threads that might invite OOM errors—it reinforces that stability should always come first.
This exchange has given me a much deeper appreciation for the design choices behind the scenes. Sometimes the most robust systems are the ones that refuse to compromise on their fundamental constraints, even if it means occasionally surprising the user.
Thanks again for sharing your thoughts—I've learned a lot from this discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org