You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by "awesomeossdataengineer (via GitHub)" <gi...@apache.org> on 2026/09/15 05:49:20 UTC
[D] How to manipulate rowset size configuration at transform level to tackle JVM OOME (hop)
GitHub user awesomeossdataengineer created a discussion: How to manipulate rowset size configuration at transform level to tackle JVM OOME
Hello Apache Hop Community,
I would like to get few suggestions around tuning Hop pipelines so that my pipeline does not fail with JVM OOME.
**Scenario:**
Hop pipelines run with 'Native Local' pipeline runtime configuration with default rowset size = 10000
The source transform pumps 2 MB worth data for each page/row on high speed
The downstream transformations take time, meanwhile source transform keeps pumping next pages at high speed until Hop backpressure mechanism for 10k rowset is applied.
This behavior ideally leads to the pipeline from crashing with JVM OOME as it expects 2MB * 10000 = 20GB memory.
**Constraints:**
1. I cannot simply raise JVM memory to 20GB because of server limitations and other pipelines running on the same servers
2. I don't want to spin up new pipeline runtime configuration with lower rowset size, as it affects all the transforms in the pipeline, making the pipeline much slower.
**Question:**
**Is there a way to manage rowset size at a transform level so that I can apply back pressure on specific transforms?**
GitHub link: https://github.com/apache/hop/discussions/8378
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] How to manipulate rowset size configuration at transform level to tackle JVM OOME (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: How to manipulate rowset size configuration at transform level to tackle JVM OOME
You can't change rowset size at the transform level, but changing it at the pipeline level should have largely the same effect, just on all transforms.
Decreasing the rowset size will reduce your memory footprint in Apache Hop, but also means you'll need to do more round trips to the database.
GitHub link: https://github.com/apache/hop/discussions/8378#discussioncomment-18445154
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] How to manipulate rowset size configuration at transform level to tackle JVM OOME (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
GitHub user mattcasters added a comment to the discussion: How to manipulate rowset size configuration at transform level to tackle JVM OOME
If the rows are really that big, lower the fetch size in the JDBC driver options of you database connection. Otherwise that's the place where you'll run out of memory.
A lower row set size isn't really a bad thing. For high throughput scenarios it can force higher parallelism because it forces "lazy" transform copies to get in on the action. So lower it to 500 rows or so and set a fetch size of 250 or so.
GitHub link: https://github.com/apache/hop/discussions/8378#discussioncomment-18684877
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] How to manipulate rowset size configuration at transform level to tackle JVM OOME (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
GitHub user mattcasters edited a comment on the discussion: How to manipulate rowset size configuration at transform level to tackle JVM OOME
If the rows are really that big, lower the fetch size in the JDBC driver options of you database connection. Otherwise that's the place where you'll run out of memory.
A lower row set size isn't really a bad thing. For high throughput scenarios it can force higher parallelism because it forces "lazy" transform copies to get in on the action. So lower it to 500 rows or so and set a fetch size of 250 or so.
Oh, and if you're loading the rows in a database somewhere, don't set huge batch sizes either, as that's where memory is used as well, in the JDBC driver itself.
GitHub link: https://github.com/apache/hop/discussions/8378#discussioncomment-18684877
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] How to manipulate rowset size configuration at transform level to tackle JVM OOME (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: How to manipulate rowset size configuration at transform level to tackle JVM OOME
Do you need the full 2MB of data or are you immediately dropping a significant amount? If that is the case you could split off this part to a sub pipeline with a lower row set configuration.
GitHub link: https://github.com/apache/hop/discussions/8378#discussioncomment-18445386
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org