You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "joaomalho (via GitHub)" <gi...@apache.org> on 2024/08/14 09:30:25 UTC
[I] [Bug]: (hop)
joaomalho opened a new issue, #4212:
URL: https://github.com/apache/hop/issues/4212
### Apache Hop version?
2.9.0
### Java version?
java 22.0.2 2024-07-16
### Operating system
Windows
### What happened?
When we have a data migration pipeline of any size open in Apache Hop GUI, if we go to \metadata\pipeline-run-configuration\local.json and change the values of "feedback_size" and "rowset_size" to a number greater than 1M, running the pipeline will not start, and no error is returned. The solution is to revert the values to a slightly lower number.
Additionally, it is observed that if we set a value greater than 10B (for example), it reverts to the standard values of 10k..
This impact in huge data migrations.
### Issue Priority
Priority: 3
### Issue Component
Component: Hop Config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: (hop)
Posted by "joaomalho (via GitHub)" <gi...@apache.org>.
joaomalho commented on issue #4212:
URL: https://github.com/apache/hop/issues/4212#issuecomment-2294368710
Understood, thats why the process keeps running even when is stopped in 10k.. it will process the remaining eventually i suppose (i didnt test waiting more the 15min).
If i understood the impact, the processor of the machine will have a huge impact on this timing no?
Thank you very much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #4212:
URL: https://github.com/apache/hop/issues/4212#issuecomment-2294195730
rowset_size does not indicate a max number of rows Apache Hop can process.
It's how many buffer lines each transform has. I probably would not go over 100K for that. That being said, it should be possible if you want to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva closed issue #4212: [Bug]:
URL: https://github.com/apache/hop/issues/4212
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: (hop)
Posted by "joaomalho (via GitHub)" <gi...@apache.org>.
joaomalho commented on issue #4212:
URL: https://github.com/apache/hop/issues/4212#issuecomment-2294340612
My apologies for the previous explanation, i wasn't clear. I have configured the settings in my project metadata as detailed below:
These are the current configurations of the project:
C:\hop\config\projects\project_name\metadata\pipeline-run-configuration\local.json
```
{
"engineRunConfiguration": {
"Local": {
"feedback_size": "10000000",
"sample_size": "100",
"sample_type_in_gui": "Last",
"wait_time": "20",
"rowset_size": "10000000",
"safe_mode": false,
"show_feedback": false,
"topo_sort": false,
"gather_metrics": false,
"transactional": false
}
},
"defaultSelection": true,
"configurationVariables": [],
"name": "local",
"description": "",
"executionInfoLocationName": "Runs your pipelines locally with the standard local Hop pipeline engine"
}
```
By default if i dont set my "rowset_size": "" the default rows are 10K:
After got this miss configuration i set "rowset_size": "10000000" (10M), and it works at least until 400k rows (then i stop it because im in test env.).
Although if i set a very large number in "rowset_size", which will be a reality in the future once our project will collect huge databases, assuming "rowset_size" : "100000000" (10OM) will also run:
But if i set it to 1B will not work, nether run the pipeline:
I know it is a huge amount of data, and i can arrange workarounds to manage it. However i'm concerned that this might be a miss configuration on my side, and i may need to deploy it on server machine if that's the case, or, if this is a limitation in apache hop i might need to consider splitting the pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #4212:
URL: https://github.com/apache/hop/issues/4212#issuecomment-2294360116
As I said before rowset_size does not mean you can only process that amount of rows...
you can process a billion rows with rowset_size value = 10K. Apache Hop is a micro-batch/streaming application. The row set size is the amount of rows each transform can have in its waiting buffer. We have a pushback mechanism, when a buffer is full it will tell the upstream transform to stop providing new rows temporarily. In contrast to other tools all transforms are running in parallel we do not do sequential processing so our data does not have to fit in memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on issue #4212:
URL: https://github.com/apache/hop/issues/4212#issuecomment-2292979352
1M is a pretty big amount of data to have as a rowset size. can you share your JVM (`-Xmx` etc) configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: (hop)
Posted by "joaomalho (via GitHub)" <gi...@apache.org>.
joaomalho commented on issue #4212:
URL: https://github.com/apache/hop/issues/4212#issuecomment-2293885167
I agree, and I anticipate encountering scenarios in our project that will involve handling large data sizes. I'm also considering various workarounds, such as using Python processes with CSV, binary, or Feather formats or even in java. However, these are merely temporary solutions to the planned architecture. Additionally, this solution will be deployed on a server machine.
As far i understood your request:
**hop-conf.sh:**
`# Settings for all OSses
#
if [ -z "$HOP_OPTIONS" ]; then
HOP_OPTIONS="-Xmx2048m"
fi`
The rest are attached.
[hop-config-copy.json](https://github.com/user-attachments/files/16639754/hop-config-copy.json)
Thank you in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org