You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "pramodgsable (via GitHub)" <gi...@apache.org> on 2024/08/22 09:29:15 UTC
[I] [Bug]: Using Insert/Update transform is causing sudden burst in oracle memory usage(in my case 50 GB's). I doubt if there are any memory leakage on hop transform which is leading to this. (hop)
pramodgsable opened a new issue, #4234:
URL: https://github.com/apache/hop/issues/4234
### Apache Hop version?
2.8.0
### Java version?
11
### Operating system
macOS
### What happened?
I was using insert/update transform to insert data into our RDS database(Oracle 19.0). Input we have taken as 2 to 3 rows from the data grid transform and tried to insert it in RDS database table. It gave same column datatype cast errors something like unable to cast Long to BigInteger. Though it inserted the records but after few trial and error we suddenly found that out database storage utilization has increased way beyond the data we were inserting into the table.
### Issue Priority
Priority: 2
### Issue Component
Component: Database, Component: Transforms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Using Insert/Update transform is causing sudden burst in oracle memory usage(in my case 50 GB's). I doubt if there are any memory leakage on hop transform which is leading to this. (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #4234:
URL: https://github.com/apache/hop/issues/4234#issuecomment-2316127995
My guess... you have a deadlock on an insert and update at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Using Insert/Update transform is causing sudden burst in oracle memory usage(in my case 50 GB's). I doubt if there are any memory leakage on hop transform which is leading to this. (hop)
Posted by "pramodgsable (via GitHub)" <gi...@apache.org>.
pramodgsable commented on issue #4234:
URL: https://github.com/apache/hop/issues/4234#issuecomment-2312913889
<img width="457" alt="Screenshot 2024-08-27 at 9 01 21 PM" src="https://github.com/user-attachments/assets/b1676517-5da5-43e7-8b43-bf5debbc8a4d">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Using Insert/Update transform is causing sudden burst in oracle memory usage(in my case 50 GB's). I doubt if there are any memory leakage on hop transform which is leading to this. (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva closed issue #4234: [Bug]: Using Insert/Update transform is causing sudden burst in oracle memory usage(in my case 50 GB's). I doubt if there are any memory leakage on hop transform which is leading to this.
URL: https://github.com/apache/hop/issues/4234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Using Insert/Update transform is causing sudden burst in oracle memory usage(in my case 50 GB's). I doubt if there are any memory leakage on hop transform which is leading to this. (hop)
Posted by "pramodgsable (via GitHub)" <gi...@apache.org>.
pramodgsable commented on issue #4234:
URL: https://github.com/apache/hop/issues/4234#issuecomment-2315965879
Still there is an issue which I saw today insert/update state are holding connections causing long running queries. I'm damm sure its from hop only. Previously we were using the Pentaho Spoon and we migrated this perticular workflow and seeing this issue since then very often...
<img width="1263" alt="Screenshot 2024-08-28 at 2 48 18 PM" src="https://github.com/user-attachments/assets/cc1ae3cc-3e79-4662-9834-e3dc84a01275">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Using Insert/Update transform is causing sudden burst in oracle memory usage(in my case 50 GB's). I doubt if there are any memory leakage on hop transform which is leading to this. (hop)
Posted by "pramodgsable (via GitHub)" <gi...@apache.org>.
pramodgsable commented on issue #4234:
URL: https://github.com/apache/hop/issues/4234#issuecomment-2312912121
2024/08/27 06:59:00 - match_ind_candidates - Pipeline duration : 8822.215 seconds [ 2h 27' 2" ]
2024/08/27 06:59:00 - match_ind_candidates - ERROR: Error handling writing final pipeline state to location (non-fatal)
2024/08/27 06:59:00 - match_ind_candidates - ERROR: org.apache.hop.core.exception.HopException:
2024/08/27 06:59:00 - match_ind_candidates - Error storing execution data
2024/08/27 06:59:00 - match_ind_candidates - PRV_INDIVIDUAL_ID BigNumber : There was a data type error: the data type of java.lang.Long object [14218814] does not correspond to value meta [BigNumber] (through reference chain: org.apache.hop.execution.ExecutionData["rowsBinaryGzipBase64Encoded"])
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Using Insert/Update transform is causing sudden burst in oracle memory usage(in my case 50 GB's). I doubt if there are any memory leakage on hop transform which is leading to this. (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #4234:
URL: https://github.com/apache/hop/issues/4234#issuecomment-2313409052
> 2024/08/27 06:59:00 - match_ind_candidates - Pipeline duration : 8822.215 seconds [ 2h 27' 2" ] 2024/08/27 06:59:00 - match_ind_candidates - ERROR: Error handling writing final pipeline state to location (non-fatal) 2024/08/27 06:59:00 - match_ind_candidates - ERROR: org.apache.hop.core.exception.HopException: 2024/08/27 06:59:00 - match_ind_candidates - Error storing execution data 2024/08/27 06:59:00 - match_ind_candidates - PRV_INDIVIDUAL_ID BigNumber : There was a data type error: the data type of java.lang.Long object [14218814] does not correspond to value meta [BigNumber] (through reference chain: org.apache.hop.execution.ExecutionData["rowsBinaryGzipBase64Encoded"])
You can ignore that error, It comes from the execution information perspective and not from the interaction with the database. You will have to start debugging on Oracle side... what is taking so much space? Is Hop the only process interacting with this database? We generate "simple" insert into statements you should be able to track those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Using Insert/Update transform is causing sudden burst in oracle memory usage(in my case 50 GB's). I doubt if there are any memory leakage on hop transform which is leading to this. (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #4234:
URL: https://github.com/apache/hop/issues/4234#issuecomment-2379438372
Closing as this is probably a pipeline design error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Using Insert/Update transform is causing sudden burst in oracle memory usage(in my case 50 GB's). I doubt if there are any memory leakage on hop transform which is leading to this. (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on issue #4234:
URL: https://github.com/apache/hop/issues/4234#issuecomment-2304379818
"some column datatype cast errors" is not very specific. do you have an error message or stacktrace?
we'll also need some more detailed information and/or a reproduction path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Using Insert/Update transform is causing sudden burst in oracle memory usage(in my case 50 GB's). I doubt if there are any memory leakage on hop transform which is leading to this. (hop)
Posted by "pramodgsable (via GitHub)" <gi...@apache.org>.
pramodgsable commented on issue #4234:
URL: https://github.com/apache/hop/issues/4234#issuecomment-2322967439
I suspect the same, as the same pipeline is doing is executing multiple queries like select and update. As transforms in pipeline executes in parallel there is possibility that deadlock may occur.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org