You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by "chiragvats18 (via GitHub)" <gi...@apache.org> on 2026/03/20 12:35:01 UTC
[D] [Task]: Time out issue taking time to complete job (hop)
GitHub user chiragvats18 created a discussion: [Task]: Time out issue taking time to complete job
### What needs to happen?
Can anyone help as i was trying to make upsert job by pipeline by taking table input and doing upsert the data is about 2lakh something so it was taking time to get data as we have setup the timeout so it is timeout connection so is there any other way to build the job easily and fast without any error.
### Issue Priority
Priority: 2
### Issue Component
Component: Pipelines
GitHub link: https://github.com/apache/hop/discussions/6841
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] [Task]: Time out issue taking time to complete job (hop)
Posted by "chiragvats18 (via GitHub)" <gi...@apache.org>.
GitHub user chiragvats18 added a comment to the discussion: [Task]: Time out issue taking time to complete job
Dear Sir,
Is there any option to use Bulk API(like table bulk input / salesforce bulk upsert/salesforce bulk api setting on for salesforce upsert) because apache hop is currently using i think so soap api but we are unable to see bulk api setting on for this if we have to use any plugin or like that
as it has run as i have reduced batched size and increased number of copies.
GitHub link: https://github.com/apache/hop/discussions/6841#discussioncomment-16376013
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] [Task]: Time out issue taking time to complete job (hop)
Posted by "dmainou (via GitHub)" <gi...@apache.org>.
GitHub user dmainou added a comment to the discussion: [Task]: Time out issue taking time to complete job
Further, if the target table is large you may want to implement logic that reads, merges and inserts new rows followed by logic that reads merges and updates existing rows.
If the target table is humongous then you may want to avoid the upsert transform and use logic that puts the onus on the database. E.g load your rows to a table and execute a script that merges and updates inside the db.
GitHub link: https://github.com/apache/hop/discussions/6841#discussioncomment-16235373
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] [Task]: Time out issue taking time to complete job (hop)
Posted by "chiragvats18 (via GitHub)" <gi...@apache.org>.
GitHub user chiragvats18 added a comment to the discussion: [Task]: Time out issue taking time to complete job
We are not putting this in table we are taking data from table and data is bulk as an input and doing upsert to the Salesforce i have also tried by increase number of rows but buffer input and buffer output becomes fill either we have to increase buffer size but unable to find the location how to change it or if any other solution like bulk input from the table then which component should we can use?
GitHub link: https://github.com/apache/hop/discussions/6841#discussioncomment-16267157
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] [Task]: Time out issue taking time to complete job (hop)
Posted by "chiragvats18 (via GitHub)" <gi...@apache.org>.
GitHub user chiragvats18 added a comment to the discussion: [Task]: Time out issue taking time to complete job
Dear Sir,
Is there any option to use Bulk API(like table bulk input / salesforce bulk upsert/salesforce bulk api setting on for salesforce upsert) because apache hop is currently using i think so soap api but we are unable to see bulk api setting on for this if we have to use any plugin or like that
as it has run as i have reduced batched size and increased number of copies.
GitHub link: https://github.com/apache/hop/discussions/6841#discussioncomment-16376017
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] [Task]: Time out issue taking time to complete job (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: [Task]: Time out issue taking time to complete job
I don't know if I understand this correctly. If your database connection times out, there usually are JDBC properties to control timeouts etc. If your upsert is slow: check your indexes.
GitHub link: https://github.com/apache/hop/discussions/6841#discussioncomment-16225658
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] [Task]: Time out issue taking time to complete job (hop)
Posted by "dmainou (via GitHub)" <gi...@apache.org>.
GitHub user dmainou added a comment to the discussion: [Task]: Time out issue taking time to complete job
200k rows is very very little so there is some setting you need to change or Salesforce is throttling you.
GitHub link: https://github.com/apache/hop/discussions/6841#discussioncomment-16283718
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] [Task]: Time out issue taking time to complete job (hop)
Posted by "chiragvats18 (via GitHub)" <gi...@apache.org>.
GitHub user chiragvats18 added a comment to the discussion: [Task]: Time out issue taking time to complete job
Dear Sir,
Is there any option to use Bulk API(like table bulk input / salesforce bulk
upsert/salesforce bulk api setting on for salesforce upsert)
because apache hop is currently using i think so soap api but we are
unable to see bulk api setting on for this if we have to use any plugin or
like that
as it has run as i have reduced batched size and increased number of
copies.
CHIRAG VATS
8076590813
On Tue, Mar 24, 2026 at 4:38 AM Diego Mainou ***@***.***>
wrote:
> So break it down into smaller pieces.
> output your results to a table.
> Then use a different job to process on line at a time.
> It sounds like Salesforce is getting overwhelmed so send less data in one
> go.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/hop/discussions/6841?email_source=notifications&email_token=BWJJNT3V7ICRXBC5CO4GI2T4SG7VNA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNRSHAZTOMBSUZZGKYLTN5XKMYLVORUG64VFMV3GK3TUVRTG633UMVZF6Y3MNFRWW#discussioncomment-16283702>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BWJJNT6BEF44OST33OD5S2D4SG7VNAVCNFSM6AAAAACW3IAITSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTMMRYGM3TAMQ>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
GitHub link: https://github.com/apache/hop/discussions/6841#discussioncomment-16352845
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] [Task]: Time out issue taking time to complete job (hop)
Posted by "dmainou (via GitHub)" <gi...@apache.org>.
GitHub user dmainou added a comment to the discussion: [Task]: Time out issue taking time to complete job
So break it down into smaller pieces.
output your results to a table.
Then use a different job to process on line at a time.
It sounds like Salesforce is getting overwhelmed so send less data in one go.
GitHub link: https://github.com/apache/hop/discussions/6841#discussioncomment-16283702
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] [Task]: Time out issue taking time to complete job (hop)
Posted by "chiragvats18 (via GitHub)" <gi...@apache.org>.
GitHub user chiragvats18 deleted a comment on the discussion: [Task]: Time out issue taking time to complete job
Dear Sir,
Is there any option to use Bulk API(like table bulk input / salesforce bulk upsert/salesforce bulk api setting on for salesforce upsert) because apache hop is currently using i think so soap api but we are unable to see bulk api setting on for this if we have to use any plugin or like that
as it has run as i have reduced batched size and increased number of copies.
GitHub link: https://github.com/apache/hop/discussions/6841#discussioncomment-16376013
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] [Task]: Time out issue taking time to complete job (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: [Task]: Time out issue taking time to complete job
Bulk API for Salesforce is not (yet) available.
GitHub link: https://github.com/apache/hop/discussions/6841#discussioncomment-16376296
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org