[I] [Feature Request]: PostgreSQL Bulk Loader needs commit size? (hop)

["ZamanGanji (via GitHub)" <gi...@apache.org>]

ZamanGanji opened a new issue, #6088:
URL: https://github.com/apache/hop/issues/6088

   ### What would you like to happen?
   
   We are using an MQTT broker to ingest sensor data into the new PostgreSQL database. The Apache HOP (2.9) pipeline runs continuously, and I monitor the incoming data by executing a query. However, I noticed that when the ETL pipeline is running, the newly ingested data is not visible. When the pipeline is stopped, the data appears. I resolved this issue by using the Table Output step instead of the PostgreSQL Bulk Loader, with a commit size of 10. It would be better if **PostgreSQL Bulk Loader** had a commit size. 
   
   ### Issue Priority
   
   Priority: 3
   
   ### Issue Component
   
   Component: Database


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature Request]: PostgreSQL Bulk Loader needs commit size? (hop)

["hansva (via GitHub)" <gi...@apache.org>]

hansva closed issue #6088: [Feature Request]: PostgreSQL Bulk Loader needs commit size?
URL: https://github.com/apache/hop/issues/6088


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature Request]: PostgreSQL Bulk Loader needs commit size? (hop)

["hansva (via GitHub)" <gi...@apache.org>]

hansva commented on issue #6088:
URL: https://github.com/apache/hop/issues/6088#issuecomment-3585654784

   Are you expecting a volume that needs a bulk loader?
   The bulk loader creates a file in the backend and then uses the PostgreSQL COPY command. The command is executed once the input stream finishes and the file is fully created. This is why you are not seeing any records in the DB.
   
   In theory, we could add a splitting option to execute the copy command and split to a new file at a certain "commit size".
   You could also add a pipeline executor to execute for every X rows and inside the pipeline executor use the Bulk loader. This pipeline will run once per batch of records that is received.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org