[I] [Bug]: Blocking until transforms finish sometimes doesn't wait for the transform to finish (hop)

["dsanderbi (via GitHub)" <gi...@apache.org>]

dsanderbi opened a new issue, #9096:
URL: https://github.com/apache/hop/issues/9096

   ### Apache Hop version?
   
   2.19.0
   
   ### Java version?
   
   26.0.2
   
   ### Operating system
   
   Windows
   
   ### What happened?
   
   We use "Blocking until transforms finish" to run an SQL statement only after a Table output has written all its rows:
   
       Row generator (1 row) -> Blocking until transforms finish [waits for "write to table", copy 0] -> Execute SQL
       ... input rows -> "write to table" (Table output, commit size 1000)
   
   This works in almost every run, but in one of about 50 runs the blocking transform passed its row on right at the start of the pipeline, while the Table output was still running. 
   The SQL statement then ran at the same time as the Table output and failed with a primary key violation, because both inserted the same keys.
   
   Log of the failing run (basic log level):
   
       10:20:37 - wait for write to table.0 - Transform write to table status: Running
       10:20:37 - wait for write to table.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
       10:20:38 - (rows start arriving at "write to table")
       10:20:41 - Execute SQL.0 - ERROR: ... Violation of PRIMARY KEY constraint ...
       10:20:41 - write to table.0 - Finished processing (I=0, O=6000, R=6000, W=5000, U=0, E=0)
   
   The blocking transform finished at 10:20:37 and logged the status of "write to table" as "Running" at that moment. The Table output only received its rows after that and was stopped at 10:20:41 because of the error. In all other runs the blocking transform waited until the Table output had finished.
   
   We could not reproduce it on purpose. It looks like a timing issue at the start of the pipeline, when the waited-for transform is just switching from idle to running.
   
   **Expected**: the blocking transform always waits until the listed transform has finished.
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Transforms


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Blocking until transforms finish sometimes doesn't wait for the transform to finish (hop)

["hansva (via GitHub)" <gi...@apache.org>]

hansva commented on issue #9096:
URL: https://github.com/apache/hop/issues/9096#issuecomment-6419144745

   .take-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org