You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "bamaer (via GitHub)" <gi...@apache.org> on 2026/10/19 14:48:36 UTC

[I] [Bug]: Native Spark engine silently skips every output branch except one (hop)

bamaer opened a new issue, #9020:
URL: https://github.com/apache/hop/issues/9020

   ### Apache Hop version?
   
   2.20.0-SNAPSHOT
   
   ### Java version?
   
   21
   
   ### Operating system
   
   Linux
   
   ### What happened?
   
   On the native Spark engine only one branch of a pipeline runs. Output transforms on the other branches (Text File Output, Table Output, ...) write nothing, and the run ends without an error.
   
   **Reproduce**
   
   Spark File Input → Filter Rows with a true and a false target → a Text File Output on each target, on `local[*]`. Only the false branch writes a file; the log ends with `Spark pipeline finished, result row count=<rows of that branch>`. The same happens with Switch / Case.
   
   **Cause**
   
   After building the Dataset graph the engine runs one `count()`, on the last leaf transform. Generic transforms are lazy `mapPartitions`, so the other leaves never run. Spark File Output and the lake table outputs are not affected: they write while the graph is built. Every branch that writes also recomputes the transforms before it.
   
   **Fix**
   
   Run an action on every leaf transform, and persist the output of transforms that several transforms read (Filter Rows / Switch / Case targets), so they run once.
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Pipelines


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Native Spark engine silently skips every output branch except one (hop)

Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva closed issue #9020: [Bug]: Native Spark engine silently skips every output branch except one
URL: https://github.com/apache/hop/issues/9020


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: Native Spark engine silently skips every output branch except one (hop)

Posted by "bamaer (via GitHub)" <gi...@apache.org>.
bamaer commented on issue #9020:
URL: https://github.com/apache/hop/issues/9020#issuecomment-6270098466

   .take-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org