You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@hop.apache.org by "ossDataEngineer (via GitHub)" <gi...@apache.org> on 2026/02/12 21:08:18 UTC

[D] How to log current batch number and increment it in pipeline executor when using row grouping? (hop)

GitHub user ossDataEngineer created a discussion: How to log current batch number and increment it in pipeline executor when using row grouping?

Hello Hop Community,

I have a usecase where source system gives me records in huge batches. The target system accepts smaller batch size, so I need to chunk src records into smaller batches with the chunk/batch size = max allowed by target system.

Eg. SRC gives 1k records, TGT accepts 200 records, so I need to create 5 batches as folllowing

tgt_batch_1.json, tgt_batch_2.json, ....etc.

I was able to create smaller batches of the data using pipeline executor row grouping as explained in `samples/loops/child-generate-1000-rows.hpl` but now I want to maintain counter variable so I can keep track of the iteration.

How can I do this?



GitHub link: https://github.com/apache/hop/discussions/6566

----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org

Re: [D] How to log current batch number and increment it in pipeline executor when using row grouping? (hop)

Posted by "dave-csc (via GitHub)" <gi...@apache.org>.

GitHub user dave-csc added a comment to the discussion: How to log current batch number and increment it in pipeline executor when using row grouping?

Simply using Get variables and Set variables in the pipeline 3 should do the trick:

- use **Get variables** at the start of your pipeline, and retrieve both variables FAILED_RECORD_COUNT and SUCCESS_RECORD_COUNT in some field
- in the success path, [be sure to reduce the stream to a single row](https://hop.apache.org/manual/latest/pipeline/transforms/setvariable.html), then use a **Calculator** to increment the _field_ SUCCESS_RECORD_COUNT, and set the new value to the _variable_ with **Set variables**
- do the same in the failure path, using field and variable FAILED_RECORD_COUNT

Anyway, to detect if a pipeline had succeeded, I would also take a look at the [Execution results](https://hop.apache.org/manual/latest/pipeline/transforms/pipeline-executor.html#_execution_results_tab) in pipeline 2...

GitHub link: https://github.com/apache/hop/discussions/6566#discussioncomment-15856527

----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org

Re: [D] How to log current batch number and increment it in pipeline executor when using row grouping? (hop)

Posted by "ossDataEngineer (via GitHub)" <gi...@apache.org>.

GitHub user ossDataEngineer added a comment to the discussion: How to log current batch number and increment it in pipeline executor when using row grouping?

Thankyou so much @dave-csc for detailed solution, it helped. Appreciate your time on this!

GitHub link: https://github.com/apache/hop/discussions/6566#discussioncomment-15875059

----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org

Re: [D] How to log current batch number and increment it in pipeline executor when using row grouping? (hop)

Posted by "dave-csc (via GitHub)" <gi...@apache.org>.

GitHub user dave-csc added a comment to the discussion: How to log current batch number and increment it in pipeline executor when using row grouping?

Hi @ossDataEngineer,

I had a quick look of your pipelines/workflow.

First, you are not implementing the CURRENT_BATCH_COUNT as described in the [previous post](https://github.com/apache/hop/discussions/6566#discussioncomment-15792353). The value of the counter must be input from the child pipeline, not calculated into the grandchild one. If you need to use that as a "variable", it actually has to be passed as a parameter in the grandchild pipeline.

Regarding the SUCCESS_ and FAILED_RECORD_COUNT, the behaviour is coherent with the fact that you're not returning the variable to the workflow after setting it, so it is reset at each iteration. To get what you need, I would proceed like this:
- in the grandchild pipeline, add a **Group by** transform at the end of the success and the failure stream, to count the rows processed by each one
- join the two grouped streams with a **Join Rows (cartesian product)** _(don't worry, they should be a single row each...)_ and send the result to the output with **Copy rows to result**
- in the child pipeline, grab the Result rows to the next transform: you can either log directly every batch result with a **Write to log** on the _fields_ received from the grandchild, or place another **Group by** in between to calculate the cumulative sum of the success and failed number _fields_
- if you still need to have those values in some variables, use another **Group by** after logging in the child pipeline (take the sums of the success and failed fields, or the maximum values of the cumulative sums if you calculated those). Then you can finally use **Set variables**, and those can be used in the parent _workflow_

Hope this helps :)

GitHub link: https://github.com/apache/hop/discussions/6566#discussioncomment-15870140

----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org

Re: [D] How to log current batch number and increment it in pipeline executor when using row grouping? (hop)

Posted by "ossDataEngineer (via GitHub)" <gi...@apache.org>.

GitHub user ossDataEngineer edited a comment on the discussion: How to log current batch number and increment it in pipeline executor when using row grouping?

@dave-csc The problem I see is that every pipeline executor instance gets variable value = 0 which was set in the parent workflow. The state of variable is not updated across different pipeline executor instances.

For example:

Parent WF set SUCCESS_RECORD_COUNT=0 and FAILED_RECORD_COUNT=0,
Repeat Pipeline creates 5 instances of pipelines because of row-grouping
When the 3rd pipeline runs, all the Get Variables inside it get 0 value. So if the 

batch 1 had SUCCESS_RECORD_COUNT=150 and FAILED_RECORD_COUNT=50
batch 2 had SUCCESS_RECORD_COUNT=200 and FAILED_RECORD_COUNT=0
batch 3 had SUCCESS_RECORD_COUNT=150 and FAILED_RECORD_COUNT=50
batch 4 had SUCCESS_RECORD_COUNT=200 and FAILED_RECORD_COUNT=0
batch 5 had SUCCESS_RECORD_COUNT=70 and FAILED_RECORD_COUNT=130

The final count is always being set to last batch failed/success counts (i.e 70 and 130) and not adding across all batches.


GitHub link: https://github.com/apache/hop/discussions/6566#discussioncomment-15862981

----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org

Re: [D] How to log current batch number and increment it in pipeline executor when using row grouping? (hop)

Posted by "ossDataEngineer (via GitHub)" <gi...@apache.org>.

GitHub user ossDataEngineer added a comment to the discussion: How to log current batch number and increment it in pipeline executor when using row grouping?

@dave-csc The problem I see is that every pipeline executor instance gets variable value = 0 which was set in the parent workflow. The state of variable is not update across different pipeline executor instances.

For example:

Parent WF set SUCCESS_RECORD_COUNT=0 and FAILED_RECORD_COUNT=0,
Repeat Pipeline creates 5 instances of pipelines because of row-grouping
When the 3rd pipeline runs, all the Get Variables inside it get 0 value. So if the 
batch 1 had SUCCESS_RECORD_COUNT=150 and FAILED_RECORD_COUNT=50
batch 2 had SUCCESS_RECORD_COUNT=200 and FAILED_RECORD_COUNT=0
.
.
The final count is always being set to last batch failed/success counts and not adding across all batches

GitHub link: https://github.com/apache/hop/discussions/6566#discussioncomment-15862981

----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org

Re: [D] How to log current batch number and increment it in pipeline executor when using row grouping? (hop)

Posted by "ossDataEngineer (via GitHub)" <gi...@apache.org>.

GitHub user ossDataEngineer added a comment to the discussion: How to log current batch number and increment it in pipeline executor when using row grouping?

Thanks for the solution @dave-csc. I would like to additionally increment the counts of variables set in my parent workflow - FAILED_RECORD_COUNT and SUCCESS_RECORD_COUNT so that I can get wholistic idea about total failed/success records across all batches combined.

Any suggestions on how this could be achieved?

Thanks!

GitHub link: https://github.com/apache/hop/discussions/6566#discussioncomment-15852445

----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org

Re: [D] How to log current batch number and increment it in pipeline executor when using row grouping? (hop)

Posted by "dave-csc (via GitHub)" <gi...@apache.org>.

GitHub user dave-csc added a comment to the discussion: How to log current batch number and increment it in pipeline executor when using row grouping?

Your best bet would be using some transforms before entering the Pipeline Executor loop:
- **Add sequence**: use it to add a sequence number on each row, starting from zero
- **Calculator**: divide the sequence number by your batch size (200), floor it and force it to an Integer, then add 1 if necessary

The output of the Calculator will be your batch number. And you can use it to group the rows for the Pipeline Executor, too ;)

Hope this helps :)

GitHub link: https://github.com/apache/hop/discussions/6566#discussioncomment-15792353

----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org

Re: [D] How to log current batch number and increment it in pipeline executor when using row grouping? (hop)

Posted by "ossDataEngineer (via GitHub)" <gi...@apache.org>.

GitHub user ossDataEngineer added a comment to the discussion: How to log current batch number and increment it in pipeline executor when using row grouping?

[child_pipeline.hpl.txt](https://github.com/user-attachments/files/25423944/child_pipeline.hpl.txt)
[grandchild_pipeline.hpl.txt](https://github.com/user-attachments/files/25423945/grandchild_pipeline.hpl.txt)
[parent_workflow.hwf.txt](https://github.com/user-attachments/files/25423946/parent_workflow.hwf.txt)


GitHub link: https://github.com/apache/hop/discussions/6566#discussioncomment-15863096

----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org

Re: [D] How to log current batch number and increment it in pipeline executor when using row grouping? (hop)

Posted by "ossDataEngineer (via GitHub)" <gi...@apache.org>.

GitHub user ossDataEngineer edited a comment on the discussion: How to log current batch number and increment it in pipeline executor when using row grouping?

@dave-csc The problem I see is that every pipeline executor instance gets variable value = 0 which was set in the parent workflow. The state of variable is not update across different pipeline executor instances.

For example:

Parent WF set SUCCESS_RECORD_COUNT=0 and FAILED_RECORD_COUNT=0,
Repeat Pipeline creates 5 instances of pipelines because of row-grouping
When the 3rd pipeline runs, all the Get Variables inside it get 0 value. So if the 

batch 1 had SUCCESS_RECORD_COUNT=150 and FAILED_RECORD_COUNT=50
batch 2 had SUCCESS_RECORD_COUNT=200 and FAILED_RECORD_COUNT=0
batch 3 had SUCCESS_RECORD_COUNT=150 and FAILED_RECORD_COUNT=50
batch 4 had SUCCESS_RECORD_COUNT=200 and FAILED_RECORD_COUNT=0
batch 5 had SUCCESS_RECORD_COUNT=70 and FAILED_RECORD_COUNT=130

The final count is always being set to last batch failed/success counts (i.e 70 and 130) and not adding across all batches.


GitHub link: https://github.com/apache/hop/discussions/6566#discussioncomment-15862981

----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org