You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by FH <fh...@web.de> on 2021/11/16 21:21:23 UTC
problem with populating test case files
Dear all,
thank you all for your efforts on delivering HoP as an Apache project,
great work!
I have difficulties with a first unit test, however. Starting with a
simple pipeline (even simpler as in the example
https://hop.apache.org/manual/latest/pipeline/pipeline-unit-testing.html),
I managed to define the test case input and output files (they appear in
the metadata), but do not get them populated with data as described.
When _detaching_ the unit test, the pipeline runs again as expected and
the input/output counters go up (in the transform metrics). When
re-activating the unit test and starting the pipeline to "write rows to
data set", everything still looks alright, but the file does not get
filled with data (the time stamp gets updated, though) and all record
counters remain at 0. This is somewhat unexpected.
When adding a test result (golden set), Hop additionally complains on
the console that it cannot "find the sort field 'a' in transform" (which
is one column from my example input csv file), which may simply happen
because both (test input and output files) remain empty (apart from the
header line):
2021/11/16 21:44:01 - Hop - Pipeline opened.
2021/11/16 21:44:01 - Hop - Launching pipeline [first]...
2021/11/16 21:44:01 - Hop - Started the pipeline execution.
2021/11/16 21:44:01 - first - Executing this pipeline using the Local
Pipeline Engine with run configuration 'local'
2021/11/16 21:44:01 - first - Execution started for pipeline [first]
2021/11/16 21:44:01 - first - ERROR: Unable to validate unit test/golden
rows
2021/11/16 21:44:01 - first - ERROR:
org.apache.hop.core.exception.HopException:
2021/11/16 21:44:01 - first - Unable to find sort field 'a' in transform
results : []
2021/11/16 21:44:01 - first -
2021/11/16 21:44:01 - first - at
org.apache.hop.testing.util.DataSetConst.validateTransformResultAgainstUnitTest(DataSetConst.java:220)
2021/11/16 21:44:01 - first - at
org.apache.hop.testing.xp.ValidatePipelineUnitTestExtensionPoint.callExtensionPoint(ValidatePipelineUnitTestExtensionPoint.java:87)
2021/11/16 21:44:01 - first - at
org.apache.hop.testing.xp.ValidatePipelineUnitTestExtensionPoint.callExtensionPoint(ValidatePipelineUnitTestExtensionPoint.java:41)
2021/11/16 21:44:01 - first - at
org.apache.hop.core.extension.ExtensionPointMap.callExtensionPoint(ExtensionPointMap.java:149)
2021/11/16 21:44:01 - first - at
org.apache.hop.core.extension.ExtensionPointHandler.callExtensionPoint(ExtensionPointHandler.java:41)
2021/11/16 21:44:01 - first - at
org.apache.hop.pipeline.Pipeline.lambda$startThreads$1(Pipeline.java:1166)
2021/11/16 21:44:01 - first - at
org.apache.hop.pipeline.Pipeline.firePipelineExecutionFinishedListeners(Pipeline.java:1268)
2021/11/16 21:44:01 - first - at
org.apache.hop.pipeline.Pipeline.lambda$startThreads$0(Pipeline.java:1083)
2021/11/16 21:44:01 - first - at
org.apache.hop.pipeline.transform.BaseTransform.fireTransformFinishedListeners(BaseTransform.java:2729)
2021/11/16 21:44:01 - first - at
org.apache.hop.pipeline.transform.BaseTransform.markStop(BaseTransform.java:2718)
2021/11/16 21:44:01 - first - at
org.apache.hop.pipeline.transform.RunThread.run(RunThread.java:146)
2021/11/16 21:44:01 - first - at java.lang.Thread.run(Thread.java:748)
2021/11/16 21:44:01 - first - Pipeline duration : 0.097 seconds [ 0.097" ]
2021/11/16 21:44:01 - first - Execution finished on a local pipeline
engine with run configuration 'local'
What I am doing wrong? Any hint is appreciated.
All the best,
Frank
PS: Is there really no shortcut (like pressing Z when hovering over a
pipeline) to open the configuration dialog of a step? (I am constantly
trying to double click them as I get used to with kettle/pdi ;-) ).
Re: problem with populating test case files
Posted by FH <fh...@web.de>.
Dear all,
seems that I figured out what the problem was. The documentation
https://hop.apache.org/manual/latest/pipeline/pipeline-unit-testing.html
does not mention how to reply to the dialog that is attached as a
screenshot. After creating the datasets, this dialogs asks the user
whether the data set shall be used as input or golden set. Anticipating
the role of this data set in the final test case already, I made the
"correct decision" here and assigned one created file to the input and
the second to the golden test set, which makes them appear in the
pipeline as shown in the documenetation (a bit further down, right
before "RUN THE UNIT TEST"). But once this assignment has been done, the
"Write rows to data set" no longer works as expected (probably because
we are in 'test run mode' already). If I remove the assignment to
input/golden sets (and the labels disappear), the "Write rows to data
set" operates as expected. So the documentation should put some emphasis
on this dialog and advise the first time user to select "Cancel".
All the best,
Frank
On 16.11.21 22:21, FH wrote:
> Dear all,
>
> thank you all for your efforts on delivering HoP as an Apache project,
> great work!
>
> I have difficulties with a first unit test, however. Starting with a
> simple pipeline (even simpler as in the example
> https://hop.apache.org/manual/latest/pipeline/pipeline-unit-testing.html),
> I managed to define the test case input and output files (they appear in
> the metadata), but do not get them populated with data as described.
> When _detaching_ the unit test, the pipeline runs again as expected and
> the input/output counters go up (in the transform metrics). When
> re-activating the unit test and starting the pipeline to "write rows to
> data set", everything still looks alright, but the file does not get
> filled with data (the time stamp gets updated, though) and all record
> counters remain at 0. This is somewhat unexpected.
>
> When adding a test result (golden set), Hop additionally complains on
> the console that it cannot "find the sort field 'a' in transform" (which
> is one column from my example input csv file), which may simply happen
> because both (test input and output files) remain empty (apart from the
> header line):
>
> 2021/11/16 21:44:01 - Hop - Pipeline opened.
> 2021/11/16 21:44:01 - Hop - Launching pipeline [first]...
> 2021/11/16 21:44:01 - Hop - Started the pipeline execution.
> 2021/11/16 21:44:01 - first - Executing this pipeline using the Local
> Pipeline Engine with run configuration 'local'
> 2021/11/16 21:44:01 - first - Execution started for pipeline [first]
> 2021/11/16 21:44:01 - first - ERROR: Unable to validate unit test/golden
> rows
> 2021/11/16 21:44:01 - first - ERROR:
> org.apache.hop.core.exception.HopException:
> 2021/11/16 21:44:01 - first - Unable to find sort field 'a' in transform
> results : []
> 2021/11/16 21:44:01 - first -
> 2021/11/16 21:44:01 - first - at
> org.apache.hop.testing.util.DataSetConst.validateTransformResultAgainstUnitTest(DataSetConst.java:220)
>
> 2021/11/16 21:44:01 - first - at
> org.apache.hop.testing.xp.ValidatePipelineUnitTestExtensionPoint.callExtensionPoint(ValidatePipelineUnitTestExtensionPoint.java:87)
>
> 2021/11/16 21:44:01 - first - at
> org.apache.hop.testing.xp.ValidatePipelineUnitTestExtensionPoint.callExtensionPoint(ValidatePipelineUnitTestExtensionPoint.java:41)
>
> 2021/11/16 21:44:01 - first - at
> org.apache.hop.core.extension.ExtensionPointMap.callExtensionPoint(ExtensionPointMap.java:149)
>
> 2021/11/16 21:44:01 - first - at
> org.apache.hop.core.extension.ExtensionPointHandler.callExtensionPoint(ExtensionPointHandler.java:41)
>
> 2021/11/16 21:44:01 - first - at
> org.apache.hop.pipeline.Pipeline.lambda$startThreads$1(Pipeline.java:1166)
> 2021/11/16 21:44:01 - first - at
> org.apache.hop.pipeline.Pipeline.firePipelineExecutionFinishedListeners(Pipeline.java:1268)
>
> 2021/11/16 21:44:01 - first - at
> org.apache.hop.pipeline.Pipeline.lambda$startThreads$0(Pipeline.java:1083)
> 2021/11/16 21:44:01 - first - at
> org.apache.hop.pipeline.transform.BaseTransform.fireTransformFinishedListeners(BaseTransform.java:2729)
>
> 2021/11/16 21:44:01 - first - at
> org.apache.hop.pipeline.transform.BaseTransform.markStop(BaseTransform.java:2718)
>
> 2021/11/16 21:44:01 - first - at
> org.apache.hop.pipeline.transform.RunThread.run(RunThread.java:146)
> 2021/11/16 21:44:01 - first - at java.lang.Thread.run(Thread.java:748)
> 2021/11/16 21:44:01 - first - Pipeline duration : 0.097 seconds [ 0.097" ]
> 2021/11/16 21:44:01 - first - Execution finished on a local pipeline
> engine with run configuration 'local'
>
> What I am doing wrong? Any hint is appreciated.
>
> All the best,
> Frank
>
> PS: Is there really no shortcut (like pressing Z when hovering over a
> pipeline) to open the configuration dialog of a step? (I am constantly
> trying to double click them as I get used to with kettle/pdi ;-) ).