You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hop.apache.org by ha...@apache.org on 2026/08/12 14:31:57 UTC
(hop) branch main updated: Improved Fuzzy Match return distances for multiple matches(Top-K/separate rows) (#7849)
This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new e96f1ad375 Improved Fuzzy Match return distances for multiple matches(Top-K/separate rows) (#7849)
e96f1ad375 is described below
commit e96f1ad3753b0b2b2808f080e8920a4b6d348e79
Author: Lance <le...@gmail.com>
AuthorDate: Wed Aug 12 22:31:47 2026 +0800
Improved Fuzzy Match return distances for multiple matches(Top-K/separate rows) (#7849)
Signed-off-by: lance <le...@gmail.com>
---
.../ROOT/pages/how-to-guides/joins-lookups.adoc | 4 +-
.../ROOT/pages/pipeline/transforms/fuzzymatch.adoc | 63 ++-
...zzymatch.hpl => 0012-fuzzymatch-all-concat.hpl} | 248 ++++--------
...fuzzymatch.hpl => 0012-fuzzymatch-all-rows.hpl} | 250 ++++--------
...-fuzzymatch.hpl => 0012-fuzzymatch-closest.hpl} | 248 ++++--------
integration-tests/transforms/0012-fuzzymatch.hpl | 2 +
.../datasets/golden-fuzzymatch-all-concat.csv | 3 +
.../datasets/golden-fuzzymatch-all-rows.csv | 4 +
.../datasets/golden-fuzzymatch-closest.csv | 3 +
.../transforms/main-0012-fuzzymatch.hwf | 11 +-
.../dataset/golden-fuzzymatch-all-concat.json | 40 ++
.../dataset/golden-fuzzymatch-all-rows.json | 48 +++
.../dataset/golden-fuzzymatch-closest.json | 48 +++
.../unit-test/0012-fuzzymatch-all-concat UNIT.json | 40 ++
.../unit-test/0012-fuzzymatch-all-rows UNIT.json | 45 +++
.../unit-test/0012-fuzzymatch-closest UNIT.json | 44 +++
.../pipeline/transforms/fuzzymatch/FuzzyMatch.java | 355 +++++++++--------
.../transforms/fuzzymatch/FuzzyMatchData.java | 7 +-
.../transforms/fuzzymatch/FuzzyMatchDialog.java | 131 ++++---
.../transforms/fuzzymatch/FuzzyMatchMeta.java | 149 +++++--
.../fuzzymatch/LetterPairSimilarity.java | 9 +-
.../fuzzymatch/messages/messages_en_US.properties | 16 +-
.../fuzzymatch/messages/messages_zh_CN.properties | 14 +-
.../transforms/fuzzymatch/FuzzyMatchDataTest.java | 42 ++
.../transforms/fuzzymatch/FuzzyMatchMetaTest.java | 303 ++++++++++++++-
.../transforms/fuzzymatch/FuzzyMatchTest.java | 431 +++++++++++++++++++--
.../fuzzymatch/LetterPairSimilarityTest.java | 58 +++
.../src/test/resources/fuzzy-match-transform.xml | 2 +
28 files changed, 1819 insertions(+), 799 deletions(-)
diff --git a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/joins-lookups.adoc b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/joins-lookups.adoc
index e9a95fd039..245811730d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/joins-lookups.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/joins-lookups.adoc
@@ -84,7 +84,9 @@ allows you to run a query against a database using data obtained from previous t
Lookups are a way to let you enrich your pipeline's stream with additional fields from external sources. The transforms allow you to choose which additional fields will be retrieved from the lookup source.
-Contrary to joins, lookups work on the current pipeline stream and your never change the number of rows you're processing.
+Contrary to joins, most lookups work on the current pipeline stream and never change the number of rows you're processing.
+
+NOTE: xref:pipeline/transforms/fuzzymatch.adoc[Fuzzy match] is an exception when *Match mode* is set to *All matches (separate rows)*: one main stream row can produce multiple output rows (Top-K).
There are a number of transforms in the `Lookup` category that are considered self-explanatory in a lookup context and won't be included here. These transforms are:
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fuzzymatch.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fuzzymatch.adoc
index 76f19e414b..cb97f7f3ae 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fuzzymatch.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/fuzzymatch.adoc
@@ -27,7 +27,8 @@ under the License.
The Fuzzy Match transform finds strings that potentially match using duplicate-detecting algorithms that calculate the similarity of two streams of data.
-This transform returns matching values as a separated list as specified by user-defined minimal or maximal values.
+For each row on the main stream, the transform compares the main stream field against values from the lookup stream and returns matches within the configured score range.
+Depending on the selected match mode, it can return the single best match, multiple matches as separate output rows (Top-K), or multiple matches as a concatenated list.
|
== Supported Engines
@@ -55,13 +56,35 @@ This transform returns matching values as a separated list as specified by user-
|Main stream field|Identifies the primary stream to match the Lookup field with
|Algorithm|Identifies which string-matching algorithm to use---options include Levenshtein, Damerau-Levenshtein, Needleman Wunsch, Jaro, Jaro Winkler, Pair letters similarity, Metaphone, Double Metaphone, SoundEx, or Refined SoundEx
|Case sensitive|Identifies if streams can or cannot differ based on the use of uppercase and lowercase letters---only for use with the Levenshtein algorithms
-|Get closer value|When checked, returns a single result with the highest similarity score---when unchecked, returns all matches that satisfy the minimal and maximal value setting as a separated list, separated by the values separator
-|Minimum value|Identifies the lowest possible similarity score
-|Maximal value|Identifies the highest possible similarity score
-|Values separator|Identifies the string that separate the matches.
-Only available for specific algorithms and when the Get closer value option is unchecked.
+|Match mode|Controls how matches are returned (see <<match-mode>> below)
+|Minimum value|Identifies the lowest possible similarity score or distance that is accepted as a match
+|Maximal value|Identifies the highest possible similarity score or distance that is accepted as a match
+|Max matches|Top-K limit for multi-match modes.
+Only the best matches (by distance or similarity) are kept.
+The default is `10`.
+Values are hard-capped at `100` to avoid runaway memory and row explosion.
+Only available when Match mode is *All matches (separate rows)* or *All matches (concatenated)*.
+|Values separator|Identifies the string that separates matches (and measure values) when Match mode is *All matches (concatenated)*.
|===
+[[match-mode]]
+*Match mode*
+
+|===
+|Mode|Description
+|Closest match|Returns a single result: the best match for the current main stream value.
+For distance algorithms (Levenshtein, Damerau-Levenshtein, Needleman Wunsch), the best match has the *smallest* distance.
+For similarity algorithms (Jaro, Jaro Winkler, Pair letters similarity), the best match has the *highest* similarity.
+|All matches (separate rows)|Returns one output row per match, each with its own match value and measure value.
+Matches are ranked (best first) and limited by *Max matches* (Top-K).
+This is the recommended way to retrieve multiple matches together with their distances or similarity scores.
+|All matches (concatenated)|Returns a single output row where all matches are joined with the *Values separator*.
+When a Value field is configured, the corresponding measure values are joined in the same order with the same separator (for example `A,B,C` with measures `1,2,3`).
+Results are also limited by *Max matches* (Top-K).
+|===
+
+TIP: Existing pipelines that used the former *Get closer value* checkbox map as follows: checked → *Closest match*; unchecked → *All matches (concatenated)*.
+
*Algorithm Definitions*
Within the Algorithm field, there are several options available to compare and match strings.
@@ -97,8 +120,30 @@ The Fields tab enables you to define how to return the results of a comparison.
[options="header"]
|===
|Option|Description
-|Match field|Defines the name of the column that contains the comparison value
-|Value field|Defines the similarity score for which to return a value
+|Match field|Defines the name of the column that contains the comparison value (the matched lookup string).
+In *All matches (concatenated)* mode this can contain multiple values joined by the values separator.
+|Value field|Defines the name of the column that contains the measure for the match.
+The measure type depends on the algorithm and match mode:
+
+* Distance algorithms (Levenshtein, Damerau-Levenshtein, Needleman Wunsch): integer distance for *Closest match* and *All matches (separate rows)*; a separator-joined string of distances for *All matches (concatenated)*.
+* Similarity algorithms (Jaro, Jaro Winkler, Pair letters similarity): numeric similarity for *Closest match* and *All matches (separate rows)*; a separator-joined string of scores for *All matches (concatenated)*.
+* Phonetic algorithms: the encoded phonetic value as a string.
+|===
+
+You can also specify the list of additional fields to retrieve from the lookup stream.
+
+* Additional fields are available for *Closest match* and *All matches (separate rows)* (one lookup field value per output row).
+* They are also available for phonetic algorithms.
+* They are not returned in *All matches (concatenated)* mode for distance and similarity algorithms, because multiple matches are collapsed into a single string cell.
+
+== Output behavior
+
+[options="header"]
+|===
+|Match mode|Output shape|Measure (Value field)
+|Closest match|One output row per main stream row|Single typed measure (integer / number / string)
+|All matches (separate rows)|Up to *Max matches* output rows per main stream row (one per match), ranked best-first|One typed measure per output row
+|All matches (concatenated)|One output row per main stream row|Separator-joined string of measures, aligned with the match list
|===
-You can also specify the list of additional fields to retrieve from the lookup stream.
\ No newline at end of file
+If no lookup value falls within the configured minimum and maximal range, the transform still forwards the main stream row with empty match and measure fields.
diff --git a/integration-tests/transforms/0012-fuzzymatch.hpl b/integration-tests/transforms/0012-fuzzymatch-all-concat.hpl
similarity index 50%
copy from integration-tests/transforms/0012-fuzzymatch.hpl
copy to integration-tests/transforms/0012-fuzzymatch-all-concat.hpl
index bafb0af4f3..5117927004 100644
--- a/integration-tests/transforms/0012-fuzzymatch.hpl
+++ b/integration-tests/transforms/0012-fuzzymatch-all-concat.hpl
@@ -19,9 +19,9 @@ limitations under the License.
-->
<pipeline>
<info>
- <name>0012-fuzzymatch</name>
+ <name>0012-fuzzymatch-all-concat</name>
<name_sync_with_filename>Y</name_sync_with_filename>
- <description/>
+ <description>Fuzzy match All matches (concatenated) Top-K with joined distances</description>
<extended_description/>
<pipeline_version/>
<pipeline_type>Normal</pipeline_type>
@@ -31,31 +31,31 @@ limitations under the License.
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
<created_user>-</created_user>
- <created_date>2021/04/19 12:06:20.350</created_date>
+ <created_date>2026/03/08 12:00:00.000</created_date>
<modified_user>-</modified_user>
- <modified_date>2021/04/19 12:06:20.350</modified_date>
+ <modified_date>2026/03/08 12:00:00.000</modified_date>
</info>
<notepads>
</notepads>
<order>
<hop>
- <from>Fuzzy match</from>
- <to>Output</to>
+ <from>Main</from>
+ <to>Fuzzy match</to>
<enabled>Y</enabled>
</hop>
<hop>
- <from>files/customers-100.txt</from>
+ <from>Lookup</from>
<to>Fuzzy match</to>
<enabled>Y</enabled>
</hop>
<hop>
- <from>Data grid</from>
- <to>Fuzzy match</to>
+ <from>Fuzzy match</from>
+ <to>Output</to>
<enabled>Y</enabled>
</hop>
</order>
<transform>
- <name>Data grid</name>
+ <name>Main</name>
<type>DataGrid</type>
<description/>
<distribute>Y</distribute>
@@ -67,13 +67,22 @@ limitations under the License.
</partitioning>
<data>
<line>
- <item>jwcdf-name</item>
+ <item>1</item>
+ <item>Catrine</item>
</line>
<line>
- <item>rdjin-name</item>
+ <item>2</item>
+ <item>Zzzz</item>
</line>
</data>
<fields>
+ <field>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ <name>id</name>
+ <type>Integer</type>
+ </field>
<field>
<length>-1</length>
<precision>-1</precision>
@@ -84,13 +93,13 @@ limitations under the License.
</fields>
<attributes/>
<GUI>
- <xloc>303</xloc>
- <yloc>162</yloc>
+ <xloc>96</xloc>
+ <yloc>64</yloc>
</GUI>
</transform>
<transform>
- <name>Fuzzy match</name>
- <type>FuzzyMatch</type>
+ <name>Lookup</name>
+ <type>DataGrid</type>
<description/>
<distribute>Y</distribute>
<custom_distribution/>
@@ -99,32 +108,49 @@ limitations under the License.
<method>none</method>
<schema_name/>
</partitioning>
- <from>Data grid</from>
- <lookupfield>name</lookupfield>
- <mainstreamfield>name</mainstreamfield>
- <outputmatchfield>match</outputmatchfield>
- <outputvaluefield>measure value</outputvaluefield>
- <caseSensitive>N</caseSensitive>
- <closervalue>Y</closervalue>
- <minimalValue>0</minimalValue>
- <maximalValue>1</maximalValue>
- <separator>,</separator>
- <algorithm>soundex</algorithm>
- <lookup>
- <value>
+ <data>
+ <line>
+ <item>Catrine</item>
+ <item>A</item>
+ </line>
+ <line>
+ <item>Catriny</item>
+ <item>B</item>
+ </line>
+ <line>
+ <item>Catherine</item>
+ <item>C</item>
+ </line>
+ <line>
+ <item>John</item>
+ <item>D</item>
+ </line>
+ </data>
+ <fields>
+ <field>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
<name>name</name>
- <rename>lookupName</rename>
- </value>
- </lookup>
+ <type>String</type>
+ </field>
+ <field>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ <name>code</name>
+ <type>String</type>
+ </field>
+ </fields>
<attributes/>
<GUI>
- <xloc>304</xloc>
- <yloc>64</yloc>
+ <xloc>96</xloc>
+ <yloc>192</yloc>
</GUI>
</transform>
<transform>
- <name>Output</name>
- <type>Dummy</type>
+ <name>Fuzzy match</name>
+ <type>FuzzyMatch</type>
<description/>
<distribute>Y</distribute>
<custom_distribution/>
@@ -133,15 +159,28 @@ limitations under the License.
<method>none</method>
<schema_name/>
</partitioning>
+ <from>Lookup</from>
+ <lookupfield>name</lookupfield>
+ <mainstreamfield>name</mainstreamfield>
+ <outputmatchfield>match</outputmatchfield>
+ <outputvaluefield>measure value</outputvaluefield>
+ <caseSensitive>N</caseSensitive>
+ <closervalue>N</closervalue>
+ <matchMode>all_concat</matchMode>
+ <maxMatches>2</maxMatches>
+ <minimalValue>0</minimalValue>
+ <maximalValue>2</maximalValue>
+ <separator>|</separator>
+ <algorithm>levenshtein</algorithm>
<attributes/>
<GUI>
- <xloc>528</xloc>
- <yloc>64</yloc>
+ <xloc>304</xloc>
+ <yloc>128</yloc>
</GUI>
</transform>
<transform>
- <name>files/customers-100.txt</name>
- <type>CSVInput</type>
+ <name>Output</name>
+ <type>Dummy</type>
<description/>
<distribute>Y</distribute>
<custom_distribution/>
@@ -150,135 +189,10 @@ limitations under the License.
<method>none</method>
<schema_name/>
</partitioning>
- <filename>${PROJECT_HOME}/files/customers-100.txt</filename>
- <filename_field/>
- <rownum_field/>
- <include_filename>N</include_filename>
- <separator>;</separator>
- <enclosure>"</enclosure>
- <header>Y</header>
- <buffer_size>50000</buffer_size>
- <lazy_conversion>N</lazy_conversion>
- <add_filename_result>N</add_filename_result>
- <parallel>N</parallel>
- <newline_possible>N</newline_possible>
- <encoding/>
- <fields>
- <field>
- <name>id</name>
- <type>Integer</type>
- <format> #</format>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>15</length>
- <precision>0</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>name</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>50</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>firstname</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>50</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>zip</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>30</length>
- <precision>0</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>city</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>8</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>birthdate</name>
- <type>Date</type>
- <format>yyyy/MM/dd</format>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>-1</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>street</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>11</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>housenr</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>50</length>
- <precision>0</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>stateCode</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>10</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>state</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>50</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- </fields>
<attributes/>
<GUI>
- <xloc>112</xloc>
- <yloc>64</yloc>
+ <xloc>512</xloc>
+ <yloc>128</yloc>
</GUI>
</transform>
<transform_error_handling>
diff --git a/integration-tests/transforms/0012-fuzzymatch.hpl b/integration-tests/transforms/0012-fuzzymatch-all-rows.hpl
similarity index 51%
copy from integration-tests/transforms/0012-fuzzymatch.hpl
copy to integration-tests/transforms/0012-fuzzymatch-all-rows.hpl
index bafb0af4f3..c8101e9adb 100644
--- a/integration-tests/transforms/0012-fuzzymatch.hpl
+++ b/integration-tests/transforms/0012-fuzzymatch-all-rows.hpl
@@ -19,9 +19,9 @@ limitations under the License.
-->
<pipeline>
<info>
- <name>0012-fuzzymatch</name>
+ <name>0012-fuzzymatch-all-rows</name>
<name_sync_with_filename>Y</name_sync_with_filename>
- <description/>
+ <description>Fuzzy match All matches (separate rows) Top-K with Levenshtein distance</description>
<extended_description/>
<pipeline_version/>
<pipeline_type>Normal</pipeline_type>
@@ -31,31 +31,74 @@ limitations under the License.
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
<created_user>-</created_user>
- <created_date>2021/04/19 12:06:20.350</created_date>
+ <created_date>2026/03/08 12:00:00.000</created_date>
<modified_user>-</modified_user>
- <modified_date>2021/04/19 12:06:20.350</modified_date>
+ <modified_date>2026/03/08 12:00:00.000</modified_date>
</info>
<notepads>
</notepads>
<order>
<hop>
- <from>Fuzzy match</from>
- <to>Output</to>
+ <from>Main</from>
+ <to>Fuzzy match</to>
<enabled>Y</enabled>
</hop>
<hop>
- <from>files/customers-100.txt</from>
+ <from>Lookup</from>
<to>Fuzzy match</to>
<enabled>Y</enabled>
</hop>
<hop>
- <from>Data grid</from>
- <to>Fuzzy match</to>
+ <from>Fuzzy match</from>
+ <to>Output</to>
<enabled>Y</enabled>
</hop>
</order>
<transform>
- <name>Data grid</name>
+ <name>Main</name>
+ <type>DataGrid</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <data>
+ <line>
+ <item>1</item>
+ <item>Catrine</item>
+ </line>
+ <line>
+ <item>2</item>
+ <item>Zzzz</item>
+ </line>
+ </data>
+ <fields>
+ <field>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ <name>id</name>
+ <type>Integer</type>
+ </field>
+ <field>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ <name>name</name>
+ <type>String</type>
+ </field>
+ </fields>
+ <attributes/>
+ <GUI>
+ <xloc>96</xloc>
+ <yloc>64</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>Lookup</name>
<type>DataGrid</type>
<description/>
<distribute>Y</distribute>
@@ -67,10 +110,20 @@ limitations under the License.
</partitioning>
<data>
<line>
- <item>jwcdf-name</item>
+ <item>Catrine</item>
+ <item>A</item>
+ </line>
+ <line>
+ <item>Catriny</item>
+ <item>B</item>
</line>
<line>
- <item>rdjin-name</item>
+ <item>Catherine</item>
+ <item>C</item>
+ </line>
+ <line>
+ <item>John</item>
+ <item>D</item>
</line>
</data>
<fields>
@@ -81,11 +134,18 @@ limitations under the License.
<name>name</name>
<type>String</type>
</field>
+ <field>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ <name>code</name>
+ <type>String</type>
+ </field>
</fields>
<attributes/>
<GUI>
- <xloc>303</xloc>
- <yloc>162</yloc>
+ <xloc>96</xloc>
+ <yloc>192</yloc>
</GUI>
</transform>
<transform>
@@ -99,27 +159,29 @@ limitations under the License.
<method>none</method>
<schema_name/>
</partitioning>
- <from>Data grid</from>
+ <from>Lookup</from>
<lookupfield>name</lookupfield>
<mainstreamfield>name</mainstreamfield>
<outputmatchfield>match</outputmatchfield>
<outputvaluefield>measure value</outputvaluefield>
<caseSensitive>N</caseSensitive>
- <closervalue>Y</closervalue>
+ <closervalue>N</closervalue>
+ <matchMode>all_rows</matchMode>
+ <maxMatches>2</maxMatches>
<minimalValue>0</minimalValue>
- <maximalValue>1</maximalValue>
+ <maximalValue>2</maximalValue>
<separator>,</separator>
- <algorithm>soundex</algorithm>
+ <algorithm>levenshtein</algorithm>
<lookup>
<value>
- <name>name</name>
- <rename>lookupName</rename>
+ <name>code</name>
+ <rename/>
</value>
</lookup>
<attributes/>
<GUI>
<xloc>304</xloc>
- <yloc>64</yloc>
+ <yloc>128</yloc>
</GUI>
</transform>
<transform>
@@ -135,150 +197,8 @@ limitations under the License.
</partitioning>
<attributes/>
<GUI>
- <xloc>528</xloc>
- <yloc>64</yloc>
- </GUI>
- </transform>
- <transform>
- <name>files/customers-100.txt</name>
- <type>CSVInput</type>
- <description/>
- <distribute>Y</distribute>
- <custom_distribution/>
- <copies>1</copies>
- <partitioning>
- <method>none</method>
- <schema_name/>
- </partitioning>
- <filename>${PROJECT_HOME}/files/customers-100.txt</filename>
- <filename_field/>
- <rownum_field/>
- <include_filename>N</include_filename>
- <separator>;</separator>
- <enclosure>"</enclosure>
- <header>Y</header>
- <buffer_size>50000</buffer_size>
- <lazy_conversion>N</lazy_conversion>
- <add_filename_result>N</add_filename_result>
- <parallel>N</parallel>
- <newline_possible>N</newline_possible>
- <encoding/>
- <fields>
- <field>
- <name>id</name>
- <type>Integer</type>
- <format> #</format>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>15</length>
- <precision>0</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>name</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>50</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>firstname</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>50</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>zip</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>30</length>
- <precision>0</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>city</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>8</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>birthdate</name>
- <type>Date</type>
- <format>yyyy/MM/dd</format>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>-1</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>street</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>11</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>housenr</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>50</length>
- <precision>0</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>stateCode</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>10</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>state</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>50</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- </fields>
- <attributes/>
- <GUI>
- <xloc>112</xloc>
- <yloc>64</yloc>
+ <xloc>512</xloc>
+ <yloc>128</yloc>
</GUI>
</transform>
<transform_error_handling>
diff --git a/integration-tests/transforms/0012-fuzzymatch.hpl b/integration-tests/transforms/0012-fuzzymatch-closest.hpl
similarity index 52%
copy from integration-tests/transforms/0012-fuzzymatch.hpl
copy to integration-tests/transforms/0012-fuzzymatch-closest.hpl
index bafb0af4f3..51011f949e 100644
--- a/integration-tests/transforms/0012-fuzzymatch.hpl
+++ b/integration-tests/transforms/0012-fuzzymatch-closest.hpl
@@ -19,9 +19,9 @@ limitations under the License.
-->
<pipeline>
<info>
- <name>0012-fuzzymatch</name>
+ <name>0012-fuzzymatch-closest</name>
<name_sync_with_filename>Y</name_sync_with_filename>
- <description/>
+ <description>Fuzzy match Closest mode with Levenshtein distance and additional lookup fields</description>
<extended_description/>
<pipeline_version/>
<pipeline_type>Normal</pipeline_type>
@@ -31,31 +31,74 @@ limitations under the License.
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
<created_user>-</created_user>
- <created_date>2021/04/19 12:06:20.350</created_date>
+ <created_date>2026/03/08 12:00:00.000</created_date>
<modified_user>-</modified_user>
- <modified_date>2021/04/19 12:06:20.350</modified_date>
+ <modified_date>2026/03/08 12:00:00.000</modified_date>
</info>
<notepads>
</notepads>
<order>
<hop>
- <from>Fuzzy match</from>
- <to>Output</to>
+ <from>Main</from>
+ <to>Fuzzy match</to>
<enabled>Y</enabled>
</hop>
<hop>
- <from>files/customers-100.txt</from>
+ <from>Lookup</from>
<to>Fuzzy match</to>
<enabled>Y</enabled>
</hop>
<hop>
- <from>Data grid</from>
- <to>Fuzzy match</to>
+ <from>Fuzzy match</from>
+ <to>Output</to>
<enabled>Y</enabled>
</hop>
</order>
<transform>
- <name>Data grid</name>
+ <name>Main</name>
+ <type>DataGrid</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <data>
+ <line>
+ <item>1</item>
+ <item>Catrine</item>
+ </line>
+ <line>
+ <item>2</item>
+ <item>Zzzz</item>
+ </line>
+ </data>
+ <fields>
+ <field>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ <name>id</name>
+ <type>Integer</type>
+ </field>
+ <field>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ <name>name</name>
+ <type>String</type>
+ </field>
+ </fields>
+ <attributes/>
+ <GUI>
+ <xloc>96</xloc>
+ <yloc>64</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>Lookup</name>
<type>DataGrid</type>
<description/>
<distribute>Y</distribute>
@@ -67,10 +110,20 @@ limitations under the License.
</partitioning>
<data>
<line>
- <item>jwcdf-name</item>
+ <item>Catrine</item>
+ <item>A</item>
+ </line>
+ <line>
+ <item>Catriny</item>
+ <item>B</item>
</line>
<line>
- <item>rdjin-name</item>
+ <item>Catherine</item>
+ <item>C</item>
+ </line>
+ <line>
+ <item>John</item>
+ <item>D</item>
</line>
</data>
<fields>
@@ -81,11 +134,18 @@ limitations under the License.
<name>name</name>
<type>String</type>
</field>
+ <field>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ <name>code</name>
+ <type>String</type>
+ </field>
</fields>
<attributes/>
<GUI>
- <xloc>303</xloc>
- <yloc>162</yloc>
+ <xloc>96</xloc>
+ <yloc>192</yloc>
</GUI>
</transform>
<transform>
@@ -99,27 +159,29 @@ limitations under the License.
<method>none</method>
<schema_name/>
</partitioning>
- <from>Data grid</from>
+ <from>Lookup</from>
<lookupfield>name</lookupfield>
<mainstreamfield>name</mainstreamfield>
<outputmatchfield>match</outputmatchfield>
<outputvaluefield>measure value</outputvaluefield>
<caseSensitive>N</caseSensitive>
<closervalue>Y</closervalue>
+ <matchMode>closest</matchMode>
+ <maxMatches>10</maxMatches>
<minimalValue>0</minimalValue>
- <maximalValue>1</maximalValue>
+ <maximalValue>2</maximalValue>
<separator>,</separator>
- <algorithm>soundex</algorithm>
+ <algorithm>levenshtein</algorithm>
<lookup>
<value>
- <name>name</name>
- <rename>lookupName</rename>
+ <name>code</name>
+ <rename/>
</value>
</lookup>
<attributes/>
<GUI>
<xloc>304</xloc>
- <yloc>64</yloc>
+ <yloc>128</yloc>
</GUI>
</transform>
<transform>
@@ -135,150 +197,8 @@ limitations under the License.
</partitioning>
<attributes/>
<GUI>
- <xloc>528</xloc>
- <yloc>64</yloc>
- </GUI>
- </transform>
- <transform>
- <name>files/customers-100.txt</name>
- <type>CSVInput</type>
- <description/>
- <distribute>Y</distribute>
- <custom_distribution/>
- <copies>1</copies>
- <partitioning>
- <method>none</method>
- <schema_name/>
- </partitioning>
- <filename>${PROJECT_HOME}/files/customers-100.txt</filename>
- <filename_field/>
- <rownum_field/>
- <include_filename>N</include_filename>
- <separator>;</separator>
- <enclosure>"</enclosure>
- <header>Y</header>
- <buffer_size>50000</buffer_size>
- <lazy_conversion>N</lazy_conversion>
- <add_filename_result>N</add_filename_result>
- <parallel>N</parallel>
- <newline_possible>N</newline_possible>
- <encoding/>
- <fields>
- <field>
- <name>id</name>
- <type>Integer</type>
- <format> #</format>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>15</length>
- <precision>0</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>name</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>50</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>firstname</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>50</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>zip</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>30</length>
- <precision>0</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>city</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>8</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>birthdate</name>
- <type>Date</type>
- <format>yyyy/MM/dd</format>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>-1</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>street</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>11</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>housenr</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>50</length>
- <precision>0</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>stateCode</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>10</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- <field>
- <name>state</name>
- <type>String</type>
- <format/>
- <currency>$</currency>
- <decimal>.</decimal>
- <group>,</group>
- <length>50</length>
- <precision>-1</precision>
- <trim_type>none</trim_type>
- </field>
- </fields>
- <attributes/>
- <GUI>
- <xloc>112</xloc>
- <yloc>64</yloc>
+ <xloc>512</xloc>
+ <yloc>128</yloc>
</GUI>
</transform>
<transform_error_handling>
diff --git a/integration-tests/transforms/0012-fuzzymatch.hpl b/integration-tests/transforms/0012-fuzzymatch.hpl
index bafb0af4f3..811ea6c544 100644
--- a/integration-tests/transforms/0012-fuzzymatch.hpl
+++ b/integration-tests/transforms/0012-fuzzymatch.hpl
@@ -106,6 +106,8 @@ limitations under the License.
<outputvaluefield>measure value</outputvaluefield>
<caseSensitive>N</caseSensitive>
<closervalue>Y</closervalue>
+ <matchMode>closest</matchMode>
+ <maxMatches>10</maxMatches>
<minimalValue>0</minimalValue>
<maximalValue>1</maximalValue>
<separator>,</separator>
diff --git a/integration-tests/transforms/datasets/golden-fuzzymatch-all-concat.csv b/integration-tests/transforms/datasets/golden-fuzzymatch-all-concat.csv
new file mode 100644
index 0000000000..5e96bf628b
--- /dev/null
+++ b/integration-tests/transforms/datasets/golden-fuzzymatch-all-concat.csv
@@ -0,0 +1,3 @@
+id,name,match,measure value
+1,Catrine,Catrine|Catriny,0|1
+2,Zzzz,,
diff --git a/integration-tests/transforms/datasets/golden-fuzzymatch-all-rows.csv b/integration-tests/transforms/datasets/golden-fuzzymatch-all-rows.csv
new file mode 100644
index 0000000000..4c3e770cd0
--- /dev/null
+++ b/integration-tests/transforms/datasets/golden-fuzzymatch-all-rows.csv
@@ -0,0 +1,4 @@
+id,name,match,measure value,code
+1,Catrine,Catrine,0,A
+1,Catrine,Catriny,1,B
+2,Zzzz,,,
diff --git a/integration-tests/transforms/datasets/golden-fuzzymatch-closest.csv b/integration-tests/transforms/datasets/golden-fuzzymatch-closest.csv
new file mode 100644
index 0000000000..e0095b76ec
--- /dev/null
+++ b/integration-tests/transforms/datasets/golden-fuzzymatch-closest.csv
@@ -0,0 +1,3 @@
+id,name,match,measure value,code
+1,Catrine,Catrine,0,A
+2,Zzzz,,,
diff --git a/integration-tests/transforms/main-0012-fuzzymatch.hwf b/integration-tests/transforms/main-0012-fuzzymatch.hwf
index bfdbe7c272..faaff5d560 100644
--- a/integration-tests/transforms/main-0012-fuzzymatch.hwf
+++ b/integration-tests/transforms/main-0012-fuzzymatch.hwf
@@ -49,7 +49,7 @@ limitations under the License.
<attributes_hac/>
</action>
<action>
- <name>Run Group By tests</name>
+ <name>Run Fuzzy match tests</name>
<description/>
<type>RunPipelineTests</type>
<attributes/>
@@ -57,6 +57,15 @@ limitations under the License.
<test_name>
<name>0012-fuzzymatch UNIT</name>
</test_name>
+ <test_name>
+ <name>0012-fuzzymatch-closest UNIT</name>
+ </test_name>
+ <test_name>
+ <name>0012-fuzzymatch-all-rows UNIT</name>
+ </test_name>
+ <test_name>
+ <name>0012-fuzzymatch-all-concat UNIT</name>
+ </test_name>
</test_names>
<parallel>N</parallel>
<xloc>304</xloc>
diff --git a/integration-tests/transforms/metadata/dataset/golden-fuzzymatch-all-concat.json b/integration-tests/transforms/metadata/dataset/golden-fuzzymatch-all-concat.json
new file mode 100644
index 0000000000..f485675e83
--- /dev/null
+++ b/integration-tests/transforms/metadata/dataset/golden-fuzzymatch-all-concat.json
@@ -0,0 +1,40 @@
+{
+ "base_filename": "golden-fuzzymatch-all-concat.csv",
+ "name": "golden-fuzzymatch-all-concat",
+ "description": "Golden data for Fuzzy match All matches (concatenated) with joined distances",
+ "dataset_fields": [
+ {
+ "field_comment": "",
+ "field_length": 15,
+ "field_type": 5,
+ "field_precision": 0,
+ "field_format": "#",
+ "field_name": "id"
+ },
+ {
+ "field_comment": "",
+ "field_length": 50,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_format": "",
+ "field_name": "name"
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_format": "",
+ "field_name": "match"
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_format": "",
+ "field_name": "measure value"
+ }
+ ],
+ "folder_name": ""
+}
diff --git a/integration-tests/transforms/metadata/dataset/golden-fuzzymatch-all-rows.json b/integration-tests/transforms/metadata/dataset/golden-fuzzymatch-all-rows.json
new file mode 100644
index 0000000000..373f8ead8f
--- /dev/null
+++ b/integration-tests/transforms/metadata/dataset/golden-fuzzymatch-all-rows.json
@@ -0,0 +1,48 @@
+{
+ "base_filename": "golden-fuzzymatch-all-rows.csv",
+ "name": "golden-fuzzymatch-all-rows",
+ "description": "Golden data for Fuzzy match All matches (separate rows) Top-K",
+ "dataset_fields": [
+ {
+ "field_comment": "",
+ "field_length": 15,
+ "field_type": 5,
+ "field_precision": 0,
+ "field_format": "#",
+ "field_name": "id"
+ },
+ {
+ "field_comment": "",
+ "field_length": 50,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_format": "",
+ "field_name": "name"
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_format": "",
+ "field_name": "match"
+ },
+ {
+ "field_comment": "",
+ "field_length": 15,
+ "field_type": 5,
+ "field_precision": 0,
+ "field_format": "#",
+ "field_name": "measure value"
+ },
+ {
+ "field_comment": "",
+ "field_length": 10,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_format": "",
+ "field_name": "code"
+ }
+ ],
+ "folder_name": ""
+}
diff --git a/integration-tests/transforms/metadata/dataset/golden-fuzzymatch-closest.json b/integration-tests/transforms/metadata/dataset/golden-fuzzymatch-closest.json
new file mode 100644
index 0000000000..0340cc0821
--- /dev/null
+++ b/integration-tests/transforms/metadata/dataset/golden-fuzzymatch-closest.json
@@ -0,0 +1,48 @@
+{
+ "base_filename": "golden-fuzzymatch-closest.csv",
+ "name": "golden-fuzzymatch-closest",
+ "description": "Golden data for Fuzzy match Closest mode (Levenshtein)",
+ "dataset_fields": [
+ {
+ "field_comment": "",
+ "field_length": 15,
+ "field_type": 5,
+ "field_precision": 0,
+ "field_format": "#",
+ "field_name": "id"
+ },
+ {
+ "field_comment": "",
+ "field_length": 50,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_format": "",
+ "field_name": "name"
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_format": "",
+ "field_name": "match"
+ },
+ {
+ "field_comment": "",
+ "field_length": 15,
+ "field_type": 5,
+ "field_precision": 0,
+ "field_format": "#",
+ "field_name": "measure value"
+ },
+ {
+ "field_comment": "",
+ "field_length": 10,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_format": "",
+ "field_name": "code"
+ }
+ ],
+ "folder_name": ""
+}
diff --git a/integration-tests/transforms/metadata/unit-test/0012-fuzzymatch-all-concat UNIT.json b/integration-tests/transforms/metadata/unit-test/0012-fuzzymatch-all-concat UNIT.json
new file mode 100644
index 0000000000..0a5f76a23c
--- /dev/null
+++ b/integration-tests/transforms/metadata/unit-test/0012-fuzzymatch-all-concat UNIT.json
@@ -0,0 +1,40 @@
+{
+ "variableValues": [],
+ "database_replacements": [],
+ "autoOpening": true,
+ "basePath": "",
+ "golden_data_sets": [
+ {
+ "field_mappings": [
+ {
+ "transform_field": "id",
+ "data_set_field": "id"
+ },
+ {
+ "transform_field": "name",
+ "data_set_field": "name"
+ },
+ {
+ "transform_field": "match",
+ "data_set_field": "match"
+ },
+ {
+ "transform_field": "measure value",
+ "data_set_field": "measure value"
+ }
+ ],
+ "field_order": [
+ "id"
+ ],
+ "data_set_name": "golden-fuzzymatch-all-concat",
+ "transform_name": "Output"
+ }
+ ],
+ "input_data_sets": [],
+ "name": "0012-fuzzymatch-all-concat UNIT",
+ "description": "All matches (concatenated) joins Top-K matches and distances with the values separator",
+ "persist_filename": "",
+ "trans_test_tweaks": [],
+ "pipeline_filename": "./0012-fuzzymatch-all-concat.hpl",
+ "test_type": "UNIT_TEST"
+}
diff --git a/integration-tests/transforms/metadata/unit-test/0012-fuzzymatch-all-rows UNIT.json b/integration-tests/transforms/metadata/unit-test/0012-fuzzymatch-all-rows UNIT.json
new file mode 100644
index 0000000000..0ce8b3d45e
--- /dev/null
+++ b/integration-tests/transforms/metadata/unit-test/0012-fuzzymatch-all-rows UNIT.json
@@ -0,0 +1,45 @@
+{
+ "variableValues": [],
+ "database_replacements": [],
+ "autoOpening": true,
+ "basePath": "",
+ "golden_data_sets": [
+ {
+ "field_mappings": [
+ {
+ "transform_field": "id",
+ "data_set_field": "id"
+ },
+ {
+ "transform_field": "name",
+ "data_set_field": "name"
+ },
+ {
+ "transform_field": "match",
+ "data_set_field": "match"
+ },
+ {
+ "transform_field": "measure value",
+ "data_set_field": "measure value"
+ },
+ {
+ "transform_field": "code",
+ "data_set_field": "code"
+ }
+ ],
+ "field_order": [
+ "id",
+ "measure value"
+ ],
+ "data_set_name": "golden-fuzzymatch-all-rows",
+ "transform_name": "Output"
+ }
+ ],
+ "input_data_sets": [],
+ "name": "0012-fuzzymatch-all-rows UNIT",
+ "description": "All matches (separate rows) emits Top-K ranked rows with individual distances",
+ "persist_filename": "",
+ "trans_test_tweaks": [],
+ "pipeline_filename": "./0012-fuzzymatch-all-rows.hpl",
+ "test_type": "UNIT_TEST"
+}
diff --git a/integration-tests/transforms/metadata/unit-test/0012-fuzzymatch-closest UNIT.json b/integration-tests/transforms/metadata/unit-test/0012-fuzzymatch-closest UNIT.json
new file mode 100644
index 0000000000..905c45fa39
--- /dev/null
+++ b/integration-tests/transforms/metadata/unit-test/0012-fuzzymatch-closest UNIT.json
@@ -0,0 +1,44 @@
+{
+ "variableValues": [],
+ "database_replacements": [],
+ "autoOpening": true,
+ "basePath": "",
+ "golden_data_sets": [
+ {
+ "field_mappings": [
+ {
+ "transform_field": "id",
+ "data_set_field": "id"
+ },
+ {
+ "transform_field": "name",
+ "data_set_field": "name"
+ },
+ {
+ "transform_field": "match",
+ "data_set_field": "match"
+ },
+ {
+ "transform_field": "measure value",
+ "data_set_field": "measure value"
+ },
+ {
+ "transform_field": "code",
+ "data_set_field": "code"
+ }
+ ],
+ "field_order": [
+ "id"
+ ],
+ "data_set_name": "golden-fuzzymatch-closest",
+ "transform_name": "Output"
+ }
+ ],
+ "input_data_sets": [],
+ "name": "0012-fuzzymatch-closest UNIT",
+ "description": "Closest match mode returns the single best Levenshtein match with measure and lookup fields",
+ "persist_filename": "",
+ "trans_test_tweaks": [],
+ "pipeline_filename": "./0012-fuzzymatch-closest.hpl",
+ "test_type": "UNIT_TEST"
+}
diff --git a/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatch.java b/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatch.java
index 9c0c697658..dc20a23fbe 100644
--- a/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatch.java
+++ b/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatch.java
@@ -19,11 +19,16 @@ package org.apache.hop.pipeline.transforms.fuzzymatch;
import static org.apache.hop.pipeline.transforms.fuzzymatch.FuzzyMatchMeta.Algorithm;
import static org.apache.hop.pipeline.transforms.fuzzymatch.FuzzyMatchMeta.FMLookupValue;
+import static org.apache.hop.pipeline.transforms.fuzzymatch.FuzzyMatchMeta.MatchMode;
import com.wcohen.ss.Jaro;
import com.wcohen.ss.JaroWinkler;
import com.wcohen.ss.NeedlemanWunsch;
-import java.util.Iterator;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.PriorityQueue;
import org.apache.commons.codec.language.DoubleMetaphone;
import org.apache.commons.codec.language.Metaphone;
import org.apache.commons.codec.language.RefinedSoundex;
@@ -173,7 +178,7 @@ public class FuzzyMatch extends BaseTransform<FuzzyMatchMeta, FuzzyMatchData> {
return true;
}
- private Object[] lookupValues(IRowMeta rowMeta, Object[] row) throws HopException {
+ private List<Object[]> lookupValues(IRowMeta rowMeta, Object[] row) throws HopException {
if (first) {
first = false;
@@ -195,17 +200,16 @@ public class FuzzyMatch extends BaseTransform<FuzzyMatchMeta, FuzzyMatchData> {
PKG, "FuzzyMatch.Exception.CouldnotFindMainField", meta.getMainStreamField()));
}
}
- Object[] add;
if (row[data.indexOfMainField] == null) {
- add = RowDataUtil.allocateRowData(data.outputRowMeta.size());
- } else {
- try {
- add = getFromCache(row);
- } catch (Exception e) {
- throw new HopTransformException(e);
- }
+ List<Object[]> empty = new ArrayList<>(1);
+ empty.add(RowDataUtil.allocateRowData(data.outputRowMeta.size()));
+ return empty;
+ }
+ try {
+ return getFromCache(row);
+ } catch (Exception e) {
+ throw new HopTransformException(e);
}
- return RowDataUtil.addRowData(row, rowMeta.size(), add);
}
private void addToCache(Object[] value) throws HopException {
@@ -218,47 +222,28 @@ public class FuzzyMatch extends BaseTransform<FuzzyMatchMeta, FuzzyMatchData> {
}
}
- private Object[] getFromCache(Object[] keyRow) throws HopValueException {
+ private List<Object[]> getFromCache(Object[] keyRow) throws HopValueException {
if (isDebug()) {
logDebug(
BaseMessages.getString(
PKG, "FuzzyMatch.Log.ReadingMainStreamRow", getInputRowMeta().getString(keyRow)));
}
- Object[] retval = null;
- switch (meta.getAlgorithm()) {
- case LEVENSHTEIN, DAMERAU_LEVENSHTEIN, NEEDLEMAN_WUNSH:
- retval = doDistance(keyRow);
- break;
- case DOUBLE_METAPHONE, METAPHONE, SOUNDEX, REFINED_SOUNDEX:
- retval = doPhonetic(keyRow);
- break;
- case JARO, JARO_WINKLER, PAIR_SIMILARITY:
- retval = doSimilarity(keyRow);
- break;
- default:
- break;
- }
-
- return retval;
+ return switch (meta.getAlgorithm()) {
+ case LEVENSHTEIN, DAMERAU_LEVENSHTEIN, NEEDLEMAN_WUNSH -> doDistance(keyRow);
+ case DOUBLE_METAPHONE, METAPHONE, SOUNDEX, REFINED_SOUNDEX -> doPhonetic(keyRow);
+ case JARO, JARO_WINKLER, PAIR_SIMILARITY -> doSimilarity(keyRow);
+ default -> List.<Object[]>of(RowDataUtil.allocateRowData(data.outputRowMeta.size()));
+ };
}
- private Object[] doDistance(Object[] row) throws HopValueException {
- // Reserve room
- Object[] rowData = RowDataUtil.allocateRowData(data.outputRowMeta.size());
-
- Iterator<Object[]> it = data.look.iterator();
-
- long distance = -1;
-
+ private List<Object[]> doDistance(Object[] row) throws HopValueException {
+ boolean lowerIsBetter = true;
+ PriorityQueue<ScoredMatch> topK = newTopKHeap(lowerIsBetter);
String lookupValueString = getInputRowMeta().getString(row, data.indexOfMainField);
- while (it.hasNext()) {
- // Get cached row data
- Object[] cachedData = it.next();
- // Key value is the first value
+ for (Object[] cachedData : data.look) {
String cacheValue = (String) cachedData[0];
- int cDistance;
String useCacheValue = cacheValue;
String useLookupvalue = lookupValueString;
if (!meta.isCaseSensitive()) {
@@ -266,7 +251,7 @@ public class FuzzyMatch extends BaseTransform<FuzzyMatchMeta, FuzzyMatchData> {
useLookupvalue = lookupValueString.toLowerCase();
}
- cDistance =
+ int cDistance =
switch (meta.getAlgorithm()) {
case DAMERAU_LEVENSHTEIN ->
Utils.getDamerauLevenshteinDistance(useCacheValue, useLookupvalue);
@@ -276,81 +261,43 @@ public class FuzzyMatch extends BaseTransform<FuzzyMatchMeta, FuzzyMatchData> {
};
if (data.minimalDistance <= cDistance && cDistance <= data.maximalDistance) {
- if (meta.isCloserValue()) {
- if (cDistance < distance || distance == -1) {
- // Get closer value
- // minimal distance
- distance = cDistance;
- int index = 0;
- rowData[index++] = cacheValue;
- // Add metric value?
- if (data.addValueFieldName) {
- rowData[index++] = distance;
- }
- // Add additional return values?
- if (data.addAdditionalFields) {
- for (int i = 0; i < meta.getLookupValues().size(); i++) {
- int nr = i + 1;
- int nf = i + index;
- rowData[nf] = cachedData[nr];
- }
- }
- }
- } else {
- // get all values separated by values separator
- if (rowData[0] == null) {
- rowData[0] = cacheValue;
- } else {
- rowData[0] = rowData[0] + data.valueSeparator + cacheValue;
- }
- }
+ offerTopK(topK, new ScoredMatch(cachedData, cDistance, (long) cDistance), lowerIsBetter);
}
}
- return rowData;
+ return buildMatchResults(topK, lowerIsBetter);
}
- private Object[] doPhonetic(Object[] row) {
- // Reserve room
- Object[] rowData = RowDataUtil.allocateRowData(data.outputRowMeta.size());
-
- Iterator<Object[]> it = data.look.iterator();
+ private List<Object[]> doPhonetic(Object[] row) {
+ boolean lowerIsBetter = true; // equal scores; first-seen kept by heap capacity
+ PriorityQueue<ScoredMatch> topK = newTopKHeap(lowerIsBetter);
Object o = row[data.indexOfMainField];
String lookupvalue = (String) o;
-
String lookupValueMF = getEncodedMF(lookupvalue, meta.getAlgorithm());
- while (it.hasNext()) {
- // Get cached row data
- Object[] cachedData = it.next();
- // Key value is the first value
+ for (Object[] cachedData : data.look) {
String cacheValue = (String) cachedData[0];
-
String cacheValueMF = getEncodedMF(cacheValue, meta.getAlgorithm());
-
if (lookupValueMF.equals(cacheValueMF)) {
+ offerTopK(topK, new ScoredMatch(cachedData, 0, cacheValueMF), lowerIsBetter);
+ }
+ }
- // Add match value
- int index = 0;
- rowData[index++] = cacheValue;
-
- // Add metric value?
- if (data.addValueFieldName) {
- rowData[index++] = cacheValueMF;
- }
- // Add additional return values?
- if (data.addAdditionalFields) {
- for (int i = 0; i < meta.getLookupValues().size(); i++) {
- int nf = i + index;
- int nr = i + 1;
- rowData[nf] = cachedData[nr];
- }
+ if (meta.isCloserValue()) {
+ // Preserve legacy behaviour: last matching phonetic wins
+ Object[] rowData = RowDataUtil.allocateRowData(data.outputRowMeta.size());
+ for (Object[] cachedData : data.look) {
+ String cacheValue = (String) cachedData[0];
+ String cacheValueMF = getEncodedMF(cacheValue, meta.getAlgorithm());
+ if (lookupValueMF.equals(cacheValueMF)) {
+ fillMatchRow(rowData, cachedData, cacheValueMF);
}
}
+ return List.<Object[]>of(rowData);
}
- return rowData;
+ return buildMatchResults(topK, lowerIsBetter);
}
private String getEncodedMF(String value, Algorithm algorithmType) {
@@ -374,69 +321,129 @@ public class FuzzyMatch extends BaseTransform<FuzzyMatchMeta, FuzzyMatchData> {
return encodedValueMF;
}
- private Object[] doSimilarity(Object[] row) {
- // Reserve room
- Object[] rowData = RowDataUtil.allocateRowData(data.outputRowMeta.size());
- // prepare to read from cache ...
- Iterator<Object[]> it = data.look.iterator();
- double similarity = 0;
+ private List<Object[]> doSimilarity(Object[] row) {
+ boolean lowerIsBetter = false;
+ PriorityQueue<ScoredMatch> topK = newTopKHeap(lowerIsBetter);
- // get current value from main stream
Object o = row[data.indexOfMainField];
-
String lookupValueString = o == null ? "" : (String) o;
- while (it.hasNext()) {
- // Get cached row data
- Object[] cachedData = it.next();
- // Key value is the first value
+ for (Object[] cachedData : data.look) {
String cacheValue = (String) cachedData[0];
double cSimilarity =
switch (meta.getAlgorithm()) {
case JARO -> new Jaro().score(cacheValue, lookupValueString);
case JARO_WINKLER -> new JaroWinkler().score(cacheValue, lookupValueString);
- default ->
- // Letters pair similarity
- LetterPairSimilarity.getSimiliarity(cacheValue, lookupValueString);
+ default -> LetterPairSimilarity.getSimilarity(cacheValue, lookupValueString);
};
if (data.minimalSimilarity <= cSimilarity && cSimilarity <= data.maximalSimilarity) {
- if (meta.isCloserValue()) {
- if (cSimilarity > similarity
- || (cSimilarity == 0 && cacheValue.equals(lookupValueString))) {
- similarity = cSimilarity;
- // Update match value
- int index = 0;
- rowData[index++] = cacheValue;
- // Add metric value?
- if (data.addValueFieldName) {
- rowData[index++] = similarity;
- }
-
- // Add additional return values?
- if (data.addAdditionalFields) {
- for (int i = 0; i < meta.getLookupValues().size(); i++) {
- int nf = i + index;
- int nr = i + 1;
- rowData[nf] = cachedData[nr];
- }
- }
- }
- } else {
- // get all values separated by values separator
- if (rowData[0] == null) {
- rowData[0] = cacheValue;
- } else {
- rowData[0] = rowData[0] + data.valueSeparator + cacheValue;
- }
+ // Exact empty-string edge case from legacy closer-value logic
+ double score = cSimilarity;
+ if (cSimilarity == 0 && cacheValue.equals(lookupValueString)) {
+ score = 0;
}
+ offerTopK(topK, new ScoredMatch(cachedData, score, cSimilarity), lowerIsBetter);
}
}
+ return buildMatchResults(topK, lowerIsBetter);
+ }
+
+ private PriorityQueue<ScoredMatch> newTopKHeap(boolean lowerIsBetter) {
+ // Worst candidate at the head so it can be evicted.
+ Comparator<ScoredMatch> worstFirst =
+ lowerIsBetter
+ ? Comparator.comparingDouble((ScoredMatch m) -> m.score).reversed()
+ : Comparator.comparingDouble(m -> m.score);
+ return new PriorityQueue<>(Math.max(1, data.maxMatches), worstFirst);
+ }
+
+ private void offerTopK(
+ PriorityQueue<ScoredMatch> heap, ScoredMatch candidate, boolean lowerIsBetter) {
+ int k = meta.isCloserValue() ? 1 : data.maxMatches;
+ if (k <= 0) {
+ return;
+ }
+ if (heap.size() < k) {
+ heap.offer(candidate);
+ return;
+ }
+ ScoredMatch worst = heap.peek();
+ if (worst == null) {
+ return;
+ }
+ boolean better = lowerIsBetter ? candidate.score < worst.score : candidate.score > worst.score;
+ if (better) {
+ heap.poll();
+ heap.offer(candidate);
+ }
+ }
+
+ private List<Object[]> buildMatchResults(PriorityQueue<ScoredMatch> heap, boolean lowerIsBetter) {
+ if (heap.isEmpty()) {
+ return List.<Object[]>of(RowDataUtil.allocateRowData(data.outputRowMeta.size()));
+ }
+
+ List<ScoredMatch> ranked = new ArrayList<>(heap);
+ ranked.sort(
+ lowerIsBetter
+ ? Comparator.comparingDouble(m -> m.score)
+ : Comparator.comparingDouble((ScoredMatch m) -> m.score).reversed());
+
+ MatchMode mode = meta.getMatchMode();
+ if (mode == MatchMode.ALL_CONCAT) {
+ return List.<Object[]>of(buildConcatRow(ranked));
+ }
+
+ // CLOSEST (k=1) and ALL_ROWS: one output addition per match
+ List<Object[]> results = new ArrayList<>(ranked.size());
+ for (ScoredMatch match : ranked) {
+ Object[] rowData = RowDataUtil.allocateRowData(data.outputRowMeta.size());
+ fillMatchRow(rowData, match.cachedData, match.measureValue);
+ results.add(rowData);
+ }
+ return results;
+ }
+
+ private Object[] buildConcatRow(List<ScoredMatch> ranked) {
+ Object[] rowData = RowDataUtil.allocateRowData(data.outputRowMeta.size());
+ StringBuilder matches = new StringBuilder();
+ StringBuilder measures = new StringBuilder();
+ for (int i = 0; i < ranked.size(); i++) {
+ ScoredMatch match = ranked.get(i);
+ if (i > 0) {
+ matches.append(data.valueSeparator);
+ if (data.addValueFieldName) {
+ measures.append(data.valueSeparator);
+ }
+ }
+ matches.append(match.cachedData[0]);
+ if (data.addValueFieldName) {
+ measures.append(match.measureValue);
+ }
+ }
+ rowData[0] = matches.toString();
+ if (data.addValueFieldName) {
+ rowData[1] = measures.toString();
+ }
return rowData;
}
+ private void fillMatchRow(Object[] rowData, Object[] cachedData, Object measureValue) {
+ int index = 0;
+ rowData[index++] = cachedData[0];
+ if (data.addValueFieldName) {
+ rowData[index++] = measureValue;
+ }
+ if (data.addAdditionalFields) {
+ for (int i = 0; i < meta.getLookupValues().size(); i++) {
+ rowData[index + i] = cachedData[i + 1];
+ }
+ }
+ }
+
@Override
public boolean processRow() throws HopException {
if (data.readLookupValues) {
@@ -468,14 +475,12 @@ public class FuzzyMatch extends BaseTransform<FuzzyMatchMeta, FuzzyMatchData> {
}
try {
-
- // Do the actual lookup in the hastable.
- Object[] outputRow = lookupValues(getInputRowMeta(), r);
- if (outputRow == null) {
- setOutputDone(); // signal end to receiver(s)
- return false;
+ List<Object[]> additions = lookupValues(getInputRowMeta(), r);
+ int inputSize = getInputRowMeta().size();
+ for (Object[] add : additions) {
+ Object[] outputRow = RowDataUtil.addRowData(Arrays.copyOf(r, r.length), inputSize, add);
+ putRow(data.outputRowMeta, outputRow);
}
- putRow(data.outputRowMeta, outputRow); // copy row to output rowset(s)
if (checkFeedback(getLinesRead()) && isBasic()) {
logBasic(BaseMessages.getString(PKG, "FuzzyMatch.Log.LineNumber") + getLinesRead());
@@ -520,30 +525,42 @@ public class FuzzyMatch extends BaseTransform<FuzzyMatchMeta, FuzzyMatchData> {
return false;
}
- // We need to add metrics (distance, similarity, ...)
- // only when the field name is provided
- // and user want to return the closer value.
- //
- data.addValueFieldName =
- (StringUtils.isNotEmpty(resolve(meta.getOutputValueField())) && meta.isCloserValue());
+ // Metrics (distance, similarity, ...) when an output field name is provided
+ data.addValueFieldName = StringUtils.isNotEmpty(resolve(meta.getOutputValueField()));
// Set the number of fields to cache
// default value is one
//
int nrFields = 1;
- if (!meta.getLookupValues().isEmpty()
- && (meta.isCloserValue()
- || meta.getAlgorithm() == Algorithm.DOUBLE_METAPHONE
- || meta.getAlgorithm() == Algorithm.SOUNDEX
- || meta.getAlgorithm() == Algorithm.REFINED_SOUNDEX
- || meta.getAlgorithm() == Algorithm.METAPHONE)) {
+ if (!meta.getLookupValues().isEmpty() && meta.supportsAdditionalFields()) {
// cache also additional fields
data.addAdditionalFields = true;
nrFields += meta.getLookupValues().size();
}
data.indexOfCachedFields = new int[nrFields];
+ // Top-K size: default 10, hard-capped at 100
+ int maxMatches = Const.toInt(resolve(meta.getMaxMatches()), FuzzyMatchMeta.DEFAULT_MAX_MATCHES);
+ if (maxMatches < 1) {
+ maxMatches = 1;
+ }
+ if (maxMatches > FuzzyMatchMeta.HARD_MAX_MATCHES) {
+ if (isBasic()) {
+ logBasic(
+ BaseMessages.getString(
+ PKG,
+ "FuzzyMatch.Log.MaxMatchesCapped",
+ maxMatches,
+ FuzzyMatchMeta.HARD_MAX_MATCHES));
+ }
+ maxMatches = FuzzyMatchMeta.HARD_MAX_MATCHES;
+ }
+ data.maxMatches = maxMatches;
+ if (isDetailed() && !meta.isCloserValue()) {
+ logDetailed(BaseMessages.getString(PKG, "FuzzyMatch.Log.MaxMatches", data.maxMatches));
+ }
+
switch (meta.getAlgorithm()) {
case LEVENSHTEIN, DAMERAU_LEVENSHTEIN, NEEDLEMAN_WUNSH:
data.minimalDistance = Const.toInt(resolve(meta.getMinimalValue()), 0);
@@ -556,7 +573,7 @@ public class FuzzyMatch extends BaseTransform<FuzzyMatchMeta, FuzzyMatchData> {
logDetailed(
BaseMessages.getString(PKG, "FuzzyMatch.Log.MaximalDistance", data.maximalDistance));
}
- if (!meta.isCloserValue()) {
+ if (meta.isAllConcatMode()) {
data.valueSeparator = resolve(meta.getSeparator());
if (isDetailed()) {
logDetailed(
@@ -577,7 +594,7 @@ public class FuzzyMatch extends BaseTransform<FuzzyMatchMeta, FuzzyMatchData> {
BaseMessages.getString(
PKG, "FuzzyMatch.Log.MaximalSimilarity", data.maximalSimilarity));
}
- if (!meta.isCloserValue()) {
+ if (meta.isAllConcatMode()) {
data.valueSeparator = resolve(meta.getSeparator());
if (isDetailed()) {
logDetailed(
@@ -599,4 +616,16 @@ public class FuzzyMatch extends BaseTransform<FuzzyMatchMeta, FuzzyMatchData> {
data.look.clear();
super.dispose();
}
+
+ private static final class ScoredMatch {
+ private final Object[] cachedData;
+ private final double score;
+ private final Object measureValue;
+
+ private ScoredMatch(Object[] cachedData, double score, Object measureValue) {
+ this.cachedData = cachedData;
+ this.score = score;
+ this.measureValue = measureValue;
+ }
+ }
}
diff --git a/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchData.java b/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchData.java
index 479510cdc5..5466397f53 100644
--- a/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchData.java
+++ b/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchData.java
@@ -18,6 +18,7 @@
package org.apache.hop.pipeline.transforms.fuzzymatch;
import java.util.HashSet;
+import java.util.Set;
import org.apache.hop.core.row.IRowMeta;
import org.apache.hop.pipeline.transform.BaseTransformData;
import org.apache.hop.pipeline.transform.ITransformData;
@@ -29,7 +30,7 @@ public class FuzzyMatchData extends BaseTransformData implements ITransformData
public IRowMeta outputRowMeta;
/** used to store values in used to look up things */
- public HashSet<Object[]> look;
+ public Set<Object[]> look;
public boolean readLookupValues;
@@ -53,6 +54,9 @@ public class FuzzyMatchData extends BaseTransformData implements ITransformData
public boolean addValueFieldName;
public boolean addAdditionalFields;
+ /** Top-K limit for multi-match modes (already clamped). */
+ public int maxMatches;
+
/** index of return fields from lookup stream */
public int[] indexOfCachedFields;
@@ -67,5 +71,6 @@ public class FuzzyMatchData extends BaseTransformData implements ITransformData
this.valueSeparator = "";
this.nrCachedFields = 1;
this.addAdditionalFields = false;
+ this.maxMatches = FuzzyMatchMeta.DEFAULT_MAX_MATCHES;
}
}
diff --git a/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchDialog.java b/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchDialog.java
index 532acbc097..5d9e0a6dcb 100644
--- a/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchDialog.java
+++ b/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchDialog.java
@@ -92,8 +92,11 @@ public class FuzzyMatchDialog extends BaseTransformDialog {
private Label wlCaseSensitive;
private Button wCaseSensitive;
- private Label wlGetCloserValue;
- private Button wGetCloserValue;
+ private Label wlMatchMode;
+ private CCombo wMatchMode;
+
+ private Label wlMaxMatches;
+ private TextVar wMaxMatches;
private Label wlMinValue;
private TextVar wMinValue;
@@ -330,32 +333,33 @@ public class FuzzyMatchDialog extends BaseTransformDialog {
fdcaseSensitive.top = new FormAttachment(wlCaseSensitive, 0, SWT.CENTER);
wCaseSensitive.setLayoutData(fdcaseSensitive);
- // Is get closer value
- wlGetCloserValue = new Label(wSettingsGroup, SWT.RIGHT);
- wlGetCloserValue.setText(BaseMessages.getString(PKG, "FuzzyMatchDialog.getCloserValue.Label"));
- PropsUi.setLook(wlGetCloserValue);
- FormData fdlGetCloserValue = new FormData();
- fdlGetCloserValue.left = new FormAttachment(0, 0);
- fdlGetCloserValue.top = new FormAttachment(wCaseSensitive, margin);
- fdlGetCloserValue.right = new FormAttachment(middle, -margin);
- wlGetCloserValue.setLayoutData(fdlGetCloserValue);
-
- wGetCloserValue = new Button(wSettingsGroup, SWT.CHECK);
- PropsUi.setLook(wGetCloserValue);
- wGetCloserValue.setToolTipText(
- BaseMessages.getString(PKG, "FuzzyMatchDialog.getCloserValue.Tooltip"));
- FormData fdgetCloserValue = new FormData();
- fdgetCloserValue.left = new FormAttachment(middle, 0);
- fdgetCloserValue.top = new FormAttachment(wlGetCloserValue, 0, SWT.CENTER);
- wGetCloserValue.setLayoutData(fdgetCloserValue);
- wGetCloserValue.addListener(SWT.Selection, e -> activeGetCloserValue());
+ // Match mode (closest / all rows / concat)
+ wlMatchMode = new Label(wSettingsGroup, SWT.RIGHT);
+ wlMatchMode.setText(BaseMessages.getString(PKG, "FuzzyMatchDialog.matchMode.Label"));
+ PropsUi.setLook(wlMatchMode);
+ FormData fdlMatchMode = new FormData();
+ fdlMatchMode.left = new FormAttachment(0, 0);
+ fdlMatchMode.top = new FormAttachment(wCaseSensitive, margin);
+ fdlMatchMode.right = new FormAttachment(middle, -margin);
+ wlMatchMode.setLayoutData(fdlMatchMode);
+
+ wMatchMode = new CCombo(wSettingsGroup, SWT.BORDER | SWT.READ_ONLY);
+ PropsUi.setLook(wMatchMode);
+ wMatchMode.setItems(FuzzyMatchMeta.MatchMode.getDescriptions());
+ wMatchMode.setToolTipText(BaseMessages.getString(PKG, "FuzzyMatchDialog.matchMode.Tooltip"));
+ FormData fdMatchMode = new FormData();
+ fdMatchMode.left = new FormAttachment(middle, 0);
+ fdMatchMode.top = new FormAttachment(wlMatchMode, 0, SWT.CENTER);
+ fdMatchMode.right = new FormAttachment(100, -margin);
+ wMatchMode.setLayoutData(fdMatchMode);
+ wMatchMode.addListener(SWT.Selection, e -> activeMatchMode());
wlMinValue = new Label(wSettingsGroup, SWT.RIGHT);
wlMinValue.setText(BaseMessages.getString(PKG, "FuzzyMatchDialog.minValue.Label"));
PropsUi.setLook(wlMinValue);
FormData fdlminValue = new FormData();
fdlminValue.left = new FormAttachment(0, 0);
- fdlminValue.top = new FormAttachment(wGetCloserValue, margin);
+ fdlminValue.top = new FormAttachment(wMatchMode, margin);
fdlminValue.right = new FormAttachment(middle, -margin);
wlMinValue.setLayoutData(fdlminValue);
wMinValue = new TextVar(variables, wSettingsGroup, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
@@ -363,7 +367,7 @@ public class FuzzyMatchDialog extends BaseTransformDialog {
wMinValue.setToolTipText(BaseMessages.getString(PKG, "FuzzyMatchDialog.minValue.Tooltip"));
FormData fdminValue = new FormData();
fdminValue.left = new FormAttachment(middle, 0);
- fdminValue.top = new FormAttachment(wGetCloserValue, margin);
+ fdminValue.top = new FormAttachment(wMatchMode, margin);
fdminValue.right = new FormAttachment(100, 0);
wMinValue.setLayoutData(fdminValue);
@@ -384,19 +388,37 @@ public class FuzzyMatchDialog extends BaseTransformDialog {
fdmaxValue.right = new FormAttachment(100, 0);
wMaxValue.setLayoutData(fdmaxValue);
+ wlMaxMatches = new Label(wSettingsGroup, SWT.RIGHT);
+ wlMaxMatches.setText(BaseMessages.getString(PKG, "FuzzyMatchDialog.maxMatches.Label"));
+ PropsUi.setLook(wlMaxMatches);
+ FormData fdlMaxMatches = new FormData();
+ fdlMaxMatches.left = new FormAttachment(0, 0);
+ fdlMaxMatches.top = new FormAttachment(wMaxValue, margin);
+ fdlMaxMatches.right = new FormAttachment(middle, -margin);
+ wlMaxMatches.setLayoutData(fdlMaxMatches);
+ wMaxMatches = new TextVar(variables, wSettingsGroup, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
+ PropsUi.setLook(wMaxMatches);
+ wMaxMatches.setToolTipText(BaseMessages.getString(PKG, "FuzzyMatchDialog.maxMatches.Tooltip"));
+ FormData fdMaxMatches = new FormData();
+ fdMaxMatches.left = new FormAttachment(middle, 0);
+ fdMaxMatches.top = new FormAttachment(wMaxValue, margin);
+ fdMaxMatches.right = new FormAttachment(100, 0);
+ wMaxMatches.setLayoutData(fdMaxMatches);
+
wlSeparator = new Label(wSettingsGroup, SWT.RIGHT);
wlSeparator.setText(BaseMessages.getString(PKG, "FuzzyMatchDialog.separator.Label"));
PropsUi.setLook(wlSeparator);
FormData fdlSeparator = new FormData();
fdlSeparator.left = new FormAttachment(0, 0);
- fdlSeparator.top = new FormAttachment(wMaxValue, margin);
+ fdlSeparator.top = new FormAttachment(wMaxMatches, margin);
fdlSeparator.right = new FormAttachment(middle, -margin);
wlSeparator.setLayoutData(fdlSeparator);
wSeparator = new TextVar(variables, wSettingsGroup, SWT.SINGLE | SWT.LEFT | SWT.BORDER);
PropsUi.setLook(wSeparator);
+ wSeparator.setToolTipText(BaseMessages.getString(PKG, "FuzzyMatchDialog.separator.Tooltip"));
FormData fdSeparator = new FormData();
fdSeparator.left = new FormAttachment(middle, 0);
- fdSeparator.top = new FormAttachment(wMaxValue, margin);
+ fdSeparator.top = new FormAttachment(wMaxMatches, margin);
fdSeparator.right = new FormAttachment(100, 0);
wSeparator.setLayoutData(fdSeparator);
@@ -564,7 +586,7 @@ public class FuzzyMatchDialog extends BaseTransformDialog {
fdTabFolder.left = new FormAttachment(0, 0);
fdTabFolder.top = new FormAttachment(wContentTop, margin);
fdTabFolder.right = new FormAttachment(100, 0);
- fdTabFolder.bottom = new FormAttachment(100, -50);
+ fdTabFolder.bottom = new FormAttachment(100, 0);
wTabFolder.setLayoutData(fdTabFolder);
wTabFolder.setSelection(0);
@@ -576,7 +598,7 @@ public class FuzzyMatchDialog extends BaseTransformDialog {
getData();
setComboBoxesLookup();
activeAlgorithm();
- activeGetCloserValue();
+ activeMatchMode();
focusTransformName();
BaseDialog.defaultShellHandling(shell, c -> ok(), c -> cancel());
@@ -595,9 +617,11 @@ public class FuzzyMatchDialog extends BaseTransformDialog {
wMainStreamField.setText(Const.NVL(input.getMainStreamField(), ""));
wLookupField.setText(Const.NVL(input.getLookupField(), ""));
wCaseSensitive.setSelection(input.isCaseSensitive());
- wGetCloserValue.setSelection(input.isCloserValue());
+ wMatchMode.setText(input.getMatchMode().getDescription());
wMinValue.setText(Const.NVL(input.getMinimalValue(), ""));
wMaxValue.setText(Const.NVL(input.getMaximalValue(), ""));
+ wMaxMatches.setText(
+ Const.NVL(input.getMaxMatches(), String.valueOf(FuzzyMatchMeta.DEFAULT_MAX_MATCHES)));
wMatchField.setText(Const.NVL(input.getOutputMatchField(), ""));
wValueField.setText(Const.NVL(input.getOutputValueField(), ""));
wSeparator.setText(Const.NVL(input.getSeparator(), ""));
@@ -631,9 +655,10 @@ public class FuzzyMatchDialog extends BaseTransformDialog {
input.setAlgorithm(lookupDescription(wAlgorithm.getText()));
input.setCaseSensitive(wCaseSensitive.getSelection());
- input.setCloserValue(wGetCloserValue.getSelection());
+ input.setMatchMode(FuzzyMatchMeta.MatchMode.lookupDescription(wMatchMode.getText()));
input.setMaximalValue(wMaxValue.getText());
input.setMinimalValue(wMinValue.getText());
+ input.setMaxMatches(wMaxMatches.getText());
input.setOutputMatchField(wMatchField.getText());
input.setOutputValueField(wValueField.getText());
@@ -700,31 +725,43 @@ public class FuzzyMatchDialog extends BaseTransformDialog {
}
}
- private void activeGetCloserValue() {
+ private void activeMatchMode() {
+ FuzzyMatchMeta.MatchMode matchMode =
+ FuzzyMatchMeta.MatchMode.lookupDescription(wMatchMode.getText());
FuzzyMatchMeta.Algorithm algorithm = lookupDescription(wAlgorithm.getText());
- boolean enableRange =
- (algorithm == LEVENSHTEIN
- || algorithm == NEEDLEMAN_WUNSH
- || algorithm == DAMERAU_LEVENSHTEIN
- || algorithm == JARO
- || algorithm == JARO_WINKLER
- || algorithm == PAIR_SIMILARITY)
- && !wGetCloserValue.getSelection();
-
- wlSeparator.setEnabled(enableRange);
- wSeparator.setEnabled(enableRange);
- wlValueField.setEnabled(wGetCloserValue.getSelection());
- wValueField.setEnabled(wGetCloserValue.getSelection());
+ boolean distanceOrSimilarity =
+ algorithm == LEVENSHTEIN
+ || algorithm == NEEDLEMAN_WUNSH
+ || algorithm == DAMERAU_LEVENSHTEIN
+ || algorithm == JARO
+ || algorithm == JARO_WINKLER
+ || algorithm == PAIR_SIMILARITY;
+
+ boolean multiMatch = matchMode != FuzzyMatchMeta.MatchMode.CLOSEST && distanceOrSimilarity;
+ wlMaxMatches.setEnabled(multiMatch);
+ wMaxMatches.setEnabled(multiMatch);
+
+ boolean enableSeparator =
+ matchMode == FuzzyMatchMeta.MatchMode.ALL_CONCAT && distanceOrSimilarity;
+ wlSeparator.setEnabled(enableSeparator);
+ wSeparator.setEnabled(enableSeparator);
+
+ // Value field is available for all match modes when a measure makes sense
+ wlValueField.setEnabled(true);
+ wValueField.setEnabled(true);
activeAddFields();
}
private void activeAddFields() {
+ FuzzyMatchMeta.MatchMode matchMode =
+ FuzzyMatchMeta.MatchMode.lookupDescription(wMatchMode.getText());
FuzzyMatchMeta.Algorithm algorithm = lookupDescription(wAlgorithm.getText());
boolean activate =
- wGetCloserValue.getSelection()
+ matchMode == FuzzyMatchMeta.MatchMode.CLOSEST
+ || matchMode == FuzzyMatchMeta.MatchMode.ALL_ROWS
|| algorithm == DOUBLE_METAPHONE
|| algorithm == SOUNDEX
|| algorithm == REFINED_SOUNDEX
@@ -746,8 +783,8 @@ public class FuzzyMatchDialog extends BaseTransformDialog {
|| algorithm == JARO_WINKLER
|| algorithm == PAIR_SIMILARITY);
- wlGetCloserValue.setEnabled(enable);
- wGetCloserValue.setEnabled(enable);
+ wlMatchMode.setEnabled(enable);
+ wMatchMode.setEnabled(enable);
wlMinValue.setEnabled(enable);
wMinValue.setEnabled(enable);
wlMaxValue.setEnabled(enable);
@@ -765,7 +802,7 @@ public class FuzzyMatchDialog extends BaseTransformDialog {
boolean enableCaseSensitive = (algorithm == LEVENSHTEIN || algorithm == DAMERAU_LEVENSHTEIN);
wlCaseSensitive.setEnabled(enableCaseSensitive);
wCaseSensitive.setEnabled(enableCaseSensitive);
- activeGetCloserValue();
+ activeMatchMode();
}
private void getlookup() {
diff --git a/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchMeta.java b/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchMeta.java
index 07e021e6ea..4bd759ff27 100644
--- a/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchMeta.java
+++ b/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchMeta.java
@@ -27,6 +27,7 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Objects;
+import lombok.AccessLevel;
import lombok.Getter;
import lombok.Setter;
import org.apache.commons.lang3.StringUtils;
@@ -71,6 +72,10 @@ public class FuzzyMatchMeta extends BaseTransformMeta<FuzzyMatch, FuzzyMatchData
private static final Class<?> PKG = FuzzyMatchMeta.class;
public static final String DEFAULT_SEPARATOR = ",";
+ public static final int DEFAULT_MAX_MATCHES = 10;
+
+ /** Hard safety cap for Top-K to avoid runaway memory / row explosion. */
+ public static final int HARD_MAX_MATCHES = 100;
/** Algorithms type */
@HopMetadataProperty(key = "algorithm", storeWithCode = true)
@@ -111,10 +116,24 @@ public class FuzzyMatchMeta extends BaseTransformMeta<FuzzyMatch, FuzzyMatchData
@HopMetadataProperty(key = "separator")
private String separator;
- /** get closer matching value */
+ /**
+ * Legacy flag kept for backward compatibility with existing pipelines. Prefer {@link #matchMode}.
+ */
+ @Getter(AccessLevel.NONE)
+ @Setter(AccessLevel.NONE)
@HopMetadataProperty(key = "closervalue")
private boolean closerValue;
+ /** How multiple matches are returned. */
+ @Getter(AccessLevel.NONE)
+ @Setter(AccessLevel.NONE)
+ @HopMetadataProperty(key = "matchMode", storeWithCode = true)
+ private MatchMode matchMode;
+
+ /** Max matches for Top-K (ALL_ROWS / ALL_CONCAT). Default 10, hard-capped at 100. */
+ @HopMetadataProperty(key = "maxMatches")
+ private String maxMatches;
+
/** return these field values from lookup */
@HopMetadataProperty(groupKey = "lookup", key = "value")
private List<FMLookupValue> lookupValues;
@@ -123,6 +142,9 @@ public class FuzzyMatchMeta extends BaseTransformMeta<FuzzyMatch, FuzzyMatchData
super();
this.algorithm = NONE;
this.lookupValues = new ArrayList<>();
+ this.matchMode = MatchMode.CLOSEST;
+ this.closerValue = true;
+ this.maxMatches = String.valueOf(DEFAULT_MAX_MATCHES);
}
public FuzzyMatchMeta(FuzzyMatchMeta m) {
@@ -136,7 +158,8 @@ public class FuzzyMatchMeta extends BaseTransformMeta<FuzzyMatch, FuzzyMatchData
this.minimalValue = m.minimalValue;
this.maximalValue = m.maximalValue;
this.separator = m.separator;
- this.closerValue = m.closerValue;
+ setMatchMode(m.getMatchMode());
+ this.maxMatches = m.maxMatches;
m.lookupValues.forEach(v -> this.lookupValues.add(new FMLookupValue(v)));
}
@@ -149,7 +172,8 @@ public class FuzzyMatchMeta extends BaseTransformMeta<FuzzyMatch, FuzzyMatchData
public void setDefault() {
algorithm = NONE;
separator = DEFAULT_SEPARATOR;
- closerValue = true;
+ setMatchMode(MatchMode.CLOSEST);
+ maxMatches = String.valueOf(DEFAULT_MAX_MATCHES);
minimalValue = "0";
maximalValue = "1";
caseSensitive = false;
@@ -159,6 +183,50 @@ public class FuzzyMatchMeta extends BaseTransformMeta<FuzzyMatch, FuzzyMatchData
outputValueField = BaseMessages.getString(PKG, "FuzzyMatchMeta.OutputValueFieldname");
}
+ public MatchMode getMatchMode() {
+ if (matchMode != null) {
+ return matchMode;
+ }
+ // Legacy pipelines only have closer value
+ return closerValue ? MatchMode.CLOSEST : MatchMode.ALL_CONCAT;
+ }
+
+ public void setMatchMode(MatchMode matchMode) {
+ this.matchMode = matchMode == null ? MatchMode.CLOSEST : matchMode;
+ this.closerValue = this.matchMode == MatchMode.CLOSEST;
+ }
+
+ /** use {@link #getMatchMode()} */
+ public boolean isCloserValue() {
+ return getMatchMode() == MatchMode.CLOSEST;
+ }
+
+ /** use {@link #setMatchMode(MatchMode)} */
+ public void setCloserValue(boolean closerValue) {
+ this.closerValue = closerValue;
+ // When loading legacy XML, matchMode may still be unset.
+ if (matchMode == null || matchMode == MatchMode.CLOSEST || matchMode == MatchMode.ALL_CONCAT) {
+ this.matchMode = closerValue ? MatchMode.CLOSEST : MatchMode.ALL_CONCAT;
+ }
+ }
+
+ public boolean isAllRowsMode() {
+ return getMatchMode() == MatchMode.ALL_ROWS;
+ }
+
+ public boolean isAllConcatMode() {
+ return getMatchMode() == MatchMode.ALL_CONCAT;
+ }
+
+ public boolean supportsAdditionalFields() {
+ return isCloserValue()
+ || isAllRowsMode()
+ || getAlgorithm() == DOUBLE_METAPHONE
+ || getAlgorithm() == SOUNDEX
+ || getAlgorithm() == REFINED_SOUNDEX
+ || getAlgorithm() == METAPHONE;
+ }
+
@Override
public void getFields(
IRowMeta inputRowMeta,
@@ -175,36 +243,34 @@ public class FuzzyMatchMeta extends BaseTransformMeta<FuzzyMatch, FuzzyMatchData
inputRowMeta.addValueMeta(v);
String mainField = variables.resolve(getOutputValueField());
- if (StringUtils.isNotEmpty(mainField) && isCloserValue()) {
- switch (getAlgorithm()) {
- case NONE:
- throw new HopTransformException("Please specify the matching algorithm to use");
- case LEVENSHTEIN, DAMERAU_LEVENSHTEIN, NEEDLEMAN_WUNSH:
- // Distance algorithms return an integer measure
- v = new ValueMetaInteger(mainField);
- v.setLength(IValueMeta.DEFAULT_INTEGER_LENGTH);
- break;
- case JARO, JARO_WINKLER, PAIR_SIMILARITY:
- v = new ValueMetaNumber(mainField);
- break;
- default:
- // Phonetic algorithms
- v = new ValueMetaString(mainField);
- break;
+ if (StringUtils.isNotEmpty(mainField)) {
+ if (isAllConcatMode()) {
+ // Concatenated measures are always a string list
+ v = new ValueMetaString(mainField);
+ } else {
+ switch (getAlgorithm()) {
+ case NONE:
+ throw new HopTransformException("Please specify the matching algorithm to use");
+ case LEVENSHTEIN, DAMERAU_LEVENSHTEIN, NEEDLEMAN_WUNSH:
+ // Distance algorithms return an integer measure
+ v = new ValueMetaInteger(mainField);
+ v.setLength(IValueMeta.DEFAULT_INTEGER_LENGTH);
+ break;
+ case JARO, JARO_WINKLER, PAIR_SIMILARITY:
+ v = new ValueMetaNumber(mainField);
+ break;
+ default:
+ // Phonetic algorithms
+ v = new ValueMetaString(mainField);
+ break;
+ }
}
v.setStorageType(IValueMeta.STORAGE_TYPE_NORMAL);
v.setOrigin(name);
inputRowMeta.addValueMeta(v);
}
- boolean activateAdditionalFields =
- isCloserValue()
- || (getAlgorithm() == DOUBLE_METAPHONE)
- || (getAlgorithm() == SOUNDEX)
- || (getAlgorithm() == REFINED_SOUNDEX)
- || (getAlgorithm() == METAPHONE);
-
- if (activateAdditionalFields) {
+ if (supportsAdditionalFields()) {
if (info != null && info.length == 1 && info[0] != null) {
for (FMLookupValue lookupValue : lookupValues) {
v = info[0].searchValueMeta(lookupValue.getName());
@@ -475,6 +541,35 @@ public class FuzzyMatchMeta extends BaseTransformMeta<FuzzyMatch, FuzzyMatchData
return ioMeta;
}
+ @Getter
+ public enum MatchMode implements IEnumHasCodeAndDescription {
+ CLOSEST("closest", BaseMessages.getString(PKG, "FuzzyMatchMeta.matchMode.Closest")),
+ ALL_ROWS("all_rows", BaseMessages.getString(PKG, "FuzzyMatchMeta.matchMode.AllRows")),
+ ALL_CONCAT("all_concat", BaseMessages.getString(PKG, "FuzzyMatchMeta.matchMode.AllConcat")),
+ ;
+ private final String code;
+ private final String description;
+
+ MatchMode(String code, String description) {
+ this.code = code;
+ this.description = description;
+ }
+
+ public static String[] getDescriptions() {
+ return Arrays.stream(MatchMode.values())
+ .map(MatchMode::getDescription)
+ .toArray(String[]::new);
+ }
+
+ public static MatchMode lookupDescription(String description) {
+ return IEnumHasCodeAndDescription.lookupDescription(MatchMode.class, description, CLOSEST);
+ }
+
+ public static MatchMode lookupCode(String code) {
+ return IEnumHasCode.lookupCode(MatchMode.class, code, CLOSEST);
+ }
+ }
+
@Getter
public enum Algorithm implements IEnumHasCodeAndDescription {
NONE("", ""),
diff --git a/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/LetterPairSimilarity.java b/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/LetterPairSimilarity.java
index c94e41c60e..afcc5355d5 100644
--- a/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/LetterPairSimilarity.java
+++ b/plugins/transforms/fuzzymatch/src/main/java/org/apache/hop/pipeline/transforms/fuzzymatch/LetterPairSimilarity.java
@@ -18,8 +18,11 @@
package org.apache.hop.pipeline.transforms.fuzzymatch;
import java.util.ArrayList;
+import java.util.Collections;
+import lombok.experimental.UtilityClass;
import org.apache.hop.core.util.Utils;
+@UtilityClass
public class LetterPairSimilarity {
/*
@@ -64,9 +67,7 @@ public class LetterPairSimilarity {
for (String word : words) {
// Find the pairs of characters
String[] pairsInWord = letterPairs(word);
- for (String s : pairsInWord) {
- allPairs.add(s);
- }
+ Collections.addAll(allPairs, pairsInWord);
}
return allPairs;
}
@@ -74,7 +75,7 @@ public class LetterPairSimilarity {
/**
* @return lexical similarity value in the range [0,1]
*/
- public static double getSimiliarity(String str1, String str2) {
+ public static double getSimilarity(String str1, String str2) {
if (Utils.isEmpty(str1) && Utils.isEmpty(str2)) {
return 1.0;
}
diff --git a/plugins/transforms/fuzzymatch/src/main/resources/org/apache/hop/pipeline/transforms/fuzzymatch/messages/messages_en_US.properties b/plugins/transforms/fuzzymatch/src/main/resources/org/apache/hop/pipeline/transforms/fuzzymatch/messages/messages_en_US.properties
index 8fe1906605..16f581f2d5 100644
--- a/plugins/transforms/fuzzymatch/src/main/resources/org/apache/hop/pipeline/transforms/fuzzymatch/messages/messages_en_US.properties
+++ b/plugins/transforms/fuzzymatch/src/main/resources/org/apache/hop/pipeline/transforms/fuzzymatch/messages/messages_en_US.properties
@@ -15,7 +15,7 @@
# limitations under the License.
#
-FuzzyMatch.Description=Finding approximate matches to a string using matching algorithms.\nRead a field from a main stream and output approximative value from lookup stream.
+FuzzyMatch.Description=Finding approximate matches to a string using matching algorithms.\nRead a field from a main stream and output approximative value(s) from the lookup stream.\nSupports closest match, multiple matches as separate rows (Top-K), or concatenated match lists with measure values.
FuzzyMatch.Error.JavaHeap=Java heap space\! {0}
FuzzyMatch.Error.LookupStreamFieldMissing=Lookup stream fieldname is missing\!
FuzzyMatch.Error.MainStreamFieldMissing=Main stream fieldname is missing\!
@@ -27,6 +27,8 @@ FuzzyMatch.Log.ErrorInTransformRunning=Because of an error, this transform can''
FuzzyMatch.Log.LineNumber=linenr
FuzzyMatch.Log.MaximalDistance=Maximal distance is {0}
FuzzyMatch.Log.MaximalSimilarity=Maximal similarity is {0}
+FuzzyMatch.Log.MaxMatches=Max matches (Top-K) is {0}
+FuzzyMatch.Log.MaxMatchesCapped=Configured max matches {0} exceeds hard limit, capped to {1}
FuzzyMatch.Log.MinimalDistance=Minimal distance is {0}
FuzzyMatch.Log.MinimalSimilarity=Minimal similarity is {0}
FuzzyMatch.Log.NoLookupTransformSpecified=No lookup transform specified.
@@ -53,6 +55,10 @@ FuzzyMatchDialog.Fields.Tab=Fields
FuzzyMatchDialog.General.Tab=General
FuzzyMatchDialog.getCloserValue.Label=Get closer value
FuzzyMatchDialog.getCloserValue.Tooltip=Get closer value.\nFor distance algorithms like levenshtein and DamerauLevenshtein, the closer value have the smaller distance.\nFor similitary algorithms like Jaro or JaroWinkler, the closer value have the higher similarity.
+FuzzyMatchDialog.matchMode.Label=Match mode
+FuzzyMatchDialog.matchMode.Tooltip=Closest match\: return the single best match.\nAll matches (separate rows)\: emit one output row per match (Top-K), each with its own measure value.\nAll matches (concatenated)\: join matches (and measures) with the values separator.
+FuzzyMatchDialog.maxMatches.Label=Max matches
+FuzzyMatchDialog.maxMatches.Tooltip=Top-K limit for multi-match modes. Default 10, hard-capped at 100 to avoid row explosion.
FuzzyMatchDialog.GetLookupFields.Button=Get fields
FuzzyMatchDialog.Group.Lookup.Label=Lookup stream (source)
FuzzyMatchDialog.Group.MainStreamGroup.Label=Main stream
@@ -66,8 +72,9 @@ FuzzyMatchDialog.maxValue.Tooltip=The maximal value is a distance (integer) for
FuzzyMatchDialog.minValue.Label=Minimal value
FuzzyMatchDialog.minValue.Tooltip=The minimal value is a distance (integer) for algorithms like levenshtein and DamerauLevenshtein\n and similarity (decimal between 0.0 and 1.0) for algorithms like Jaro and JaroWinkler.
FuzzyMatchDialog.NotTransformSpecified.DialogMessage=You must specify a source transform\!
-FuzzyMatchDialog.ReturnFields.Label=Specify the additional fields (not mandatory) to retrieve from lookup stream
+FuzzyMatchDialog.ReturnFields.Label=Specify the additional fields (not mandatory) to retrieve from the lookup stream.\nAvailable for Closest match, All matches (separate rows), and phonetic algorithms.
FuzzyMatchDialog.separator.Label=Values separator
+FuzzyMatchDialog.separator.Tooltip=Separator used to join match values and measure values in All matches (concatenated) mode.
FuzzyMatchDialog.Shell.Title=Fuzzy match
FuzzyMatchDialog.SourceTransform.Label=Lookup transform
FuzzyMatchDialog.TransformCanNotFound.DialogMessage=transform [{0}] can''t be found\: please select an existing Lookup transform\!
@@ -76,7 +83,7 @@ FuzzyMatchDialog.TransformName.Label=Transform name
FuzzyMatchDialog.TransformNameRequired.DialogMessage=Please select a transform name\!
FuzzyMatchDialog.TransformNameRequired.DialogTitle=ERROR
FuzzyMatchDialog.valueField.Label=Value field
-FuzzyMatchDialog.valueField.Tooltip=The value is the distance (integer) for algorithms like levenshtein and DamerauLevenshtein\n and similarity (decimal between 0.0 and 1.0) for algorithms like Jaro and JaroWinkler and string for metaphone.
+FuzzyMatchDialog.valueField.Tooltip=The measure for each match\: distance (integer) for algorithms like Levenshtein and DamerauLevenshtein,\nsimilarity (decimal between 0.0 and 1.0) for Jaro, JaroWinkler and Pair letters similarity,\nand a phonetic encoding string for Metaphone/SoundEx.\nIn All matches (concatenated) mode, multiple measures are joined with the values separator.
FuzzyMatchDialog.wlLookupField.Label=Lookup field
FuzzyMatchDialog.wlMainStreamField.Label=Main stream field
FuzzyMatchMeta.algorithm.DamerauLevenshtein=Damerau Levenshtein
@@ -111,3 +118,6 @@ FuzzyMatchMeta.InfoStream.Description=These rows are loaded into memory and used
FuzzyMatchMeta.keyword=fuzzy,levenshtein,soundex,similarity,approximate
FuzzyMatchMeta.OutputMatchFieldname=match
FuzzyMatchMeta.OutputValueFieldname=measure value
+FuzzyMatchMeta.matchMode.Closest=Closest match
+FuzzyMatchMeta.matchMode.AllRows=All matches (separate rows)
+FuzzyMatchMeta.matchMode.AllConcat=All matches (concatenated)
diff --git a/plugins/transforms/fuzzymatch/src/main/resources/org/apache/hop/pipeline/transforms/fuzzymatch/messages/messages_zh_CN.properties b/plugins/transforms/fuzzymatch/src/main/resources/org/apache/hop/pipeline/transforms/fuzzymatch/messages/messages_zh_CN.properties
index 35b21c92af..aff08304fc 100644
--- a/plugins/transforms/fuzzymatch/src/main/resources/org/apache/hop/pipeline/transforms/fuzzymatch/messages/messages_zh_CN.properties
+++ b/plugins/transforms/fuzzymatch/src/main/resources/org/apache/hop/pipeline/transforms/fuzzymatch/messages/messages_zh_CN.properties
@@ -17,7 +17,7 @@
#
#
-FuzzyMatch.Description=\u4F7F\u7528\u5339\u914D\u7B97\u6CD5\u67E5\u627E\u5B57\u7B26\u4E32\u7684\u8FD1\u4F3C\u5339\u914D.\n\u6839\u636E\u4E3B\u8F93\u5165\u6D41\u4E2D\u8BFB\u53D6\u5230\u7684\u5B57\u6BB5\u5728\u67E5\u627E\u6D41\u4E2D\u68C0\u7D22\u5E76\u8F93\u51FA\u8FD1\u4F3C\u503C.
+FuzzyMatch.Description=\u4F7F\u7528\u5339\u914D\u7B97\u6CD5\u67E5\u627E\u5B57\u7B26\u4E32\u7684\u8FD1\u4F3C\u5339\u914D.\n\u6839\u636E\u4E3B\u8F93\u5165\u6D41\u4E2D\u8BFB\u53D6\u5230\u7684\u5B57\u6BB5\u5728\u67E5\u627E\u6D41\u4E2D\u68C0\u7D22\u5E76\u8F93\u51FA\u8FD1\u4F3C\u503C.\n\u652F\u6301\u6700\u8FD1\u5339\u914D\u3001\u591A\u5339\u914D\u5206\u884C\u8F93\u51FA\uff08Top-K\uff09\u6216\u62FC\u63A5\u5217\u8868\uff08\u542B\u8DDD\u79BB/\u76F8\u4F3C\u5EA6\uff09.
FuzzyMatch.Error.JavaHeap=Java heap space\! {0}
FuzzyMatch.Error.LookupStreamFieldMissing=Lookup stream fieldname is missing\!
FuzzyMatch.Error.MainStreamFieldMissing=Main stream fieldname is missing\!
@@ -29,6 +29,8 @@ FuzzyMatch.Log.ErrorInTransformRunning=Because of an error, this transform can''
FuzzyMatch.Log.LineNumber=\u884C\u53F7
FuzzyMatch.Log.MaximalDistance=Maximal distance is {0}
FuzzyMatch.Log.MaximalSimilarity=Maximal similarity is {0}
+FuzzyMatch.Log.MaxMatches=Max matches (Top-K) is {0}
+FuzzyMatch.Log.MaxMatchesCapped=Configured max matches {0} exceeds hard limit, capped to {1}
FuzzyMatch.Log.MinimalDistance=Minimal distance is {0}
FuzzyMatch.Log.MinimalSimilarity=Minimal similarity is {0}
FuzzyMatch.Log.NoLookupTransformSpecified=No lookup transform specified.
@@ -55,6 +57,10 @@ FuzzyMatchDialog.Fields.Tab=\u5B57\u6BB5
FuzzyMatchDialog.General.Tab=\u4E00\u822C
FuzzyMatchDialog.getCloserValue.Label=\u83B7\u53D6\u8FD1\u4F3C\u503C\:
FuzzyMatchDialog.getCloserValue.Tooltip=Get closer value.\nFor distance algorithms like levenshtein and DamerauLevenshtein, the closer value have the smaller distance.\nFor similitary algorithms like Jaro or JaroWinkler, the closer value have the higher similarity.
+FuzzyMatchDialog.matchMode.Label=\u5339\u914D\u6A21\u5F0F\:
+FuzzyMatchDialog.matchMode.Tooltip=\u6700\u8FD1\u5339\u914D\uff1a\u53ea\u8fd4\u56de\u6700\u4f18\u4e00\u4e2a\u5339\u914d\u3002\n\u5168\u90e8\u5339\u914d\uff08\u5206\u884c\uff09\uff1a\u6bcf\u4e2a\u5339\u914d\u8f93\u51fa\u4e00\u884c\uff08Top-K\uff09\uff0c\u5e76\u5e26\u72ec\u7acb\u8ddd\u79bb/\u76f8\u4f3c\u5ea6\u3002\n\u5168\u90e8\u5339\u914d\uff08\u62fc\u63a5\uff09\uff1a\u7528\u5206\u9694\u7b26\u62fc\u63a5\u591a\u4e2a\u5339\u914d\u503c\u4e0e measure\u3002
+FuzzyMatchDialog.maxMatches.Label=\u6700\u5927\u5339\u914D\u6570\:
+FuzzyMatchDialog.maxMatches.Tooltip=Top-K \u4e0a\u9650\uff0c\u9ed8\u8ba4 10\uff0c\u786c\u4e0a\u9650 100\uff0c\u9632\u6b62\u884c\u6570\u7206\u70b8\u3002
FuzzyMatchDialog.GetLookupFields.Button=\ \u83B7\u53D6\u5B57\u6BB5
FuzzyMatchDialog.Group.Lookup.Label=\u5339\u914D\u6D41 (\u6E90)\:
FuzzyMatchDialog.Group.MainStreamGroup.Label=\u4E3B\u6570\u636E\u6D41
@@ -70,6 +76,7 @@ FuzzyMatchDialog.minValue.Tooltip=The minimal value is a distance (integer) for
FuzzyMatchDialog.NotTransformSpecified.DialogMessage=You must specify a source transform\!
FuzzyMatchDialog.ReturnFields.Label=\u6307\u5B9A\u989D\u5916\u7684\u5728\u5339\u914D\u6D41\u4E2D\u7684\u5B57\u6BB5\:
FuzzyMatchDialog.separator.Label=\u503C\u5206\u9694\u7B26\:
+FuzzyMatchDialog.separator.Tooltip=\u5728\u300C\u5168\u90E8\u5339\u914D\uff08\u62FC\u63A5\uff09\u300D\u6A21\u5F0F\u4E0B\u7528\u4E8E\u62FC\u63A5\u5339\u914D\u503C\u4E0E measure \u7684\u5206\u9694\u7B26\u3002
FuzzyMatchDialog.Shell.Title=\u6A21\u7CCA\u5339\u914D
FuzzyMatchDialog.SourceTransform.Label=\u5339\u914D Transform:
FuzzyMatchDialog.TransformCanNotFound.DialogMessage=transform [{0}] can''t be found\: please select an existing Lookup transform\!
@@ -78,7 +85,7 @@ FuzzyMatchDialog.TransformName.Label=Transform \u540D\u79F0\:
FuzzyMatchDialog.TransformNameRequired.DialogMessage=Please select a transform name\!
FuzzyMatchDialog.TransformNameRequired.DialogTitle=ERROR
FuzzyMatchDialog.valueField.Label=\u503C\u5B57\u6BB5\:
-FuzzyMatchDialog.valueField.Tooltip=The value is the distance (integer) for algorithms like levenshtein and DamerauLevenshtein\n and similarity (decimal between 0.0 and 1.0) for algorithms like Jaro and JaroWinkler and string for metaphone.
+FuzzyMatchDialog.valueField.Tooltip=\u6BCF\u4E2A\u5339\u914D\u7684 measure\uff1aLevenshtein \u7B49\u8DDD\u79BB\u7B97\u6CD5\u4E3A\u6574\u6570\uff1bJaro \u7B49\u76F8\u4F3C\u5EA6\u7B97\u6CD5\u4E3A 0.0\u20131.0\u7684\u5C0F\u6570\uff1bMetaphone/SoundEx \u4E3A\u7F16\u7801\u5B57\u7B26\u4E32\u3002\n\u5728\u300C\u5168\u90E8\u5339\u914D\uff08\u62FC\u63A5\uff09\u300D\u6A21\u5F0F\u4E0B\uff0C\u591A\u4E2A measure \u4F1A\u7528\u5206\u9694\u7B26\u62FC\u63A5\u3002
FuzzyMatchDialog.wlLookupField.Label=\u5339\u914D\u5B57\u6BB5\:
FuzzyMatchDialog.wlMainStreamField.Label=\u4E3B\u8981\u6D41\u5B57\u6BB5\:
FuzzyMatchMeta.algorithm.DamerauLevenshtein=Damerau Levenshtein
@@ -113,3 +120,6 @@ FuzzyMatchMeta.InfoStream.Description=These rows are loaded into memory and used
FuzzyMatchMeta.keyword=fuzzy,match
FuzzyMatchMeta.OutputMatchFieldname=match
FuzzyMatchMeta.OutputValueFieldname=measure value
+FuzzyMatchMeta.matchMode.Closest=\u6700\u8FD1\u5339\u914D
+FuzzyMatchMeta.matchMode.AllRows=\u5168\u90E8\u5339\u914D\uff08\u5206\u884C\uff09
+FuzzyMatchMeta.matchMode.AllConcat=\u5168\u90E8\u5339\u914D\uff08\u62FC\u63A5\uff09
diff --git a/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchDataTest.java b/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchDataTest.java
new file mode 100644
index 0000000000..0071f75b21
--- /dev/null
+++ b/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchDataTest.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.pipeline.transforms.fuzzymatch;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import org.junit.jupiter.api.Test;
+
+/** Unit tests for {@link FuzzyMatchData}. */
+class FuzzyMatchDataTest {
+
+ @Test
+ void testDefaults() {
+ FuzzyMatchData data = new FuzzyMatchData();
+
+ assertNotNull(data.look);
+ assertTrue(data.look.isEmpty());
+ assertEquals(-1, data.indexOfMainField);
+ assertFalse(data.addValueFieldName);
+ assertFalse(data.addAdditionalFields);
+ assertEquals("", data.valueSeparator);
+ assertEquals(1, data.nrCachedFields);
+ assertEquals(FuzzyMatchMeta.DEFAULT_MAX_MATCHES, data.maxMatches);
+ }
+}
diff --git a/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchMetaTest.java b/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchMetaTest.java
index 883fba5fb4..ce68284156 100644
--- a/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchMetaTest.java
+++ b/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchMetaTest.java
@@ -19,9 +19,13 @@ package org.apache.hop.pipeline.transforms.fuzzymatch;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.util.List;
+import org.apache.hop.core.exception.HopTransformException;
import org.apache.hop.core.row.IRowMeta;
import org.apache.hop.core.row.IValueMeta;
import org.apache.hop.core.row.RowMeta;
@@ -46,6 +50,8 @@ class FuzzyMatchMetaTest {
assertEquals("measure value", meta.getOutputValueField());
assertFalse(meta.isCaseSensitive());
assertTrue(meta.isCloserValue());
+ assertEquals(FuzzyMatchMeta.MatchMode.CLOSEST, meta.getMatchMode());
+ assertEquals("10", meta.getMaxMatches());
assertEquals("0", meta.getMinimalValue());
assertEquals("1", meta.getMaximalValue());
assertEquals(",", meta.getSeparator());
@@ -55,6 +61,163 @@ class FuzzyMatchMetaTest {
assertEquals("lookupName", meta.getLookupValues().get(0).getRename());
}
+ @Test
+ void testSetDefault() {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_ROWS);
+ meta.setMaxMatches("3");
+ meta.setCaseSensitive(true);
+
+ meta.setDefault();
+
+ assertEquals(FuzzyMatchMeta.Algorithm.NONE, meta.getAlgorithm());
+ assertEquals(FuzzyMatchMeta.MatchMode.CLOSEST, meta.getMatchMode());
+ assertTrue(meta.isCloserValue());
+ assertEquals(String.valueOf(FuzzyMatchMeta.DEFAULT_MAX_MATCHES), meta.getMaxMatches());
+ assertEquals(FuzzyMatchMeta.DEFAULT_SEPARATOR, meta.getSeparator());
+ assertEquals("0", meta.getMinimalValue());
+ assertEquals("1", meta.getMaximalValue());
+ assertFalse(meta.isCaseSensitive());
+ assertNotNull(meta.getOutputMatchField());
+ assertNotNull(meta.getOutputValueField());
+ }
+
+ @Test
+ void testCloneCopiesMatchModeAndLookupValues() {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.JARO);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_ROWS);
+ meta.setMaxMatches("7");
+ meta.setMainStreamField("main");
+ meta.setLookupField("lookup");
+ meta.setLookupValues(List.of(new FuzzyMatchMeta.FMLookupValue("id", "rid")));
+
+ FuzzyMatchMeta copy = meta.clone();
+
+ assertNotSame(meta, copy);
+ assertEquals(FuzzyMatchMeta.MatchMode.ALL_ROWS, copy.getMatchMode());
+ assertEquals("7", copy.getMaxMatches());
+ assertEquals("main", copy.getMainStreamField());
+ assertEquals(1, copy.getLookupValues().size());
+ assertEquals("id", copy.getLookupValues().get(0).getName());
+ assertEquals("rid", copy.getLookupValues().get(0).getRename());
+ assertNotSame(meta.getLookupValues().get(0), copy.getLookupValues().get(0));
+ }
+
+ @Test
+ void testLegacyCloserValueMapsToMatchMode() {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setCloserValue(false);
+ assertEquals(FuzzyMatchMeta.MatchMode.ALL_CONCAT, meta.getMatchMode());
+ assertTrue(meta.isAllConcatMode());
+ assertFalse(meta.isCloserValue());
+
+ meta.setCloserValue(true);
+ assertEquals(FuzzyMatchMeta.MatchMode.CLOSEST, meta.getMatchMode());
+ assertTrue(meta.isCloserValue());
+ assertFalse(meta.isAllRowsMode());
+ }
+
+ @Test
+ void testSetCloserValueDoesNotOverwriteAllRows() {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_ROWS);
+ meta.setCloserValue(true);
+
+ assertEquals(FuzzyMatchMeta.MatchMode.ALL_ROWS, meta.getMatchMode());
+ assertTrue(meta.isAllRowsMode());
+ assertFalse(meta.isCloserValue());
+ }
+
+ @Test
+ void testSetMatchModeNullFallsBackToClosest() {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setMatchMode(null);
+ assertEquals(FuzzyMatchMeta.MatchMode.CLOSEST, meta.getMatchMode());
+ assertTrue(meta.isCloserValue());
+ }
+
+ @Test
+ void testSupportsAdditionalFields() {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.CLOSEST);
+ assertTrue(meta.supportsAdditionalFields());
+
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_ROWS);
+ assertTrue(meta.supportsAdditionalFields());
+
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_CONCAT);
+ assertFalse(meta.supportsAdditionalFields());
+
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.SOUNDEX);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_CONCAT);
+ assertTrue(meta.supportsAdditionalFields());
+ }
+
+ @Test
+ void testGetFieldsAllRowsIncludesMeasureAndAdditionalFields() throws Exception {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_ROWS);
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+ meta.setMaxMatches("10");
+ meta.setLookupValues(List.of(new FuzzyMatchMeta.FMLookupValue("id", "lookup_id")));
+
+ IRowMeta inputRowMeta = new RowMeta();
+ IRowMeta lookupRowMeta = new RowMeta();
+ lookupRowMeta.addValueMeta(new ValueMetaString("id"));
+
+ meta.getFields(
+ inputRowMeta, "FuzzyMatch", new IRowMeta[] {lookupRowMeta}, null, new Variables(), null);
+
+ assertNotNull(inputRowMeta.searchValueMeta("match"));
+ IValueMeta measure = inputRowMeta.searchValueMeta("measure value");
+ assertNotNull(measure);
+ assertEquals(IValueMeta.TYPE_INTEGER, measure.getType());
+ assertNotNull(inputRowMeta.searchValueMeta("lookup_id"));
+ }
+
+ @Test
+ void testGetFieldsConcatMeasureIsStringAndSkipsAdditionalFields() throws Exception {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_CONCAT);
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+ meta.setLookupValues(List.of(new FuzzyMatchMeta.FMLookupValue("id", "lookup_id")));
+
+ IRowMeta inputRowMeta = new RowMeta();
+ IRowMeta lookupRowMeta = new RowMeta();
+ lookupRowMeta.addValueMeta(new ValueMetaString("id"));
+
+ meta.getFields(
+ inputRowMeta, "FuzzyMatch", new IRowMeta[] {lookupRowMeta}, null, new Variables(), null);
+
+ IValueMeta measure = inputRowMeta.searchValueMeta("measure value");
+ assertNotNull(measure);
+ assertEquals(IValueMeta.TYPE_STRING, measure.getType());
+ assertNull(inputRowMeta.searchValueMeta("lookup_id"));
+ }
+
+ @Test
+ void testGetFieldsWithoutMeasureFieldName() throws Exception {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_ROWS);
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("");
+
+ IRowMeta inputRowMeta = new RowMeta();
+ meta.getFields(inputRowMeta, "FuzzyMatch", null, null, new Variables(), null);
+
+ assertEquals(1, inputRowMeta.size());
+ assertEquals("match", inputRowMeta.getValueMeta(0).getName());
+ }
+
@Test
void testGetFieldsDistanceMeasureIsInteger() throws Exception {
for (FuzzyMatchMeta.Algorithm algorithm :
@@ -80,6 +243,89 @@ class FuzzyMatchMetaTest {
}
}
+ @Test
+ void testGetFieldsSimilarityMeasureIsNumber() throws Exception {
+ for (FuzzyMatchMeta.Algorithm algorithm :
+ List.of(
+ FuzzyMatchMeta.Algorithm.JARO,
+ FuzzyMatchMeta.Algorithm.JARO_WINKLER,
+ FuzzyMatchMeta.Algorithm.PAIR_SIMILARITY)) {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.CLOSEST);
+ meta.setAlgorithm(algorithm);
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+
+ IRowMeta inputRowMeta = new RowMeta();
+ meta.getFields(inputRowMeta, "FuzzyMatch", null, null, new Variables(), null);
+
+ IValueMeta measure = inputRowMeta.searchValueMeta("measure value");
+ assertNotNull(measure);
+ assertEquals(
+ IValueMeta.TYPE_NUMBER,
+ measure.getType(),
+ "Algorithm " + algorithm + " should output Number measure value");
+ }
+ }
+
+ @Test
+ void testGetFieldsPhoneticMeasureIsString() throws Exception {
+ for (FuzzyMatchMeta.Algorithm algorithm :
+ List.of(
+ FuzzyMatchMeta.Algorithm.METAPHONE,
+ FuzzyMatchMeta.Algorithm.DOUBLE_METAPHONE,
+ FuzzyMatchMeta.Algorithm.SOUNDEX,
+ FuzzyMatchMeta.Algorithm.REFINED_SOUNDEX)) {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.CLOSEST);
+ meta.setAlgorithm(algorithm);
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+
+ IRowMeta inputRowMeta = new RowMeta();
+ meta.getFields(inputRowMeta, "FuzzyMatch", null, null, new Variables(), null);
+
+ IValueMeta measure = inputRowMeta.searchValueMeta("measure value");
+ assertNotNull(measure);
+ assertEquals(
+ IValueMeta.TYPE_STRING,
+ measure.getType(),
+ "Algorithm " + algorithm + " should output String measure value");
+ }
+ }
+
+ @Test
+ void testGetFieldsThrowsWhenAlgorithmMissing() {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.CLOSEST);
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.NONE);
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+
+ assertThrows(
+ HopTransformException.class,
+ () -> meta.getFields(new RowMeta(), "FuzzyMatch", null, null, new Variables(), null));
+ }
+
+ @Test
+ void testGetFieldsThrowsWhenLookupValueMissingInInfo() {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.CLOSEST);
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+ meta.setLookupValues(List.of(new FuzzyMatchMeta.FMLookupValue("missing", null)));
+
+ IRowMeta info = new RowMeta();
+ info.addValueMeta(new ValueMetaString("other"));
+
+ assertThrows(
+ HopTransformException.class,
+ () ->
+ meta.getFields(
+ new RowMeta(), "FuzzyMatch", new IRowMeta[] {info}, null, new Variables(), null));
+ }
+
@Test
void testGetFieldsRename() throws Exception {
FuzzyMatchMeta meta = new FuzzyMatchMeta();
@@ -92,33 +338,70 @@ class FuzzyMatchMetaTest {
String newName = "new_name";
String noChangeName = "noChangeName";
- // lookup name="oldName", rename="newName"
FuzzyMatchMeta.FMLookupValue lookupValue = new FuzzyMatchMeta.FMLookupValue(oldName, newName);
FuzzyMatchMeta.FMLookupValue noChange = new FuzzyMatchMeta.FMLookupValue(noChangeName, null);
meta.setLookupValues(List.of(lookupValue, noChange));
- // input main row meta
IRowMeta inputRowMeta = new RowMeta();
-
- // lookup info row meta
IRowMeta lookupRowMeta = new RowMeta();
lookupRowMeta.addValueMeta(new ValueMetaString(oldName));
lookupRowMeta.addValueMeta(new ValueMetaString(noChangeName));
- IRowMeta[] info = new IRowMeta[] {lookupRowMeta};
-
- // execute getFields methods.
- meta.getFields(inputRowMeta, "FuzzyMatch", info, null, new Variables(), null);
+ meta.getFields(
+ inputRowMeta, "FuzzyMatch", new IRowMeta[] {lookupRowMeta}, null, new Variables(), null);
- // valid rename
IValueMeta result = inputRowMeta.searchValueMeta(newName);
assertNotNull(result);
assertEquals(newName, result.getName());
assertEquals("FuzzyMatch", result.getOrigin());
- // not change.
result = inputRowMeta.searchValueMeta(noChangeName);
assertNotNull(result);
assertEquals(noChangeName, result.getName());
}
+
+ @Test
+ void testMatchModeEnumHelpers() {
+ assertEquals(3, FuzzyMatchMeta.MatchMode.getDescriptions().length);
+ assertEquals(
+ FuzzyMatchMeta.MatchMode.ALL_ROWS,
+ FuzzyMatchMeta.MatchMode.lookupDescription(
+ FuzzyMatchMeta.MatchMode.ALL_ROWS.getDescription()));
+ assertEquals(
+ FuzzyMatchMeta.MatchMode.ALL_CONCAT, FuzzyMatchMeta.MatchMode.lookupCode("all_concat"));
+ assertEquals(FuzzyMatchMeta.MatchMode.CLOSEST, FuzzyMatchMeta.MatchMode.lookupCode("unknown"));
+ }
+
+ @Test
+ void testAlgorithmEnumHelpers() {
+ assertTrue(FuzzyMatchMeta.Algorithm.getDescriptions().length > 0);
+ assertEquals(
+ FuzzyMatchMeta.Algorithm.LEVENSHTEIN,
+ FuzzyMatchMeta.Algorithm.lookupDescription(
+ FuzzyMatchMeta.Algorithm.LEVENSHTEIN.getDescription()));
+ assertEquals(
+ FuzzyMatchMeta.Algorithm.DAMERAU_LEVENSHTEIN,
+ FuzzyMatchMeta.Algorithm.lookupCode("dameraulevenshtein"));
+ assertEquals(FuzzyMatchMeta.Algorithm.NONE, FuzzyMatchMeta.Algorithm.lookupCode("nope"));
+ }
+
+ @Test
+ void testFmLookupValueConstructors() {
+ FuzzyMatchMeta.FMLookupValue empty = new FuzzyMatchMeta.FMLookupValue();
+ assertNull(empty.getName());
+
+ FuzzyMatchMeta.FMLookupValue source = new FuzzyMatchMeta.FMLookupValue("a", "b");
+ FuzzyMatchMeta.FMLookupValue copy = new FuzzyMatchMeta.FMLookupValue(source);
+ assertEquals("a", copy.getName());
+ assertEquals("b", copy.getRename());
+ }
+
+ @Test
+ void testSupportsErrorHandlingAndExcludeRowLayout() {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ assertTrue(meta.supportsErrorHandling());
+ assertTrue(meta.excludeFromRowLayoutVerification());
+ assertNotNull(meta.getTransformIOMeta());
+ assertEquals(1, meta.getTransformIOMeta().getInfoStreams().size());
+ }
}
diff --git a/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchTest.java b/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchTest.java
index 6bd5dbf567..660bad836f 100644
--- a/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchTest.java
+++ b/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/FuzzyMatchTest.java
@@ -18,6 +18,8 @@
package org.apache.hop.pipeline.transforms.fuzzymatch;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
@@ -27,8 +29,10 @@ import static org.mockito.Mockito.when;
import java.util.ArrayList;
import java.util.Arrays;
+import java.util.Collections;
import java.util.HashSet;
import java.util.List;
+import java.util.Objects;
import org.apache.hop.core.IRowSet;
import org.apache.hop.core.exception.HopTransformException;
import org.apache.hop.core.logging.ILoggingObject;
@@ -47,20 +51,21 @@ import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.mockito.InjectMocks;
+/** Unit test for {@link FuzzyMatch} */
class FuzzyMatchTest {
@InjectMocks private FuzzyMatchHandler fuzzyMatch;
private TransformMockHelper<FuzzyMatchMeta, FuzzyMatchData> mockHelper;
- private Object[] row = new Object[] {"Catrine"};
- private Object[] rowB = new Object[] {"Catrine".getBytes()};
- private Object[] row2 = new Object[] {"John"};
- private Object[] row2B = new Object[] {"John".getBytes()};
- private Object[] row3 = new Object[] {"Catriny"};
- private Object[] row3B = new Object[] {"Catriny".getBytes()};
- private List<Object[]> rows = new ArrayList<>();
- private List<Object[]> binaryRows = new ArrayList<>();
- private List<Object[]> lookupRows = new ArrayList<>();
- private List<Object[]> binaryLookupRows = new ArrayList<>();
+ private final Object[] row = new Object[] {"Catrine"};
+ private final Object[] rowB = new Object[] {"Catrine".getBytes()};
+ private final Object[] row2 = new Object[] {"John"};
+ private final Object[] row2B = new Object[] {"John".getBytes()};
+ private final Object[] row3 = new Object[] {"Catriny"};
+ private final Object[] row3B = new Object[] {"Catriny".getBytes()};
+ private final List<Object[]> rows = new ArrayList<>();
+ private final List<Object[]> binaryRows = new ArrayList<>();
+ private final List<Object[]> lookupRows = new ArrayList<>();
+ private final List<Object[]> binaryLookupRows = new ArrayList<>();
{
rows.add(row);
@@ -71,8 +76,9 @@ class FuzzyMatchTest {
binaryLookupRows.add(row3B);
}
- private class FuzzyMatchHandler extends FuzzyMatch {
+ private static class FuzzyMatchHandler extends FuzzyMatch {
private Object[] resultRow = null;
+ private final List<Object[]> resultRows = new ArrayList<>();
private IRowSet rowset = null;
public FuzzyMatchHandler(
@@ -88,15 +94,9 @@ class FuzzyMatchTest {
@Override
public void putRow(IRowMeta rowMeta, Object[] row) throws HopTransformException {
resultRow = row;
+ resultRows.add(row);
}
- /**
- * Find input row set.
- *
- * @param sourceTransformName the source transform
- * @return the row set
- * @throws HopTransformException the hop transform exception
- */
@Override
public IRowSet findInputRowSet(String sourceTransformName) throws HopTransformException {
return rowset;
@@ -117,26 +117,387 @@ class FuzzyMatchTest {
mockHelper.cleanUp();
}
+ private FuzzyMatchHandler createHandler(FuzzyMatchMeta meta, FuzzyMatchData data) {
+ return new FuzzyMatchHandler(
+ mockHelper.transformMeta, meta, data, 0, mockHelper.pipelineMeta, mockHelper.pipeline);
+ }
+
+ private FuzzyMatchData prepareDistanceData(
+ FuzzyMatchMeta meta, boolean addMeasure, Object[]... lookupValues) throws Exception {
+ FuzzyMatchData data = new FuzzyMatchData();
+ data.readLookupValues = false;
+ data.indexOfMainField = 0;
+ data.minimalDistance =
+ Integer.parseInt(meta.getMinimalValue() == null ? "0" : meta.getMinimalValue());
+ data.maximalDistance =
+ Integer.parseInt(meta.getMaximalValue() == null ? "5" : meta.getMaximalValue());
+ data.maxMatches =
+ Integer.parseInt(
+ meta.getMaxMatches() == null
+ ? String.valueOf(FuzzyMatchMeta.DEFAULT_MAX_MATCHES)
+ : meta.getMaxMatches());
+ data.addValueFieldName = addMeasure;
+ data.valueSeparator = meta.getSeparator() == null ? "," : meta.getSeparator();
+ data.look = new HashSet<>();
+ Collections.addAll(data.look, lookupValues);
+
+ IRowMeta inputMeta = new RowMeta();
+ inputMeta.addValueMeta(new ValueMetaString("name"));
+ fuzzyMatch = createHandler(meta, data);
+ fuzzyMatch.setInputRowMeta(inputMeta);
+ data.outputRowMeta = inputMeta.clone();
+ meta.getFields(data.outputRowMeta, "FuzzyMatch", null, null, fuzzyMatch, null);
+ fuzzyMatch.first = false;
+ return data;
+ }
+
@Test
void testProcessRow() throws Exception {
- fuzzyMatch =
- new FuzzyMatchHandler(
- mockHelper.transformMeta,
- mockHelper.iTransformMeta,
- mockHelper.iTransformData,
- 0,
- mockHelper.pipelineMeta,
- mockHelper.pipeline);
- fuzzyMatch.init();
- fuzzyMatch.addRowSetToInputRowSets(mockHelper.getMockInputRowSet(rows));
- fuzzyMatch.addRowSetToInputRowSets(mockHelper.getMockInputRowSet(lookupRows));
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.SOUNDEX);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.CLOSEST);
+ meta.setMainStreamField("name");
+ meta.setLookupField("name");
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
- when(mockHelper.iTransformMeta.getAlgorithm()).thenReturn(FuzzyMatchMeta.Algorithm.SOUNDEX);
- mockHelper.iTransformData.look = mock(HashSet.class);
- when(mockHelper.iTransformData.look.iterator()).thenReturn(lookupRows.iterator());
+ FuzzyMatchData data = new FuzzyMatchData();
+ data.readLookupValues = false;
+ data.indexOfMainField = 0;
+ data.addValueFieldName = true;
+ data.look = new HashSet<>();
+ data.look.add(new Object[] {"John"});
+ data.look.add(new Object[] {"Catriny"});
- fuzzyMatch.processRow();
- assertEquals(fuzzyMatch.resultRow[0], row3[0]);
+ fuzzyMatch = createHandler(meta, data);
+ IRowMeta inputMeta = new RowMeta();
+ inputMeta.addValueMeta(new ValueMetaString("name"));
+ fuzzyMatch.setInputRowMeta(inputMeta);
+ data.outputRowMeta = inputMeta.clone();
+ meta.getFields(data.outputRowMeta, "FuzzyMatch", null, null, fuzzyMatch, null);
+ fuzzyMatch.first = false;
+ fuzzyMatch.addRowSetToInputRowSets(mockHelper.getMockInputRowSet(new Object[] {"Catrine"}));
+
+ assertTrue(fuzzyMatch.processRow());
+ assertEquals(1, fuzzyMatch.resultRows.size());
+ // main stream value + phonetic match (Soundex of Catrine ~= Catriny)
+ assertEquals("Catrine", fuzzyMatch.resultRows.getFirst()[0]);
+ assertEquals("Catriny", fuzzyMatch.resultRows.getFirst()[1]);
+ assertNotNull(fuzzyMatch.resultRows.getFirst()[2]); // soundex code
+ }
+
+ @Test
+ void testClosestDistanceReturnsBestMatch() throws Exception {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.CLOSEST);
+ meta.setMainStreamField("name");
+ meta.setLookupField("name");
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+ meta.setMinimalValue("0");
+ meta.setMaximalValue("5");
+
+ prepareDistanceData(meta, true, new Object[] {"Catriny"}, new Object[] {"John"});
+ fuzzyMatch.addRowSetToInputRowSets(mockHelper.getMockInputRowSet(new Object[] {"Catrine"}));
+
+ assertTrue(fuzzyMatch.processRow());
+ assertEquals(1, fuzzyMatch.resultRows.size());
+ assertEquals("Catriny", fuzzyMatch.resultRows.getFirst()[1]);
+ assertEquals(1L, fuzzyMatch.resultRows.getFirst()[2]);
+ }
+
+ @Test
+ void testAllRowsTopKEmitsSeparateRowsWithDistance() throws Exception {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_ROWS);
+ meta.setMainStreamField("name");
+ meta.setLookupField("name");
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+ meta.setMinimalValue("0");
+ meta.setMaximalValue("5");
+ meta.setMaxMatches("2");
+ meta.setCaseSensitive(false);
+
+ prepareDistanceData(
+ meta, true, new Object[] {"Catrine"}, new Object[] {"Catriny"}, new Object[] {"John"});
+ fuzzyMatch.addRowSetToInputRowSets(mockHelper.getMockInputRowSet(new Object[] {"Catrine"}));
+
+ assertTrue(fuzzyMatch.processRow());
+ assertEquals(2, fuzzyMatch.resultRows.size());
+ assertEquals("Catrine", fuzzyMatch.resultRows.get(0)[1]);
+ assertEquals(0L, fuzzyMatch.resultRows.get(0)[2]);
+ assertEquals("Catriny", fuzzyMatch.resultRows.get(1)[1]);
+ assertEquals(1L, fuzzyMatch.resultRows.get(1)[2]);
+ }
+
+ @Test
+ void testAllConcatReturnsJoinedMatchesAndMeasures() throws Exception {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_CONCAT);
+ meta.setMainStreamField("name");
+ meta.setLookupField("name");
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+ meta.setMinimalValue("0");
+ meta.setMaximalValue("5");
+ meta.setMaxMatches("10");
+ meta.setSeparator("|");
+
+ prepareDistanceData(meta, true, new Object[] {"Catrine"}, new Object[] {"Catriny"});
+ fuzzyMatch.addRowSetToInputRowSets(mockHelper.getMockInputRowSet(new Object[] {"Catrine"}));
+
+ assertTrue(fuzzyMatch.processRow());
+ assertEquals(1, fuzzyMatch.resultRows.size());
+ assertEquals("Catrine|Catriny", fuzzyMatch.resultRows.getFirst()[1]);
+ assertEquals("0|1", fuzzyMatch.resultRows.getFirst()[2]);
+ }
+
+ @Test
+ void testAllRowsRespectsDistanceThreshold() throws Exception {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_ROWS);
+ meta.setMainStreamField("name");
+ meta.setLookupField("name");
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+ meta.setMinimalValue("0");
+ meta.setMaximalValue("1");
+ meta.setMaxMatches("10");
+
+ prepareDistanceData(meta, true, new Object[] {"Catriny"}, new Object[] {"John"});
+ fuzzyMatch.addRowSetToInputRowSets(mockHelper.getMockInputRowSet(new Object[] {"Catrine"}));
+
+ assertTrue(fuzzyMatch.processRow());
+ assertEquals(1, fuzzyMatch.resultRows.size());
+ assertEquals("Catriny", fuzzyMatch.resultRows.getFirst()[1]);
+ }
+
+ @Test
+ void testNoMatchStillEmitsRowWithNullMatch() throws Exception {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_ROWS);
+ meta.setMainStreamField("name");
+ meta.setLookupField("name");
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+ meta.setMinimalValue("0");
+ meta.setMaximalValue("1");
+ meta.setMaxMatches("5");
+
+ prepareDistanceData(meta, true, new Object[] {"zzzz"});
+ fuzzyMatch.addRowSetToInputRowSets(mockHelper.getMockInputRowSet(new Object[] {"Catrine"}));
+
+ assertTrue(fuzzyMatch.processRow());
+ assertEquals(1, fuzzyMatch.resultRows.size());
+ assertEquals("Catrine", fuzzyMatch.resultRows.getFirst()[0]);
+ assertNull(fuzzyMatch.resultRows.getFirst()[1]);
+ }
+
+ @Test
+ void testCaseSensitiveDistance() throws Exception {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.CLOSEST);
+ meta.setMainStreamField("name");
+ meta.setLookupField("name");
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+ meta.setMinimalValue("0");
+ meta.setMaximalValue("0");
+ meta.setCaseSensitive(true);
+
+ prepareDistanceData(meta, true, new Object[] {"catrine"});
+ fuzzyMatch.addRowSetToInputRowSets(mockHelper.getMockInputRowSet(new Object[] {"Catrine"}));
+
+ assertTrue(fuzzyMatch.processRow());
+ assertNull(fuzzyMatch.resultRows.getFirst()[1]);
+ }
+
+ @Test
+ void testJaroClosestReturnsSimilarityMeasure() throws Exception {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.JARO);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.CLOSEST);
+ meta.setMainStreamField("name");
+ meta.setLookupField("name");
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+ meta.setMinimalValue("0");
+ meta.setMaximalValue("1");
+
+ FuzzyMatchData data = new FuzzyMatchData();
+ data.readLookupValues = false;
+ data.indexOfMainField = 0;
+ data.minimalSimilarity = 0;
+ data.maximalSimilarity = 1;
+ data.maxMatches = 1;
+ data.addValueFieldName = true;
+ data.look = new HashSet<>();
+ data.look.add(new Object[] {"Martha"});
+ data.look.add(new Object[] {"Marhta"});
+
+ fuzzyMatch = createHandler(meta, data);
+ IRowMeta inputMeta = new RowMeta();
+ inputMeta.addValueMeta(new ValueMetaString("name"));
+ fuzzyMatch.setInputRowMeta(inputMeta);
+ data.outputRowMeta = inputMeta.clone();
+ meta.getFields(data.outputRowMeta, "FuzzyMatch", null, null, fuzzyMatch, null);
+ fuzzyMatch.first = false;
+ fuzzyMatch.addRowSetToInputRowSets(mockHelper.getMockInputRowSet(new Object[] {"Martha"}));
+
+ assertTrue(fuzzyMatch.processRow());
+ assertEquals(1, fuzzyMatch.resultRows.size());
+ assertEquals("Martha", fuzzyMatch.resultRows.getFirst()[1]);
+ assertEquals(1.0, ((Number) fuzzyMatch.resultRows.getFirst()[2]).doubleValue(), 1e-9);
+ }
+
+ @Test
+ void testInitCapsMaxMatchesAndRequiresFields() {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setMainStreamField("name");
+ meta.setLookupField("name");
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_ROWS);
+ meta.setMaxMatches("999");
+ meta.setMinimalValue("0");
+ meta.setMaximalValue("5");
+
+ FuzzyMatchData data = new FuzzyMatchData();
+ fuzzyMatch = createHandler(meta, data);
+ assertTrue(fuzzyMatch.init());
+ assertEquals(FuzzyMatchMeta.HARD_MAX_MATCHES, data.maxMatches);
+ assertTrue(data.addValueFieldName);
+
+ FuzzyMatchMeta missing = new FuzzyMatchMeta();
+ missing.setOutputMatchField("match");
+ FuzzyMatchData data2 = new FuzzyMatchData();
+ FuzzyMatchHandler handler2 = createHandler(missing, data2);
+ assertFalse(handler2.init());
+ }
+
+ @Test
+ void testInitRejectsEmptyOutputMatchField() {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setMainStreamField("name");
+ meta.setLookupField("name");
+ meta.setOutputMatchField("");
+ FuzzyMatchHandler handler = createHandler(meta, new FuzzyMatchData());
+ assertFalse(handler.init());
+ }
+
+ @Test
+ void testProcessRowEndsWhenInputExhausted() throws Exception {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.CLOSEST);
+ meta.setMainStreamField("name");
+ meta.setLookupField("name");
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+
+ FuzzyMatchData data = prepareDistanceData(meta, true, new Object[] {"Catrine"});
+ // empty input
+ fuzzyMatch.addRowSetToInputRowSets(mockHelper.getMockInputRowSet());
+
+ assertFalse(fuzzyMatch.processRow());
+ assertTrue(fuzzyMatch.resultRows.isEmpty());
+ assertNotNull(data);
+ }
+
+ @Test
+ void testPairSimilarityAllRows() throws Exception {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.PAIR_SIMILARITY);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_ROWS);
+ meta.setMainStreamField("name");
+ meta.setLookupField("name");
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+ meta.setMinimalValue("0.1");
+ meta.setMaximalValue("1");
+ meta.setMaxMatches("2");
+
+ FuzzyMatchData data = new FuzzyMatchData();
+ data.readLookupValues = false;
+ data.indexOfMainField = 0;
+ data.minimalSimilarity = 0.1;
+ data.maximalSimilarity = 1;
+ data.maxMatches = 2;
+ data.addValueFieldName = true;
+ data.look = new HashSet<>();
+ data.look.add(new Object[] {"France"});
+ data.look.add(new Object[] {"French"});
+ data.look.add(new Object[] {"zzzz"});
+
+ fuzzyMatch = createHandler(meta, data);
+ IRowMeta inputMeta = new RowMeta();
+ inputMeta.addValueMeta(new ValueMetaString("name"));
+ fuzzyMatch.setInputRowMeta(inputMeta);
+ data.outputRowMeta = inputMeta.clone();
+ meta.getFields(data.outputRowMeta, "FuzzyMatch", null, null, fuzzyMatch, null);
+ fuzzyMatch.first = false;
+ fuzzyMatch.addRowSetToInputRowSets(mockHelper.getMockInputRowSet(new Object[] {"France"}));
+
+ assertTrue(fuzzyMatch.processRow());
+ assertEquals(2, fuzzyMatch.resultRows.size());
+ assertEquals("France", fuzzyMatch.resultRows.getFirst()[1]);
+ assertTrue(((Number) fuzzyMatch.resultRows.getFirst()[2]).doubleValue() >= 0.1);
+ }
+
+ @Test
+ void testAllRowsWithAdditionalLookupFields() throws Exception {
+ FuzzyMatchMeta meta = new FuzzyMatchMeta();
+ meta.setAlgorithm(FuzzyMatchMeta.Algorithm.LEVENSHTEIN);
+ meta.setMatchMode(FuzzyMatchMeta.MatchMode.ALL_ROWS);
+ meta.setMainStreamField("name");
+ meta.setLookupField("name");
+ meta.setOutputMatchField("match");
+ meta.setOutputValueField("measure value");
+ meta.setMinimalValue("0");
+ meta.setMaximalValue("5");
+ meta.setMaxMatches("2");
+ meta.setLookupValues(List.of(new FuzzyMatchMeta.FMLookupValue("id", "lookup_id")));
+
+ FuzzyMatchData data = new FuzzyMatchData();
+ data.readLookupValues = false;
+ data.indexOfMainField = 0;
+ data.minimalDistance = 0;
+ data.maximalDistance = 5;
+ data.maxMatches = 2;
+ data.addValueFieldName = true;
+ data.addAdditionalFields = true;
+ data.look = new HashSet<>();
+ data.look.add(new Object[] {"Catrine", "1"});
+ data.look.add(new Object[] {"Catriny", "2"});
+
+ fuzzyMatch = createHandler(meta, data);
+ IRowMeta inputMeta = new RowMeta();
+ inputMeta.addValueMeta(new ValueMetaString("name"));
+ fuzzyMatch.setInputRowMeta(inputMeta);
+
+ IRowMeta infoMeta = new RowMeta();
+ infoMeta.addValueMeta(new ValueMetaString("id"));
+ data.outputRowMeta = inputMeta.clone();
+ meta.getFields(
+ data.outputRowMeta, "FuzzyMatch", new IRowMeta[] {infoMeta}, null, fuzzyMatch, null);
+ fuzzyMatch.first = false;
+ fuzzyMatch.addRowSetToInputRowSets(mockHelper.getMockInputRowSet(new Object[] {"Catrine"}));
+
+ assertTrue(fuzzyMatch.processRow());
+ assertEquals(2, fuzzyMatch.resultRows.size());
+ assertEquals("Catrine", fuzzyMatch.resultRows.getFirst()[1]);
+ assertEquals(0L, fuzzyMatch.resultRows.get(0)[2]);
+ assertEquals("1", fuzzyMatch.resultRows.get(0)[3]);
+ assertEquals("Catriny", fuzzyMatch.resultRows.get(1)[1]);
+ assertEquals("2", fuzzyMatch.resultRows.get(1)[3]);
}
@Test
@@ -214,6 +575,6 @@ class FuzzyMatchTest {
assertNull(fuzzyMatch.resultRow[1]);
assertTrue(
Arrays.stream(fuzzyMatch.resultRow, 3, fuzzyMatch.resultRow.length)
- .allMatch(val -> val == null));
+ .allMatch(Objects::isNull));
}
}
diff --git a/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/LetterPairSimilarityTest.java b/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/LetterPairSimilarityTest.java
new file mode 100644
index 0000000000..70ed0600e0
--- /dev/null
+++ b/plugins/transforms/fuzzymatch/src/test/java/org/apache/hop/pipeline/transforms/fuzzymatch/LetterPairSimilarityTest.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.pipeline.transforms.fuzzymatch;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import org.junit.jupiter.api.Test;
+
+/** Unit tests for {@link LetterPairSimilarity}. */
+class LetterPairSimilarityTest {
+
+ @Test
+ void testIdenticalStringsHaveSimilarityOne() {
+ assertEquals(1.0, LetterPairSimilarity.getSimilarity("Hop", "Hop"), 1e-9);
+ assertEquals(1.0, LetterPairSimilarity.getSimilarity("apache hop", "apache hop"), 1e-9);
+ }
+
+ @Test
+ void testBothEmptyHaveSimilarityOne() {
+ assertEquals(1.0, LetterPairSimilarity.getSimilarity("", ""), 1e-9);
+ assertEquals(1.0, LetterPairSimilarity.getSimilarity(null, null), 1e-9);
+ }
+
+ @Test
+ void testCompletelyDifferentStringsHaveLowSimilarity() {
+ double score = LetterPairSimilarity.getSimilarity("abc", "xyz");
+ assertEquals(0.0, score, 1e-9);
+ }
+
+ @Test
+ void testPartialOverlapIsBetweenZeroAndOne() {
+ double score = LetterPairSimilarity.getSimilarity("france", "french");
+ assertTrue(score > 0.0 && score < 1.0, "expected partial similarity, got " + score);
+ }
+
+ @Test
+ void testSimilarityIsCaseInsensitive() {
+ assertEquals(
+ LetterPairSimilarity.getSimilarity("Hop", "HOP"),
+ LetterPairSimilarity.getSimilarity("hop", "hop"),
+ 1e-9);
+ }
+}
diff --git a/plugins/transforms/fuzzymatch/src/test/resources/fuzzy-match-transform.xml b/plugins/transforms/fuzzymatch/src/test/resources/fuzzy-match-transform.xml
index a8b37fd014..367b50f011 100644
--- a/plugins/transforms/fuzzymatch/src/test/resources/fuzzy-match-transform.xml
+++ b/plugins/transforms/fuzzymatch/src/test/resources/fuzzy-match-transform.xml
@@ -34,6 +34,8 @@
<outputvaluefield>measure value</outputvaluefield>
<caseSensitive>N</caseSensitive>
<closervalue>Y</closervalue>
+ <matchMode>closest</matchMode>
+ <maxMatches>10</maxMatches>
<minimalValue>0</minimalValue>
<maximalValue>1</maximalValue>
<separator>,</separator>