You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Greg Rahn (Code Review)" <ge...@cloudera.org> on 2019/07/03 04:45:53 UTC

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Greg Rahn has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13794


Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................

IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

This patch adds new built-in functions to calculate Damerau-Levenshtein
edit distance. Implmented as dle_dst() and damerau_levenshtein(). If
either value is NULL or both values are NULL returns NULL which differs
from Netezza's dle_dst() which returns the length of the not NULL value
or 0 if both values are NULL. The NULL behavior matches the existing
levenshtein() function.

Testing:
- Added unit tests to expr-test.cc
- Manualal testing on over 1400 string pairs from
  http://marvin.cs.uidaho.edu/misspell.html and results match Netezza

Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
---
M be/src/exprs/expr-test.cc
M be/src/exprs/string-functions-ir.cc
M be/src/exprs/string-functions.h
M common/function-registry/impala_functions.py
4 files changed, 110 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/13794/1
-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 1
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 9:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5279/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 9
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Nov 2019 09:46:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Greg Rahn (Code Review)" <ge...@cloudera.org>.
Greg Rahn has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 1:

> (1 comment)

I'll clarify in the commit msg and docs -- this is the restricted DL (or OSA) as you pointed out, not the full DL.


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 1
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jul 2019 17:41:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Greg Rahn (Code Review)" <ge...@cloudera.org>.
Greg Rahn has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................

IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

This patch adds new built-in functions to calculate restricted
Damerau-Levenshtein edit distance (optimal string alignment).
Implmented as dle_dst() and damerau_levenshtein(). If either value is
NULL or both values are NULL returns NULL which differs from Netezza's
dle_dst() which returns the length of the not NULL value or 0 if both
values are NULL. The NULL behavior matches the existing levenshtein()
function.

Also cleans up levenshtein tests.

Testing:
- Added unit tests to expr-test.cc
- Manual testing on over 1400 string pairs from
  http://marvin.cs.uidaho.edu/misspell.html and results match Netezza

Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
---
M be/src/exprs/expr-test.cc
M be/src/exprs/string-functions-ir.cc
M be/src/exprs/string-functions.h
M common/function-registry/impala_functions.py
4 files changed, 142 insertions(+), 37 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/13794/7
-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 7
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/3811/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 1
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jul 2019 05:25:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Norbert Luksa (Code Review)" <ge...@cloudera.org>.
Norbert Luksa has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 1:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/string-functions-ir.cc
File be/src/exprs/string-functions-ir.cc:

http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/string-functions-ir.cc@1177
PS1, Line 1177:   int d[s1len + 1][s2len + 1];
> Yeah, usually we don't allocate that much memory on the stack because we re
Changed VLA on stack to dynamic allocation.


http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/string-functions-ir.cc@1218
PS1, Line 1218:         d[i][j] = std::min(d[i][j],
              :                            d[i - 2][j - 2] + l_cost // transposition
              :         );
> nit: fits single line
Done


http://gerrit.cloudera.org:8080/#/c/13794/3/be/src/exprs/string-functions-ir.cc
File be/src/exprs/string-functions-ir.cc:

http://gerrit.cloudera.org:8080/#/c/13794/3/be/src/exprs/string-functions-ir.cc@1220
PS3, Line 1220: );
> nit: you could just put it before the comment from the prev line.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 1
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 25 Oct 2019 12:12:06 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 3:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/string-functions-ir.cc
File be/src/exprs/string-functions-ir.cc:

http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/string-functions-ir.cc@1177
PS1, Line 1177:   int s2len = s2.len;
> This worries me a bit:
Yeah, usually we don't allocate that much memory on the stack because we really don't want a stack overflow. Though the default stack size on Linux/x64 is fairly large, 8MB, but, it isn't committed memory from the beginning. And a 64K allocation will likely commit some new memory to your process that will stay committed until your thread exits.

Also, VLAs are non-standard. Heap allocation with TCMalloc should be fairly efficient, but you might wanna do some benchmarks.

If you turn to heap allocation, then long ago I wrote a simple Matrix class that I've just uploaded here:
https://github.com/boroknagyz/UtilityClasses/blob/master/include/matrix.h

It allocates a big std::vector to hold all the elements and provides a 2D array-like interface, e.g. you can access an element via 'matrix[2][3]'. The coding style doesn't confirm completely to the Impala style, e.g. the use of camel case is different.


http://gerrit.cloudera.org:8080/#/c/13794/3/be/src/exprs/string-functions-ir.cc
File be/src/exprs/string-functions-ir.cc:

http://gerrit.cloudera.org:8080/#/c/13794/3/be/src/exprs/string-functions-ir.cc@1220
PS3, Line 1220: );
nit: you could just put it before the comment from the prev line.



-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 3
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Jul 2019 12:38:36 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 8: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 8
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Nov 2019 09:45:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/4974/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 8
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Nov 2019 17:14:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 10: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 10
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Nov 2019 16:56:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Greg Rahn (Code Review)" <ge...@cloudera.org>.
Greg Rahn has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................

IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

This patch adds new built-in functions to calculate restricted
Damerau-Levenshtein edit distance (optimal string alignment).
Implmented as dle_dst() and damerau_levenshtein(). If either value is
NULL or both values are NULL returns NULL which differs from Netezza's
dle_dst() which returns the length of the not NULL value or 0 if both
values are NULL. The NULL behavior matches the existing levenshtein()
function.

Also clean up levenshtein tests.

Testing:
- Added unit tests to expr-test.cc
- Manualal testing on over 1400 string pairs from
  http://marvin.cs.uidaho.edu/misspell.html and results match Netezza

Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
---
M be/src/exprs/expr-test.cc
M be/src/exprs/string-functions-ir.cc
M be/src/exprs/string-functions.h
M common/function-registry/impala_functions.py
4 files changed, 107 insertions(+), 30 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/13794/3
-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 3
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 9: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 9
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Nov 2019 09:46:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................

IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

This patch adds new built-in functions to calculate restricted
Damerau-Levenshtein edit distance (optimal string alignment).
Implmented as dle_dst() and damerau_levenshtein(). If either value is
NULL or both values are NULL returns NULL which differs from Netezza's
dle_dst() which returns the length of the not NULL value or 0 if both
values are NULL. The NULL behavior matches the existing levenshtein()
function.

Also cleans up levenshtein tests.

Testing:
- Added unit tests to expr-test.cc
- Manual testing on over 1400 string pairs from
  http://marvin.cs.uidaho.edu/misspell.html and results match Netezza

Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Reviewed-on: http://gerrit.cloudera.org:8080/13794
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exprs/expr-test.cc
M be/src/exprs/string-functions-ir.cc
M be/src/exprs/string-functions.h
M common/function-registry/impala_functions.py
4 files changed, 152 insertions(+), 37 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified
  Csaba Ringhofer: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 11
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 9: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5279/


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 9
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Nov 2019 14:27:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/expr-test.cc
File be/src/exprs/expr-test.cc:

http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/expr-test.cc@3999
PS1, Line 3999:   TestValue("damerau_levenshtein('CA', 'ABC')", TYPE_INT, 3);
Based on https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance#Algorithm the used algorithm computes "optimal string alignment distance" or "restricted edit distance" which is not the "true" Damerau–Levenshtein distance because that should return 2 in this case (CA -> AC -> ABC).

I'm not sure how to resolve it, I think you know it best, but I think at least some comments needed to clarify the semantics.



-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 1
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jul 2019 12:37:17 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5280/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 10
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Nov 2019 16:56:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 10: Code-Review+2

Running the build again as the issue seems unrelated: https://issues.apache.org/jira/browse/IMPALA-9165


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 10
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Nov 2019 16:57:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Greg Rahn (Code Review)" <ge...@cloudera.org>.
Greg Rahn has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................

IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

This patch adds new built-in functions to calculate restricted
Damerau-Levenshtein edit distance (optimal string alignment).
Implmented as dle_dst() and damerau_levenshtein(). If either value is
NULL or both values are NULL returns NULL which differs from Netezza's
dle_dst() which returns the length of the not NULL value or 0 if both
values are NULL. The NULL behavior matches the existing levenshtein()
function.

Also cleans up levenshtein tests.

Testing:
- Added unit tests to expr-test.cc
- Manual testing on over 1400 string pairs from
  http://marvin.cs.uidaho.edu/misspell.html and results match Netezza

Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
---
M be/src/exprs/expr-test.cc
M be/src/exprs/string-functions-ir.cc
M be/src/exprs/string-functions.h
M common/function-registry/impala_functions.py
4 files changed, 152 insertions(+), 37 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/13794/8
-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 8
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/expr-test.cc
File be/src/exprs/expr-test.cc:

http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/expr-test.cc@4003
PS1, Line 4003:   TestErrorString("damerau_levenshtein('z', repeat('x', 256))",
              :       "damerau-levenshtein argument exceeds maximum length of 255 characters\n");
              :   TestErrorString("damerau_levenshtein(repeat('x', 256), 'z')",
              :       "damerau-levenshtein argument exceeds maximum length of 255 characters\n");
It could be useful to test the 255*255 case to ensure that there are no allocation issues.


http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/string-functions-ir.cc
File be/src/exprs/string-functions-ir.cc:

http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/string-functions-ir.cc@1177
PS1, Line 1177:   int d[s1len + 1][s2len + 1];
This worries me a bit:
1. The s1len > 255 || s2len > 255 does not protect from the allocation itself, so it is possible that a call with huge strings will lead to stack overflow. A solution that seems safe enough to me is creating an inner function that is called after the check and declare / use the array there,

2. Even if we exclude the s1len > 255 || s2len > 255 case, 64K on the stack seems too much to me, StringFunctions::Levenshtein uses dynamic allocation for much shorter arrays.



-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 1
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jul 2019 14:42:09 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 10: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 10
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Nov 2019 21:39:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 1: Code-Review+1

(2 comments)

Found some nits, but LGTM.

http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/expr-test.cc
File be/src/exprs/expr-test.cc:

http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/expr-test.cc@3984
PS1, Line 3984:   TestValue("damerau_levenshtein('', '')", TYPE_INT, 0);
              :   TestValue("damerau_levenshtein('abc', 'abc')", TYPE_INT, 0);
              :   TestValue("damerau_levenshtein('a', 'b')", TYPE_INT, 1);
              :   TestValue("damerau_levenshtein('a', '')", TYPE_INT, 1);
              :   TestValue("damerau_levenshtein('aabc', 'abc')", TYPE_INT, 1);
              :   TestValue("damerau_levenshtein('abcc', 'abc')", TYPE_INT, 1);
              :   TestValue("damerau_levenshtein('', 'a')", TYPE_INT, 1);
              :   TestValue("damerau_levenshtein('abc', 'abcc')", TYPE_INT, 1);
              :   TestValue("damerau_levenshtein('abc', 'aabc')", TYPE_INT, 1);
              :   TestValue("damerau_levenshtein('teh', 'the')", TYPE_INT, 1);
              :   TestValue("damerau_levenshtein('tets', 'test')", TYPE_INT, 1);
              :   TestValue("damerau_levenshtein('fuor', 'four')", TYPE_INT, 1);
              :   TestValue("damerau_levenshtein('kitten', 'sitting')", TYPE_INT, 3);
              :   TestValue("damerau_levenshtein('Saturday', 'Sunday')", TYPE_INT, 3);
              :   TestValue("damerau_levenshtein('rosettacode', 'raisethysword')", TYPE_INT, 8);
              :   TestValue("damerau_levenshtein('CA', 'ABC')", TYPE_INT, 3);
              :   TestIsNull("damerau_levenshtein('foo', NULL)", TYPE_INT);
              :   TestIsNull("damerau_levenshtein(NULL, 'foo')", TYPE_INT);
              :   TestIsNull("damerau_levenshtein(NULL, NULL)", TYPE_INT);
              :   TestErrorString("damerau_levenshtein('z', repeat('x', 256))",
              :       "damerau-levenshtein argument exceeds maximum length of 255 characters\n");
              :   TestErrorString("damerau_levenshtein(repeat('x', 256), 'z')",
              :       "damerau-levenshtein argument exceeds maximum length of 255 characters\n");
              : 
              :   TestValue("dle_dst('', '')", TYPE_INT, 0);
              :   TestValue("dle_dst('abc', 'abc')", TYPE_INT, 0);
              :   TestValue("dle_dst('a', 'b')", TYPE_INT, 1);
              :   TestValue("dle_dst('a', '')", TYPE_INT, 1);
              :   TestValue("dle_dst('aabc', 'abc')", TYPE_INT, 1);
              :   TestValue("dle_dst('abcc', 'abc')", TYPE_INT, 1);
              :   TestValue("dle_dst('', 'a')", TYPE_INT, 1);
              :   TestValue("dle_dst('abc', 'abcc')", TYPE_INT, 1);
              :   TestValue("dle_dst('abc', 'aabc')", TYPE_INT, 1);
              :   TestValue("dle_dst('teh', 'the')", TYPE_INT, 1);
              :   TestValue("dle_dst('tets', 'test')", TYPE_INT, 1);
              :   TestValue("dle_dst('fuor', 'four')", TYPE_INT, 1);
              :   TestValue("dle_dst('kitten', 'sitting')", TYPE_INT, 3);
              :   TestValue("dle_dst('Saturday', 'Sunday')", TYPE_INT, 3);
              :   TestValue("dle_dst('rosettacode', 'raisethysword')", TYPE_INT, 8);
              :   TestValue("dle_dst('CA', 'ABC')", TYPE_INT, 3);
              :   TestIsNull("dle_dst('foo', NULL)", TYPE_INT);
              :   TestIsNull("dle_dst(NULL, 'foo')", TYPE_INT);
              :   TestIsNull("dle_dst(NULL, NULL)", TYPE_INT);
              :   TestErrorString("dle_dst('z', repeat('x', 256))",
              :       "damerau-levenshtein argument exceeds maximum length of 255 characters\n");
              :   TestErrorString("dle_dst(repeat('x', 256), 'z')",
              :       "damerau-levenshtein argument exceeds maximum length of 255 characters\n");
nit: instead of duplicating maybe you could just write something like

 for (string fn_name : {"damerau_levenshtein", "dle_dst"}) {
   TestValue(fn_name + "('', '')" ...);
   ...
 }


http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/string-functions-ir.cc
File be/src/exprs/string-functions-ir.cc:

http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/string-functions-ir.cc@1218
PS1, Line 1218:         d[i][j] = std::min(d[i][j],
              :                            d[i - 2][j - 2] + l_cost // transposition
              :         );
nit: fits single line



-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 1
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jul 2019 12:04:51 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/3827/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 3
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Jul 2019 20:40:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/4906/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 7
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Oct 2019 18:00:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Norbert Luksa (Code Review)" <ge...@cloudera.org>.
Norbert Luksa has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13794/4/be/src/exprs/string-functions-ir.cc
File be/src/exprs/string-functions-ir.cc:

http://gerrit.cloudera.org:8080/#/c/13794/4/be/src/exprs/string-functions-ir.cc@1239
PS4, Line 1239:   if (matching_characters == 0) return DoubleVal(0.0);
> Alternatively, you could use MakeScopeExitTrigger().
Since it applies to more functions of this file, it would be more consistent to change there as well. But then it would be clearer to apply these changes in a separate commit, I think .



-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 4
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Nov 2019 09:13:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Greg Rahn (Code Review)" <ge...@cloudera.org>.
Greg Rahn has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................

IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

This patch adds new built-in functions to calculate restricted
Damerau-Levenshtein edit distance (optimal string alignment).
Implmented as dle_dst() and damerau_levenshtein(). If either value is
NULL or both values are NULL returns NULL which differs from Netezza's
dle_dst() which returns the length of the not NULL value or 0 if both
values are NULL. The NULL behavior matches the existing levenshtein()
function.

Also cleans up levenshtein tests.

Testing:
- Added unit tests to expr-test.cc
- Manualal testing on over 1400 string pairs from
  http://marvin.cs.uidaho.edu/misspell.html and results match Netezza

Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
---
M be/src/exprs/expr-test.cc
M be/src/exprs/string-functions-ir.cc
M be/src/exprs/string-functions.h
M common/function-registry/impala_functions.py
4 files changed, 123 insertions(+), 35 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/13794/4
-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 4
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 )

Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/4886/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13794
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d
Gerrit-Change-Number: 13794
Gerrit-PatchSet: 4
Gerrit-Owner: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Greg Rahn <gr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Norbert Luksa <no...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 25 Oct 2019 19:14:51 +0000
Gerrit-HasComments: No