Using Lucene for Moderate Similarity Check..

[RaviK Thakur <Ra...@lntinfotech.com>]

Hello All,
      I want to check the feasibility of using Lucene for similarity check
between the two flat csv files. The actual requirement is like this: We
have two files each containing the information of customers like their
name, address, pin code etc. Some customers may be in common in both the
files. We want to find the customer that are common in these files. But the
match should be on attribute basis. If the name of the customer matches in
one file to the name of the customer in another file, then match the
address, if it matches then match pin code and so on. But the main
consideration is that this matching is not exact. If the name of customer
matches say 80% then it may be termed as match. For example, if ABDUL is
matched with ABDULLAH, it should be termed as a match. In this fashion each
record of one file will be matched with each record of another file. The
output of this procedure will be another file containing the matched
record.

Can anyone please suggest the applicability of lucene for this requirement.
May in the form of Pros n Cons.

Thanks in advance:-)
Ravi


______________________________________________________________________

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Using Lucene for Moderate Similarity Check..

[Grant Ingersoll <gs...@apache.org>]

Hi Ravi,

Lucene can enable this, but you will have some work to do on top of  
it.  If you search the archives for record linkage (http://www.lucidimagination.com/search/?q=record+linkage 
) you will find a fair amount of discussion on this.  Also, in  
somewhat shameless marketing mode, my co-author, Tom Morton, is just  
putting the finishing touches on a chapter in our book called Taming  
Text (http://www.manning.com/ingersoll) which discusses some of the  
techniques involved in making this stuff happen.  That chapter should  
be released in the next few weeks.

You likely can get a basic system working pretty quickly with what is  
in Lucene, but then the next level is often more difficult.  You often  
end up with a rules system that can become brittle with this  
approach.  An alternative is to apply some type of machine learning  
approach.  You could also look at this as a clustering problem, which  
Mahout (or other clustering tools) could be helpful in solving.

Finally, just know there will be a human in the loop with any  
approach.  The goal is to minimize the number of matches that a person  
has to check.

Hope this helps,
Grant

On Jun 9, 2009, at 4:16 AM, RaviK Thakur wrote:

>
> Hello All,
>      I want to check the feasibility of using Lucene for similarity  
> check
> between the two flat csv files. The actual requirement is like this:  
> We
> have two files each containing the information of customers like their
> name, address, pin code etc. Some customers may be in common in both  
> the
> files. We want to find the customer that are common in these files.  
> But the
> match should be on attribute basis. If the name of the customer  
> matches in
> one file to the name of the customer in another file, then match the
> address, if it matches then match pin code and so on. But the main
> consideration is that this matching is not exact. If the name of  
> customer
> matches say 80% then it may be termed as match. For example, if  
> ABDUL is
> matched with ABDULLAH, it should be termed as a match. In this  
> fashion each
> record of one file will be matched with each record of another file.  
> The
> output of this procedure will be another file containing the matched
> record.
>
> Can anyone please suggest the applicability of lucene for this  
> requirement.
> May in the form of Pros n Cons.
>
> Thanks in advance:-)
> Ravi
>
>
> ______________________________________________________________________
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)  
using Solr/Lucene:
http://www.lucidimagination.com/search


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org