You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@uima.apache.org by Grant Ingersoll <gs...@apache.org> on 2007/06/06 13:44:54 UTC
Lucene and UIMA was Re: Human annotation tool for UIMA
On Jun 6, 2007, at 7:31 AM, Thilo Goetz wrote:
> Julien Nioche wrote:
>> Hi,
>>> Re funding: we are optimistic that there will be more UIMA
>>> innovation
>>> awards by IBM this year. Watch this space for the announcement.
>>> I would
>>> think that a Knowtator/UIMA integration could be a candidate.
>>>
>> what about a UIMA / Lucene integration? Would that also be a good
>> candidate for funding as well?
>
> Well, I'm not giving away those grants ;-) In my very personal
> opinion, it
> depends on what kind of integration. The simple integration where
> UIMA
> annotations are indexed as Lucene fields has probably been done
> half a dozen
> times, though I haven't seen an open source/Apache version. If
> you're talking
> about a deep integration for semantic search, using the new Lucene
> payloads,
> now there's an interesting project. Again, my personal opinion.
I'm pretty new to UIMA, but know a thing or two about Lucene. Care
to share more about what kinds of things you are interested in? Are
you talking Semantic Web type stuff or things like enhanced NLP
search? I see from the Javadocs that there is a place to hook in
search implementations, but haven't dug deeper than that.
--------------------------
Grant Ingersoll
Center for Natural Language Processing
http://www.cnlp.org/tech/lucene.asp
Read the Lucene Java FAQ at http://wiki.apache.org/jakarta-lucene/
LuceneFAQ
Re: Lucene and UIMA - we have it!- was Re: Human annotation tool for UIMA
Posted by Benjamin Sznajder <be...@il.ibm.com>.
Hi Andrew,
Good point! :)
There is indeed a strong relationship between our "Semantic Search upon
Lucene" and the IBM "Semantic Search for Apache UIMA".
The IBM Search engine has been developped by team of our group here -IBM
Haifa labs-!
There are, anyway, some real differences between the two products:
- IBM Semantic Search has been developped upon the Juru Search engine when
ours has been developped upon Apache Lucene.
- We are supporting a strong XML Fragments query language version (see the
paper of RIAO) and a subset of XPath when IBM's former version supports
only a former XML Fragments version. In addition, our package let in an
easy way the support of additional future Semantic Query Languages.
- The algorithm of retrieval of the twig query is different in the both,
when we strengthened the efficiency by reducing in a drastic way the IO and
avoiding any post-processing.
Best regards,
Benjamin
"Andrew
Borthwick"
<andrew@corp.spoc To
k.com> uima-user@incubator.apache.org
cc
18/06/2007 20:06
Subject
Re: Lucene and UIMA - we have it!-
Please respond to was Re: Human annotation tool for
uima-user@incubat UIMA
or.apache.org
Benjamin,
Thanks for the update. I'll look forward to reading the paper.
One question: is there any relationship between your "Semantic Search
upon Lucene" tool and the IBM "Semantic Search for Apache UIMA" tool
which is posted at
http://incubator.apache.org/uima/external-resources.html?
Thanks,
Andrew
On 6/18/07, Benjamin Sznajder <be...@il.ibm.com> wrote:
> Thank you Thilo!!
>
> The whole title is "XML Fragments Extended with Database Operators" by
Yosi
> Mass, Dafna Sheinwald, Sivan Yogev and Benjamin Sznajder
>
> It will be soon online, I guess.
> Waiting for, any interested person can write to my personal mail...
>
> Benjamin
>
>
>
> Thilo Goetz
> <tw...@gmx.de>
>
To
> 18/06/2007 18:08 uima-user@incubator.apache.org
>
cc
>
> Please respond to
Subject
> uima-user@incubat Re: Lucene and UIMA - we have it!-
> or.apache.org was Re: Human annotation tool for
> UIMA
>
>
>
>
>
>
>
>
>
>
> Benjamin Sznajder wrote:
> > Hi Andrew,
> >
> > Very motivating to get a so rapid reply!! Thanks!
> >
> > About your questions:
> > - I join here a paper we have recently presented at RIAO conference. It
> > describes the Query Languages we support in our package and some
> > implementation highlights. We have not yet official documentation for
our
> > package itself. (See attached file: XMLFragment-camera-ready.pdf)
>
> Benjamin, I don't see any attachments. I think maybe the mailing list
> doesn't allow them.
>
> --Thilo
>
>
>
>
Re: Lucene and UIMA - we have it!- was Re: Human annotation tool for UIMA
Posted by Andrew Borthwick <an...@corp.spock.com>.
Benjamin,
Thanks for the update. I'll look forward to reading the paper.
One question: is there any relationship between your "Semantic Search
upon Lucene" tool and the IBM "Semantic Search for Apache UIMA" tool
which is posted at
http://incubator.apache.org/uima/external-resources.html?
Thanks,
Andrew
On 6/18/07, Benjamin Sznajder <be...@il.ibm.com> wrote:
> Thank you Thilo!!
>
> The whole title is "XML Fragments Extended with Database Operators" by Yosi
> Mass, Dafna Sheinwald, Sivan Yogev and Benjamin Sznajder
>
> It will be soon online, I guess.
> Waiting for, any interested person can write to my personal mail...
>
> Benjamin
>
>
>
> Thilo Goetz
> <tw...@gmx.de>
> To
> 18/06/2007 18:08 uima-user@incubator.apache.org
> cc
>
> Please respond to Subject
> uima-user@incubat Re: Lucene and UIMA - we have it!-
> or.apache.org was Re: Human annotation tool for
> UIMA
>
>
>
>
>
>
>
>
>
>
> Benjamin Sznajder wrote:
> > Hi Andrew,
> >
> > Very motivating to get a so rapid reply!! Thanks!
> >
> > About your questions:
> > - I join here a paper we have recently presented at RIAO conference. It
> > describes the Query Languages we support in our package and some
> > implementation highlights. We have not yet official documentation for our
> > package itself. (See attached file: XMLFragment-camera-ready.pdf)
>
> Benjamin, I don't see any attachments. I think maybe the mailing list
> doesn't allow them.
>
> --Thilo
>
>
>
>
Re: Lucene and UIMA - we have it!- was Re: Human annotation tool for UIMA
Posted by Benjamin Sznajder <be...@il.ibm.com>.
Thank you Thilo!!
The whole title is "XML Fragments Extended with Database Operators" by Yosi
Mass, Dafna Sheinwald, Sivan Yogev and Benjamin Sznajder
It will be soon online, I guess.
Waiting for, any interested person can write to my personal mail...
Benjamin
Thilo Goetz
<tw...@gmx.de>
To
18/06/2007 18:08 uima-user@incubator.apache.org
cc
Please respond to Subject
uima-user@incubat Re: Lucene and UIMA - we have it!-
or.apache.org was Re: Human annotation tool for
UIMA
Benjamin Sznajder wrote:
> Hi Andrew,
>
> Very motivating to get a so rapid reply!! Thanks!
>
> About your questions:
> - I join here a paper we have recently presented at RIAO conference. It
> describes the Query Languages we support in our package and some
> implementation highlights. We have not yet official documentation for our
> package itself. (See attached file: XMLFragment-camera-ready.pdf)
Benjamin, I don't see any attachments. I think maybe the mailing list
doesn't allow them.
--Thilo
Re: Lucene and UIMA - we have it!- was Re: Human annotation tool
for UIMA
Posted by Thilo Goetz <tw...@gmx.de>.
Benjamin Sznajder wrote:
> Hi Andrew,
>
> Very motivating to get a so rapid reply!! Thanks!
>
> About your questions:
> - I join here a paper we have recently presented at RIAO conference. It
> describes the Query Languages we support in our package and some
> implementation highlights. We have not yet official documentation for our
> package itself. (See attached file: XMLFragment-camera-ready.pdf)
Benjamin, I don't see any attachments. I think maybe the mailing list
doesn't allow them.
--Thilo
Re: Lucene and UIMA - we have it!- was Re: Human annotation tool for UIMA
Posted by Benjamin Sznajder <be...@il.ibm.com>.
Hi Andrew,
Very motivating to get a so rapid reply!! Thanks!
About your questions:
- I join here a paper we have recently presented at RIAO conference. It
describes the Query Languages we support in our package and some
implementation highlights. We have not yet official documentation for our
package itself. (See attached file: XMLFragment-camera-ready.pdf)
- The licensign terms issue is still an open question for us. It is not yet
widely available. We are exploring several options. Obviously, we will let
know as soon it is decided.
I would be interested to hear from you if you already know some use cases
you consider with such capability? Which kind of data and which kind of
queries would you be interested to use with such package? We are, here,
very interested to understand scenarios needed by UIMA users!
Best regards,
Benjamin Sznajder
IBM Haifa Research Laboratory
Information Retrieval Group.
"Andrew
Borthwick"
<andrew@corp.spoc To
k.com> uima-user@incubator.apache.org
cc
14/06/2007 17:21
Subject
Re: Lucene and UIMA - we have it!-
Please respond to was Re: Human annotation tool for
uima-user@incubat UIMA
or.apache.org
Benjamin,
Sounds good!
A couple questions:
--Do you have a paper or documentation on this package?
--What sort of licensing terms is the package available under?
Thanks,
Andrew Borthwick, Ph.D.
Principal Scientist
Spock Networks
On 6/14/07, Benjamin Sznajder <be...@il.ibm.com> wrote:
> Hi,
>
> We have developped in IBM research lab in Haifa a tool corresponding to
> your description! :)
> Our package -we have named Semantic Search upon Lucene- lets the user
> indexing the annotations of the text, and querying some "Semantic
Queries"
> including constraints over the Annotations and the free-text in the span
of
> these Annotations.
> Our implementation uses the Lucene's payloads and we have written an
> efficient algorithm for efficiently retrieving "Twig" queries of
Structure
> (Annotations) and content (free-text or boolean constraints).
>
> The kind of queries you can ask are "free terms in the span of an
> annotation, that is itself in the span of another annotation etc.."
> including boolean constraints and attribute constraints.
> Two query languages are currently supported : XML Fragments and a subset
of
> XPath. The XML Fragments possesses operators especially designed for
> querying the intersection of annotations that can occur with UIMA.
>
> Currently, our package is not available as Open Source.
>
> I will answer with pleasure to any questions relative to this project!
>
> Best regards,
>
> Benjamin Sznajder
> IBM Haifa Research Laboratory
> Information Retrieval Group.
>
>
>
>
>
>
> Thilo Goetz
> <tw...@gmx.de>
>
To
> 08/06/2007 14:59 uima-user@incubator.apache.org
>
cc
>
> Please respond to
Subject
> uima-user@incubat Re: Lucene and UIMA was Re: Human
> or.apache.org annotation tool for UIMA
>
>
>
>
>
>
>
>
>
>
> Grant Ingersoll wrote:
> [...]
> > I'm pretty new to UIMA, but know a thing or two about Lucene. Care to
> > share more about what kinds of things you are interested in? Are you
> > talking Semantic Web type stuff or things like enhanced NLP search? I
> > see from the Javadocs that there is a place to hook in search
> > implementations, but haven't dug deeper than that.
> >
> >
> > --------------------------
> > Grant Ingersoll
> > Center for Natural Language Processing
> > http://www.cnlp.org/tech/lucene.asp
> >
> > Read the Lucene Java FAQ at
> http://wiki.apache.org/jakarta-lucene/LuceneFAQ
> >
>
> The IBM version of UIMA comes with a search engine that lets you search
for
> UIMA annotations. Suppose you have book description annotations, and
> inside
> those, author, title etc. annotations. With this search engine, you are
> able
> to, for example, search for title and author words that occur within the
> same
> book annotation (and there could be many in a given document). It's like
> searching XML documents with XPath and ftcontains. You didn't use to be
> able
> to do this sort of thing with Lucene, but with the new payloads, you
could
> implement something like it. That's what I would call a non-trivial
> integration
> of UIMA into Lucene.
>
> --Thilo
>
>
>
>
Re: Lucene and UIMA - we have it!- was Re: Human annotation tool for UIMA
Posted by Andrew Borthwick <an...@corp.spock.com>.
Benjamin,
Sounds good!
A couple questions:
--Do you have a paper or documentation on this package?
--What sort of licensing terms is the package available under?
Thanks,
Andrew Borthwick, Ph.D.
Principal Scientist
Spock Networks
On 6/14/07, Benjamin Sznajder <be...@il.ibm.com> wrote:
> Hi,
>
> We have developped in IBM research lab in Haifa a tool corresponding to
> your description! :)
> Our package -we have named Semantic Search upon Lucene- lets the user
> indexing the annotations of the text, and querying some "Semantic Queries"
> including constraints over the Annotations and the free-text in the span of
> these Annotations.
> Our implementation uses the Lucene's payloads and we have written an
> efficient algorithm for efficiently retrieving "Twig" queries of Structure
> (Annotations) and content (free-text or boolean constraints).
>
> The kind of queries you can ask are "free terms in the span of an
> annotation, that is itself in the span of another annotation etc.."
> including boolean constraints and attribute constraints.
> Two query languages are currently supported : XML Fragments and a subset of
> XPath. The XML Fragments possesses operators especially designed for
> querying the intersection of annotations that can occur with UIMA.
>
> Currently, our package is not available as Open Source.
>
> I will answer with pleasure to any questions relative to this project!
>
> Best regards,
>
> Benjamin Sznajder
> IBM Haifa Research Laboratory
> Information Retrieval Group.
>
>
>
>
>
>
> Thilo Goetz
> <tw...@gmx.de>
> To
> 08/06/2007 14:59 uima-user@incubator.apache.org
> cc
>
> Please respond to Subject
> uima-user@incubat Re: Lucene and UIMA was Re: Human
> or.apache.org annotation tool for UIMA
>
>
>
>
>
>
>
>
>
>
> Grant Ingersoll wrote:
> [...]
> > I'm pretty new to UIMA, but know a thing or two about Lucene. Care to
> > share more about what kinds of things you are interested in? Are you
> > talking Semantic Web type stuff or things like enhanced NLP search? I
> > see from the Javadocs that there is a place to hook in search
> > implementations, but haven't dug deeper than that.
> >
> >
> > --------------------------
> > Grant Ingersoll
> > Center for Natural Language Processing
> > http://www.cnlp.org/tech/lucene.asp
> >
> > Read the Lucene Java FAQ at
> http://wiki.apache.org/jakarta-lucene/LuceneFAQ
> >
>
> The IBM version of UIMA comes with a search engine that lets you search for
> UIMA annotations. Suppose you have book description annotations, and
> inside
> those, author, title etc. annotations. With this search engine, you are
> able
> to, for example, search for title and author words that occur within the
> same
> book annotation (and there could be many in a given document). It's like
> searching XML documents with XPath and ftcontains. You didn't use to be
> able
> to do this sort of thing with Lucene, but with the new payloads, you could
> implement something like it. That's what I would call a non-trivial
> integration
> of UIMA into Lucene.
>
> --Thilo
>
>
>
>
Re: Lucene and UIMA - we have it!- was Re: Human annotation tool for UIMA
Posted by Benjamin Sznajder <be...@il.ibm.com>.
Hi,
We have developped in IBM research lab in Haifa a tool corresponding to
your description! :)
Our package -we have named Semantic Search upon Lucene- lets the user
indexing the annotations of the text, and querying some "Semantic Queries"
including constraints over the Annotations and the free-text in the span of
these Annotations.
Our implementation uses the Lucene's payloads and we have written an
efficient algorithm for efficiently retrieving "Twig" queries of Structure
(Annotations) and content (free-text or boolean constraints).
The kind of queries you can ask are "free terms in the span of an
annotation, that is itself in the span of another annotation etc.."
including boolean constraints and attribute constraints.
Two query languages are currently supported : XML Fragments and a subset of
XPath. The XML Fragments possesses operators especially designed for
querying the intersection of annotations that can occur with UIMA.
Currently, our package is not available as Open Source.
I will answer with pleasure to any questions relative to this project!
Best regards,
Benjamin Sznajder
IBM Haifa Research Laboratory
Information Retrieval Group.
Thilo Goetz
<tw...@gmx.de>
To
08/06/2007 14:59 uima-user@incubator.apache.org
cc
Please respond to Subject
uima-user@incubat Re: Lucene and UIMA was Re: Human
or.apache.org annotation tool for UIMA
Grant Ingersoll wrote:
[...]
> I'm pretty new to UIMA, but know a thing or two about Lucene. Care to
> share more about what kinds of things you are interested in? Are you
> talking Semantic Web type stuff or things like enhanced NLP search? I
> see from the Javadocs that there is a place to hook in search
> implementations, but haven't dug deeper than that.
>
>
> --------------------------
> Grant Ingersoll
> Center for Natural Language Processing
> http://www.cnlp.org/tech/lucene.asp
>
> Read the Lucene Java FAQ at
http://wiki.apache.org/jakarta-lucene/LuceneFAQ
>
The IBM version of UIMA comes with a search engine that lets you search for
UIMA annotations. Suppose you have book description annotations, and
inside
those, author, title etc. annotations. With this search engine, you are
able
to, for example, search for title and author words that occur within the
same
book annotation (and there could be many in a given document). It's like
searching XML documents with XPath and ftcontains. You didn't use to be
able
to do this sort of thing with Lucene, but with the new payloads, you could
implement something like it. That's what I would call a non-trivial
integration
of UIMA into Lucene.
--Thilo
Re: Lucene and UIMA was Re: Human annotation tool for UIMA
Posted by Thilo Goetz <tw...@gmx.de>.
Grant Ingersoll wrote:
[...]
> I'm pretty new to UIMA, but know a thing or two about Lucene. Care to
> share more about what kinds of things you are interested in? Are you
> talking Semantic Web type stuff or things like enhanced NLP search? I
> see from the Javadocs that there is a place to hook in search
> implementations, but haven't dug deeper than that.
>
>
> --------------------------
> Grant Ingersoll
> Center for Natural Language Processing
> http://www.cnlp.org/tech/lucene.asp
>
> Read the Lucene Java FAQ at http://wiki.apache.org/jakarta-lucene/LuceneFAQ
>
The IBM version of UIMA comes with a search engine that lets you search for
UIMA annotations. Suppose you have book description annotations, and inside
those, author, title etc. annotations. With this search engine, you are able
to, for example, search for title and author words that occur within the same
book annotation (and there could be many in a given document). It's like
searching XML documents with XPath and ftcontains. You didn't use to be able
to do this sort of thing with Lucene, but with the new payloads, you could
implement something like it. That's what I would call a non-trivial integration
of UIMA into Lucene.
--Thilo