You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Otis Gospodnetic <ot...@yahoo.com> on 2012/09/11 17:51:07 UTC
Semantic document format... standards?
Hello,
If I'm extracting named entities, topics, key phrases/tags, etc. from documents and I want to have a representation of this document, what format should I use? Are there any standard or at least common formats or approaches people use in such situations?
For example, the most straight forward format might be something like this:
<document>
<title>doc title</title>
<keywords>meta keywords coming from the web page</keywords>
<content>page meat</content>
<entities>name entities recognized in the document</entities>
<topics>topics extracted by the annotator</topics>
<tags>tags extracted by the annotator</tags>
<relations>relations extracted by the annotator</relations>
</document>
But this is a made up format - the XML tags above are just what somebody happened to pick.
Are there any standard or at least common formats for this?
Thanks,
Otis
----
Performance Monitoring - Solr - ElasticSearch - HBase - http://sematext.com/spm
Search Analytics - http://sematext.com/search-analytics/index.html
Re: Semantic document format... standards?
Posted by Paul Libbrecht <pa...@hoplahup.net>.
As Michael hinted, I believe RDF would be the de-factor answer.
Within it, things such as OWL or SKOS certainly represent classical formats.
Processors such as OWLAPI can go pretty far there.
As Péter hinted, schema.org might provide a way to complement an existing XML with semantic information. The big support everyone talks about (because apparently the big names speak out), I haven't seen yet very present; in particular in terms of shared toolset.
There's many many many alternatives. We've been recently touching with a format call DITA which is an XML format for annotated documents and it also claims to provide semantic support (e.g. with taxonomies).
Is your goal to serve these as food for solr to index?
Paul
Le 11 sept. 2012 à 17:51, Otis Gospodnetic a écrit :
> Hello,
>
> If I'm extracting named entities, topics, key phrases/tags, etc. from documents and I want to have a representation of this document, what format should I use? Are there any standard or at least common formats or approaches people use in such situations?
>
> For example, the most straight forward format might be something like this:
>
>
> <document>
> <title>doc title</title>
> <keywords>meta keywords coming from the web page</keywords>
> <content>page meat</content>
> <entities>name entities recognized in the document</entities>
> <topics>topics extracted by the annotator</topics>
> <tags>tags extracted by the annotator</tags>
> <relations>relations extracted by the annotator</relations>
> </document>
>
> But this is a made up format - the XML tags above are just what somebody happened to pick.
>
> Are there any standard or at least common formats for this?
Re: Semantic document format... standards?
Posted by Michael Della Bitta <mi...@appinions.com>.
Actually at my company, we do a lot of NLP work and we've ended up
using bespoke formats, formerly a FeatureStructure serialized to JSON,
but most recently in protobufs. Possibly not the answer you were
looking for, Otis, but at least it's a datapoint.
Michael Della Bitta
------------------------------------------------
Appinions | 18 East 41st St., Suite 1806 | New York, NY 10017
www.appinions.com
Where Influence Isn’t a Game
On Wed, Sep 12, 2012 at 7:36 AM, Alexandre Rafalovitch
<ar...@gmail.com> wrote:
> Otis,
>
> If you are doing Named Entity Recognition, you may want to look at the
> research area concerned with Named Entity Recognition. :-) In general,
> there is inline markup and standoff markup. You seem to be going for
> standoff/stand-alone markup. I am not clear though whether it is just
> 'discovery' format or actual annotation format (with reference to
> where in the sentence it is with offsets or token ids).
>
> UIMA (which Solr integrate with already, right?), does NER so it must
> be using some sort of format.
>
> Also, TREC is one of the competitions and they provide marked-up
> datasets you might be able to learn something from:
> http://ilps.science.uva.nl/trec-entity/
>
> If you are not sure where to start with NER, you can look at my
> collection of papers, though most of them are probably too specific:
> http://www.citeulike.org/user/arafalov
>
> Finally, if you have to deal with overlapping entities, there was an
> article about a month about some sort of general format. I can't seem
> to find the article right now, but I could try digging if you are
> still stuck.
>
> Regards,
> Alex.
> Personal blog: http://blog.outerthoughts.com/
> LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
> - Time is the quality of nature that keeps events from happening all
> at once. Lately, it doesn't seem to be working. (Anonymous - via GTD
> book)
>
>
> On Tue, Sep 11, 2012 at 11:51 AM, Otis Gospodnetic
> <ot...@yahoo.com> wrote:
>> Hello,
>>
>> If I'm extracting named entities, topics, key phrases/tags, etc. from documents and I want to have a representation of this document, what format should I use? Are there any standard or at least common formats or approaches people use in such situations?
>>
>> For example, the most straight forward format might be something like this:
>>
>>
>> <document>
>> <title>doc title</title>
>> <keywords>meta keywords coming from the web page</keywords>
>> <content>page meat</content>
>> <entities>name entities recognized in the document</entities>
>> <topics>topics extracted by the annotator</topics>
>> <tags>tags extracted by the annotator</tags>
>> <relations>relations extracted by the annotator</relations>
>> </document>
>>
>> But this is a made up format - the XML tags above are just what somebody happened to pick.
>>
>> Are there any standard or at least common formats for this?
>>
>>
>> Thanks,
>> Otis
>> ----
>> Performance Monitoring - Solr - ElasticSearch - HBase - http://sematext.com/spm
>>
>> Search Analytics - http://sematext.com/search-analytics/index.html
Re: Semantic document format... standards?
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Otis,
If you are doing Named Entity Recognition, you may want to look at the
research area concerned with Named Entity Recognition. :-) In general,
there is inline markup and standoff markup. You seem to be going for
standoff/stand-alone markup. I am not clear though whether it is just
'discovery' format or actual annotation format (with reference to
where in the sentence it is with offsets or token ids).
UIMA (which Solr integrate with already, right?), does NER so it must
be using some sort of format.
Also, TREC is one of the competitions and they provide marked-up
datasets you might be able to learn something from:
http://ilps.science.uva.nl/trec-entity/
If you are not sure where to start with NER, you can look at my
collection of papers, though most of them are probably too specific:
http://www.citeulike.org/user/arafalov
Finally, if you have to deal with overlapping entities, there was an
article about a month about some sort of general format. I can't seem
to find the article right now, but I could try digging if you are
still stuck.
Regards,
Alex.
Personal blog: http://blog.outerthoughts.com/
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
- Time is the quality of nature that keeps events from happening all
at once. Lately, it doesn't seem to be working. (Anonymous - via GTD
book)
On Tue, Sep 11, 2012 at 11:51 AM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
> Hello,
>
> If I'm extracting named entities, topics, key phrases/tags, etc. from documents and I want to have a representation of this document, what format should I use? Are there any standard or at least common formats or approaches people use in such situations?
>
> For example, the most straight forward format might be something like this:
>
>
> <document>
> <title>doc title</title>
> <keywords>meta keywords coming from the web page</keywords>
> <content>page meat</content>
> <entities>name entities recognized in the document</entities>
> <topics>topics extracted by the annotator</topics>
> <tags>tags extracted by the annotator</tags>
> <relations>relations extracted by the annotator</relations>
> </document>
>
> But this is a made up format - the XML tags above are just what somebody happened to pick.
>
> Are there any standard or at least common formats for this?
>
>
> Thanks,
> Otis
> ----
> Performance Monitoring - Solr - ElasticSearch - HBase - http://sematext.com/spm
>
> Search Analytics - http://sematext.com/search-analytics/index.html
Re: Semantic document format... standards?
Posted by Paul Libbrecht <pa...@hoplahup.net>.
Otis,
if you have a bit of time to research, I think your document may look a lot like the documents processed by:
http://langtech.jrc.it/
which is a flagship "multilingual technology" implementation and includes a fair amount of entity disambiguation as far as I could hear in Ralph's talk.
I do not have a more concrete pointer, sorry, and I would love to read something more concretely closer to solr about them.
Paul
Le 12 sept. 2012 à 00:46, Jack Krupansky a écrit :
> My standard question for such a situation: How are you expecting your users to query this data? Are they expecting simple English/natural language text, or are they expecting structured "identifiers" that can be keys into other data sources.
>
> For example, are your "entities" simple text literal names, or might they be Dublin Core (DC) "Agent" URI identifiers?
>
> Ditto for topics - free text vs. some SKOS vocabulary or other form of taxonomy.
>
> In other words, clue us in as to your client "requirements".
>
> -- Jack Krupansky
>
> -----Original Message----- From: Otis Gospodnetic
> Sent: Tuesday, September 11, 2012 11:51 AM
> To: solr-user@lucene.apache.org
> Subject: Semantic document format... standards?
>
> Hello,
>
> If I'm extracting named entities, topics, key phrases/tags, etc. from documents and I want to have a representation of this document, what format should I use? Are there any standard or at least common formats or approaches people use in such situations?
>
> For example, the most straight forward format might be something like this:
>
>
> <document>
> <title>doc title</title>
> <keywords>meta keywords coming from the web page</keywords>
> <content>page meat</content>
> <entities>name entities recognized in the document</entities>
> <topics>topics extracted by the annotator</topics>
> <tags>tags extracted by the annotator</tags>
> <relations>relations extracted by the annotator</relations>
> </document>
>
> But this is a made up format - the XML tags above are just what somebody happened to pick.
>
> Are there any standard or at least common formats for this?
>
>
> Thanks,
> Otis
> ----
> Performance Monitoring - Solr - ElasticSearch - HBase - http://sematext.com/spm
>
> Search Analytics - http://sematext.com/search-analytics/index.html
Re: Semantic document format... standards?
Posted by Jack Krupansky <ja...@basetechnology.com>.
My standard question for such a situation: How are you expecting your users
to query this data? Are they expecting simple English/natural language text,
or are they expecting structured "identifiers" that can be keys into other
data sources.
For example, are your "entities" simple text literal names, or might they be
Dublin Core (DC) "Agent" URI identifiers?
Ditto for topics - free text vs. some SKOS vocabulary or other form of
taxonomy.
In other words, clue us in as to your client "requirements".
-- Jack Krupansky
-----Original Message-----
From: Otis Gospodnetic
Sent: Tuesday, September 11, 2012 11:51 AM
To: solr-user@lucene.apache.org
Subject: Semantic document format... standards?
Hello,
If I'm extracting named entities, topics, key phrases/tags, etc. from
documents and I want to have a representation of this document, what format
should I use? Are there any standard or at least common formats or
approaches people use in such situations?
For example, the most straight forward format might be something like this:
<document>
<title>doc title</title>
<keywords>meta keywords coming from the web page</keywords>
<content>page meat</content>
<entities>name entities recognized in the document</entities>
<topics>topics extracted by the annotator</topics>
<tags>tags extracted by the annotator</tags>
<relations>relations extracted by the annotator</relations>
</document>
But this is a made up format - the XML tags above are just what somebody
happened to pick.
Are there any standard or at least common formats for this?
Thanks,
Otis
----
Performance Monitoring - Solr - ElasticSearch - HBase -
http://sematext.com/spm
Search Analytics - http://sematext.com/search-analytics/index.html
Re: Semantic document format... standards?
Posted by Péter Király <ki...@gmail.com>.
Hi,
I guess the most common format today is using the schema.org's ontologies.
It provides a couple of definitions, and it is supported by big players,
such as Google, Yahoo, Microsoft. See http://schema.org/.
Hope it helps,
Péter
<ot...@yahoo.com> wrote:
> > Hello,
> >
> > If I'm extracting named entities, topics, key phrases/tags, etc. from
> documents and I want to have a representation of this document, what format
> should I use? Are there any standard or at least common formats or
> approaches people use in such situations?
> >
> > For example, the most straight forward format might be something like
> this:
> >
> >
> > <document>
> > <title>doc title</title>
> > <keywords>meta keywords coming from the web page</keywords>
> > <content>page meat</content>
> > <entities>name entities recognized in the document</entities>
> > <topics>topics extracted by the annotator</topics>
> > <tags>tags extracted by the annotator</tags>
> > <relations>relations extracted by the annotator</relations>
> > </document>
> >
> > But this is a made up format - the XML tags above are just what somebody
> happened to pick.
> >
> > Are there any standard or at least common formats for this?
> >
> >
> > Thanks,
> > Otis
> > ----
> > Performance Monitoring - Solr - ElasticSearch - HBase -
> http://sematext.com/spm
> >
> > Search Analytics - http://sematext.com/search-analytics/index.html
>
On Tue, Sep 11, 2012 at 11:51 AM, Otis Gospodnetic
--
Péter Király
eXtensible Catalog
http://eXtensibleCatalog.org
http://drupal.org/project/xc
Re: Semantic document format... standards?
Posted by Michael Della Bitta <mi...@appinions.com>.
I'm probably a little unclear about the breadth of what you want to
do, but I would recommend DC at the extremely lightweight end, and TEI
at the very heavyweight end. Perhaps you could come up with a mashup
of DC and your own fields in RDF as well.
Michael Della Bitta
------------------------------------------------
Appinions | 18 East 41st St., Suite 1806 | New York, NY 10017
www.appinions.com
Where Influence Isn’t a Game
On Tue, Sep 11, 2012 at 11:51 AM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
> Hello,
>
> If I'm extracting named entities, topics, key phrases/tags, etc. from documents and I want to have a representation of this document, what format should I use? Are there any standard or at least common formats or approaches people use in such situations?
>
> For example, the most straight forward format might be something like this:
>
>
> <document>
> <title>doc title</title>
> <keywords>meta keywords coming from the web page</keywords>
> <content>page meat</content>
> <entities>name entities recognized in the document</entities>
> <topics>topics extracted by the annotator</topics>
> <tags>tags extracted by the annotator</tags>
> <relations>relations extracted by the annotator</relations>
> </document>
>
> But this is a made up format - the XML tags above are just what somebody happened to pick.
>
> Are there any standard or at least common formats for this?
>
>
> Thanks,
> Otis
> ----
> Performance Monitoring - Solr - ElasticSearch - HBase - http://sematext.com/spm
>
> Search Analytics - http://sematext.com/search-analytics/index.html