You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tika.apache.org by Ista Pouss <is...@gmail.com> on 2010/10/23 14:18:50 UTC
Tika for mediawiki ?
Hi,
Two newbis questions :
- Is there a tika parser for mediawiki
(http://www.mediawiki.org/wiki/MediaWiki) ?
- Is it possible to write in the supported formats with tika ?
Thanks.
Re: Tika for mediawiki ?
Posted by Ista Pouss <is...@gmail.com>.
2010/10/25 Paul Jakubik <pa...@purediscovery.com>:
>
> If your goal is to index or perform any kind of text analysis of mediawiki
> pages, I understand why you want to parse the page since the markup tends to
> mess up text analysis.
My goal is to perform some text analysis, and some structure analysis
on the pages.
It's for text search, and also to reformat these pages ; something
like a mediawiki bots.
It's also to write some wikipedia pages, but I understand it's outside
the tika scope.
I'cant' download all wikipedia pages : I need to work with the mediawiki api.
If I undestand, there is two big difficulties : no mime type, and a
complicated markup. I'm going to sit down and think a little bit.
Thanks.
Re: Tika for mediawiki ?
Posted by Paul Jakubik <pa...@purediscovery.com>.
On Mon, Oct 25, 2010 at 3:14 AM, Jan Høydahl / Cominvent <
jan.asf@cominvent.com> wrote:
>
> If I were you, I'd build a standalone program which interfaces your wiki
> (If wikipedia, perhaps download it at
> http://en.wikipedia.org/wiki/Wikipedia:Database_download), parses and feed
> to your index or whatever you need.
>
>
If your goal is to index or perform any kind of text analysis of mediawiki
pages, I understand why you want to parse the page since the markup tends to
mess up text analysis.
If your goal is to look at wikipedia pages, I recommend downloading the
Freebase Wikipedia Extraction (WEX) (http://download.freebase.com/wex/)
instead of the wikipedia database download. If you download the articles
(current latest at
http://download.freebase.com/wex/latest/freebase-wex-2010-10-09-articles.tsv.bz2)
one of the fields for each article is the text extracted from the wikipedia
article. One of the original mediawiki developers wrote the text extractor
for freebase, and if nothing else it does a better job of extracting text
than the code I wrote a few years ago :-)
While I still do a little bit of cleanup before performing text analysis on
WEX, that cleanup is nothing compared to what I had to do to try to get
clean text out of mediawiki formatted text.
Paul
Re: Tika for mediawiki ?
Posted by Jan Høydahl / Cominvent <ja...@cominvent.com>.
Here's an Apache2 licensed parser: http://code.google.com/p/wikimodel/source/browse/trunk/org.wikimodel.wem/src/main/java/org/wikimodel/wem/mediawiki/
But I think too there's probably no need to involve Tika, unless you have a file system with tons of different files where plaintext wiki files are some of them. It is hard to detect the type of a mediawiki file, as I don't think they have a standard filename suffix or magic byte sequence. You'd need to start scanning for parts of the markup.
If I were you, I'd build a standalone program which interfaces your wiki (If wikipedia, perhaps download it at http://en.wikipedia.org/wiki/Wikipedia:Database_download), parses and feed to your index or whatever you need.
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
On 25. okt. 2010, at 09.27, Ista Pouss wrote:
> Hi,
>
> I thank Mattmann Chris and Paul Jakubik.
>
> There is no official spec of the markup langage. There are some
> parsers... I find "Wiki2HtmlJavaProgram"
> (http://community.jboss.org/wiki/Wiki2HtmlJavaProgram) and "jwpl"
> (http://code.google.com/p/jwpl/). Perhaps it's best to start from
> scratch with antlr ?
>
> My use case is to catch and analyse wikipedia pages about natural life theme.
>
> I think there is no mime type for wikipedia source code page. M.
> Mattman says if "someone... wants to throw out there a best practice
> on the MIME spec"... what is that ?
>
> Perhaps I'm going to start My Little Tika Project on that, but I'm
> afraid I cant't do all the mess and stuff and foo about mediawiki
> markup...
>
> Thanks.
>
>
> 2010/10/24 Mattmann, Chris A (388J) <ch...@jpl.nasa.gov>:
>> Hi Guys,
>>
>>> [...]
>>> Until there is a complete spec for parsing media wiki markup, or a java
>>> library that does a good job of extracting text from documents formatted with
>>> media wiki markup, I don't think extracting text from media wiki markup
>>> documents is in scope for Tika.
>>
>> I'd disagree with that. We never have complete specs for *many* of the
>> existing formats we tackle in Tika, and there are exceptions and bugs and
>> platform-specific things that are found all the time that require
>> accommodations.
>>
>> I'd say if someone can find a parsing library for Media-wiki format, and
>> wants to throw out there a best practice on the MIME spec, or if someone was
>> even willing to roll their own parsing library, I'd welcome the
>> contribution.
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: Chris.Mattmann@jpl.nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
Re: Tika for mediawiki ?
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Mon, Oct 25, 2010 at 10:27 AM, Ista Pouss <is...@gmail.com> wrote:
> There is no official spec of the markup langage. There are some
> parsers... I find "Wiki2HtmlJavaProgram"
> (http://community.jboss.org/wiki/Wiki2HtmlJavaProgram) and "jwpl"
> (http://code.google.com/p/jwpl/). Perhaps it's best to start from
> scratch with antlr ?
Note that since the MediaWiki markup is practically plain text with
some structural formatting rules, you can get pretty far with Tika's
normal plain text parser unless you really need the structural
information.
Or if you already have the markup of a wiki page available as a string
or a character stream (for example if you're accessing the underlying
database or JSON exports directly), then there may be no need to
involve Tika in the process.
BR,
Jukka Zitting
Re: Tika for mediawiki ?
Posted by Ista Pouss <is...@gmail.com>.
Hi,
I thank Mattmann Chris and Paul Jakubik.
There is no official spec of the markup langage. There are some
parsers... I find "Wiki2HtmlJavaProgram"
(http://community.jboss.org/wiki/Wiki2HtmlJavaProgram) and "jwpl"
(http://code.google.com/p/jwpl/). Perhaps it's best to start from
scratch with antlr ?
My use case is to catch and analyse wikipedia pages about natural life theme.
I think there is no mime type for wikipedia source code page. M.
Mattman says if "someone... wants to throw out there a best practice
on the MIME spec"... what is that ?
Perhaps I'm going to start My Little Tika Project on that, but I'm
afraid I cant't do all the mess and stuff and foo about mediawiki
markup...
Thanks.
2010/10/24 Mattmann, Chris A (388J) <ch...@jpl.nasa.gov>:
> Hi Guys,
>
>> [...]
>> Until there is a complete spec for parsing media wiki markup, or a java
>> library that does a good job of extracting text from documents formatted with
>> media wiki markup, I don't think extracting text from media wiki markup
>> documents is in scope for Tika.
>
> I'd disagree with that. We never have complete specs for *many* of the
> existing formats we tackle in Tika, and there are exceptions and bugs and
> platform-specific things that are found all the time that require
> accommodations.
>
> I'd say if someone can find a parsing library for Media-wiki format, and
> wants to throw out there a best practice on the MIME spec, or if someone was
> even willing to roll their own parsing library, I'd welcome the
> contribution.
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: Chris.Mattmann@jpl.nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
Re: Tika for mediawiki ?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Guys,
> [...]
> Until there is a complete spec for parsing media wiki markup, or a java
> library that does a good job of extracting text from documents formatted with
> media wiki markup, I don't think extracting text from media wiki markup
> documents is in scope for Tika.
I'd disagree with that. We never have complete specs for *many* of the
existing formats we tackle in Tika, and there are exceptions and bugs and
platform-specific things that are found all the time that require
accommodations.
I'd say if someone can find a parsing library for Media-wiki format, and
wants to throw out there a best practice on the MIME spec, or if someone was
even willing to roll their own parsing library, I'd welcome the
contribution.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Tika for mediawiki ?
Posted by Paul Jakubik <pa...@purediscovery.com>.
On Sun, Oct 24, 2010 at 8:32 AM, Ista Pouss <is...@gmail.com> wrote:
> 2010/10/24 Jukka Zitting <ju...@gmail.com>:
> >
> > No. MediaWiki uses a database backend instead of a special file format
> > for storing data, so you'd need to use something like the ManifoldCF
> > (http://incubator.apache.org/connectors/) to extract information from
> > a MediaWiki installation.
> >
>
> Yes, but it's also possible to use the media wiki API
> (http://www.mediawiki.org/wiki/API) and read json, yaml, xml etc
> format. It's also possible to read the mediawiki code of a simple page
> (http://en.wikipedia.org/wiki/Lucene?action=raw, to get the mediawiki
> source of Lucene page). Is it possible to make an extractor with that,
> or is it best to do with Manifold ?
>
>
I think using the media wiki API is outside of the scope of Tika.
Parsing media wiki markup, and extracting text from a document that is
formatted using media wiki markup is potentially inside the scope of Tika.
This text extraction would not load templates or other content that the
markup referred to and would normally appear as part of the page, but it
could extract the text present in a raw media wiki formatted document. Even
if this limitation is acceptable, I think we would also have to answer the
following questions before media wiki parsing could be added to Tika:
- How would Tika know that it needs to perform media wiki markup parsing?
Is there a mime type for that?
- Is there a parsing library available for extracting text from media
wiki markup documents?
I think the second item is the more difficult part of extracting text from
media wiki markup. While many uses of media wiki markup are simple and
straightforward, the entire markup language is not. As far as I can tell,
there is still no spec for the language though this page appears to be the
current best attempt at a spec http://www.mediawiki.org/wiki/Markup_spec.
Until there is a complete spec for parsing media wiki markup, or a java
library that does a good job of extracting text from documents formatted
with media wiki markup, I don't think extracting text from media wiki markup
documents is in scope for Tika.
Paul
Re: Tika for mediawiki ?
Posted by Ista Pouss <is...@gmail.com>.
2010/10/24 Jukka Zitting <ju...@gmail.com>:
>
> No. MediaWiki uses a database backend instead of a special file format
> for storing data, so you'd need to use something like the ManifoldCF
> (http://incubator.apache.org/connectors/) to extract information from
> a MediaWiki installation.
>
Yes, but it's also possible to use the media wiki API
(http://www.mediawiki.org/wiki/API) and read json, yaml, xml etc
format. It's also possible to read the mediawiki code of a simple page
(http://en.wikipedia.org/wiki/Lucene?action=raw, to get the mediawiki
source of Lucene page). Is it possible to make an extractor with that,
or is it best to do with Manifold ?
Thanks.
Re: Tika for mediawiki ?
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Sat, Oct 23, 2010 at 2:18 PM, Ista Pouss <is...@gmail.com> wrote:
> - Is there a tika parser for mediawiki
> (http://www.mediawiki.org/wiki/MediaWiki) ?
No. MediaWiki uses a database backend instead of a special file format
for storing data, so you'd need to use something like the ManifoldCF
(http://incubator.apache.org/connectors/) to extract information from
a MediaWiki installation.
> - Is it possible to write in the supported formats with tika ?
No. Tika only supports extracting information from documents, not
writing them. You can use the underlying parser libraries like POI or
PDFBox directly to produce documents in a specific format.
BR,
Jukka Zitting