You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Allison, Timothy B." <ta...@mitre.org> on 2015/11/30 13:24:07 UTC
more modular parser bundles
All,
I'm extremely grateful for all of the new nlp +image processing parsers that we're adding. Might it be time to start down the implementation path to more modular parser bundles?
Perhaps we could start with a tika-advanced-bundle to gather all of the nlp/advanced parsers? Or would this have to wait for Tika 2.0?
Bob got us off to a great start. There hasn't been much discussion since August. I think my email from 24 Aug [1] was the last?
Cheers,
Tim
[1] https://mail-archives.apache.org/mod_mbox/tika-dev/201508.mbox/%3cDM2PR09MB071305DFD203E21BFBE7A63AC7620@DM2PR09MB0713.namprd09.prod.outlook.com%3e
-----Original Message-----
From: Madhav Sharan (JIRA) [mailto:jira@apache.org]
Sent: Wednesday, November 25, 2015 6:16 PM
To: dev@tika.apache.org
Subject: [jira] [Created] (TIKA-1803) Use lucene-geo-gazetteer REST API in GeoTopicParser
Madhav Sharan created TIKA-1803:
-----------------------------------
Summary: Use lucene-geo-gazetteer REST API in GeoTopicParser
Key: TIKA-1803
URL: https://issues.apache.org/jira/browse/TIKA-1803
Project: Tika
Issue Type: Sub-task
Components: parser
Reporter: Madhav Sharan
As of now tika uses lucene-geo-gazetteer CLI to extract co-ordinates of a location. CLI requires jvm and lucene to instantiate for every request. With all new REST api it will be possible to gain improvement in this space.
Idea is to create a client of lucene-geo-gazetteer in tika and use it in GeoTopicParser
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: more modular parser bundles
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Tim,
Fully agreed. One solution that presents itself to me is to finish
up the Git discuss (which was overwhelmingly positive, and I need
to write a wiki page for Nick), get that VOTE out of the way, move
to Git, then basically have two main branches of development. I’d
like 1.x to continue as-is, and then to create a 2.x branch and
let Bob go to town on it (and others that are interested). Then,
once you guys are ready you can release on it out of 2.x, while
1.x maintenance and existing architecture keep pressing forward.
What do you think?
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: "Allison, Timothy B." <ta...@mitre.org>
Reply-To: "dev@tika.apache.org" <de...@tika.apache.org>
Date: Monday, November 30, 2015 at 4:24 AM
To: "dev@tika.apache.org" <de...@tika.apache.org>
Subject: more modular parser bundles
>All,
>
> I'm extremely grateful for all of the new nlp +image processing parsers
>that we're adding. Might it be time to start down the implementation
>path to more modular parser bundles?
>
> Perhaps we could start with a tika-advanced-bundle to gather all of the
>nlp/advanced parsers? Or would this have to wait for Tika 2.0?
>
> Bob got us off to a great start. There hasn't been much discussion
>since August. I think my email from 24 Aug [1] was the last?
>
> Cheers,
>
> Tim
>
>[1]
>https://mail-archives.apache.org/mod_mbox/tika-dev/201508.mbox/%3cDM2PR09M
>B071305DFD203E21BFBE7A63AC7620@DM2PR09MB0713.namprd09.prod.outlook.com%3e
>
>-----Original Message-----
>From: Madhav Sharan (JIRA) [mailto:jira@apache.org]
>Sent: Wednesday, November 25, 2015 6:16 PM
>To: dev@tika.apache.org
>Subject: [jira] [Created] (TIKA-1803) Use lucene-geo-gazetteer REST API
>in GeoTopicParser
>
>Madhav Sharan created TIKA-1803:
>-----------------------------------
>
> Summary: Use lucene-geo-gazetteer REST API in GeoTopicParser
> Key: TIKA-1803
> URL: https://issues.apache.org/jira/browse/TIKA-1803
> Project: Tika
> Issue Type: Sub-task
> Components: parser
> Reporter: Madhav Sharan
>
>
>As of now tika uses lucene-geo-gazetteer CLI to extract co-ordinates of a
>location. CLI requires jvm and lucene to instantiate for every request.
>With all new REST api it will be possible to gain improvement in this
>space.
>
>Idea is to create a client of lucene-geo-gazetteer in tika and use it in
>GeoTopicParser
>
>
>
>--
>This message was sent by Atlassian JIRA
>(v6.3.4#6332)
Re: more modular parser bundles
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Sure that’s fine Bob - we don’t need it to be gated on Git.
Create a 2.x branch and go to town, +1 from me :)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Bob Paulin <bo...@bobpaulin.com>
Reply-To: "dev@tika.apache.org" <de...@tika.apache.org>
Date: Monday, November 30, 2015 at 7:08 AM
To: "dev@tika.apache.org" <de...@tika.apache.org>
Subject: Re: more modular parser bundles
>Hi,
>
>I think Chris actually mentioned that this could be something targeted for
>a 2.0 release. The first step towards that would be to create the 2.0
>branch since I think this might be a big enough effort to not want to
>block
>the trunk ( or master if we move to git). Would the list agree that now
>would be a good time to branch?
>
>- Bob
>
>On Mon, Nov 30, 2015 at 6:24 AM, Allison, Timothy B. <ta...@mitre.org>
>wrote:
>
>> All,
>>
>> I'm extremely grateful for all of the new nlp +image processing
>>parsers
>> that we're adding. Might it be time to start down the implementation
>>path
>> to more modular parser bundles?
>>
>> Perhaps we could start with a tika-advanced-bundle to gather all of
>>the
>> nlp/advanced parsers? Or would this have to wait for Tika 2.0?
>>
>> Bob got us off to a great start. There hasn't been much discussion
>> since August. I think my email from 24 Aug [1] was the last?
>>
>> Cheers,
>>
>> Tim
>>
>> [1]
>>
>>https://mail-archives.apache.org/mod_mbox/tika-dev/201508.mbox/%3cDM2PR09
>>MB071305DFD203E21BFBE7A63AC7620@DM2PR09MB0713.namprd09.prod.outlook.com%3
>>e
>>
>> -----Original Message-----
>> From: Madhav Sharan (JIRA) [mailto:jira@apache.org]
>> Sent: Wednesday, November 25, 2015 6:16 PM
>> To: dev@tika.apache.org
>> Subject: [jira] [Created] (TIKA-1803) Use lucene-geo-gazetteer REST API
>>in
>> GeoTopicParser
>>
>> Madhav Sharan created TIKA-1803:
>> -----------------------------------
>>
>> Summary: Use lucene-geo-gazetteer REST API in
>>GeoTopicParser
>> Key: TIKA-1803
>> URL: https://issues.apache.org/jira/browse/TIKA-1803
>> Project: Tika
>> Issue Type: Sub-task
>> Components: parser
>> Reporter: Madhav Sharan
>>
>>
>> As of now tika uses lucene-geo-gazetteer CLI to extract co-ordinates of
>>a
>> location. CLI requires jvm and lucene to instantiate for every request.
>> With all new REST api it will be possible to gain improvement in this
>>space.
>>
>> Idea is to create a client of lucene-geo-gazetteer in tika and use it in
>> GeoTopicParser
>>
>>
>>
>> --
>> This message was sent by Atlassian JIRA
>> (v6.3.4#6332)
>>
Re: more modular parser bundles
Posted by Bob Paulin <bo...@bobpaulin.com>.
Chris,
Looks like we're on the same page :).
- Bob
On Mon, Nov 30, 2015 at 9:08 AM, Bob Paulin <bo...@bobpaulin.com> wrote:
> Hi,
>
> I think Chris actually mentioned that this could be something targeted for
> a 2.0 release. The first step towards that would be to create the 2.0
> branch since I think this might be a big enough effort to not want to block
> the trunk ( or master if we move to git). Would the list agree that now
> would be a good time to branch?
>
> - Bob
>
> On Mon, Nov 30, 2015 at 6:24 AM, Allison, Timothy B. <ta...@mitre.org>
> wrote:
>
>> All,
>>
>> I'm extremely grateful for all of the new nlp +image processing parsers
>> that we're adding. Might it be time to start down the implementation path
>> to more modular parser bundles?
>>
>> Perhaps we could start with a tika-advanced-bundle to gather all of the
>> nlp/advanced parsers? Or would this have to wait for Tika 2.0?
>>
>> Bob got us off to a great start. There hasn't been much discussion
>> since August. I think my email from 24 Aug [1] was the last?
>>
>> Cheers,
>>
>> Tim
>>
>> [1]
>> https://mail-archives.apache.org/mod_mbox/tika-dev/201508.mbox/%3cDM2PR09MB071305DFD203E21BFBE7A63AC7620@DM2PR09MB0713.namprd09.prod.outlook.com%3e
>>
>> -----Original Message-----
>> From: Madhav Sharan (JIRA) [mailto:jira@apache.org]
>> Sent: Wednesday, November 25, 2015 6:16 PM
>> To: dev@tika.apache.org
>> Subject: [jira] [Created] (TIKA-1803) Use lucene-geo-gazetteer REST API
>> in GeoTopicParser
>>
>> Madhav Sharan created TIKA-1803:
>> -----------------------------------
>>
>> Summary: Use lucene-geo-gazetteer REST API in GeoTopicParser
>> Key: TIKA-1803
>> URL: https://issues.apache.org/jira/browse/TIKA-1803
>> Project: Tika
>> Issue Type: Sub-task
>> Components: parser
>> Reporter: Madhav Sharan
>>
>>
>> As of now tika uses lucene-geo-gazetteer CLI to extract co-ordinates of a
>> location. CLI requires jvm and lucene to instantiate for every request.
>> With all new REST api it will be possible to gain improvement in this space.
>>
>> Idea is to create a client of lucene-geo-gazetteer in tika and use it in
>> GeoTopicParser
>>
>>
>>
>> --
>> This message was sent by Atlassian JIRA
>> (v6.3.4#6332)
>>
>
>
RE: more modular parser bundles
Posted by Ken Krugler <kk...@transpac.com>.
+1
Thanks to Bob for moving forward on this, it's definitely needed.
As one example, the growing list of dependencies is making it increasingly hard to build a reasonable size job jar for processing the sub-set of all docs we care about in a web crawl.
-- Ken
> From: Bob Paulin
> Sent: November 30, 2015 7:08:17am PST
> To: dev@tika.apache.org
> Subject: Re: more modular parser bundles
>
> Hi,
>
> I think Chris actually mentioned that this could be something targeted for
> a 2.0 release. The first step towards that would be to create the 2.0
> branch since I think this might be a big enough effort to not want to block
> the trunk ( or master if we move to git). Would the list agree that now
> would be a good time to branch?
>
> - Bob
>
> On Mon, Nov 30, 2015 at 6:24 AM, Allison, Timothy B. <ta...@mitre.org>
> wrote:
>
>> All,
>>
>> I'm extremely grateful for all of the new nlp +image processing parsers
>> that we're adding. Might it be time to start down the implementation path
>> to more modular parser bundles?
>>
>> Perhaps we could start with a tika-advanced-bundle to gather all of the
>> nlp/advanced parsers? Or would this have to wait for Tika 2.0?
>>
>> Bob got us off to a great start. There hasn't been much discussion
>> since August. I think my email from 24 Aug [1] was the last?
>>
>> Cheers,
>>
>> Tim
>>
>> [1]
>> https://mail-archives.apache.org/mod_mbox/tika-dev/201508.mbox/%3cDM2PR09MB071305DFD203E21BFBE7A63AC7620@DM2PR09MB0713.namprd09.prod.outlook.com%3e
>>
>> -----Original Message-----
>> From: Madhav Sharan (JIRA) [mailto:jira@apache.org]
>> Sent: Wednesday, November 25, 2015 6:16 PM
>> To: dev@tika.apache.org
>> Subject: [jira] [Created] (TIKA-1803) Use lucene-geo-gazetteer REST API in
>> GeoTopicParser
>>
>> Madhav Sharan created TIKA-1803:
>> -----------------------------------
>>
>> Summary: Use lucene-geo-gazetteer REST API in GeoTopicParser
>> Key: TIKA-1803
>> URL: https://issues.apache.org/jira/browse/TIKA-1803
>> Project: Tika
>> Issue Type: Sub-task
>> Components: parser
>> Reporter: Madhav Sharan
>>
>>
>> As of now tika uses lucene-geo-gazetteer CLI to extract co-ordinates of a
>> location. CLI requires jvm and lucene to instantiate for every request.
>> With all new REST api it will be possible to gain improvement in this space.
>>
>> Idea is to create a client of lucene-geo-gazetteer in tika and use it in
>> GeoTopicParser
>>
>>
>>
>> --
>> This message was sent by Atlassian JIRA
>> (v6.3.4#6332)
>>
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
Re: more modular parser bundles
Posted by Bob Paulin <bo...@bobpaulin.com>.
Hi,
I think Chris actually mentioned that this could be something targeted for
a 2.0 release. The first step towards that would be to create the 2.0
branch since I think this might be a big enough effort to not want to block
the trunk ( or master if we move to git). Would the list agree that now
would be a good time to branch?
- Bob
On Mon, Nov 30, 2015 at 6:24 AM, Allison, Timothy B. <ta...@mitre.org>
wrote:
> All,
>
> I'm extremely grateful for all of the new nlp +image processing parsers
> that we're adding. Might it be time to start down the implementation path
> to more modular parser bundles?
>
> Perhaps we could start with a tika-advanced-bundle to gather all of the
> nlp/advanced parsers? Or would this have to wait for Tika 2.0?
>
> Bob got us off to a great start. There hasn't been much discussion
> since August. I think my email from 24 Aug [1] was the last?
>
> Cheers,
>
> Tim
>
> [1]
> https://mail-archives.apache.org/mod_mbox/tika-dev/201508.mbox/%3cDM2PR09MB071305DFD203E21BFBE7A63AC7620@DM2PR09MB0713.namprd09.prod.outlook.com%3e
>
> -----Original Message-----
> From: Madhav Sharan (JIRA) [mailto:jira@apache.org]
> Sent: Wednesday, November 25, 2015 6:16 PM
> To: dev@tika.apache.org
> Subject: [jira] [Created] (TIKA-1803) Use lucene-geo-gazetteer REST API in
> GeoTopicParser
>
> Madhav Sharan created TIKA-1803:
> -----------------------------------
>
> Summary: Use lucene-geo-gazetteer REST API in GeoTopicParser
> Key: TIKA-1803
> URL: https://issues.apache.org/jira/browse/TIKA-1803
> Project: Tika
> Issue Type: Sub-task
> Components: parser
> Reporter: Madhav Sharan
>
>
> As of now tika uses lucene-geo-gazetteer CLI to extract co-ordinates of a
> location. CLI requires jvm and lucene to instantiate for every request.
> With all new REST api it will be possible to gain improvement in this space.
>
> Idea is to create a client of lucene-geo-gazetteer in tika and use it in
> GeoTopicParser
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>
RE: more modular parser bundles
Posted by "Allison, Timothy B." <ta...@mitre.org>.
Woohoo!!!
So much to do...
Thank you!
-----Original Message-----
From: Bob Paulin [mailto:bob@bobpaulin.com]
Sent: Monday, November 30, 2015 10:49 PM
To: dev@tika.apache.org
Subject: Re: more modular parser bundles
Created 2.x Branch.
https://svn.apache.org/repos/asf/tika/branches/2.x
On 11/30/2015 3:12 PM, Bob Paulin wrote:
> This makes sense. I think providing an "all" jar with all the parsers
> will be convenient for new developers. The modular parsers would give
> more developers a means to insulate themselves from changes and
> upgrades to other parsers. This is currently not available when all
> of the parsers are combine. So my expectation would be that the jar
> with all the parsers would be good for general applications or POC.
> While the modules would target production deployments where developers
> know what they want and would like to limit risk. Also agree that new
> documentation will be required!
>
> - Bob
>
> On Mon, Nov 30, 2015 at 2:50 PM, Nick Burch <apache@gagravarr.org
> <ma...@gagravarr.org>> wrote:
>
> On Mon, 30 Nov 2015, Allison, Timothy B. wrote:
>
> Perhaps we could start with a tika-advanced-bundle to gather
> all of the nlp/advanced parsers? Or would this have to wait
> for Tika 2.0?
>
>
> I've noticed that there have been a lot fewer queries (on our
> list, on stackoverflow, at events etc) caused by people missing
> jars of late. Not sure of the message has got out there better,
> the right posts are getting to the top of google, the
> troubleshooting page has done its magic, or something else
> entirely! But I'm now less worried about the impact of modular
> parsers on newbies that I have been before
>
> To try to avoid all the existing guidance (most of it external)
> from going stale, I'd lean towards either keeping "tika-parsers"
> as the full version, or make "tika-parsers" be an alias to
> "tika-parsers-all", so that current behaviour remains
>
> I'd also probably suggest we change the default load error handler
> to warn/log, so that people by default will find out more quickly
> that they've missed jars, and probably also have an extra load
> error log/check which triggers in the event of 0 parser
> definitions being found. People can turn that off if they want, as
> now, but maybe the new default should be so that newbies tend to
> get told quickly what they've done wrong!
>
> Oh, and we'll need to update the troubleshooting page too for the
> new bundles world :)
>
> Nick
>
>
Re: more modular parser bundles
Posted by Bob Paulin <bo...@bobpaulin.com>.
Created 2.x Branch.
https://svn.apache.org/repos/asf/tika/branches/2.x
On 11/30/2015 3:12 PM, Bob Paulin wrote:
> This makes sense. I think providing an "all" jar with all the parsers
> will be convenient for new developers. The modular parsers would give
> more developers a means to insulate themselves from changes and
> upgrades to other parsers. This is currently not available when all
> of the parsers are combine. So my expectation would be that the jar
> with all the parsers would be good for general applications or POC.
> While the modules would target production deployments where developers
> know what they want and would like to limit risk. Also agree that new
> documentation will be required!
>
> - Bob
>
> On Mon, Nov 30, 2015 at 2:50 PM, Nick Burch <apache@gagravarr.org
> <ma...@gagravarr.org>> wrote:
>
> On Mon, 30 Nov 2015, Allison, Timothy B. wrote:
>
> Perhaps we could start with a tika-advanced-bundle to gather
> all of the nlp/advanced parsers? Or would this have to wait
> for Tika 2.0?
>
>
> I've noticed that there have been a lot fewer queries (on our
> list, on stackoverflow, at events etc) caused by people missing
> jars of late. Not sure of the message has got out there better,
> the right posts are getting to the top of google, the
> troubleshooting page has done its magic, or something else
> entirely! But I'm now less worried about the impact of modular
> parsers on newbies that I have been before
>
> To try to avoid all the existing guidance (most of it external)
> from going stale, I'd lean towards either keeping "tika-parsers"
> as the full version, or make "tika-parsers" be an alias to
> "tika-parsers-all", so that current behaviour remains
>
> I'd also probably suggest we change the default load error handler
> to warn/log, so that people by default will find out more quickly
> that they've missed jars, and probably also have an extra load
> error log/check which triggers in the event of 0 parser
> definitions being found. People can turn that off if they want, as
> now, but maybe the new default should be so that newbies tend to
> get told quickly what they've done wrong!
>
> Oh, and we'll need to update the troubleshooting page too for the
> new bundles world :)
>
> Nick
>
>
Re: more modular parser bundles
Posted by Bob Paulin <bo...@bobpaulin.com>.
This makes sense. I think providing an "all" jar with all the parsers will
be convenient for new developers. The modular parsers would give more
developers a means to insulate themselves from changes and upgrades to
other parsers. This is currently not available when all of the parsers are
combine. So my expectation would be that the jar with all the parsers
would be good for general applications or POC. While the modules would
target production deployments where developers know what they want and
would like to limit risk. Also agree that new documentation will be
required!
- Bob
On Mon, Nov 30, 2015 at 2:50 PM, Nick Burch <ap...@gagravarr.org> wrote:
> On Mon, 30 Nov 2015, Allison, Timothy B. wrote:
>
>> Perhaps we could start with a tika-advanced-bundle to gather all of the
>> nlp/advanced parsers? Or would this have to wait for Tika 2.0?
>>
>
> I've noticed that there have been a lot fewer queries (on our list, on
> stackoverflow, at events etc) caused by people missing jars of late. Not
> sure of the message has got out there better, the right posts are getting
> to the top of google, the troubleshooting page has done its magic, or
> something else entirely! But I'm now less worried about the impact of
> modular parsers on newbies that I have been before
>
> To try to avoid all the existing guidance (most of it external) from going
> stale, I'd lean towards either keeping "tika-parsers" as the full version,
> or make "tika-parsers" be an alias to "tika-parsers-all", so that current
> behaviour remains
>
> I'd also probably suggest we change the default load error handler to
> warn/log, so that people by default will find out more quickly that they've
> missed jars, and probably also have an extra load error log/check which
> triggers in the event of 0 parser definitions being found. People can turn
> that off if they want, as now, but maybe the new default should be so that
> newbies tend to get told quickly what they've done wrong!
>
> Oh, and we'll need to update the troubleshooting page too for the new
> bundles world :)
>
> Nick
>
Re: more modular parser bundles
Posted by Nick Burch <ap...@gagravarr.org>.
On Mon, 30 Nov 2015, Allison, Timothy B. wrote:
>Perhaps we could start with a tika-advanced-bundle to gather all of the
>nlp/advanced parsers? Or would this have to wait for Tika 2.0?
I've noticed that there have been a lot fewer queries (on our list, on
stackoverflow, at events etc) caused by people missing jars of late. Not
sure of the message has got out there better, the right posts are getting
to the top of google, the troubleshooting page has done its magic, or
something else entirely! But I'm now less worried about the impact of
modular parsers on newbies that I have been before
To try to avoid all the existing guidance (most of it external) from going
stale, I'd lean towards either keeping "tika-parsers" as the full version,
or make "tika-parsers" be an alias to "tika-parsers-all", so that current
behaviour remains
I'd also probably suggest we change the default load error handler to
warn/log, so that people by default will find out more quickly that
they've missed jars, and probably also have an extra load error log/check
which triggers in the event of 0 parser definitions being found. People
can turn that off if they want, as now, but maybe the new default should
be so that newbies tend to get told quickly what they've done wrong!
Oh, and we'll need to update the troubleshooting page too for the new
bundles world :)
Nick