You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Allison, Timothy B." <ta...@mitre.org> on 2015/08/03 20:56:26 UTC
RE: [DISCUSS] A more modular parser project
Bob,
Thank you for leading this effort. Please continue to forgive my lack of OSGi knowledge in the following :).
I just built your strawman, and it looks great...I think(?)
To confirm that I understand correctly...
For non-OSGi folks, let's say I know that I'm only going to be parsing images, to replicate the features of tika-app but with only the image parsers, would I put these three jars in a bin directory (or add them via maven to my project):
original-tika-app-1.10-SNAPSHOT.jar
tika-image-parser-bundle-1.10-SNAPSHOT.jar
tika-core-1.10-SNAPSHOT.jar
and then type:
java -cp "bin/*" org.apache.tika.cli.TikaCLI -then -the -commandline -options
If this is the goal, why is tika-parsers "provided" in tika-image-parser-bundle's pom? Aren't all dependencies of the image parsers packaged within that bundle?
When I actually tried this, I got a noclassdeferror for Apache logging. Should we add that within the image-parser-bundle? Or should I be doing something else (answer= very likely :) ).
Thank you, again.
Best,
Tim
-----Original Message-----
From: Bob Paulin [mailto:bob@apache.org]
Sent: Saturday, July 25, 2015 9:07 PM
To: dev@tika.apache.org
Subject: [DISCUSS] A more modular parser project
Hi,
At Apache Con this year there was some discussion of some woes with OSGi
and Tika.
The TL;DR version of this email is I think the pain with tika-bundle is
due to it's attempt to encompass all the parsers in one Uber jar. I
think breaking down the tika-bundle into smaller jars (like Apache
Camel) might provide a more maintainable approach going forward for OSGi
and non-OSGi developers alike.
I've put up a straw man implementation of smaller bundles here:
https://github.com/bobpaulin/tika
(specifically in
https://github.com/bobpaulin/tika/tree/trunk/tika-parser-bundles)
For the impatient please check out the code above. The approach has
zero impact on any of the current tika jar files so nothing would have
to change for current users but they could chose to move to the new
approach if it better fit their needs. It's a straw man so feel free to
beat it up :).
For the rest that have time for an explanation....
The straw man implementation does the following:
1) Copies and inlines the tika-core.jar in the tika-osgi-bundle. (see
pom.xml)
2) The tika-osgi-bundle registers a Tika Service that contains a Default
Parser that is composed of all Tika Parsers registered as OSGi services
as well as available Detectors. This extends the functionality in the
tika-core TikaActivator class.
3) Copies and inlines classes from the tika-parser project into specific
bundles (see individual pom.xml specifically in the maven-bundle-plugin).
4) Each bundle registers services for each parser and provides
configuration for ranking (see individual Activator.java). This could
expand considerably if we wanted to provide additional properties to
toggle features on and off.
This provides the following advantages to users and developers in both
an OSGi and standard deployment:
1) These bundles could be evolved separately. The community could drive
how finely grained the bundles were. For example I bundled the image
and jpeg packages together but put tesseract ocr separate.
2) Users can just select the parsers they want to include in there
projects.
3) Each parser project could maintain it's own optional and embedded
bundle dependencies. Currently the tika-bundle project has a marathon
of these entries with optional and embedded dependencies clocking in at
~200 line and ~20 lines respectively.
If you made it this far thanks for reading :). As an OSGi developer I
think the implementation provides a better experience but could also
make things better for non-OSGi developers as well. And best of all it
works without requiring changes to any of the existing released jars.
Let me know what you think.
- Bob Paulin
Re: [DISCUSS] A more modular parser project
Posted by Bob Paulin <bo...@bobpaulin.com>.
FYI my classpath looks like this.
08/03/2015 08:09 PM 185,140 commons-io-2.4.jar
08/03/2015 08:15 PM 16,618 jcl-over-slf4j-1.7.12.jar
08/03/2015 08:10 PM 489,884 log4j-1.2.17.jar
08/03/2015 05:35 PM 115,613 original-tika-app-1.10-SNAPSHOT.jar
08/03/2015 08:16 PM 32,127 slf4j-api-1.7.12.jar
07/30/2015 08:30 AM 562,352 tika-core-1.10-SNAPSHOT.jar
07/25/2015 08:11 AM 2,717,249
tika-image-parser-bundle-1.10-SNAPSHOT.jar
This got the CLI to run and detect but only the empty parser came up.
Dynamic classloading works fine in OSGi. Any thoughts on either doing
bundle specific static loading or perhaps trying to do a non-OSGi
dynamic loader?
- Bob
On 8/3/2015 6:58 PM, Allison, Timothy B. wrote:
>>> If we actually moved the source files out of
> tika-parsers and into the modules this would not be required.
>
> Of course, got it...now. Thank you. In Tika 2.0, would we want to actually move the source code for the bundles into the bundles, I wonder. Or, would we still want to have one massive tika-parsers module as we do now?
>
>>> Can you provide the source for your test?
> Source?! Ha. No, I literally/manually copied the jars into bin... the horror! I'm sure maven would have added the transitive dependencies. I'll give that a try. Thank you, again.
>
> Best,
>
> Tim
>
> On Mon, Aug 3, 2015 at 1:56 PM, Allison, Timothy B. <ta...@mitre.org>
> wrote:
>
>> Bob,
>> Thank you for leading this effort. Please continue to forgive my lack
>> of OSGi knowledge in the following :).
>> I just built your strawman, and it looks great...I think(?)
>>
>> To confirm that I understand correctly...
>>
>> For non-OSGi folks, let's say I know that I'm only going to be parsing
>> images, to replicate the features of tika-app but with only the image
>> parsers, would I put these three jars in a bin directory (or add them via
>> maven to my project):
>>
>> original-tika-app-1.10-SNAPSHOT.jar
>> tika-image-parser-bundle-1.10-SNAPSHOT.jar
>> tika-core-1.10-SNAPSHOT.jar
>>
>> and then type:
>> java -cp "bin/*" org.apache.tika.cli.TikaCLI -then -the -commandline
>> -options
>>
>>
>> If this is the goal, why is tika-parsers "provided" in
>> tika-image-parser-bundle's pom? Aren't all dependencies of the image
>> parsers packaged within that bundle?
>>
>> When I actually tried this, I got a noclassdeferror for Apache logging.
>> Should we add that within the image-parser-bundle? Or should I be doing
>> something else (answer= very likely :) ).
>>
>> Thank you, again.
>>
>> Best,
>>
>> Tim
>>
>>
>> -----Original Message-----
>> From: Bob Paulin [mailto:bob@apache.org]
>> Sent: Saturday, July 25, 2015 9:07 PM
>> To: dev@tika.apache.org
>> Subject: [DISCUSS] A more modular parser project
>>
>> Hi,
>>
>> At Apache Con this year there was some discussion of some woes with OSGi
>> and Tika.
>>
>> The TL;DR version of this email is I think the pain with tika-bundle is
>> due to it's attempt to encompass all the parsers in one Uber jar. I
>> think breaking down the tika-bundle into smaller jars (like Apache
>> Camel) might provide a more maintainable approach going forward for OSGi
>> and non-OSGi developers alike.
>>
>> I've put up a straw man implementation of smaller bundles here:
>>
>> https://github.com/bobpaulin/tika
>> (specifically in
>> https://github.com/bobpaulin/tika/tree/trunk/tika-parser-bundles)
>>
>> For the impatient please check out the code above. The approach has
>> zero impact on any of the current tika jar files so nothing would have
>> to change for current users but they could chose to move to the new
>> approach if it better fit their needs. It's a straw man so feel free to
>> beat it up :).
>>
>> For the rest that have time for an explanation....
>> The straw man implementation does the following:
>>
>> 1) Copies and inlines the tika-core.jar in the tika-osgi-bundle. (see
>> pom.xml)
>>
>> 2) The tika-osgi-bundle registers a Tika Service that contains a Default
>> Parser that is composed of all Tika Parsers registered as OSGi services
>> as well as available Detectors. This extends the functionality in the
>> tika-core TikaActivator class.
>>
>> 3) Copies and inlines classes from the tika-parser project into specific
>> bundles (see individual pom.xml specifically in the maven-bundle-plugin).
>>
>> 4) Each bundle registers services for each parser and provides
>> configuration for ranking (see individual Activator.java). This could
>> expand considerably if we wanted to provide additional properties to
>> toggle features on and off.
>>
>> This provides the following advantages to users and developers in both
>> an OSGi and standard deployment:
>>
>> 1) These bundles could be evolved separately. The community could drive
>> how finely grained the bundles were. For example I bundled the image
>> and jpeg packages together but put tesseract ocr separate.
>>
>> 2) Users can just select the parsers they want to include in there
>> projects.
>>
>> 3) Each parser project could maintain it's own optional and embedded
>> bundle dependencies. Currently the tika-bundle project has a marathon
>> of these entries with optional and embedded dependencies clocking in at
>> ~200 line and ~20 lines respectively.
>>
>> If you made it this far thanks for reading :). As an OSGi developer I
>> think the implementation provides a better experience but could also
>> make things better for non-OSGi developers as well. And best of all it
>> works without requiring changes to any of the existing released jars.
>> Let me know what you think.
>>
>>
>> - Bob Paulin
>>
RE: [DISCUSS] A more modular parser project
Posted by "Allison, Timothy B." <ta...@mitre.org>.
Apologies for my delay. If anyone else wants to chime in on potential subgroupings of parsers, that'd be great!
>>So just to understand the break downs. When you say:
tika-classic-parser-bundle/
tika-office-parser-bundle/ (including microsoft, opendocument, pst, rtf, iwork? Has dependency on html/text)
tika-pdf-parser-bundle/
tika-text-parser-bundle (including txt,chm, rfc822, html, xml, kml, feed, iptc, crypto, etc?)/
tika-sourcecode-parser-bundle (parsers that handle source code)
tika-package-parser-bundle (all zip/tar/etc)
>>Does that indicate 6 bundles? 5 individuals that could wrap into 1 uber jar?
Y, this is pure strawman, without knowledge of OSGi conventions/best practices. So, guidance on best practices is very much needed! My thought, which may be crazy in an OSGi environment, was to have 6 bundles that consumers could use: the overall tika-classic or any combination of the 5 child bundles. For the most basic use case, y, there'd be one uber-jar for tika-classic.
>>Breaking things down at different levels will add to maintenance effort so it may be better to start with the broad strokes like tika-classic-parser-bundle.
Yes, makes sense. I broke out the pdf parser, for example, only because I heard that there are some users who really only want that. Any division/subdivision of this sort will be based on subjective criteria, I think. More feedback from the community will be important on these decisions.
>>I think this approach is fine but it does mean we're taking an opinion on what most of Tika's basic users want for simple usecases.
Y. I agree. My simple use case will be different from someone else's.
>>Another approach could be grouping the parsers by similar dependencies which I think the tika-multimedia-parser-bundle does fairly well. From a dependence management perspective this is desirable.
I really like this because it is defensible and predictable, and y, from a dependency management perspective, this is great. However, from a use case perspective, I think users will care much more about file types handled than jars required. I could very well be wrong though. I have to look more carefully at the dependency report link that you sent...I suspect that there may be some clear overlap between file formats and dependencies that may be enough to drive this type of configuration (logging and commons-x aside :) ).
>>With respect to bundles that don't fit perhaps those live on their own until an obvious emerges. It's much harder to remove something from a bundle than to add it later. I think this may apply to native bundles too.
Oh, very good to know. Thank you.
Cheers,
Tim
On 8/4/2015 8:32 AM, Allison, Timothy B. wrote:
> Bob,
> Thank you, again. This looks promising at first glance!
>
> To continue down the strawman path and to start discussion on the elephant in the room...
>
> We'd want bundles that allow enough control for users but aren't too much of a hassle to configure. There will be trade-offs.
>
> So, what do we think of this strawman for proposed bundles:
>
> tika-classic-parser-bundle/
> Tika-office-parser-bundle/ (including microsoft, opendocument, pst, rtf, iwork? Has dependency on html/text)
> Tika-pdf-parser-bundle/
> Tika-text-parser-bundle (including txt,chm, rfc822, html, xml, kml, feed, iptc, crypto, etc?)/
> Tika-sourcecode-parser-bundle (parsers that handle source code)
> Tika-package-parser-bundle (all zip/tar/etc)
>
> tika-multimedia-parser-bundle/ (parsers that pull metadata out of image, audio, audio+video files)
> Tika-image-parser-bundle
> Tika-image-ocr-parser-bundle
> Tika-audio-parser-bundle
> Tika-video-parser-bundle
>
> tika-scientific-parser-bundle/ (all parsers that handle scientific
> data sets (grib, isatab,gdal,hdf,netcdf,geoinfo,dif...much
> hand-waving...input, Chris?)
>
> tika-nativelib-parser-bundle/ (sqlite...any others at the moment? all
> parsers that rely on native libs...unfortunately, this doesn't fit
> well thematically...)
>
> tika-advanced-bundle/ (all parsers that rely on nlp or other advanced techniques for extraction of information...
> these aren't really just pulling text and metadata out, but are operating on the text/metadata
> once it has been pulled out. We may need separate bundles for each?)
> Tika-nlp-parser-bundle/ (ctakes, phone number, geo.topic, grobid(?) etc.
> ...or maybe we want separate bundles for each?)
> Tika-sentiment-parser-bundle (imaginary...?)
> Tika-object-parser-bundle
>
> Where to put?
> font parser
> executable
> mat
> prt
> strings
>
>
> Cheers,
>
> Tim
>
>
>
> -----Original Message-----
> From: Bob Paulin [mailto:bob@bobpaulin.com]
> Sent: Tuesday, August 04, 2015 8:56 AM
> To: dev@tika.apache.org
> Subject: Re: [DISCUSS] A more modular parser project
>
> So I just tried adding a
> META-INF/services/org.apache.tika.parser.Parser
> file to each bundle in the straw man implementation and it seemed to
> do the trick. Looks like the ServiceLoader code searches the
> classloader for all of these files and iterates through them to pick
> up each jar's META-INF/services/org.apache.tika.parser.Parser entries
> and adds them to the list. I've updated the code on github to include one per bundle.
> This might be the way to go.
>
> ex.
> https://github.com/bobpaulin/tika/tree/trunk/tika-parser-bundles/tika-
> image-parser-bundle/src/main/resources/META-INF/services
>
>
> - Bob
>
> On 8/3/2015 9:21 PM, Allison, Timothy B. wrote:
>>>> +1 to moving the source to bundles. I think for a 2.0 would be
>>>> +easier
>> to consolidate into a parser uber jar than trying to tease things out
>> like I did in the straw man impl. However deciding how to break
>> things up might take some experimentation.
>>
>> Y, and the strawman is a great easy entry down this path towards 2.0. I think the main hangup will be coming to consensus about granularity and nature of the packages, but we can burn that bridge when we get to it. There are some dependencies between parsers, but we can work through that.
>>
>>>> 1) To spin up the GUI you need org.apache.tika.parser.util (perhaps
>> consider moving this up to core).
>> Y, I put that in tika-parsers because it relies on commons codec, and I wanted to keep that dependency out of tika-core. But, I'm willing to add it to tika-core if there aren't objections.
>>
>>
>>>> 2) Since the META-INF/services/org.apache.tika.parser.Parser is in
>> tika-parser we'd need to rethink the static ServiceLoader strategy to
>> either always be dynamic or figure out a way to have each jar bring
>> there own static loader.
>>
>> Hmmm...is there a way to specify this in one overall tika-config file or in separate configs in each bundle (yuck)...
>>
Re: [DISCUSS] A more modular parser project
Posted by Bob Paulin <bo...@bobpaulin.com>.
Hi,
So just to understand the break downs. When you say:
tika-classic-parser-bundle/
Tika-office-parser-bundle/ (including microsoft, opendocument, pst, rtf, iwork? Has dependency on html/text)
Tika-pdf-parser-bundle/
Tika-text-parser-bundle (including txt,chm, rfc822, html, xml, kml, feed, iptc, crypto, etc?)/
Tika-sourcecode-parser-bundle (parsers that handle source code)
Tika-package-parser-bundle (all zip/tar/etc)
Does that indicate 6 bundles? 5 individuals that could wrap into 1 uber
jar? Breaking things down at different levels will add to maintenance
effort so it may be better to start with the broad strokes like
tika-classic-parser-bundle. But if we just created a
tika-classic-parser-bundle are we attempting to group the bundles by a
type of usecase? I think this approach is fine but it does mean we're
taking an opinion on what most of Tika's basic users want for simple
usecases.
Another approach could be grouping the parsers by similar dependencies
which I think the tika-multimedia-parser-bundle does fairly well. From
a dependence management perspective this is desirable. I've used tools
like JDepend to break down which packages use which dependencies. Also
determining package based dependencies within tika-parsers can be seen
here in sonar:
http://nemo.sonarqube.org/design/index/253571
With respect to bundles that don't fit perhaps those live on their own
until an obvious emerges. It's much harder to remove something from a
bundle than to add it later. I think this may apply to native bundles too.
- Bob
On 8/4/2015 8:32 AM, Allison, Timothy B. wrote:
> Bob,
> Thank you, again. This looks promising at first glance!
>
> To continue down the strawman path and to start discussion on the elephant in the room...
>
> We'd want bundles that allow enough control for users but aren't too much of a hassle to configure. There will be trade-offs.
>
> So, what do we think of this strawman for proposed bundles:
>
> tika-classic-parser-bundle/
> Tika-office-parser-bundle/ (including microsoft, opendocument, pst, rtf, iwork? Has dependency on html/text)
> Tika-pdf-parser-bundle/
> Tika-text-parser-bundle (including txt,chm, rfc822, html, xml, kml, feed, iptc, crypto, etc?)/
> Tika-sourcecode-parser-bundle (parsers that handle source code)
> Tika-package-parser-bundle (all zip/tar/etc)
>
> tika-multimedia-parser-bundle/ (parsers that pull metadata out of image, audio, audio+video files)
> Tika-image-parser-bundle
> Tika-image-ocr-parser-bundle
> Tika-audio-parser-bundle
> Tika-video-parser-bundle
>
> tika-scientific-parser-bundle/ (all parsers that handle scientific data sets (grib, isatab,gdal,hdf,netcdf,geoinfo,dif...much hand-waving...input, Chris?)
>
> tika-nativelib-parser-bundle/ (sqlite...any others at the moment? all parsers that rely on native libs...unfortunately, this doesn't fit well thematically...)
>
> tika-advanced-bundle/ (all parsers that rely on nlp or other advanced techniques for extraction of information...
> these aren't really just pulling text and metadata out, but are operating on the text/metadata
> once it has been pulled out. We may need separate bundles for each?)
> Tika-nlp-parser-bundle/ (ctakes, phone number, geo.topic, grobid(?) etc.
> ...or maybe we want separate bundles for each?)
> Tika-sentiment-parser-bundle (imaginary...?)
> Tika-object-parser-bundle
>
> Where to put?
> font parser
> executable
> mat
> prt
> strings
>
>
> Cheers,
>
> Tim
>
>
>
> -----Original Message-----
> From: Bob Paulin [mailto:bob@bobpaulin.com]
> Sent: Tuesday, August 04, 2015 8:56 AM
> To: dev@tika.apache.org
> Subject: Re: [DISCUSS] A more modular parser project
>
> So I just tried adding a META-INF/services/org.apache.tika.parser.Parser
> file to each bundle in the straw man implementation and it seemed to do
> the trick. Looks like the ServiceLoader code searches the classloader
> for all of these files and iterates through them to pick up each jar's
> META-INF/services/org.apache.tika.parser.Parser entries and adds them to
> the list. I've updated the code on github to include one per bundle.
> This might be the way to go.
>
> ex.
> https://github.com/bobpaulin/tika/tree/trunk/tika-parser-bundles/tika-image-parser-bundle/src/main/resources/META-INF/services
>
>
> - Bob
>
> On 8/3/2015 9:21 PM, Allison, Timothy B. wrote:
>>>> +1 to moving the source to bundles. I think for a 2.0 would be easier
>> to consolidate into a parser uber jar than trying to tease things out
>> like I did in the straw man impl. However deciding how to break things
>> up might take some experimentation.
>>
>> Y, and the strawman is a great easy entry down this path towards 2.0. I think the main hangup will be coming to consensus about granularity and nature of the packages, but we can burn that bridge when we get to it. There are some dependencies between parsers, but we can work through that.
>>
>>>> 1) To spin up the GUI you need org.apache.tika.parser.util (perhaps
>> consider moving this up to core).
>> Y, I put that in tika-parsers because it relies on commons codec, and I wanted to keep that dependency out of tika-core. But, I'm willing to add it to tika-core if there aren't objections.
>>
>>
>>>> 2) Since the META-INF/services/org.apache.tika.parser.Parser is in
>> tika-parser we'd need to rethink the static ServiceLoader strategy to
>> either always be dynamic or figure out a way to have each jar bring
>> there own static loader.
>>
>> Hmmm...is there a way to specify this in one overall tika-config file or in separate configs in each bundle (yuck)...
>>
RE: [DISCUSS] A more modular parser project
Posted by "Allison, Timothy B." <ta...@mitre.org>.
Bob,
Thank you, again. This looks promising at first glance!
To continue down the strawman path and to start discussion on the elephant in the room...
We'd want bundles that allow enough control for users but aren't too much of a hassle to configure. There will be trade-offs.
So, what do we think of this strawman for proposed bundles:
tika-classic-parser-bundle/
Tika-office-parser-bundle/ (including microsoft, opendocument, pst, rtf, iwork? Has dependency on html/text)
Tika-pdf-parser-bundle/
Tika-text-parser-bundle (including txt,chm, rfc822, html, xml, kml, feed, iptc, crypto, etc?)/
Tika-sourcecode-parser-bundle (parsers that handle source code)
Tika-package-parser-bundle (all zip/tar/etc)
tika-multimedia-parser-bundle/ (parsers that pull metadata out of image, audio, audio+video files)
Tika-image-parser-bundle
Tika-image-ocr-parser-bundle
Tika-audio-parser-bundle
Tika-video-parser-bundle
tika-scientific-parser-bundle/ (all parsers that handle scientific data sets (grib, isatab,gdal,hdf,netcdf,geoinfo,dif...much hand-waving...input, Chris?)
tika-nativelib-parser-bundle/ (sqlite...any others at the moment? all parsers that rely on native libs...unfortunately, this doesn't fit well thematically...)
tika-advanced-bundle/ (all parsers that rely on nlp or other advanced techniques for extraction of information...
these aren't really just pulling text and metadata out, but are operating on the text/metadata
once it has been pulled out. We may need separate bundles for each?)
Tika-nlp-parser-bundle/ (ctakes, phone number, geo.topic, grobid(?) etc.
...or maybe we want separate bundles for each?)
Tika-sentiment-parser-bundle (imaginary...?)
Tika-object-parser-bundle
Where to put?
font parser
executable
mat
prt
strings
Cheers,
Tim
-----Original Message-----
From: Bob Paulin [mailto:bob@bobpaulin.com]
Sent: Tuesday, August 04, 2015 8:56 AM
To: dev@tika.apache.org
Subject: Re: [DISCUSS] A more modular parser project
So I just tried adding a META-INF/services/org.apache.tika.parser.Parser
file to each bundle in the straw man implementation and it seemed to do
the trick. Looks like the ServiceLoader code searches the classloader
for all of these files and iterates through them to pick up each jar's
META-INF/services/org.apache.tika.parser.Parser entries and adds them to
the list. I've updated the code on github to include one per bundle.
This might be the way to go.
ex.
https://github.com/bobpaulin/tika/tree/trunk/tika-parser-bundles/tika-image-parser-bundle/src/main/resources/META-INF/services
- Bob
On 8/3/2015 9:21 PM, Allison, Timothy B. wrote:
>>> +1 to moving the source to bundles. I think for a 2.0 would be easier
> to consolidate into a parser uber jar than trying to tease things out
> like I did in the straw man impl. However deciding how to break things
> up might take some experimentation.
>
> Y, and the strawman is a great easy entry down this path towards 2.0. I think the main hangup will be coming to consensus about granularity and nature of the packages, but we can burn that bridge when we get to it. There are some dependencies between parsers, but we can work through that.
>
>>> 1) To spin up the GUI you need org.apache.tika.parser.util (perhaps
> consider moving this up to core).
> Y, I put that in tika-parsers because it relies on commons codec, and I wanted to keep that dependency out of tika-core. But, I'm willing to add it to tika-core if there aren't objections.
>
>
>>> 2) Since the META-INF/services/org.apache.tika.parser.Parser is in
> tika-parser we'd need to rethink the static ServiceLoader strategy to
> either always be dynamic or figure out a way to have each jar bring
> there own static loader.
>
> Hmmm...is there a way to specify this in one overall tika-config file or in separate configs in each bundle (yuck)...
>
FW: [DISCUSS] A more modular parser project
Posted by "Allison, Timothy B." <ta...@mitre.org>.
Bob,
Thank you, again. This looks promising!
To continue down the strawman path and to start discussion on the elephant in the room...
We'd want bundles that allow enough control for users but aren't too much of a hassle to configure. There will be trade-offs.
So, what do we think of this strawman for proposed bundles:
tika-classic-parser-bundle/
Tika-office-parser-bundle/ (including microsoft, opendocument, pst, rtf, iwork? Has dependency on html/text)
Tika-pdf-parser-bundle/
Tika-text-parser-bundle (including txt,chm, rfc822, html, xml, kml, feed, iptc, crypto, etc?)/
Tika-sourcecode-parser-bundle (parsers that handle source code)
Tika-package-parser-bundle (all zip/tar/etc)
tika-multimedia-parser-bundle/ (parsers that pull metadata out of image, audio, audio+video files)
Tika-image-parser-bundle
Tika-image-ocr-parser-bundle
Tika-audio-parser-bundle
Tika-video-parser-bundle
tika-scientific-parser-bundle/ (all parsers that handle scientific data sets
(grib, isatab,gdal,hdf,netcdf,geoinfo,dif...much hand-waving...input, Chris?)
tika-nativelib-parser-bundle/ (sqlite...any others at the moment? all parsers that rely on native libs...unfortunately, this doesn't fit well thematically...)
tika-advanced-bundle/ (all parsers that rely on nlp or other advanced techniques for extraction of information...
these aren't really just pulling text and metadata out, but are operating on the text/metadata
once it has been pulled out. We may need separate bundles for each?)
Tika-nlp-parser-bundle/ (ctakes, phone number, geo.topic, grobid(?) etc.
...or maybe we want separate bundles for each?)
Tika-sentiment-parser-bundle (imaginary...?)
Tika-object-parser-bundle
Where to put these?
font parser
executable
mat
prt
strings
Cheers,
Tim
-----Original Message-----
From: Bob Paulin [mailto:bob@bobpaulin.com]
Sent: Tuesday, August 04, 2015 8:56 AM
To: dev@tika.apache.org
Subject: Re: [DISCUSS] A more modular parser project
So I just tried adding a META-INF/services/org.apache.tika.parser.Parser
file to each bundle in the straw man implementation and it seemed to do
the trick. Looks like the ServiceLoader code searches the classloader
for all of these files and iterates through them to pick up each jar's
META-INF/services/org.apache.tika.parser.Parser entries and adds them to
the list. I've updated the code on github to include one per bundle.
This might be the way to go.
ex.
https://github.com/bobpaulin/tika/tree/trunk/tika-parser-bundles/tika-image-parser-bundle/src/main/resources/META-INF/services
- Bob
On 8/3/2015 9:21 PM, Allison, Timothy B. wrote:
>>> +1 to moving the source to bundles. I think for a 2.0 would be easier
> to consolidate into a parser uber jar than trying to tease things out
> like I did in the straw man impl. However deciding how to break things
> up might take some experimentation.
>
> Y, and the strawman is a great easy entry down this path towards 2.0. I think the main hangup will be coming to consensus about granularity and nature of the packages, but we can burn that bridge when we get to it. There are some dependencies between parsers, but we can work through that.
>
>>> 1) To spin up the GUI you need org.apache.tika.parser.util (perhaps
> consider moving this up to core).
> Y, I put that in tika-parsers because it relies on commons codec, and I wanted to keep that dependency out of tika-core. But, I'm willing to add it to tika-core if there aren't objections.
>
>
>>> 2) Since the META-INF/services/org.apache.tika.parser.Parser is in
> tika-parser we'd need to rethink the static ServiceLoader strategy to
> either always be dynamic or figure out a way to have each jar bring
> there own static loader.
>
> Hmmm...is there a way to specify this in one overall tika-config file or in separate configs in each bundle (yuck)...
>
Re: [DISCUSS] A more modular parser project
Posted by Bob Paulin <bo...@bobpaulin.com>.
So I just tried adding a META-INF/services/org.apache.tika.parser.Parser
file to each bundle in the straw man implementation and it seemed to do
the trick. Looks like the ServiceLoader code searches the classloader
for all of these files and iterates through them to pick up each jar's
META-INF/services/org.apache.tika.parser.Parser entries and adds them to
the list. I've updated the code on github to include one per bundle.
This might be the way to go.
ex.
https://github.com/bobpaulin/tika/tree/trunk/tika-parser-bundles/tika-image-parser-bundle/src/main/resources/META-INF/services
- Bob
On 8/3/2015 9:21 PM, Allison, Timothy B. wrote:
>>> +1 to moving the source to bundles. I think for a 2.0 would be easier
> to consolidate into a parser uber jar than trying to tease things out
> like I did in the straw man impl. However deciding how to break things
> up might take some experimentation.
>
> Y, and the strawman is a great easy entry down this path towards 2.0. I think the main hangup will be coming to consensus about granularity and nature of the packages, but we can burn that bridge when we get to it. There are some dependencies between parsers, but we can work through that.
>
>>> 1) To spin up the GUI you need org.apache.tika.parser.util (perhaps
> consider moving this up to core).
> Y, I put that in tika-parsers because it relies on commons codec, and I wanted to keep that dependency out of tika-core. But, I'm willing to add it to tika-core if there aren't objections.
>
>
>>> 2) Since the META-INF/services/org.apache.tika.parser.Parser is in
> tika-parser we'd need to rethink the static ServiceLoader strategy to
> either always be dynamic or figure out a way to have each jar bring
> there own static loader.
>
> Hmmm...is there a way to specify this in one overall tika-config file or in separate configs in each bundle (yuck)...
>
RE: [DISCUSS] A more modular parser project
Posted by "Allison, Timothy B." <ta...@mitre.org>.
>>+1 to moving the source to bundles. I think for a 2.0 would be easier
to consolidate into a parser uber jar than trying to tease things out
like I did in the straw man impl. However deciding how to break things
up might take some experimentation.
Y, and the strawman is a great easy entry down this path towards 2.0. I think the main hangup will be coming to consensus about granularity and nature of the packages, but we can burn that bridge when we get to it. There are some dependencies between parsers, but we can work through that.
>>1) To spin up the GUI you need org.apache.tika.parser.util (perhaps
consider moving this up to core).
Y, I put that in tika-parsers because it relies on commons codec, and I wanted to keep that dependency out of tika-core. But, I'm willing to add it to tika-core if there aren't objections.
>>2) Since the META-INF/services/org.apache.tika.parser.Parser is in
tika-parser we'd need to rethink the static ServiceLoader strategy to
either always be dynamic or figure out a way to have each jar bring
there own static loader.
Hmmm...is there a way to specify this in one overall tika-config file or in separate configs in each bundle (yuck)...
Re: [DISCUSS] A more modular parser project
Posted by Bob Paulin <bo...@bobpaulin.com>.
Responses inline.
- Bob
On 8/3/2015 6:58 PM, Allison, Timothy B. wrote:
>>> If we actually moved the source files out of
> tika-parsers and into the modules this would not be required.
>
> Of course, got it...now. Thank you. In Tika 2.0, would we want to actually move the source code for the bundles into the bundles, I wonder. Or, would we still want to have one massive tika-parsers module as we do now?
+1 to moving the source to bundles. I think for a 2.0 would be easier
to consolidate into a parser uber jar than trying to tease things out
like I did in the straw man impl. However deciding how to break things
up might take some experimentation.
I added the Logging Jars to your test and I discovered 2 things.
1) To spin up the GUI you need org.apache.tika.parser.util (perhaps
consider moving this up to core).
2) Since the META-INF/services/org.apache.tika.parser.Parser is in
tika-parser we'd need to rethink the static ServiceLoader strategy to
either always be dynamic or figure out a way to have each jar bring
there own static loader.
>>> Can you provide the source for your test?
> Source?! Ha. No, I literally/manually copied the jars into bin... the horror! I'm sure maven would have added the transitive dependencies. I'll give that a try. Thank you, again.
You crack me up :).
>
> Best,
>
> Tim
>
> On Mon, Aug 3, 2015 at 1:56 PM, Allison, Timothy B. <ta...@mitre.org>
> wrote:
>
>> Bob,
>> Thank you for leading this effort. Please continue to forgive my lack
>> of OSGi knowledge in the following :).
>> I just built your strawman, and it looks great...I think(?)
>>
>> To confirm that I understand correctly...
>>
>> For non-OSGi folks, let's say I know that I'm only going to be parsing
>> images, to replicate the features of tika-app but with only the image
>> parsers, would I put these three jars in a bin directory (or add them via
>> maven to my project):
>>
>> original-tika-app-1.10-SNAPSHOT.jar
>> tika-image-parser-bundle-1.10-SNAPSHOT.jar
>> tika-core-1.10-SNAPSHOT.jar
>>
>> and then type:
>> java -cp "bin/*" org.apache.tika.cli.TikaCLI -then -the -commandline
>> -options
>>
>>
>> If this is the goal, why is tika-parsers "provided" in
>> tika-image-parser-bundle's pom? Aren't all dependencies of the image
>> parsers packaged within that bundle?
>>
>> When I actually tried this, I got a noclassdeferror for Apache logging.
>> Should we add that within the image-parser-bundle? Or should I be doing
>> something else (answer= very likely :) ).
>>
>> Thank you, again.
>>
>> Best,
>>
>> Tim
>>
>>
>> -----Original Message-----
>> From: Bob Paulin [mailto:bob@apache.org]
>> Sent: Saturday, July 25, 2015 9:07 PM
>> To: dev@tika.apache.org
>> Subject: [DISCUSS] A more modular parser project
>>
>> Hi,
>>
>> At Apache Con this year there was some discussion of some woes with OSGi
>> and Tika.
>>
>> The TL;DR version of this email is I think the pain with tika-bundle is
>> due to it's attempt to encompass all the parsers in one Uber jar. I
>> think breaking down the tika-bundle into smaller jars (like Apache
>> Camel) might provide a more maintainable approach going forward for OSGi
>> and non-OSGi developers alike.
>>
>> I've put up a straw man implementation of smaller bundles here:
>>
>> https://github.com/bobpaulin/tika
>> (specifically in
>> https://github.com/bobpaulin/tika/tree/trunk/tika-parser-bundles)
>>
>> For the impatient please check out the code above. The approach has
>> zero impact on any of the current tika jar files so nothing would have
>> to change for current users but they could chose to move to the new
>> approach if it better fit their needs. It's a straw man so feel free to
>> beat it up :).
>>
>> For the rest that have time for an explanation....
>> The straw man implementation does the following:
>>
>> 1) Copies and inlines the tika-core.jar in the tika-osgi-bundle. (see
>> pom.xml)
>>
>> 2) The tika-osgi-bundle registers a Tika Service that contains a Default
>> Parser that is composed of all Tika Parsers registered as OSGi services
>> as well as available Detectors. This extends the functionality in the
>> tika-core TikaActivator class.
>>
>> 3) Copies and inlines classes from the tika-parser project into specific
>> bundles (see individual pom.xml specifically in the maven-bundle-plugin).
>>
>> 4) Each bundle registers services for each parser and provides
>> configuration for ranking (see individual Activator.java). This could
>> expand considerably if we wanted to provide additional properties to
>> toggle features on and off.
>>
>> This provides the following advantages to users and developers in both
>> an OSGi and standard deployment:
>>
>> 1) These bundles could be evolved separately. The community could drive
>> how finely grained the bundles were. For example I bundled the image
>> and jpeg packages together but put tesseract ocr separate.
>>
>> 2) Users can just select the parsers they want to include in there
>> projects.
>>
>> 3) Each parser project could maintain it's own optional and embedded
>> bundle dependencies. Currently the tika-bundle project has a marathon
>> of these entries with optional and embedded dependencies clocking in at
>> ~200 line and ~20 lines respectively.
>>
>> If you made it this far thanks for reading :). As an OSGi developer I
>> think the implementation provides a better experience but could also
>> make things better for non-OSGi developers as well. And best of all it
>> works without requiring changes to any of the existing released jars.
>> Let me know what you think.
>>
>>
>> - Bob Paulin
>>
RE: [DISCUSS] A more modular parser project
Posted by "Allison, Timothy B." <ta...@mitre.org>.
>>If we actually moved the source files out of
tika-parsers and into the modules this would not be required.
Of course, got it...now. Thank you. In Tika 2.0, would we want to actually move the source code for the bundles into the bundles, I wonder. Or, would we still want to have one massive tika-parsers module as we do now?
>> Can you provide the source for your test?
Source?! Ha. No, I literally/manually copied the jars into bin... the horror! I'm sure maven would have added the transitive dependencies. I'll give that a try. Thank you, again.
Best,
Tim
On Mon, Aug 3, 2015 at 1:56 PM, Allison, Timothy B. <ta...@mitre.org>
wrote:
> Bob,
> Thank you for leading this effort. Please continue to forgive my lack
> of OSGi knowledge in the following :).
> I just built your strawman, and it looks great...I think(?)
>
> To confirm that I understand correctly...
>
> For non-OSGi folks, let's say I know that I'm only going to be parsing
> images, to replicate the features of tika-app but with only the image
> parsers, would I put these three jars in a bin directory (or add them via
> maven to my project):
>
> original-tika-app-1.10-SNAPSHOT.jar
> tika-image-parser-bundle-1.10-SNAPSHOT.jar
> tika-core-1.10-SNAPSHOT.jar
>
> and then type:
> java -cp "bin/*" org.apache.tika.cli.TikaCLI -then -the -commandline
> -options
>
>
> If this is the goal, why is tika-parsers "provided" in
> tika-image-parser-bundle's pom? Aren't all dependencies of the image
> parsers packaged within that bundle?
>
> When I actually tried this, I got a noclassdeferror for Apache logging.
> Should we add that within the image-parser-bundle? Or should I be doing
> something else (answer= very likely :) ).
>
> Thank you, again.
>
> Best,
>
> Tim
>
>
> -----Original Message-----
> From: Bob Paulin [mailto:bob@apache.org]
> Sent: Saturday, July 25, 2015 9:07 PM
> To: dev@tika.apache.org
> Subject: [DISCUSS] A more modular parser project
>
> Hi,
>
> At Apache Con this year there was some discussion of some woes with OSGi
> and Tika.
>
> The TL;DR version of this email is I think the pain with tika-bundle is
> due to it's attempt to encompass all the parsers in one Uber jar. I
> think breaking down the tika-bundle into smaller jars (like Apache
> Camel) might provide a more maintainable approach going forward for OSGi
> and non-OSGi developers alike.
>
> I've put up a straw man implementation of smaller bundles here:
>
> https://github.com/bobpaulin/tika
> (specifically in
> https://github.com/bobpaulin/tika/tree/trunk/tika-parser-bundles)
>
> For the impatient please check out the code above. The approach has
> zero impact on any of the current tika jar files so nothing would have
> to change for current users but they could chose to move to the new
> approach if it better fit their needs. It's a straw man so feel free to
> beat it up :).
>
> For the rest that have time for an explanation....
> The straw man implementation does the following:
>
> 1) Copies and inlines the tika-core.jar in the tika-osgi-bundle. (see
> pom.xml)
>
> 2) The tika-osgi-bundle registers a Tika Service that contains a Default
> Parser that is composed of all Tika Parsers registered as OSGi services
> as well as available Detectors. This extends the functionality in the
> tika-core TikaActivator class.
>
> 3) Copies and inlines classes from the tika-parser project into specific
> bundles (see individual pom.xml specifically in the maven-bundle-plugin).
>
> 4) Each bundle registers services for each parser and provides
> configuration for ranking (see individual Activator.java). This could
> expand considerably if we wanted to provide additional properties to
> toggle features on and off.
>
> This provides the following advantages to users and developers in both
> an OSGi and standard deployment:
>
> 1) These bundles could be evolved separately. The community could drive
> how finely grained the bundles were. For example I bundled the image
> and jpeg packages together but put tesseract ocr separate.
>
> 2) Users can just select the parsers they want to include in there
> projects.
>
> 3) Each parser project could maintain it's own optional and embedded
> bundle dependencies. Currently the tika-bundle project has a marathon
> of these entries with optional and embedded dependencies clocking in at
> ~200 line and ~20 lines respectively.
>
> If you made it this far thanks for reading :). As an OSGi developer I
> think the implementation provides a better experience but could also
> make things better for non-OSGi developers as well. And best of all it
> works without requiring changes to any of the existing released jars.
> Let me know what you think.
>
>
> - Bob Paulin
>
Re: [DISCUSS] A more modular parser project
Posted by Bob Paulin <bo...@bobpaulin.com>.
Hey Tim,
This is a really good question on the provided scope. In maven I need to
have some sort of dependency on tika-parsers so that I can inline some of
the classes from in the jar. If we actually moved the source files out of
tika-parsers and into the modules this would not be required.
In the OSGi environment I provided my own logging in the runtime but in a
standard deployment (like what you're doing with tika app) I would assume
that logging would be pulled in as a transitive dependency. If this was
happening automatically before perhaps there's something I need to add to
image parser. Can you provide the source for your test?
- Bob
On Mon, Aug 3, 2015 at 1:56 PM, Allison, Timothy B. <ta...@mitre.org>
wrote:
> Bob,
> Thank you for leading this effort. Please continue to forgive my lack
> of OSGi knowledge in the following :).
> I just built your strawman, and it looks great...I think(?)
>
> To confirm that I understand correctly...
>
> For non-OSGi folks, let's say I know that I'm only going to be parsing
> images, to replicate the features of tika-app but with only the image
> parsers, would I put these three jars in a bin directory (or add them via
> maven to my project):
>
> original-tika-app-1.10-SNAPSHOT.jar
> tika-image-parser-bundle-1.10-SNAPSHOT.jar
> tika-core-1.10-SNAPSHOT.jar
>
> and then type:
> java -cp "bin/*" org.apache.tika.cli.TikaCLI -then -the -commandline
> -options
>
>
> If this is the goal, why is tika-parsers "provided" in
> tika-image-parser-bundle's pom? Aren't all dependencies of the image
> parsers packaged within that bundle?
>
> When I actually tried this, I got a noclassdeferror for Apache logging.
> Should we add that within the image-parser-bundle? Or should I be doing
> something else (answer= very likely :) ).
>
> Thank you, again.
>
> Best,
>
> Tim
>
>
> -----Original Message-----
> From: Bob Paulin [mailto:bob@apache.org]
> Sent: Saturday, July 25, 2015 9:07 PM
> To: dev@tika.apache.org
> Subject: [DISCUSS] A more modular parser project
>
> Hi,
>
> At Apache Con this year there was some discussion of some woes with OSGi
> and Tika.
>
> The TL;DR version of this email is I think the pain with tika-bundle is
> due to it's attempt to encompass all the parsers in one Uber jar. I
> think breaking down the tika-bundle into smaller jars (like Apache
> Camel) might provide a more maintainable approach going forward for OSGi
> and non-OSGi developers alike.
>
> I've put up a straw man implementation of smaller bundles here:
>
> https://github.com/bobpaulin/tika
> (specifically in
> https://github.com/bobpaulin/tika/tree/trunk/tika-parser-bundles)
>
> For the impatient please check out the code above. The approach has
> zero impact on any of the current tika jar files so nothing would have
> to change for current users but they could chose to move to the new
> approach if it better fit their needs. It's a straw man so feel free to
> beat it up :).
>
> For the rest that have time for an explanation....
> The straw man implementation does the following:
>
> 1) Copies and inlines the tika-core.jar in the tika-osgi-bundle. (see
> pom.xml)
>
> 2) The tika-osgi-bundle registers a Tika Service that contains a Default
> Parser that is composed of all Tika Parsers registered as OSGi services
> as well as available Detectors. This extends the functionality in the
> tika-core TikaActivator class.
>
> 3) Copies and inlines classes from the tika-parser project into specific
> bundles (see individual pom.xml specifically in the maven-bundle-plugin).
>
> 4) Each bundle registers services for each parser and provides
> configuration for ranking (see individual Activator.java). This could
> expand considerably if we wanted to provide additional properties to
> toggle features on and off.
>
> This provides the following advantages to users and developers in both
> an OSGi and standard deployment:
>
> 1) These bundles could be evolved separately. The community could drive
> how finely grained the bundles were. For example I bundled the image
> and jpeg packages together but put tesseract ocr separate.
>
> 2) Users can just select the parsers they want to include in there
> projects.
>
> 3) Each parser project could maintain it's own optional and embedded
> bundle dependencies. Currently the tika-bundle project has a marathon
> of these entries with optional and embedded dependencies clocking in at
> ~200 line and ~20 lines respectively.
>
> If you made it this far thanks for reading :). As an OSGi developer I
> think the implementation provides a better experience but could also
> make things better for non-OSGi developers as well. And best of all it
> works without requiring changes to any of the existing released jars.
> Let me know what you think.
>
>
> - Bob Paulin
>