You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tika.apache.org by Marc C Ubaldino <ub...@mitre.org> on 2023/03/07 00:32:27 UTC
Re: [EXT] Re: Best practice for extracting content and metadata repeatedly
Tim, all
Greetings.
I ran into this same thing this week – I had been processing Images no problem with Tika 1.x.
Moving to 2.x metadata and output for images silently went away. I reviewed my tika-config and realized I was missing:
1. Dependencies in POM
2. Parser inclusion in tika-config.xml
What befuddled me was how obscure the discovery process was to map file extension, mime-type, class for Parser and required dependency in POM – those four things all need to line up. It feels like a simple tabulation that y’all know. It was all a bit of a research project to figure it all out -- that the Getting Started chapter glosses over.
Also maybe its described some place, but there are Maven items “tika-xxx-module” and “tika-xxx-parser”; Not clear about the the differences.
Its all great; I think the layout of plugin and mime type could be more linear.
// From there which class do I configure from which plugin would I get “MP4Parser” to parse MP4s? is answered by looking at a table.
// Model: Maven Plugins table is like this, https://maven.apache.org/plugins/index.html
Thanks,
Marc
From: Tim Allison <ta...@apache.org>
Date: Monday, March 6, 2023 at 2:46 PM
To: user@tika.apache.org <us...@tika.apache.org>, Chris Bamford <cb...@mimecast.com>
Subject: [EXT] Re: Best practice for extracting content and metadata repeatedly
I looked at our 2.x MP4Parser and it should handle video/quicktime. Are you using tika-parsers-standard-package in your dependencies? Which other parser dependencies are you adding?
On Mon, Mar 6, 2023 at 2:11 PM Tim Allison <ta...@apache.org>> wrote:
<facepalm/>
Narrator: Tika did used to have a NoakesMP4Parser... in 1.x
Sorry. Let me take a look at `video/quicktime`.
On Mon, Mar 6, 2023 at 2:02 PM Tim Allison <ta...@apache.org>> wrote:
Is NoakesMP4Parser your own custom parser? We don't have one with that name, I don't think?
On Mon, Mar 6, 2023 at 12:59 PM Chris Bamford via user <us...@tika.apache.org>> wrote:
Thanks Nick, that’s helpful.
On a related note, I seem to have a regression with the move to 2.6.0 from 1.28.1, I’m trying to extract metadata from a video/quicktime file which now results in nothing as it uses the EmptyParser.
The previous version used the CompositeParser / DefaultParser / NoakesMP4Parser and works.
The 1.28.1 version produces this metadata:
date=2015-03-22T18:17:53Z X-Parsed-By=org.apache.tika.parser.CompositeParser X-Parsed-By=org.apache.tika.parser.DefaultParser X-Parsed-By=org.apache.tika.parser.mp4.NoakesMP4Parser meta:creation-date=2015-03-22T18:17:34Z geo:lat=51.4937 X-TIKA:EXCEPTION:warn=End of data reached. Creation-Date=2015-03-22T18:17:34Z tiff:ImageLength=320 dcterms:created=2015-03-22T18:17:34Z dcterms:modified=2015-03-22T18:17:53Z Last-Modified=2015-03-22T18:17:53Z xmpDM:audioChannelType=Mono Last-Save-Date=2015-03-22T18:17:53Z xmpDM:audioSampleRate=44100 meta:save-date=2015-03-22T18:17:53Z xmpDM:videoCompressor=H.264 modified=2015-03-22T18:17:53Z geo:long=-0.1572 tiff:ImageWidth=568 xmpDM:duration=54.03 Content-Type=video/quicktime
The 2.6.0 version produces this metadata:
X-TIKA:Parsed-By=org.apache.tika.parser.EmptyParser X-TIKA:Parsed-By-Full-Set=org.apache.tika.parser.EmptyParser Content-Type=video/quicktime
Do I need to do something different when I set up my AutoDetectParser or something?
Thanks
Chris
Chris Bamford
m:
www.mimecast.com<https://www.mimecast.com/>
Principal Research Engineer
p: +44 207 847 8700
Address click here<https://www.mimecast.com/company/contact/>
________________________________
[Mimecast]<https://www.mimecast.com/>
[Linked In]<https://eu-api.mimecast.com/s/click/XujAZpejvFW2OIhYbUKIG1VF3MnXfLsZwisGiGbmHoz8st5DQYye4WcXuFSWn58fDmTi3NRm8XU1JsiYMKppMXVPCRBYg4JXq_Wd9owjxjfh4R2PlEAcZSGpm-yAxDIQ-1-DXs8HNoBxB7OUVIfjBbm80zerQX9iyu2hUqSsBeorOQA5m0DSs02m-WfDE0D83w1g2w9_z36jCiA03nm2wQ>
[You Tube]<https://eu-api.mimecast.com/s/click/1K7xTdhoqgjnB3PEFCIbOROQ0sKMO9fxVPBOuSDKWe6XVJZrcKzMoHRotnZNSRHkUU8AQ9RSCt_9sx_GtrTtEeNCG2swydD93ADB_9_8TnxB1M0_DwHy9MGqDI9AEJNIYfAkrLU-N8ixSUeIWv9S6D3ZDvbJ75yF5X__jZqhXBHfmDdODW0-7_V7tYxjVVWW>
[Facebook]<https://eu-api.mimecast.com/s/click/BBdygDA5Eu0J_lRKuagyozUY2VSEGpXcxrEKXclPgSxYK0ik3Z19QlRGbLX0NMkOLIYxn-D-RPObgfWxO8QhtDGw6dYV6hoP2RH4qcZ49_81_KLsWMnwzBkCdnMrfKBZKaU9GmaCv7bsfiPrc1vDeGCxeQMhwA0psDQtcRA8W1XN9-wp8VMNdVE7HtaiqcNTgEE0pJQ-WxBJMbAchLm44Q>
[Blog]<https://eu-api.mimecast.com/s/click/Hkg07htbWBc7_Em75CosQze0cQCUZKrLxeSRZQg9xGMp2Ojs4p5Jo-vFGIbqFJ-cKOfG-DLOcJyfn_f8Fl9vvckxGONf9uqzeT_AAZpfhuTiX7KidmoN6GzMjMQWumYanHZewo0YErIe-q4WFPDD_qBCBA6_ECyFJQP61dl1VNKfXwH2vxDINii3gO0GZD_jUWsThNMfSE_TxYg9ZhC-Fg>
[Twitter]<https://eu-api.mimecast.com/s/click/0ChSNgfhxT33DvPLIaGrHYoZTcWuX9gWREcUN01Z52BQebcK4pc67SGPzi3NU-Hg1krTEjUc3eHf2F6iVVKzfrHd3rhUiqTKaBAJsmUjHAa1yoN_6hkApFMwJc-m_dIQAYJxPcuj3-4ZW8r5Lkxbr3NNNgP-H8ZSK9uaCYn8cQy1L_1sz7M7y1Bd_i_wQkNHb4nQcAFT3HGZXZ-jRRaPFw>
[Mimecast Presents: 2023 State of Email Security]<https://eu-api.mimecast.com/s/click/XujAZpejvFW2OIhYbUKIG3RLt0NMPcOMk5p8FZuqJcyHEeZmgi-TGh_KoxnfFutNzmTmZgajDwrot2Scp36nw3VPCRBYg4JXq_Wd9owjxjfh4R2PlEAcZSGpm-yAxDIQ-1-DXs8HNoBxB7OUVIfjBbm80zerQX9iyu2hUqSsBeorOQA5m0DSs02m-WfDE0D8cI4NUPmas9gcSMVqgbIs4w>
Disclaimer
The information contained in this communication from cbamford@mimecast.com<ma...@mimecast.com> sent at 2023-03-06 17:58:23 is confidential and may be legally privileged. It is intended solely for use by user@tika.apache.org<ma...@tika.apache.org> and others authorized to receive it. If you are not user@tika.apache.org<ma...@tika.apache.org> you are hereby notified that any disclosure, copying, distribution or taking action in reliance of the contents of this information is strictly prohibited and may be unlawful.
This email message has been scanned for viruses by Mimecast. Mimecast delivers a complete managed email solution from a single web based platform. For more information please visit https://www.mimecast.com
Visit our preference center to change how often you hear from us: Preference Center<https://info.mimecast.com/Subscription-Management.html?utm_source=EmailStationary>.
From: Nick Burch <ni...@apache.org>>
Date: Monday, 6 March 2023 at 14:15
To: Chris Bamford <cb...@mimecast.com>>
Cc: user@tika.apache.org<ma...@tika.apache.org> <us...@tika.apache.org>>
Subject: Re: Best practice for extracting content and metadata repeatedly
On Mon, 6 Mar 2023, Chris Bamford via user wrote:
> From both performance and thread safety points of view what is the best
> approach for the use / reuse of the following objects:
>
> Tika
> ParseContext
> Parser
> Metadata
The Tika object and/or TikaConfig object should only be created once and
then re-used. Same for any Parser or Detector instances
ParseContext ought to be fine to re-use, but it's such a light-weight
thing I normally create one fresh.
Metadata is normally created from scratch each time - all the entries need
to be removed and recreating is typically a lot less work then removing
everything
Depending on how untrustworth / variable the input you're receiving is,
and the impact of an OOM or similar, you might want to look into using the
Tika Server or Fork Mode or Batch Mode.
Nick
Re: [EXT] Re: Best practice for extracting content and metadata repeatedly
Posted by Tim Allison <ta...@apache.org>.
Thank you, Marc.
Please let us know how we can improve the documentation here:
https://cwiki.apache.org/confluence/display/TIKA/Migrating+to+Tika+2.0.0
and/or if we need to add documentation elsewhere.
On Mon, Mar 6, 2023 at 7:32 PM Marc C Ubaldino <ub...@mitre.org> wrote:
> Tim, all
>
>
>
> Greetings.
>
>
>
> I ran into this same thing this week – I had been processing Images no
> problem with Tika 1.x.
>
> Moving to 2.x metadata and output for images silently went away. I
> reviewed my tika-config and realized I was missing:
>
> 1. Dependencies in POM
> 2. Parser inclusion in tika-config.xml
>
>
>
> What befuddled me was how obscure the discovery process was to map file
> extension, mime-type, class for Parser and required dependency in POM –
> those four things all need to line up. It feels like a simple tabulation
> that y’all know. It was all a bit of a research project to figure it all
> out -- that the Getting Started chapter glosses over.
>
>
>
> Also maybe its described some place, but there are Maven items
> “tika-xxx-module” and “tika-xxx-parser”; Not clear about the the
> differences.
>
>
>
> Its all great; I think the layout of plugin and mime type could be more
> linear.
>
>
>
> // From there which class do I configure from which plugin would I get “MP4Parser”
> to parse MP4s? is answered by looking at a table.
>
> // Model: Maven Plugins table is like this,
> https://maven.apache.org/plugins/index.html
>
>
>
> Thanks,
>
> Marc
>
>
>
>
>
> *From: *Tim Allison <ta...@apache.org>
> *Date: *Monday, March 6, 2023 at 2:46 PM
> *To: *user@tika.apache.org <us...@tika.apache.org>, Chris Bamford <
> cbamford@mimecast.com>
> *Subject: *[EXT] Re: Best practice for extracting content and metadata
> repeatedly
>
> I looked at our 2.x MP4Parser and it should handle video/quicktime. Are
> you using tika-parsers-standard-package in your dependencies? Which other
> parser dependencies are you adding?
>
>
>
> On Mon, Mar 6, 2023 at 2:11 PM Tim Allison <ta...@apache.org> wrote:
>
> <facepalm/>
>
>
>
> Narrator: Tika did used to have a NoakesMP4Parser... in 1.x
>
>
>
> Sorry. Let me take a look at `video/quicktime`.
>
>
>
> On Mon, Mar 6, 2023 at 2:02 PM Tim Allison <ta...@apache.org> wrote:
>
> Is NoakesMP4Parser your own custom parser? We don't have one with that
> name, I don't think?
>
>
>
> On Mon, Mar 6, 2023 at 12:59 PM Chris Bamford via user <
> user@tika.apache.org> wrote:
>
>
>
> Thanks Nick, that’s helpful.
>
>
>
> On a related note, I seem to have a regression with the move to 2.6.0 from
> 1.28.1, I’m trying to extract metadata from a video/quicktime file which
> now results in nothing as it uses the EmptyParser.
>
> The previous version used the CompositeParser / DefaultParser /
> NoakesMP4Parser and works.
>
>
>
> The 1.28.1 version produces this metadata:
>
> *date=2015-03-22T18:17:53Z
> X-Parsed-By=org.apache.tika.parser.CompositeParser
> X-Parsed-By=org.apache.tika.parser.DefaultParser
> X-Parsed-By=org.apache.tika.parser.mp4.NoakesMP4Parser
> meta:creation-date=2015-03-22T18:17:34Z geo:lat=51.4937
> X-TIKA:EXCEPTION:warn=End of data reached.
> Creation-Date=2015-03-22T18:17:34Z tiff:ImageLength=320
> dcterms:created=2015-03-22T18:17:34Z dcterms:modified=2015-03-22T18:17:53Z
> Last-Modified=2015-03-22T18:17:53Z xmpDM:audioChannelType=Mono
> Last-Save-Date=2015-03-22T18:17:53Z xmpDM:audioSampleRate=44100
> meta:save-date=2015-03-22T18:17:53Z xmpDM:videoCompressor=H.264
> modified=2015-03-22T18:17:53Z geo:long=-0.1572 tiff:ImageWidth=568
> xmpDM:duration=54.03 Content-Type=video/quicktime*
>
>
>
> The 2.6.0 version produces this metadata:
>
> *X-TIKA:Parsed-By=org.apache.tika.parser.EmptyParser
> X-TIKA:Parsed-By-Full-Set=org.apache.tika.parser.EmptyParser
> Content-Type=video/quicktime*
>
>
>
> Do I need to do something different when I set up my AutoDetectParser or
> something?
>
>
>
> Thanks
>
> Chris
>
>
>
> Chris Bamford
>
> m:
>
> www.mimecast.com
>
> Principal Research Engineer
>
> p: +44 207 847 8700
>
> Address click here <https://www.mimecast.com/company/contact/>
> ------------------------------
>
> [image: Mimecast] <https://www.mimecast.com/>
>
>
>
> [image: Linked In]
> <https://eu-api.mimecast.com/s/click/XujAZpejvFW2OIhYbUKIG1VF3MnXfLsZwisGiGbmHoz8st5DQYye4WcXuFSWn58fDmTi3NRm8XU1JsiYMKppMXVPCRBYg4JXq_Wd9owjxjfh4R2PlEAcZSGpm-yAxDIQ-1-DXs8HNoBxB7OUVIfjBbm80zerQX9iyu2hUqSsBeorOQA5m0DSs02m-WfDE0D83w1g2w9_z36jCiA03nm2wQ>
>
> [image: You Tube]
> <https://eu-api.mimecast.com/s/click/1K7xTdhoqgjnB3PEFCIbOROQ0sKMO9fxVPBOuSDKWe6XVJZrcKzMoHRotnZNSRHkUU8AQ9RSCt_9sx_GtrTtEeNCG2swydD93ADB_9_8TnxB1M0_DwHy9MGqDI9AEJNIYfAkrLU-N8ixSUeIWv9S6D3ZDvbJ75yF5X__jZqhXBHfmDdODW0-7_V7tYxjVVWW>
>
> [image: Facebook]
> <https://eu-api.mimecast.com/s/click/BBdygDA5Eu0J_lRKuagyozUY2VSEGpXcxrEKXclPgSxYK0ik3Z19QlRGbLX0NMkOLIYxn-D-RPObgfWxO8QhtDGw6dYV6hoP2RH4qcZ49_81_KLsWMnwzBkCdnMrfKBZKaU9GmaCv7bsfiPrc1vDeGCxeQMhwA0psDQtcRA8W1XN9-wp8VMNdVE7HtaiqcNTgEE0pJQ-WxBJMbAchLm44Q>
>
> [image: Blog]
> <https://eu-api.mimecast.com/s/click/Hkg07htbWBc7_Em75CosQze0cQCUZKrLxeSRZQg9xGMp2Ojs4p5Jo-vFGIbqFJ-cKOfG-DLOcJyfn_f8Fl9vvckxGONf9uqzeT_AAZpfhuTiX7KidmoN6GzMjMQWumYanHZewo0YErIe-q4WFPDD_qBCBA6_ECyFJQP61dl1VNKfXwH2vxDINii3gO0GZD_jUWsThNMfSE_TxYg9ZhC-Fg>
>
> [image: Twitter]
> <https://eu-api.mimecast.com/s/click/0ChSNgfhxT33DvPLIaGrHYoZTcWuX9gWREcUN01Z52BQebcK4pc67SGPzi3NU-Hg1krTEjUc3eHf2F6iVVKzfrHd3rhUiqTKaBAJsmUjHAa1yoN_6hkApFMwJc-m_dIQAYJxPcuj3-4ZW8r5Lkxbr3NNNgP-H8ZSK9uaCYn8cQy1L_1sz7M7y1Bd_i_wQkNHb4nQcAFT3HGZXZ-jRRaPFw>
>
>
>
>
>
> [image: Mimecast Presents: 2023 State of Email Security]
> <https://eu-api.mimecast.com/s/click/XujAZpejvFW2OIhYbUKIG3RLt0NMPcOMk5p8FZuqJcyHEeZmgi-TGh_KoxnfFutNzmTmZgajDwrot2Scp36nw3VPCRBYg4JXq_Wd9owjxjfh4R2PlEAcZSGpm-yAxDIQ-1-DXs8HNoBxB7OUVIfjBbm80zerQX9iyu2hUqSsBeorOQA5m0DSs02m-WfDE0D8cI4NUPmas9gcSMVqgbIs4w>
>
>
> *Disclaimer*
> The information contained in this communication from *cbamford@mimecast.com
> <cb...@mimecast.com> *sent at 2023-03-06 17:58:23 is confidential and
> may be legally privileged. It is intended solely for use by *user@tika.apache.org
> <us...@tika.apache.org> *and others authorized to receive it. If you are
> not *user@tika.apache.org <us...@tika.apache.org> *you are hereby notified
> that any disclosure, copying, distribution or taking action in reliance of
> the contents of this information is strictly prohibited and may be unlawful.
>
> This email message has been scanned for viruses by Mimecast. Mimecast
> delivers a complete managed email solution from a single web based
> platform. For more information please visit https://www.mimecast.com
>
> Visit our preference center to change how often you hear from us: Preference
> Center
> <https://info.mimecast.com/Subscription-Management.html?utm_source=EmailStationary>.
>
>
>
>
>
>
> *From: *Nick Burch <ni...@apache.org>
> *Date: *Monday, 6 March 2023 at 14:15
> *To: *Chris Bamford <cb...@mimecast.com>
> *Cc: *user@tika.apache.org <us...@tika.apache.org>
> *Subject: *Re: Best practice for extracting content and metadata
> repeatedly
>
> On Mon, 6 Mar 2023, Chris Bamford via user wrote:
> > From both performance and thread safety points of view what is the best
> > approach for the use / reuse of the following objects:
> >
> > Tika
> > ParseContext
> > Parser
> > Metadata
>
> The Tika object and/or TikaConfig object should only be created once and
> then re-used. Same for any Parser or Detector instances
>
> ParseContext ought to be fine to re-use, but it's such a light-weight
> thing I normally create one fresh.
>
> Metadata is normally created from scratch each time - all the entries need
> to be removed and recreating is typically a lot less work then removing
> everything
>
>
> Depending on how untrustworth / variable the input you're receiving is,
> and the impact of an OOM or similar, you might want to look into using the
> Tika Server or Fork Mode or Batch Mode.
>
> Nick
>
>
>
>