You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by cnsgithub <cn...@posteo.de> on 2018/11/18 10:43:21 UTC
Dependencies needed to just use TikaFileTypeDetector
Hey guys,
we recently contributed some security improvements to the popular JSF
component library PrimeFaces to validate specified accepted content
types of uploaded files at server side. Therefore we make use of Java's
Files.probeContentType which automatically picks up registered
java.nio.file.spi.FileTypeDetector service implementations. The default
implementations however primarily check the file extension only by doing
registry lookups or something like that. That's insufficient from a
security point of view. That's why we will recommend to have Apache Tika
in the classpath, more strictly speaking the tika-java7 dependency.
Now there are two questions regarding the use of TikaFileTypeDetector
and the required dependencies:
1. Transitive dependencies of tika-java7 are really big (more than 50
megabyte). I know that Apache Tika is not just about file type detection
but very much more like meta data extraction that we don't need at all.
Would it be okay to exclude tika-parsers which makes up the biggest
module without losing file type detection abilities?
2. Unfortunately, TikaFileTypeDetector defaults to perform most
efficiently by having included a short circuit if the content type can
be guessed from the file extension. This is however insecure since we
want to protect our users from tampered file uploads. We always want to
use deep (and expensive) content type analysis by looking at the magic
bytes or something like that. We currently work around this limitation
by explicitly putting .tmp as the file name's extension to have Tika
detected application/octet-stream and force it to go ahead. But that's
some kind of white box knowledge to rely on which is not that good. Can
you somehow provide different implementations of
java.nio.file.spi.FileTypeDetector, say "eager" and "lazy"? Please note
that we are not allowed to introduce required dependencies, i.e. using
Tika directly is not an option.
Here are the related issues:
https://github.com/primefaces/primefaces/issues/2791 and
https://github.com/primefaces/primefaces/issues/4244
And the pull requests already merged into PrimeFaces 6.3:
https://github.com/primefaces/primefaces/pull/4242 and
https://github.com/primefaces/primefaces/pull/4249
Thanks for your response in advance
Kind regards, cnsgithub
Re: Dependencies needed to just use TikaFileTypeDetector
Posted by Tim Allison <ta...@apache.org>.
Sorry for the late reply!
>Would it be okay to exclude tika-parsers which makes up the biggest
module without losing file type detection abilities?
It _should work_ for straight mime/magic detection. However, you need
some of the items within tika-parsers to figure out specific POIFS
(OLE/ooxml) and package types...e.g. what type of an ooxml or OLE file
is it, really, etc. These are the key classes that you'd be missing
without tika-parsers:
org.apache.tika.parser.microsoft.POIFSContainerDetector
org.apache.tika.parser.pkg.ZipContainerDetector And, you'd also need
to include POI and commons-compress as dependencies. In
short...experiment with those file types and see if the results are ok
for you.
>But that's some kind of white box knowledge to rely on which is not that good. Can
you somehow provide different implementations of
java.nio.file.spi.FileTypeDetector, say "eager" and "lazy"?
I wouldn't object to this. To confirm, we'd have two implementations
in the java7 package and you'd do something like this in your code:
Class.forName("org.apache.tika.filetypedetector.TikaBytesFileTypeDetector");
Open a ticket on our JIRA, and I'll see what I can do.
Fellow devs, any other recommendations?
Best,
Tim
On Sun, Nov 18, 2018 at 5:44 AM cnsgithub <cn...@posteo.de> wrote:
>
> Hey guys,
>
> we recently contributed some security improvements to the popular JSF
> component library PrimeFaces to validate specified accepted content
> types of uploaded files at server side. Therefore we make use of Java's
> Files.probeContentType which automatically picks up registered
> java.nio.file.spi.FileTypeDetector service implementations. The default
> implementations however primarily check the file extension only by doing
> registry lookups or something like that. That's insufficient from a
> security point of view. That's why we will recommend to have Apache Tika
> in the classpath, more strictly speaking the tika-java7 dependency.
>
> Now there are two questions regarding the use of TikaFileTypeDetector
> and the required dependencies:
> 1. Transitive dependencies of tika-java7 are really big (more than 50
> megabyte). I know that Apache Tika is not just about file type detection
> but very much more like meta data extraction that we don't need at all.
> Would it be okay to exclude tika-parsers which makes up the biggest
> module without losing file type detection abilities?
> 2. Unfortunately, TikaFileTypeDetector defaults to perform most
> efficiently by having included a short circuit if the content type can
> be guessed from the file extension. This is however insecure since we
> want to protect our users from tampered file uploads. We always want to
> use deep (and expensive) content type analysis by looking at the magic
> bytes or something like that. We currently work around this limitation
> by explicitly putting .tmp as the file name's extension to have Tika
> detected application/octet-stream and force it to go ahead. But that's
> some kind of white box knowledge to rely on which is not that good. Can
> you somehow provide different implementations of
> java.nio.file.spi.FileTypeDetector, say "eager" and "lazy"? Please note
> that we are not allowed to introduce required dependencies, i.e. using
> Tika directly is not an option.
>
> Here are the related issues:
> https://github.com/primefaces/primefaces/issues/2791 and
> https://github.com/primefaces/primefaces/issues/4244
> And the pull requests already merged into PrimeFaces 6.3:
> https://github.com/primefaces/primefaces/pull/4242 and
> https://github.com/primefaces/primefaces/pull/4249
>
> Thanks for your response in advance
>
> Kind regards, cnsgithub