You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Allison, Timothy B." <ta...@mitre.org> on 2017/04/13 18:35:55 UTC

[COMPRESS] zip-bomb prevention for Z?

On TIKA-1631 [1], users have observed that a corrupt Z file can cause an OOM at Internal_.InternalLZWStream.initializeTable.  Should we try to protect against this at the Tika level, or should we open an issue on commons-compress's JIRA?  

A second question, we're creating a stream with the CompressorStreamFactory when all we want to do is detect.  Is there a recommended way to detect the type of compressor without creating a stream?

Thank you!

            Best,

                     Tim

[1] https://issues.apache.org/jira/browse/TIKA-1631

Re: [COMPRESS] zip-bomb prevention for Z?

Posted by Stefan Bodewig <bo...@apache.org>.
On 2017-04-14, Allison, Timothy B. wrote:

>> enum wouldn't work for formats added via ServiceLoader. LZO supports a couple of names of its own and you couldn't inject them into the enum.

> Doh!  Got it.  New code base...Sorry.

No problem :-)

I've taken a prolonged offline Easter-weekend, will look into the your
PRs and comments in JIRA the coming days.

Many thanks

     Stefan

Re: [COMPRESS] zip-bomb prevention for Z?

Posted by Stefan Bodewig <bo...@apache.org>.
On 2017-04-14, Allison, Timothy B. wrote:

>> enum wouldn't work for formats added via ServiceLoader. LZO supports a couple of names of its own and you couldn't inject them into the enum.

> Doh!  Got it.  New code base...Sorry.

No problem :-)

I've taken a prolonged offline Easter-weekend, will look into the your
PRs and comments in JIRA the coming days.

Many thanks

     Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


RE: [COMPRESS] zip-bomb prevention for Z?

Posted by "Allison, Timothy B." <ta...@mitre.org>.
>enum wouldn't work for formats added via ServiceLoader. LZO supports a couple of names of its own and you couldn't inject them into the enum.

Doh!  Got it.  New code base...Sorry.

RE: [COMPRESS] zip-bomb prevention for Z?

Posted by "Allison, Timothy B." <ta...@mitre.org>.
>enum wouldn't work for formats added via ServiceLoader. LZO supports a couple of names of its own and you couldn't inject them into the enum.

Doh!  Got it.  New code base...Sorry.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Re: [COMPRESS] zip-bomb prevention for Z?

Posted by Stefan Bodewig <bo...@apache.org>.
On 2017-04-14, Allison, Timothy B. wrote:

>>> If there is anything COMPRESS can do to detect and avoid the
>>> situation, then please open an issue over here.

> Done: COMPRESS-385, PR submitted

Thanks.

>>> If we wanted to add such a method, what would the return value be?
>>> One of the String constants contained inside the *Factory classes,
>>> likely. Tika would have to be prepared for new strings popping up
>>> when using a newer version of Compress (1.14 will add "lz4-framed"
>>> for example).

> Y, I'm ok with a String...perhaps longer term or for 2.0, move to an
> enum?  Thank you for the heads-up!

enum wouldn't work for formats added via ServiceLoader. LZO supports a
couple of names of its own and you couldn't inject them into the enum.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Re: [COMPRESS] zip-bomb prevention for Z?

Posted by Stefan Bodewig <bo...@apache.org>.
On 2017-04-14, Allison, Timothy B. wrote:

>>> If there is anything COMPRESS can do to detect and avoid the
>>> situation, then please open an issue over here.

> Done: COMPRESS-385, PR submitted

Thanks.

>>> If we wanted to add such a method, what would the return value be?
>>> One of the String constants contained inside the *Factory classes,
>>> likely. Tika would have to be prepared for new strings popping up
>>> when using a newer version of Compress (1.14 will add "lz4-framed"
>>> for example).

> Y, I'm ok with a String...perhaps longer term or for 2.0, move to an
> enum?  Thank you for the heads-up!

enum wouldn't work for formats added via ServiceLoader. LZO supports a
couple of names of its own and you couldn't inject them into the enum.

Stefan

RE: [COMPRESS] zip-bomb prevention for Z?

Posted by "Allison, Timothy B." <ta...@mitre.org>.
>> If there is anything COMPRESS can do to detect and avoid the situation, then please open an issue over here.

Done: COMPRESS-385, PR submitted

>> If we wanted to add such a method, what would the return value be? One of the String constants contained inside the *Factory classes, likely. Tika would have to be prepared for new strings popping up when using a newer version of Compress (1.14 will add "lz4-framed" for example).

Y, I'm ok with a String...perhaps longer term or for 2.0, move to an enum?  Thank you for the heads-up!

I opened COMPRESS-386 to discuss adding a threshold for the table size.

As always, thank you!

Cheers,

          Tim


RE: [COMPRESS] zip-bomb prevention for Z?

Posted by "Allison, Timothy B." <ta...@mitre.org>.
>> If there is anything COMPRESS can do to detect and avoid the situation, then please open an issue over here.

Done: COMPRESS-385, PR submitted

>> If we wanted to add such a method, what would the return value be? One of the String constants contained inside the *Factory classes, likely. Tika would have to be prepared for new strings popping up when using a newer version of Compress (1.14 will add "lz4-framed" for example).

Y, I'm ok with a String...perhaps longer term or for 2.0, move to an enum?  Thank you for the heads-up!

I opened COMPRESS-386 to discuss adding a threshold for the table size.

As always, thank you!

Cheers,

          Tim


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Re: [COMPRESS] zip-bomb prevention for Z?

Posted by Stefan Bodewig <bo...@apache.org>.
On 2017-04-13, Allison, Timothy B. wrote:

> On TIKA-1631 [1], users have observed that a corrupt Z file can cause
> an OOM at Internal_.InternalLZWStream.initializeTable.

Ouch.

> Should we try to protect against this at the Tika level, or should we
> open an issue on commons-compress's JIRA?

If there is anything COMPRESS can do to detect and avoid the situation,
then please open an issue over here.

> A second question, we're creating a stream with the
> CompressorStreamFactory when all we want to do is detect.  Is there a
> recommended way to detect the type of compressor without creating a
> stream?

This has never been a goal of the *StreamFactory's but it would be
pretty easy to add "guess" or "detect" methods to them. At least for the
formats built-in into Compress.

Since 1.13 we support extensions of the factories via ServiceLoader and
LZO for Java[2] uses it for example.  Looking into the code again, we
don't support autodetection for formats added via ServiceProvider
anyway, so this is no restriction for the Tika case.

If we wanted to add such a method, what would the return value be? One
of the String constants contained inside the *Factory classes,
likely. Tika would have to be prepared for new strings popping up when
using a newer version of Compress (1.14 will add "lz4-framed" for
example).

Stefan

> [1] https://issues.apache.org/jira/browse/TIKA-1631

[2] https://github.com/shevek/lzo-java

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Re: [COMPRESS] zip-bomb prevention for Z?

Posted by Luís Filipe Nassif <lf...@gmail.com>.
I have reported a similar issue to them, see Compress-382, maybe those
issues should be handled at Compress side, if I understood correctly the
API contract.

Luis


Em 13 de abr de 2017 3:36 PM, "Allison, Timothy B." <ta...@mitre.org>
escreveu:

On TIKA-1631 [1], users have observed that a corrupt Z file can cause an
OOM at Internal_.InternalLZWStream.initializeTable.  Should we try to
protect against this at the Tika level, or should we open an issue on
commons-compress's JIRA?

A second question, we're creating a stream with the CompressorStreamFactory
when all we want to do is detect.  Is there a recommended way to detect the
type of compressor without creating a stream?

Thank you!

            Best,

                     Tim

[1] https://issues.apache.org/jira/browse/TIKA-1631

Re: [COMPRESS] zip-bomb prevention for Z?

Posted by Luís Filipe Nassif <lf...@gmail.com>.
I have reported a similar issue to them, see Compress-382, maybe those
issues should be handled at Compress side, if I understood correctly the
API contract.

Luis


Em 13 de abr de 2017 3:36 PM, "Allison, Timothy B." <ta...@mitre.org>
escreveu:

On TIKA-1631 [1], users have observed that a corrupt Z file can cause an
OOM at Internal_.InternalLZWStream.initializeTable.  Should we try to
protect against this at the Tika level, or should we open an issue on
commons-compress's JIRA?

A second question, we're creating a stream with the CompressorStreamFactory
when all we want to do is detect.  Is there a recommended way to detect the
type of compressor without creating a stream?

Thank you!

            Best,

                     Tim

[1] https://issues.apache.org/jira/browse/TIKA-1631

Re: [COMPRESS] zip-bomb prevention for Z?

Posted by Stefan Bodewig <bo...@apache.org>.
On 2017-04-13, Allison, Timothy B. wrote:

> On TIKA-1631 [1], users have observed that a corrupt Z file can cause
> an OOM at Internal_.InternalLZWStream.initializeTable.

Ouch.

> Should we try to protect against this at the Tika level, or should we
> open an issue on commons-compress's JIRA?

If there is anything COMPRESS can do to detect and avoid the situation,
then please open an issue over here.

> A second question, we're creating a stream with the
> CompressorStreamFactory when all we want to do is detect.  Is there a
> recommended way to detect the type of compressor without creating a
> stream?

This has never been a goal of the *StreamFactory's but it would be
pretty easy to add "guess" or "detect" methods to them. At least for the
formats built-in into Compress.

Since 1.13 we support extensions of the factories via ServiceLoader and
LZO for Java[2] uses it for example.  Looking into the code again, we
don't support autodetection for formats added via ServiceProvider
anyway, so this is no restriction for the Tika case.

If we wanted to add such a method, what would the return value be? One
of the String constants contained inside the *Factory classes,
likely. Tika would have to be prepared for new strings popping up when
using a newer version of Compress (1.14 will add "lz4-framed" for
example).

Stefan

> [1] https://issues.apache.org/jira/browse/TIKA-1631

[2] https://github.com/shevek/lzo-java