You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by GitBox <gi...@apache.org> on 2020/09/11 17:47:54 UTC
[GitHub] [tika] tbentleypfpt opened a new pull request #356: Attempt to read zips with STORED data descriptors
tbentleypfpt opened a new pull request #356:
URL: https://github.com/apache/tika/pull/356
If we get an UnsupportedZipFeatureException because of the DataDescriptor feature, create a new zip input stream with reading of entries with data descriptors enabled and attempt to parse the entries again.
Apache Ticket: https://issues.apache.org/jira/browse/TIKA-3196
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt edited a comment on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt edited a comment on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-697478056
@PeterAlfredLee - I just put that zip generation code into the ZipParserTest but it's throwing a 'Truncated Zip File' exception when run.
org.apache.tika.exception.TikaException: TIKA-198: Illegal IOException from org.apache.tika.parser.pkg.PackageParser@3688eb5b
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:283)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:277)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:143)
at org.apache.tika.parser.pkg.ZipParserTest.testZipUsingStoredWithDataDescriptor(ZipParserTest.java:88)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Caused by: org.apache.tika.io.TaggedIOException: Truncated ZIP file
at org.apache.tika.io.TaggedInputStream.handleIOException(TaggedInputStream.java:133)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:106)
at org.apache.tika.io.IOUtils.read(IOUtils.java:935)
at org.apache.tika.detect.zip.DefaultZipContainerDetector.detect(DefaultZipContainerDetector.java:101)
at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:84)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:116)
at org.apache.tika.parser.DelegatingParser.parse(DelegatingParser.java:72)
at org.apache.tika.extractor.ParsingEmbeddedDocumentExtractor.parseEmbedded(ParsingEmbeddedDocumentExtractor.java:105)
at org.apache.tika.parser.pkg.PackageParser.parseEntry(PackageParser.java:351)
at org.apache.tika.parser.pkg.PackageParser.readEntries(PackageParser.java:303)
at org.apache.tika.parser.pkg.PackageParser.parse(PackageParser.java:285)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:277)
... 29 more
Caused by: org.apache.tika.io.TaggedIOException: Truncated ZIP file
at org.apache.tika.io.TaggedInputStream.handleIOException(TaggedInputStream.java:133)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:106)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:102)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:102)
... 39 more
Caused by: java.io.IOException: Truncated ZIP file
at org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.readStoredEntry(ZipArchiveInputStream.java:934)
at org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.readStored(ZipArchiveInputStream.java:544)
at org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:486)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:102)
... 41 more
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] PeterAlfredLee commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
PeterAlfredLee commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-696618151
> Do we have to reset the stream before reprocessing?
+1. The stream should be `reset` or `relocation to the beginning of the file`.
I think this is complicated here, cause the stream can may be not a `seekable` or `resetable` stream. Seems we got to write the stream to a temp file, which may cause some other problems cause zip archives may be pretty huge. Maybe you have other better ideas? @tballison
> Can you create a unit test with a small document that shows that this works?
+1. I can write a test for this. Will update when this is done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] PeterAlfredLee commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
PeterAlfredLee commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698066699
> @PeterAlfredLee - I just put that zip generation code into the ZipParserTest but it's throwing a 'Truncated Zip File' exception when run.
Oops. I forgot to add the Data Descriptor signature in the generated zip - even through it is recommanded to have a DD signature, it's a legal zip without it. The generated zip could be successfully read with latest Commons Compress code(that's why I successfully read it on my computer using latest Commons Compress) but it could not work on Compress v1.20 - because I fixed this with commit `86bb35a` in Commons Compress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt edited a comment on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt edited a comment on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698613886
> > finding out a way to reset the ContentHandler would be more easier.
>
> We were thinking about adding a resettable content handler to Tika 2.0, but it is really tricky and adds some complexity. If you can find a solution, great, but I'd be happy enough throwing the exception and then putting it on the client code to rerun the parse with PKGParser configured to allow for data descriptors.
Having the client code handle the exception / re-running the parsing seems like the better option at this point. My initial thought is to create a ZipArchiveInputStreamFactory in commons-compress in which you can set values for the various ZipArchiveInputStream parameters, and to have methods to generate a ZipArchiveInputStream using those parameters.
Then when the data descriptor exception is thrown and caught, the client can create a ZipArchiveInputStreamFactory with the data descriptor feature enabled and set that in the ParseContext.
Then in PackageParser, when we go to create an ArchiveInputStream, if the archive type is zip and a ZipArchiveInputStreamFactory is defined in the ParseContext then use the ZipArchiveInputStreamFactory to generate the ZipArchiveInputStream. Otherwise use the ArchiveStreamFactory.
Something like this:
` // in commons-compress
import java.io.InputStream;
import static org.apache.commons.compress.archivers.zip.ZipEncodingHelper.UTF8;
public class ZipArchiveInputStreamFactory {
private String encoding = UTF8;
private boolean allowStoredEntriesWithDataDescriptor = false;
private boolean useUnicodeExtraFields = true;
private boolean skipSplitSig = false;
public ZipArchiveInputStreamFactory(){}
public void setEncoding(String encoding) {
this.encoding = encoding;
}
public void setAllowStoredEntriesWithDataDescriptor(boolean allowStoredEntriesWithDataDescriptor) {
this.allowStoredEntriesWithDataDescriptor = allowStoredEntriesWithDataDescriptor;
}
public void setUseUnicodeExtraFields(boolean useUnicodeExtraFields) {
this.useUnicodeExtraFields = useUnicodeExtraFields;
}
public void setSkipSplitSig(boolean skipSplitSig) {
this.skipSplitSig = skipSplitSig;
}
public ZipArchiveInputStream createZipArchiveInputStream(InputStream stream) {
return new ZipArchiveInputStream(stream, encoding, useUnicodeExtraFields, allowStoredEntriesWithDataDescriptor, skipSplitSig);
}
}
// PackageParser.java
...
try {
ArchiveStreamFactory factory = context.get(ArchiveStreamFactory.class, new ArchiveStreamFactory());
// At the end we want to close the archive stream to release
// any associated resources, but the underlying document stream
// should not be closed
CloseShieldInputStream csis = new CloseShieldInputStream(stream);
ZipArchiveInputStreamFactory zipArchiveInputStreamFactory = context.get(ZipArchiveInputStreamFactory);
if (ArchiveStreamFactory.detect(csis).equalsIgnoreCase("zip") && zipArchiveInputStreamFactory != null) {
ais = zipArchiveInputStreamFactory.createZipArchiveInputStream(csis);
} else {
ais = factory.createArchiveInputStream(csis);
}
}
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] PeterAlfredLee commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
PeterAlfredLee commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698696817
I think I found a solution :
we can read the archive and save the count of entries we have read. If a UnsupportedZipFeatureException is met, we can reset the stream and skip the entries we have already read(these entries have already beed written to the handler), then we can continue reading the remain of the zip with allowDataDescriptor true.
I'm almost finished with this and would push another PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt edited a comment on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt edited a comment on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698613886
> > finding out a way to reset the ContentHandler would be more easier.
>
> We were thinking about adding a resettable content handler to Tika 2.0, but it is really tricky and adds some complexity. If you can find a solution, great, but I'd be happy enough throwing the exception and then putting it on the client code to rerun the parse with PKGParser configured to allow for data descriptors.
Having the client code handle the exception / re-running the parsing seems like the better option at this point. My initial thought is to create a ZipArchiveInputStreamFactory in commons-compress in which you can set values for the various ZipArchiveInputStream parameters, and to have methods to generate a ZipArchiveInputStream using those parameters.
Then when the data descriptor exception is thrown and caught, the client can create a ZipArchiveInputStreamFactory with the data descriptor feature enabled and set that in the ParseContext.
Then in PackageParser, when we go to create an ArchiveInputStream, if the archive type is zip and a ZipArchiveInputStreamFactory is defined in the ParseContext then use the ZipArchiveInputStreamFactory to generate the ZipArchiveInputStream. Otherwise use the ArchiveStreamFactory.
Something like this:
` // in commons-compress
import java.io.InputStream;
import static org.apache.commons.compress.archivers.zip.ZipEncodingHelper.UTF8;
public class ZipArchiveInputStreamFactory {
private String encoding = UTF8;
private boolean allowStoredEntriesWithDataDescriptor = false;
private boolean useUnicodeExtraFields = true;
private boolean skipSplitSig = false;
public ZipArchiveInputStreamFactory(){}
public void setEncoding(String encoding) {
this.encoding = encoding;
}
public void setAllowStoredEntriesWithDataDescriptor(boolean allowStoredEntriesWithDataDescriptor) {
this.allowStoredEntriesWithDataDescriptor = allowStoredEntriesWithDataDescriptor;
}
public void setUseUnicodeExtraFields(boolean useUnicodeExtraFields) {
this.useUnicodeExtraFields = useUnicodeExtraFields;
}
public void setSkipSplitSig(boolean skipSplitSig) {
this.skipSplitSig = skipSplitSig;
}
public ZipArchiveInputStream createZipArchiveInputStream(InputStream stream) {
return new ZipArchiveInputStream(stream, encoding, useUnicodeExtraFields, allowStoredEntriesWithDataDescriptor, skipSplitSig);
}
}
// PackageParser.java
...
try {
ArchiveStreamFactory factory = context.get(ArchiveStreamFactory.class, new ArchiveStreamFactory());
// At the end we want to close the archive stream to release
// any associated resources, but the underlying document stream
// should not be closed
CloseShieldInputStream csis = new CloseShieldInputStream(stream);
Object zipArchiveInputStreamFactory = context.get(ZipArchiveInputStreamFactory);
if (ArchiveStreamFactory.detect(csis).equalsIgnoreCase("zip") && zipArchiveInputStreamFactory != null) {
ais = zipArchiveInputStreamFactory.createZipArchiveInputStream(csis);
} else {
ais = factory.createArchiveInputStream(csis);
}
}
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] PeterAlfredLee commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
PeterAlfredLee commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698067189
I updated a little bit with the DD signature part. This works with Compress 1.20 and works fine with this PR.
```
private InputStream forgeZipInputStream() throws IOException {
// generate a zip archive in memory
try (final ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
final ZipArchiveOutputStream zo = new ZipArchiveOutputStream(byteArrayOutputStream);){
final ZipArchiveEntry entryA = new ZipArchiveEntry("foo");
entryA.setMethod(ZipEntry.STORED);
entryA.setSize(4);
entryA.setCrc(0xb63cfbcdl);
zo.putArchiveEntry(entryA);
zo.write(new byte[] { 1, 2, 3, 4 });
zo.closeArchiveEntry();
zo.close();
final byte[] zipContent = byteArrayOutputStream.toByteArray();
final byte[] old = Arrays.copyOf(zipContent, zipContent.length);
final byte[] zipContentWithDataDescriptor = new byte[zipContent.length + 16];
System.arraycopy(zipContent, 0, zipContentWithDataDescriptor, 0, 37);
// modify the general purpose bit flag
zipContentWithDataDescriptor[6] = (byte) 0x08;
// add the Data Descriptor Signature
zipContentWithDataDescriptor[37] = (byte) 0x50;
zipContentWithDataDescriptor[38] = (byte) 0x4B;
zipContentWithDataDescriptor[39] = (byte) 0x07;
zipContentWithDataDescriptor[40] = (byte) 0x08;
// copy the crc-32, compressed size and uncompressed size to the data descriptor
System.arraycopy(zipContent, 14, zipContentWithDataDescriptor, 41, 12);
// and copy the rest of the zip content
System.arraycopy(zipContent, 37, zipContentWithDataDescriptor, 53, zipContent.length - 37);
return new ByteArrayInputStream(zipContentWithDataDescriptor);
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] PeterAlfredLee edited a comment on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
PeterAlfredLee edited a comment on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-696721537
I forged a zip archive in memory that uses STORED and Data Descriptor at the same time. This could be easily used as a test case.
BTW this PR could not pass this test because the stream was not reseted.
```
@Test
public void testZipUsingStoredWithDataDescriptor() throws Exception {
ContentHandler handler = new BodyContentHandler();
Metadata metadata = new Metadata();
try (InputStream stream = forgeZipInputStream() ) {
AUTO_DETECT_PARSER.parse(stream, handler, metadata, recursingContext);
}
}
private InputStream forgeZipInputStream() throws IOException {
// generate a zip archive in memory
try (final ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
final ZipArchiveOutputStream zo = new ZipArchiveOutputStream(byteArrayOutputStream);){
final ZipArchiveEntry entryA = new ZipArchiveEntry("foo");
entryA.setMethod(ZipEntry.STORED);
entryA.setSize(4);
entryA.setCrc(0xb63cfbcdl);
zo.putArchiveEntry(entryA);
zo.write(new byte[] { 1, 2, 3, 4 });
zo.closeArchiveEntry();
zo.close();
final byte[] zipContent = byteArrayOutputStream.toByteArray();
final byte[] old = Arrays.copyOf(zipContent, zipContent.length);
final byte[] zipContentWithDataDescriptor = new byte[zipContent.length + 12];
System.arraycopy(zipContent, 0, zipContentWithDataDescriptor, 0, 37);
// modify the general purpose bit flag
zipContentWithDataDescriptor[6] = 8;
// copy the crc-32, compressed size and uncompressed size to the data descriptor
System.arraycopy(zipContent, 14, zipContentWithDataDescriptor, 37, 12);
// and copy the rest of the zip content
System.arraycopy(zipContent, 37, zipContentWithDataDescriptor, 49, zipContent.length - 37);
return new ByteArrayInputStream(zipContentWithDataDescriptor);
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] PeterAlfredLee commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
PeterAlfredLee commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698696817
I think I found a solution :
we can read the archive and save the count of entries we have read. If a UnsupportedZipFeatureException is met, we can reset the stream and skip the entries we have already read(these entries have already beed written to the handler), then we can continue reading the remain of the zip with allowDataDescriptor true.
I'm almost finished with this and would push another PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tballison commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tballison commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-694888751
Thank you for this PR. Two questions:
1. Do we have to reset the stream before reprocessing?
2. Can you create a unit test with a small document that shows that this works?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt edited a comment on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt edited a comment on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698613886
> > finding out a way to reset the ContentHandler would be more easier.
>
> We were thinking about adding a resettable content handler to Tika 2.0, but it is really tricky and adds some complexity. If you can find a solution, great, but I'd be happy enough throwing the exception and then putting it on the client code to rerun the parse with PKGParser configured to allow for data descriptors.
Having the client code handle the exception / re-running the parsing seems like the better option at this point. My initial thought is to create a ZipArchiveInputStreamFactory in commons-compress in which you can set values for the various ZipArchiveInputStream parameters, and to have methods to generate a ZipArchiveInputStream using those parameters.
Then when the data descriptor exception is thrown and caught, the client can create a ZipArchiveInputStreamFactory with the data descriptor feature enabled and set that in the ParseContext.
Then in PackageParser, when we go to create an ArchiveInputStream, if the archive type is zip and a ZipArchiveInputStreamFactory is defined in the ParseContext then use the ZipArchiveInputStreamFactory to generate the ZipArchiveInputStream. Otherwise use the ArchiveStreamFactory.
Something like this:
`
// in commons-compress
import java.io.InputStream;
import static org.apache.commons.compress.archivers.zip.ZipEncodingHelper.UTF8;
public class ZipArchiveInputStreamFactory {
private String encoding = UTF8;
private boolean allowStoredEntriesWithDataDescriptor = false;
private boolean useUnicodeExtraFields = true;
private boolean skipSplitSig = false;
public ZipArchiveInputStreamFactory(){}
public void setEncoding(String encoding) {
this.encoding = encoding;
}
public void setAllowStoredEntriesWithDataDescriptor(boolean allowStoredEntriesWithDataDescriptor) {
this.allowStoredEntriesWithDataDescriptor = allowStoredEntriesWithDataDescriptor;
}
public void setUseUnicodeExtraFields(boolean useUnicodeExtraFields) {
this.useUnicodeExtraFields = useUnicodeExtraFields;
}
public void setSkipSplitSig(boolean skipSplitSig) {
this.skipSplitSig = skipSplitSig;
}
public ZipArchiveInputStream createZipArchiveInputStream(InputStream stream) {
return new ZipArchiveInputStream(stream, encoding, useUnicodeExtraFields, allowStoredEntriesWithDataDescriptor, skipSplitSig);
}
}
// PackageParser.java
...
try {
ArchiveStreamFactory factory = context.get(ArchiveStreamFactory.class, new ArchiveStreamFactory());
// At the end we want to close the archive stream to release
// any associated resources, but the underlying document stream
// should not be closed
CloseShieldInputStream csis = new CloseShieldInputStream(stream);
Object zipArchiveInputStreamFactory = context.get(ZipArchiveInputStreamFactory);
if (ArchiveStreamFactory.detect(csis).equalsIgnoreCase("zip") && zipArchiveInputStreamFactory != null) {
ais = zipArchiveInputStreamFactory.createZipArchiveInputStream(csis);
} else {
ais = factory.createArchiveInputStream(csis);
}
}
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] PeterAlfredLee commented on a change in pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
PeterAlfredLee commented on a change in pull request #356:
URL: https://github.com/apache/tika/pull/356#discussion_r493998557
##########
File path: tika-parser-modules/tika-parser-pkg-module/src/main/java/org/apache/tika/parser/pkg/PackageParser.java
##########
@@ -287,6 +284,35 @@ public void parse(
xhtml.endDocument();
}
+ private ZipArchiveInputStream updateZipArchiveInputStream(ArchiveInputStream ais, Metadata metadata, InputStream stream, String entryEncoding) throws IOException {
+ boolean allowStoredEntriesWithDataDescriptor = false;
+ ZipArchiveInputStream zais;
+ try {
+ readEntries(metadata, ais, null, null, false);
+ } catch (UnsupportedZipFeatureException zfe) {
+ if (zfe.getFeature() == Feature.DATA_DESCRIPTOR) {
+ allowStoredEntriesWithDataDescriptor = true;
+ }
+ } catch (Exception ignored) {}
+ finally {
+ ais.close();
+ stream.reset();
Review comment:
A `mark(int limit)` should always be called before a `reset()` is called - cause we need to mark the position we want to reset to.
This is not throwing an exception because a `mark(int limit)` was called in `DefaultZipContainerDetector.detect` before this. Besides this I think there're some problems in `TikaInputStream.reset` and some other places where `mark` is not called before `reset`.
I have already finished a fix and would push a PR separatly soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] PeterAlfredLee edited a comment on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
PeterAlfredLee edited a comment on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-696618151
> Do we have to reset the stream before reprocessing?
+1. The stream should be `reset` or `relocation to the beginning of the file`.
I think this is complicated here, cause the stream can may be not a `seekable` or `resetable` stream. Seems we got to write the stream to a temp file, which may cause some other problems cause zip archives may be pretty huge. Maybe you have other better ideas? @tballison
> Can you create a unit test with a small document that shows that this works?
+1. I can help out with this. Will update when this is done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt edited a comment on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt edited a comment on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698621442
Actually an even simpler solution would be to create a ZipArchiveStreamFactory which extends ArchiveStreamFactory and simply overwrite the createArchiveInputStream function. That would make it so there wouldn't need to be any change to PackageParser at all - just a new class created in commons-compress
All the client would have to do is create that ZipArchiveStreamFactory, set the allowStoredEntriesWithDataDescriptor to true, and set that in the ParseContext and it should then use that to create the data descriptor enabled ZipArchiveInputStream.
What you think about that @tballison @PeterAlfredLee ?
` public class ZipArchiveStreamFactory extends ArchiveStreamFactory {
private boolean allowStoredEntriesWithDataDescriptor = false;
private boolean useUnicodeExtraFields = true;
private boolean skipSplitSig = false;
public void setAllowStoredEntriesWithDataDescriptor(boolean allowStoredEntriesWithDataDescriptor) {
this.allowStoredEntriesWithDataDescriptor = allowStoredEntriesWithDataDescriptor;
}
public void setUseUnicodeExtraFields(boolean useUnicodeExtraFields) {
this.useUnicodeExtraFields = useUnicodeExtraFields;
}
public void setSkipSplitSig(boolean skipSplitSig) {
this.skipSplitSig = skipSplitSig;
}
@Override
public ArchiveInputStream createArchiveInputStream(String archiverName, InputStream in, String actualEncoding) throws ArchiveException {
if (archiverName == null) {
throw new IllegalArgumentException("Archivername must not be null.");
}
if (in == null) {
throw new IllegalArgumentException("InputStream must not be null.");
}
if (ZIP.equalsIgnoreCase(archiverName)) {
Optional<String> maybeEncoding = Optional.of(actualEncoding);
return new ZipArchiveInputStream(in, maybeEncoding.orElse(ZipEncodingHelper.UTF8), useUnicodeExtraFields, allowStoredEntriesWithDataDescriptor, skipSplitSig);
} else {
return super.createArchiveInputStream(archiverName, in, actualEncoding);
}
}
}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-694533716
@PeterAlfredLee who needs to review this to get it approved and merged?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] PeterAlfredLee edited a comment on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
PeterAlfredLee edited a comment on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-696618151
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt commented on a change in pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt commented on a change in pull request #356:
URL: https://github.com/apache/tika/pull/356#discussion_r494523989
##########
File path: tika-parser-modules/tika-parser-pkg-module/src/main/java/org/apache/tika/parser/pkg/PackageParser.java
##########
@@ -287,6 +284,35 @@ public void parse(
xhtml.endDocument();
}
+ private ZipArchiveInputStream updateZipArchiveInputStream(ArchiveInputStream ais, Metadata metadata, InputStream stream, String entryEncoding) throws IOException {
+ boolean allowStoredEntriesWithDataDescriptor = false;
+ ZipArchiveInputStream zais;
+ try {
+ readEntries(metadata, ais, null, null, false);
+ } catch (UnsupportedZipFeatureException zfe) {
+ if (zfe.getFeature() == Feature.DATA_DESCRIPTOR) {
+ allowStoredEntriesWithDataDescriptor = true;
+ }
+ } catch (Exception ignored) {}
+ finally {
+ ais.close();
+ stream.reset();
Review comment:
That makes sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt commented on a change in pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt commented on a change in pull request #356:
URL: https://github.com/apache/tika/pull/356#discussion_r494523989
##########
File path: tika-parser-modules/tika-parser-pkg-module/src/main/java/org/apache/tika/parser/pkg/PackageParser.java
##########
@@ -287,6 +284,35 @@ public void parse(
xhtml.endDocument();
}
+ private ZipArchiveInputStream updateZipArchiveInputStream(ArchiveInputStream ais, Metadata metadata, InputStream stream, String entryEncoding) throws IOException {
+ boolean allowStoredEntriesWithDataDescriptor = false;
+ ZipArchiveInputStream zais;
+ try {
+ readEntries(metadata, ais, null, null, false);
+ } catch (UnsupportedZipFeatureException zfe) {
+ if (zfe.getFeature() == Feature.DATA_DESCRIPTOR) {
+ allowStoredEntriesWithDataDescriptor = true;
+ }
+ } catch (Exception ignored) {}
+ finally {
+ ais.close();
+ stream.reset();
Review comment:
That makes sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] PeterAlfredLee commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
PeterAlfredLee commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-696721537
I forged a zip archive in memory that uses STORED and Data Descriptor at the same time. This could be easily used as a test case :
```
@Test
public void testZipUsingStoredWithDataDescriptor() throws Exception {
ContentHandler handler = new BodyContentHandler();
Metadata metadata = new Metadata();
try (InputStream stream = forgeZipInputStream() ) {
AUTO_DETECT_PARSER.parse(stream, handler, metadata, recursingContext);
}
}
private InputStream forgeZipInputStream() throws IOException {
// generate a zip archive in memory
try (final ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
final ZipArchiveOutputStream zo = new ZipArchiveOutputStream(byteArrayOutputStream);){
final ZipArchiveEntry entryA = new ZipArchiveEntry("foo");
entryA.setMethod(ZipEntry.STORED);
entryA.setSize(4);
entryA.setCrc(0xb63cfbcdl);
zo.putArchiveEntry(entryA);
zo.write(new byte[] { 1, 2, 3, 4 });
zo.closeArchiveEntry();
zo.close();
final byte[] zipContent = byteArrayOutputStream.toByteArray();
final byte[] old = Arrays.copyOf(zipContent, zipContent.length);
final byte[] zipContentWithDataDescriptor = new byte[zipContent.length + 12];
System.arraycopy(zipContent, 0, zipContentWithDataDescriptor, 0, 37);
// modify the general purpose bit flag
zipContentWithDataDescriptor[6] = 8;
// copy the crc-32, compressed size and uncompressed size to the data descriptor
System.arraycopy(zipContent, 14, zipContentWithDataDescriptor, 37, 12);
// and copy the rest of the zip content
System.arraycopy(zipContent, 37, zipContentWithDataDescriptor, 49, zipContent.length - 37);
return new ByteArrayInputStream(zipContentWithDataDescriptor);
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tballison commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tballison commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-694890251
For those interested in this issue, make sure to check out the conversation here: https://github.com/apache/commons-compress/pull/137
for why we don't want to make this the default.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] PeterAlfredLee commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
PeterAlfredLee commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698075634
> In PackageParser if the ArchiveInputStream is a ZipArchiveInputStream, iterate through all the zip entries without trying to parse/read them. This will make it so we do not write any of the zip contents to the ContentHandler which will avoid writing any duplicate content to the ContentHandler if we end up having a data descriptor exception and attempt to scan again with the data descriptor support enabled.
-1. I don't like this idea. Iterating through the zip without any reading would not save more time than reading all entries - we would always extract and read the data of the entry(see also `ZipArchiveInputStream.drainCurrentEntryData` about this) even through you didn't call `ZipArchiveInputStream.read`.
Zip archives could be pretty huge. Iterating them twice may be a time consuming job. Comparing with this, finding out a way to reset the `ContentHandler` would be more easier.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698613886
I think the idea of having the client handle the exception and rerun the parse may actually be the better option at this point.
> > finding out a way to reset the ContentHandler would be more easier.
>
> We were thinking about adding a resettable content handler to Tika 2.0, but it is really tricky and adds some complexity. If you can find a solution, great, but I'd be happy enough throwing the exception and then putting it on the client code to rerun the parse with PKGParser configured to allow for data descriptors.
Having the client code handle the exception / re-running the parsing seems like the better option at this point. My initial thought is to create a ZipArchiveInputStreamFactory in commons-compress in which you can set values for the various ZipArchiveInputStream parameters, and to have methods to generate a ZipArchiveInputStream using those parameters.
Then when the data descriptor exception is thrown and caught, the client can create a ZipArchiveInputStreamFactory with the data descriptor feature enabled and set that in the ParseContext.
Then in PackageParser, when we go to create an ArchiveInputStream, if the archive type is zip and a ZipArchiveInputStreamFactory is defined in the ParseContext then use the ZipArchiveInputStreamFactory to generate the ZipArchiveInputStream. Otherwise use the ArchiveStreamFactory.
Something like this:
`
// in commons-compress
import java.io.InputStream;
import static org.apache.commons.compress.archivers.zip.ZipEncodingHelper.UTF8;
public class ZipArchiveInputStreamFactory {
private String encoding = UTF8;
private boolean allowStoredEntriesWithDataDescriptor = false;
private boolean useUnicodeExtraFields = true;
private boolean skipSplitSig = false;
public ZipArchiveInputStreamFactory(){}
public void setEncoding(String encoding) {
this.encoding = encoding;
}
public void setAllowStoredEntriesWithDataDescriptor(boolean allowStoredEntriesWithDataDescriptor) {
this.allowStoredEntriesWithDataDescriptor = allowStoredEntriesWithDataDescriptor;
}
public void setUseUnicodeExtraFields(boolean useUnicodeExtraFields) {
this.useUnicodeExtraFields = useUnicodeExtraFields;
}
public void setSkipSplitSig(boolean skipSplitSig) {
this.skipSplitSig = skipSplitSig;
}
public ZipArchiveInputStream createZipArchiveInputStream(InputStream stream) {
return new ZipArchiveInputStream(stream, encoding, useUnicodeExtraFields, allowStoredEntriesWithDataDescriptor, skipSplitSig);
}
}
// PackageParser.java
...
try {
ArchiveStreamFactory factory = context.get(ArchiveStreamFactory.class, new ArchiveStreamFactory());
// At the end we want to close the archive stream to release
// any associated resources, but the underlying document stream
// should not be closed
CloseShieldInputStream csis = new CloseShieldInputStream(stream);
Object zipArchiveInputStreamFactory = context.get(ZipArchiveInputStreamFactory);
if (ArchiveStreamFactory.detect(csis).equalsIgnoreCase("zip") && zipArchiveInputStreamFactory != null) {
ais = zipArchiveInputStreamFactory.createZipArchiveInputStream(csis);
} else {
ais = factory.createArchiveInputStream(csis);
}
}
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tballison edited a comment on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tballison edited a comment on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698525643
If we do go with the stream the full thing first to see if there are data descriptors option, we should enable users to configure the PKGParser to do that because of the hit in parsing time.
Yes, ideally, we'd be able to handle this all behind the scenes, but it is sounding complicated and there are tradeoffs we'll need to make.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt edited a comment on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt edited a comment on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698613886
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698517123
> > In PackageParser if the ArchiveInputStream is a ZipArchiveInputStream, iterate through all the zip entries without trying to parse/read them. This will make it so we do not write any of the zip contents to the ContentHandler which will avoid writing any duplicate content to the ContentHandler if we end up having a data descriptor exception and attempt to scan again with the data descriptor support enabled.
>
> -1. I don't like this idea. Iterating through the zip without any reading would not save more time than reading all entries - we would always extract and read the data of the entry(see also `ZipArchiveInputStream.drainCurrentEntryData` about this) even through you didn't call `ZipArchiveInputStream.read`.
>
> Zip archives could be pretty huge. Iterating them twice may be a time consuming job. Comparing with this, finding out a way to reset the `ContentHandler` would be more easier.
I agree. I have restored PackageParser to no longer perform the initial iteration through the zip and added a throwing and handling of TikaException when a zip entry cannot be read because of data descriptor being present. This makes it so we do not write to the content handler for this entry until we can actually read it.
I marked two of the tests as ignored for the moment until we can get the mark(int limit) and reset() to reset the stream to the right place so we can retry reading the zip starting from the entry with the data descriptor.
When I call mark before reset (before retrying with a new ZipArchiveInputStream) I get an exception with message "Unexpected record signature: 0X73696854"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt closed pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt closed pull request #356:
URL: https://github.com/apache/tika/pull/356
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt edited a comment on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt edited a comment on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-697478056
@PeterAlfredLee - I just put that zip generation code into the ZipParserTest but it's throwing a 'Truncated Zip File' exception when run.
org.apache.tika.exception.TikaException: TIKA-198: Illegal IOException from org.apache.tika.parser.pkg.PackageParser@3688eb5b
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:283)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:277)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:143)
at org.apache.tika.parser.pkg.ZipParserTest.testZipUsingStoredWithDataDescriptor(ZipParserTest.java:88)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Caused by: org.apache.tika.io.TaggedIOException: Truncated ZIP file
at org.apache.tika.io.TaggedInputStream.handleIOException(TaggedInputStream.java:133)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:106)
at org.apache.tika.io.IOUtils.read(IOUtils.java:935)
at org.apache.tika.detect.zip.DefaultZipContainerDetector.detect(DefaultZipContainerDetector.java:101)
at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:84)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:116)
at org.apache.tika.parser.DelegatingParser.parse(DelegatingParser.java:72)
at org.apache.tika.extractor.ParsingEmbeddedDocumentExtractor.parseEmbedded(ParsingEmbeddedDocumentExtractor.java:105)
at org.apache.tika.parser.pkg.PackageParser.parseEntry(PackageParser.java:351)
at org.apache.tika.parser.pkg.PackageParser.readEntries(PackageParser.java:303)
at org.apache.tika.parser.pkg.PackageParser.parse(PackageParser.java:285)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:277)
... 29 more
Caused by: org.apache.tika.io.TaggedIOException: Truncated ZIP file
at org.apache.tika.io.TaggedInputStream.handleIOException(TaggedInputStream.java:133)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:106)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:102)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:102)
... 39 more
Caused by: java.io.IOException: Truncated ZIP file
at org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.readStoredEntry(ZipArchiveInputStream.java:934)
at org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.readStored(ZipArchiveInputStream.java:544)
at org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:486)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:102)
... 41 more
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-697456940
I have added the resetting of the stream and added some unit tests. Though I will add in the zip generation code from @PeterAlfredLee in the tests instead of adding a zip archive to the project. In the meantime, here is the solution I have come up with (and have tested):
In PackageParser if the ArchiveInputStream is a ZipArchiveInputStream, iterate through all the zip entries without trying to parse/read them. This will make it so we do not write any of the zip contents to the ContentHandler which will avoid writing any duplicate content to the ContentHandler if we end up having a data descriptor exception and attempt to scan again with the data descriptor support enabled.
If in iterating through the entries there are entries that have data descriptors a data descriptor UnsupportedZipFeatureException will be thrown, which we catch and can then try reading/parsing the zip with a ZipArchiveInputStream with the data descriptor support enabled. If we get a PasswordRequiredException, or another UnsupportedZipFeatureException, that will be handled just like it was before.
If no exception is thrown, the ArchiveInputStream is closed, a new ZipArchiveInputStream is created (or else the zip doesn't get read), the stream is reset, and reading/parsing of entries happens like normal.
How does this solution sound? I know it's not great to have to iterate through a zip twice but it was the only way I could find so far of determining if the zip has entries with data descriptors. I initially wanted to do something like, read the stream like is done now, catch the data descriptor exception, reset the contents of the ContentHandler, reset the stream and try reading again with the data descriptor enabled ZipArchiveInputStream but there is no way to reset the ContentHandler. So this was the next best thing I could come up with so far.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tballison edited a comment on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tballison edited a comment on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698525643
If we do go with the stream the full thing first to see if there are data descriptors option, we should enable users to configure the PKGParser to do that because of the hit in parsing time.
Yes, ideally, we'd be able to handle this all behind the scenes, but it is sounding complicated and there are tradeoffs we'll need to make.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tballison commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tballison commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698525054
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tballison commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tballison commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698525054
>finding out a way to reset the ContentHandler would be more easier.
We were thinking about adding a resettable content handler to Tika 2.0, but it is really tricky and adds some complexity. If you can find a solution, great, but I'd be happy enough throwing the exception and then putting it on the client code to rerun the parse with PKGParser configured to allow for data descriptors.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-697478056
@PeterAlfredLee - I just put that zip generation code into the ZipParserTest but it's throwing a 'Truncated Zip File' exception when run.
`org.apache.tika.exception.TikaException: TIKA-198: Illegal IOException from org.apache.tika.parser.pkg.PackageParser@3688eb5b
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:283)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:277)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:143)
at org.apache.tika.parser.pkg.ZipParserTest.testZipUsingStoredWithDataDescriptor(ZipParserTest.java:88)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Caused by: org.apache.tika.io.TaggedIOException: Truncated ZIP file
at org.apache.tika.io.TaggedInputStream.handleIOException(TaggedInputStream.java:133)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:106)
at org.apache.tika.io.IOUtils.read(IOUtils.java:935)
at org.apache.tika.detect.zip.DefaultZipContainerDetector.detect(DefaultZipContainerDetector.java:101)
at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:84)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:116)
at org.apache.tika.parser.DelegatingParser.parse(DelegatingParser.java:72)
at org.apache.tika.extractor.ParsingEmbeddedDocumentExtractor.parseEmbedded(ParsingEmbeddedDocumentExtractor.java:105)
at org.apache.tika.parser.pkg.PackageParser.parseEntry(PackageParser.java:351)
at org.apache.tika.parser.pkg.PackageParser.readEntries(PackageParser.java:303)
at org.apache.tika.parser.pkg.PackageParser.parse(PackageParser.java:285)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:277)
... 29 more
Caused by: org.apache.tika.io.TaggedIOException: Truncated ZIP file
at org.apache.tika.io.TaggedInputStream.handleIOException(TaggedInputStream.java:133)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:106)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:102)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:102)
... 39 more
Caused by: java.io.IOException: Truncated ZIP file
at org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.readStoredEntry(ZipArchiveInputStream.java:934)
at org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.readStored(ZipArchiveInputStream.java:544)
at org.apache.commons.compress.archivers.zip.ZipArchiveInputStream.read(ZipArchiveInputStream.java:486)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
at org.apache.tika.io.ProxyInputStream.read(ProxyInputStream.java:102)
... 41 more`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] PeterAlfredLee commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
PeterAlfredLee commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-696618151
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tballison commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tballison commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698525643
If we do go with the stream the full thing first to see if there are data descriptors option, we should enable users to configure the PKGParser to do that because of the hit in parsing time.
Yes, ideally, we'd be able to handle this all behind the scenes, but it is sounding complicated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt edited a comment on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt edited a comment on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698613886
> > finding out a way to reset the ContentHandler would be more easier.
>
> We were thinking about adding a resettable content handler to Tika 2.0, but it is really tricky and adds some complexity. If you can find a solution, great, but I'd be happy enough throwing the exception and then putting it on the client code to rerun the parse with PKGParser configured to allow for data descriptors.
Having the client code handle the exception / re-running the parsing seems like the better option at this point. My initial thought is to create a ZipArchiveInputStreamFactory in commons-compress in which you can set values for the various ZipArchiveInputStream parameters, and to have methods to generate a ZipArchiveInputStream using those parameters.
Then when the data descriptor exception is thrown and caught, the client can create a ZipArchiveInputStreamFactory with the data descriptor feature enabled and set that in the ParseContext.
Then in PackageParser, when we go to create an ArchiveInputStream, if the archive type is zip and a ZipArchiveInputStreamFactory is defined in the ParseContext then use the ZipArchiveInputStreamFactory to generate the ZipArchiveInputStream. Otherwise use the ArchiveStreamFactory.
Something like this:
`
// in commons-compress
import java.io.InputStream;
import static org.apache.commons.compress.archivers.zip.ZipEncodingHelper.UTF8;
public class ZipArchiveInputStreamFactory {
private String encoding = UTF8;
private boolean allowStoredEntriesWithDataDescriptor = false;
private boolean useUnicodeExtraFields = true;
private boolean skipSplitSig = false;
public ZipArchiveInputStreamFactory(){}
public void setEncoding(String encoding) {
this.encoding = encoding;
}
public void setAllowStoredEntriesWithDataDescriptor(boolean allowStoredEntriesWithDataDescriptor) {
this.allowStoredEntriesWithDataDescriptor = allowStoredEntriesWithDataDescriptor;
}
public void setUseUnicodeExtraFields(boolean useUnicodeExtraFields) {
this.useUnicodeExtraFields = useUnicodeExtraFields;
}
public void setSkipSplitSig(boolean skipSplitSig) {
this.skipSplitSig = skipSplitSig;
}
public ZipArchiveInputStream createZipArchiveInputStream(InputStream stream) {
return new ZipArchiveInputStream(stream, encoding, useUnicodeExtraFields, allowStoredEntriesWithDataDescriptor, skipSplitSig);
}
}
// PackageParser.java
...
try {
ArchiveStreamFactory factory = context.get(ArchiveStreamFactory.class, new ArchiveStreamFactory());
// At the end we want to close the archive stream to release
// any associated resources, but the underlying document stream
// should not be closed
CloseShieldInputStream csis = new CloseShieldInputStream(stream);
Object zipArchiveInputStreamFactory = context.get(ZipArchiveInputStreamFactory);
if (ArchiveStreamFactory.detect(csis).equalsIgnoreCase("zip") && zipArchiveInputStreamFactory != null) {
ais = zipArchiveInputStreamFactory.createZipArchiveInputStream(csis);
} else {
ais = factory.createArchiveInputStream(csis);
}
}
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698621442
Actually an even simpler solution would be to create a ZipArchiveStreamFactory which extends ArchiveStreamFactory and simply overwrite the createArchiveInputStream function. That would make it so there wouldn't need to be any change to PackageParser at all.
All the client would have to do is create that ZipArchiveStreamFactory, set the allowStoredEntriesWithDataDescriptor to true, and set that in the ParseContext and it should then use that to create the data descriptor enabled ZipArchiveInputStream.
What you think about that @tballison @PeterAlfredLee ?
`
public class ZipArchiveStreamFactory extends ArchiveStreamFactory {
private boolean allowStoredEntriesWithDataDescriptor = false;
private boolean useUnicodeExtraFields = true;
private boolean skipSplitSig = false;
public void setAllowStoredEntriesWithDataDescriptor(boolean allowStoredEntriesWithDataDescriptor) {
this.allowStoredEntriesWithDataDescriptor = allowStoredEntriesWithDataDescriptor;
}
public void setUseUnicodeExtraFields(boolean useUnicodeExtraFields) {
this.useUnicodeExtraFields = useUnicodeExtraFields;
}
public void setSkipSplitSig(boolean skipSplitSig) {
this.skipSplitSig = skipSplitSig;
}
@Override
public ArchiveInputStream createArchiveInputStream(String archiverName, InputStream in, String actualEncoding) throws ArchiveException {
if (archiverName == null) {
throw new IllegalArgumentException("Archivername must not be null.");
}
if (in == null) {
throw new IllegalArgumentException("InputStream must not be null.");
}
if (ZIP.equalsIgnoreCase(archiverName)) {
Optional<String> maybeEncoding = Optional.of(actualEncoding);
return new ZipArchiveInputStream(in, maybeEncoding.orElse(ZipEncodingHelper.UTF8), useUnicodeExtraFields, allowStoredEntriesWithDataDescriptor, skipSplitSig);
} else {
return super.createArchiveInputStream(archiverName, in, actualEncoding);
}
}
}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tbentleypfpt commented on pull request #356: Attempt to read zips with STORED data descriptors
Posted by GitBox <gi...@apache.org>.
tbentleypfpt commented on pull request #356:
URL: https://github.com/apache/tika/pull/356#issuecomment-698517123
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org