You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tika.apache.org by ta...@apache.org on 2021/04/28 16:04:57 UTC
[tika] branch branch_1x updated (f7d5119 -> f414fe4)
This is an automated email from the ASF dual-hosted git repository.
tallison pushed a change to branch branch_1x
in repository https://gitbox.apache.org/repos/asf/tika.git.
from f7d5119 TIKA-3373 Add the *.yml extension for YAML, which is commonly used, along with aliases for popular alternate mimetypes for it
new cd191ff update changes for new development in 1.27...please may it never happen... 2.0.0 here we come!
new 34fb775 Merge remote-tracking branch 'origin/branch_1x' into branch_1x
new 6f2f373 Merge remote-tracking branch 'origin/branch_1x' into branch_1x
new bc551b1 TIKA-3372 -- fix write limit in recursive parser handler
new f414fe4 Merge remote-tracking branch 'origin/branch_1x' into branch_1x
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGES.txt | 1 -
.../src/main/java/org/apache/tika/cli/TikaCLI.java | 2 +-
.../batch/fs/RecursiveParserWrapperFSConsumer.java | 2 +-
.../apache/tika/parser/RecursiveParserWrapper.java | 15 +++++++------
.../sax/AbstractRecursiveParserWrapperHandler.java | 11 ++++++++--
.../tika/sax/RecursiveParserWrapperHandler.java | 9 ++++----
.../tika/parser/RecursiveParserWrapperTest.java | 2 +-
.../server/resource/RecursiveMetadataResource.java | 4 ++--
.../tika/server/RecursiveMetadataResourceTest.java | 25 ++++++++++++++++++----
.../org/apache/tika/server/TikaResourceTest.java | 13 +++++++++++
10 files changed, 61 insertions(+), 23 deletions(-)
[tika] 05/05: Merge remote-tracking branch 'origin/branch_1x' into
branch_1x
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tallison pushed a commit to branch branch_1x
in repository https://gitbox.apache.org/repos/asf/tika.git
commit f414fe48bf688a4714dd5f11b37d2cee0c5f2b08
Merge: bc551b1 f7d5119
Author: tballison <ta...@apache.org>
AuthorDate: Wed Apr 28 12:04:43 2021 -0400
Merge remote-tracking branch 'origin/branch_1x' into branch_1x
.../src/main/resources/org/apache/tika/mime/tika-mimetypes.xml | 7 +++++++
1 file changed, 7 insertions(+)
[tika] 01/05: update changes for new development in 1.27...please
may it never happen... 2.0.0 here we come!
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tallison pushed a commit to branch branch_1x
in repository https://gitbox.apache.org/repos/asf/tika.git
commit cd191ff25f003c2a44e4b4f25b716b254d670706
Author: tballison <ta...@apache.org>
AuthorDate: Tue Apr 13 12:04:30 2021 -0400
update changes for new development in 1.27...please may it never happen... 2.0.0 here we come!
---
CHANGES.txt | 3 +++
1 file changed, 3 insertions(+)
diff --git a/CHANGES.txt b/CHANGES.txt
index d3894c5..a918b4d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,3 +1,6 @@
+Release 1.27 - ???
+
+
Release 1.26 - 03/24/2021
* Fix thread safety bug in OpenOffice parser (TIKA-3334).
[tika] 02/05: Merge remote-tracking branch 'origin/branch_1x' into
branch_1x
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tallison pushed a commit to branch branch_1x
in repository https://gitbox.apache.org/repos/asf/tika.git
commit 34fb7753bbcf97c757d9bba81e9cdd42afe9d1f6
Merge: cd191ff f51a905
Author: tballison <ta...@apache.org>
AuthorDate: Tue Apr 13 12:05:03 2021 -0400
Merge remote-tracking branch 'origin/branch_1x' into branch_1x
# Conflicts:
# CHANGES.txt
CHANGES.txt | 2 +
tika-nlp/pom.xml | 2 +-
.../java/org/apache/tika/parser/pdf/PDFParser.java | 2 +-
.../apache/tika/server/resource/TikaResource.java | 49 ++++++++++++++-----
.../org/apache/tika/server/TikaResourceTest.java | 55 +++++++++++++++++++---
5 files changed, 88 insertions(+), 22 deletions(-)
diff --cc CHANGES.txt
index a918b4d,4c3464b..9735a8a
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,6 -1,9 +1,8 @@@
Release 1.27 - ???
+ * Tika's PDFParser should use the underlying file if one is passed in
+ via a TikaInputStream (TIKA-3350)
-
Release 1.26 - 03/24/2021
* Fix thread safety bug in OpenOffice parser (TIKA-3334).
[tika] 04/05: TIKA-3372 -- fix write limit in recursive parser
handler
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tallison pushed a commit to branch branch_1x
in repository https://gitbox.apache.org/repos/asf/tika.git
commit bc551b18da2fb540a360e3300660490a046b457b
Author: tballison <ta...@apache.org>
AuthorDate: Wed Apr 28 12:04:33 2021 -0400
TIKA-3372 -- fix write limit in recursive parser handler
---
.../src/main/java/org/apache/tika/cli/TikaCLI.java | 2 +-
.../batch/fs/RecursiveParserWrapperFSConsumer.java | 2 +-
.../apache/tika/parser/RecursiveParserWrapper.java | 15 +++++++------
.../sax/AbstractRecursiveParserWrapperHandler.java | 11 ++++++++--
.../tika/sax/RecursiveParserWrapperHandler.java | 9 ++++----
.../tika/parser/RecursiveParserWrapperTest.java | 2 +-
.../server/resource/RecursiveMetadataResource.java | 4 ++--
.../tika/server/RecursiveMetadataResourceTest.java | 25 ++++++++++++++++++----
.../org/apache/tika/server/TikaResourceTest.java | 13 +++++++++++
9 files changed, 61 insertions(+), 22 deletions(-)
diff --git a/tika-app/src/main/java/org/apache/tika/cli/TikaCLI.java b/tika-app/src/main/java/org/apache/tika/cli/TikaCLI.java
index 4b6c164..b115ef2 100644
--- a/tika-app/src/main/java/org/apache/tika/cli/TikaCLI.java
+++ b/tika-app/src/main/java/org/apache/tika/cli/TikaCLI.java
@@ -526,7 +526,7 @@ public class TikaCLI {
RecursiveParserWrapper wrapper = new RecursiveParserWrapper(parser);
RecursiveParserWrapperHandler handler =
new RecursiveParserWrapperHandler(getContentHandlerFactory(type),
- -1, config.getMetadataFilter());
+ -1, -1, config.getMetadataFilter());
try (InputStream input = TikaInputStream.get(url, metadata)) {
wrapper.parse(input, handler, metadata, context);
}
diff --git a/tika-batch/src/main/java/org/apache/tika/batch/fs/RecursiveParserWrapperFSConsumer.java b/tika-batch/src/main/java/org/apache/tika/batch/fs/RecursiveParserWrapperFSConsumer.java
index 9732781..aa8af8f 100644
--- a/tika-batch/src/main/java/org/apache/tika/batch/fs/RecursiveParserWrapperFSConsumer.java
+++ b/tika-batch/src/main/java/org/apache/tika/batch/fs/RecursiveParserWrapperFSConsumer.java
@@ -100,7 +100,7 @@ public class RecursiveParserWrapperFSConsumer extends AbstractFSConsumer {
List<Metadata> metadataList = null;
Metadata containerMetadata = fileResource.getMetadata();
RecursiveParserWrapperHandler handler = new RecursiveParserWrapperHandler(contentHandlerFactory,
- -1, metadataFilter);
+ -1, -1, metadataFilter);
try {
parse(fileResource.getResourceId(), parser, is, handler,
containerMetadata, context);
diff --git a/tika-core/src/main/java/org/apache/tika/parser/RecursiveParserWrapper.java b/tika-core/src/main/java/org/apache/tika/parser/RecursiveParserWrapper.java
index 021fca3..95899a6 100644
--- a/tika-core/src/main/java/org/apache/tika/parser/RecursiveParserWrapper.java
+++ b/tika-core/src/main/java/org/apache/tika/parser/RecursiveParserWrapper.java
@@ -227,17 +227,15 @@ public class RecursiveParserWrapper extends ParserDecorator {
long started = System.currentTimeMillis();
parserState.recursiveParserWrapperHandler.startDocument();
TemporaryResources tmp = new TemporaryResources();
- int writeLimit = -1;
- //TODO -- rely on a new interface WriteLimiting...?
- //It'd be better not to tie this to a specific class
- if (recursiveParserWrapperHandler instanceof BasicContentHandlerFactory) {
- writeLimit =
- ((BasicContentHandlerFactory)recursiveParserWrapperHandler).getWriteLimit();
+ int totalWriteLimit = -1;
+ if (recursiveParserWrapperHandler instanceof AbstractRecursiveParserWrapperHandler) {
+ totalWriteLimit =
+ ((AbstractRecursiveParserWrapperHandler)recursiveParserWrapperHandler).getTotalWriteLimit();
}
try {
TikaInputStream tis = TikaInputStream.get(stream, tmp);
RecursivelySecureContentHandler secureContentHandler =
- new RecursivelySecureContentHandler(localHandler, tis, writeLimit);
+ new RecursivelySecureContentHandler(localHandler, tis, totalWriteLimit);
context.set(RecursivelySecureContentHandler.class, secureContentHandler);
getWrappedParser().parse(tis, secureContentHandler, metadata, context);
} catch (SAXException e) {
@@ -493,6 +491,7 @@ public class RecursiveParserWrapper extends ParserDecorator {
}
int availableLength = Math.min(totalWriteLimit - totalChars, length);
super.characters(ch, start, availableLength);
+ totalChars += availableLength;
if (availableLength < length) {
throw new WriteLimitReached();
}
@@ -506,9 +505,11 @@ public class RecursiveParserWrapper extends ParserDecorator {
}
int availableLength = Math.min(totalWriteLimit - totalChars, length);
super.ignorableWhitespace(ch, start, availableLength);
+
if (availableLength < length) {
throw new WriteLimitReached();
}
+ totalChars += availableLength;
}
}
diff --git a/tika-core/src/main/java/org/apache/tika/sax/AbstractRecursiveParserWrapperHandler.java b/tika-core/src/main/java/org/apache/tika/sax/AbstractRecursiveParserWrapperHandler.java
index 8515f09..50c1eca 100644
--- a/tika-core/src/main/java/org/apache/tika/sax/AbstractRecursiveParserWrapperHandler.java
+++ b/tika-core/src/main/java/org/apache/tika/sax/AbstractRecursiveParserWrapperHandler.java
@@ -63,16 +63,19 @@ public abstract class AbstractRecursiveParserWrapperHandler extends DefaultHandl
private static final int MAX_DEPTH = 100;
private final int maxEmbeddedResources;
+ private final int totalWriteLimit;
private int embeddedResources = 0;
private int embeddedDepth = 0;
public AbstractRecursiveParserWrapperHandler(ContentHandlerFactory contentHandlerFactory) {
- this(contentHandlerFactory, -1);
+ this(contentHandlerFactory, -1, -1);
}
- public AbstractRecursiveParserWrapperHandler(ContentHandlerFactory contentHandlerFactory, int maxEmbeddedResources) {
+ public AbstractRecursiveParserWrapperHandler(ContentHandlerFactory contentHandlerFactory,
+ int maxEmbeddedResources, int totalWriteLimit) {
this.contentHandlerFactory = contentHandlerFactory;
this.maxEmbeddedResources = maxEmbeddedResources;
+ this.totalWriteLimit = totalWriteLimit;
}
public ContentHandler getNewContentHandler() {
@@ -142,4 +145,8 @@ public abstract class AbstractRecursiveParserWrapperHandler extends DefaultHandl
public ContentHandlerFactory getContentHandlerFactory() {
return contentHandlerFactory;
}
+
+ public int getTotalWriteLimit() {
+ return totalWriteLimit;
+ }
}
diff --git a/tika-core/src/main/java/org/apache/tika/sax/RecursiveParserWrapperHandler.java b/tika-core/src/main/java/org/apache/tika/sax/RecursiveParserWrapperHandler.java
index 50f0fb8..a0d620e 100644
--- a/tika-core/src/main/java/org/apache/tika/sax/RecursiveParserWrapperHandler.java
+++ b/tika-core/src/main/java/org/apache/tika/sax/RecursiveParserWrapperHandler.java
@@ -49,7 +49,7 @@ public class RecursiveParserWrapperHandler extends AbstractRecursiveParserWrappe
* Create a handler with no limit on the number of embedded resources
*/
public RecursiveParserWrapperHandler(ContentHandlerFactory contentHandlerFactory) {
- this(contentHandlerFactory, -1, NoOpFilter.NOOP_FILTER);
+ this(contentHandlerFactory, -1, -1, NoOpFilter.NOOP_FILTER);
}
/**
@@ -58,12 +58,13 @@ public class RecursiveParserWrapperHandler extends AbstractRecursiveParserWrappe
* @param maxEmbeddedResources number of embedded resources that will be parsed
*/
public RecursiveParserWrapperHandler(ContentHandlerFactory contentHandlerFactory, int maxEmbeddedResources) {
- this(contentHandlerFactory, maxEmbeddedResources, NoOpFilter.NOOP_FILTER);
+ this(contentHandlerFactory, maxEmbeddedResources, -1, NoOpFilter.NOOP_FILTER);
}
- public RecursiveParserWrapperHandler(ContentHandlerFactory contentHandlerFactory, int maxEmbeddedResources,
+ public RecursiveParserWrapperHandler(ContentHandlerFactory contentHandlerFactory,
+ int maxEmbeddedResources, int maxWriteLimit,
MetadataFilter metadataFilter) {
- super(contentHandlerFactory, maxEmbeddedResources);
+ super(contentHandlerFactory, maxEmbeddedResources, maxWriteLimit);
this.metadataFilter = metadataFilter;
}
diff --git a/tika-parsers/src/test/java/org/apache/tika/parser/RecursiveParserWrapperTest.java b/tika-parsers/src/test/java/org/apache/tika/parser/RecursiveParserWrapperTest.java
index 349f271..e9b74fc 100644
--- a/tika-parsers/src/test/java/org/apache/tika/parser/RecursiveParserWrapperTest.java
+++ b/tika-parsers/src/test/java/org/apache/tika/parser/RecursiveParserWrapperTest.java
@@ -381,7 +381,7 @@ public class RecursiveParserWrapperTest extends TikaTest {
-1);
RecursiveParserWrapperHandler handler = new RecursiveParserWrapperHandler(contentHandlerFactory,
- -1, tikaConfig.getMetadataFilter());
+ -1, -1, tikaConfig.getMetadataFilter());
try (InputStream is = getClass().getResourceAsStream(path)) {
wrapper.parse(is, handler, metadata, context);
}
diff --git a/tika-server/src/main/java/org/apache/tika/server/resource/RecursiveMetadataResource.java b/tika-server/src/main/java/org/apache/tika/server/resource/RecursiveMetadataResource.java
index ec37779..664f776 100644
--- a/tika-server/src/main/java/org/apache/tika/server/resource/RecursiveMetadataResource.java
+++ b/tika-server/src/main/java/org/apache/tika/server/resource/RecursiveMetadataResource.java
@@ -146,13 +146,13 @@ public class RecursiveMetadataResource {
int maxEmbeddedResources = -1;
if (httpHeaders.containsKey("maxEmbeddedResources")) {
- maxEmbeddedResources = Integer.parseInt(httpHeaders.getFirst("maxEmbeddedResources"));
+ maxEmbeddedResources = Integer.parseInt(httpHeaders.getFirst("maxEmbeddedResources"));
}
BasicContentHandlerFactory.HANDLER_TYPE type =
BasicContentHandlerFactory.parseHandlerType(handlerTypeName, DEFAULT_HANDLER_TYPE);
RecursiveParserWrapperHandler handler = new RecursiveParserWrapperHandler(
- new BasicContentHandlerFactory(type, writeLimit), maxEmbeddedResources,
+ new BasicContentHandlerFactory(type, writeLimit), maxEmbeddedResources, writeLimit,
TikaResource.getConfig().getMetadataFilter());
try {
TikaResource.parse(wrapper, LOG, info.getPath(), is, handler, metadata, context);
diff --git a/tika-server/src/test/java/org/apache/tika/server/RecursiveMetadataResourceTest.java b/tika-server/src/test/java/org/apache/tika/server/RecursiveMetadataResourceTest.java
index d0c84c7..6a70d34 100644
--- a/tika-server/src/test/java/org/apache/tika/server/RecursiveMetadataResourceTest.java
+++ b/tika-server/src/test/java/org/apache/tika/server/RecursiveMetadataResourceTest.java
@@ -18,7 +18,6 @@
package org.apache.tika.server;
import static java.nio.charset.StandardCharsets.UTF_8;
-import static org.apache.tika.TikaTest.assertNotContained;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;
@@ -29,6 +28,7 @@ import javax.ws.rs.core.Response;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
+import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
@@ -342,6 +342,22 @@ public class RecursiveMetadataResourceTest extends CXFTestBase {
}
@Test
+ public void testJsonWriteLimitEmbedded() throws Exception {
+ for (int i = 500; i < 8500; i += 500) {
+ Response response = WebClient.create(endPoint + META_PATH + "/text").accept("application/json")
+ .header("writeLimit",
+ Integer.toString(i)).put(ClassLoader.getSystemResourceAsStream(TEST_RECURSIVE_DOC));
+ List<Metadata> metadata = JsonMetadataList.fromJson(
+ new InputStreamReader(((InputStream) response.getEntity()), StandardCharsets.UTF_8));
+ int len = 0;
+ for (Metadata m : metadata) {
+ len += m.get(AbstractRecursiveParserWrapperHandler.TIKA_CONTENT).length();
+ }
+ assertEquals(i, len);
+ }
+ }
+
+ @Test
public void testWriteLimit() throws Exception {
int writeLimit = 10;
Response response = WebClient.create(endPoint + META_PATH).accept("application/json")
@@ -356,7 +372,7 @@ public class RecursiveMetadataResourceTest extends CXFTestBase {
assertEquals("true", metadataList.get(0).get(AbstractRecursiveParserWrapperHandler.WRITE_LIMIT_REACHED));
//now try with a write limit of 1000
- writeLimit = 200;
+ writeLimit = 500;
response = WebClient.create(endPoint + META_PATH).accept("application/json")
.header("writeLimit", Integer.toString(writeLimit))
.put(ClassLoader.getSystemResourceAsStream(TEST_RECURSIVE_DOC));
@@ -365,9 +381,10 @@ public class RecursiveMetadataResourceTest extends CXFTestBase {
// Check results
reader = new InputStreamReader((InputStream) response.getEntity(), UTF_8);
metadataList = JsonMetadataList.fromJson(reader);
- assertEquals(12, metadataList.size());
+ assertEquals(10, metadataList.size());
+
assertEquals("true", metadataList.get(6).get(AbstractRecursiveParserWrapperHandler.WRITE_LIMIT_REACHED));
- assertContains("When in the Course of human events it becomes necessary for one people",
+ assertContains("When in the Course of human events it becomes necessary",
metadataList.get(6).get(AbstractRecursiveParserWrapperHandler.TIKA_CONTENT));
TikaTest.assertNotContained("We hold these truths",
metadataList.get(6).get(AbstractRecursiveParserWrapperHandler.TIKA_CONTENT));
diff --git a/tika-server/src/test/java/org/apache/tika/server/TikaResourceTest.java b/tika-server/src/test/java/org/apache/tika/server/TikaResourceTest.java
index 68d537d..7511821 100644
--- a/tika-server/src/test/java/org/apache/tika/server/TikaResourceTest.java
+++ b/tika-server/src/test/java/org/apache/tika/server/TikaResourceTest.java
@@ -626,6 +626,19 @@ public class TikaResourceTest extends CXFTestBase {
assertEquals(400, response.getStatus());
}
+ @Test
+ public void testJsonWriteLimitEmbedded() throws Exception {
+ for (int i = 0; i < 8500; i += 500) {
+ Response response = WebClient.create(endPoint + TIKA_PATH + "/text").accept("application/json")
+ .header("writeLimit",
+ Integer.toString(i)).put(ClassLoader.getSystemResourceAsStream(TEST_RECURSIVE_DOC));
+ Metadata metadata = JsonMetadata.fromJson(
+ new InputStreamReader(((InputStream) response.getEntity()), StandardCharsets.UTF_8));
+ int len = metadata.get(AbstractRecursiveParserWrapperHandler.TIKA_CONTENT).length();
+ assertEquals(i, len);
+ }
+ }
+
private MultipartBody testPDFLowerCaseOCRConfigPOSTBody() throws FileNotFoundException, URISyntaxException {
ContentDisposition cd = new ContentDisposition(
"form-data; name=\"input\"; filename=\"testOCR.pdf\"");
[tika] 03/05: Merge remote-tracking branch 'origin/branch_1x' into
branch_1x
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tallison pushed a commit to branch branch_1x
in repository https://gitbox.apache.org/repos/asf/tika.git
commit 6f2f373629a11978f7add66ad302ae27597828ed
Merge: 34fb775 64cdc8e
Author: tballison <ta...@apache.org>
AuthorDate: Mon Apr 26 14:10:14 2021 -0400
Merge remote-tracking branch 'origin/branch_1x' into branch_1x
CHANGES.txt | 2 +
NOTICE.txt | 24 ++
tika-core/pom.xml | 375 ++++++++++++---------
.../main/assembly/test-jar-with-dependencies.xml | 38 +++
.../org/apache/tika/parser/mock/MockParser.java | 163 ++++++---
.../apache/tika/parser/mock/MockParserTest.java | 30 ++
.../resources/test-documents/mock_fakeload.xml | 29 ++
tika-parent/pom.xml | 1 +
.../org/apache/tika/parser/pkg/PackageParser.java | 21 +-
tika-server/pom.xml | 53 +++
.../tika/server/ProduceTypeResourceComparator.java | 145 ++++++++
.../java/org/apache/tika/server/TikaServerCli.java | 49 ++-
.../org/apache/tika/server/mbean/MBeanHelper.java | 60 ++++
.../tika/server/mbean/ServerStatusExporter.java | 1 +
.../server/mbean/ServerStatusExporterMBean.java | 1 +
.../apache/tika/server/metrics/Log4JMetrics.java | 212 ++++++++++++
.../apache/tika/server/metrics/MetricsHelper.java | 220 ++++++++++++
.../tika/server/metrics/MetricsResource.java | 53 +++
.../tika/server/metrics/ServerStatusMetrics.java | 61 ++++
.../server/resource/RecursiveMetadataResource.java | 4 +-
.../apache/tika/server/resource/TikaResource.java | 95 +++++-
.../java/org/apache/tika/server/CXFTestBase.java | 20 +-
.../apache/tika/server/MetricsResourceTest.java | 118 +++++++
.../tika/server/RecursiveMetadataResourceTest.java | 17 +
.../org/apache/tika/server/StackTraceTest.java | 17 +-
.../server/TikaResourceMetadataFilterTest.java | 83 +++++
.../tika/server/TikaResourceNoStackTest.java | 83 +++++
.../org/apache/tika/server/TikaResourceTest.java | 141 +++++++-
.../resources/configs/metadata-filter-include.xml | 30 ++
.../src/test/resources/mock/hello_world.xml | 26 ++
.../src/test/resources/mock/hello_world_long.xml | 30 ++
31 files changed, 1943 insertions(+), 259 deletions(-)