You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tika.apache.org by ta...@apache.org on 2026/08/22 01:19:20 UTC
(tika) 01/01: TIKA-4814 -- OneNote dom-order extraction and follow-ups
This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch TIKA-4814-for-3x
in repository https://gitbox.apache.org/repos/asf/tika.git
commit 9519878af1fc27a1137d165eed563869546bf5f3
Author: tallison <ta...@apache.org>
AuthorDate: Fri Aug 21 21:19:02 2026 -0400
TIKA-4814 -- OneNote dom-order extraction and follow-ups
Backport to 3.x of a9abca092c plus the TIKA-4814-followup-1 work,
adapted to the 3.x APIs: onenote: metadata key strings instead of the
4.x OneNote metadata class, the 1-arg/4-arg EmbeddedDocumentExtractor
signatures, the 2-arg XHTMLContentHandler, and no TikaComponent or
EmbeddedLimitReachedException.
Co-authored-by: Henry Lindeman <he...@glean.com>
Co-authored-by: nddipiazza <nd...@users.noreply.github.com>
---
CHANGES.txt | 11 +
.../tika/parser/microsoft/onenote/FileNodePtr.java | 15 +-
.../apache/tika/parser/microsoft/onenote/GUID.java | 40 +-
.../parser/microsoft/onenote/OneNoteDocument.java | 3 +
.../parser/microsoft/onenote/OneNoteParser.java | 150 +++-
.../tika/parser/microsoft/onenote/OneNotePtr.java | 183 +++-
.../microsoft/onenote/OneNoteTreeWalker.java | 135 ++-
.../onenote/fsshttpb/MSOneStorePackage.java | 999 ++++++++++++++++++---
.../onenote/fsshttpb/MSOneStoreParser.java | 326 +++++--
.../property/PrtArrayOfPropertyValues.java | 13 +-
.../PrtFourBytesOfLengthFollowedByData.java | 6 +
.../onenote/fsshttpb/streamobj/DataElement.java | 3 +-
.../onenote/fsshttpb/streamobj/FileDataObject.java | 18 +-
.../onenote/fsshttpb/streamobj/ObjectDataBLOB.java | 73 ++
.../streamobj/ObjectDataBLOBDataElementData.java | 48 +
.../onenote/fsshttpb/streamobj/PropertySet.java | 23 +-
...{FileDataObject.java => RevisionStoreCell.java} | 21 +-
.../streamobj/RevisionStoreObjectGroup.java | 45 +-
.../onenote/fsshttpb/streamobj/StreamObject.java | 3 +-
.../onenote/fsshttpb/streamobj/basic/CellID.java | 8 +
.../onenote/fsshttpb/streamobj/basic/ExGuid.java | 9 +
.../space/ObjectSpaceObjectStreamOfContextIDs.java | 8 +-
.../space/ObjectSpaceObjectStreamOfOIDs.java | 8 +-
.../space/ObjectSpaceObjectStreamOfOSIDs.java | 6 +
.../tika/parser/microsoft/onenote/GUIDTest.java | 70 ++
.../microsoft/onenote/OneNoteParserTest.java | 337 ++++++-
.../onenote/fsshttpb/MSOneStoreBlobTest.java | 343 +++++++
.../onenote/fsshttpb/MSOneStorePackageTest.java | 846 +++++++++++++++++
.../onenote/fsshttpb/MSOneStoreParserTest.java | 286 ++++++
.../PropertyDeserializationBoundsTest.java | 184 ++++
.../resources/test-documents/testOneNote-fuzz1.one | Bin 0 -> 6448 bytes
.../resources/test-documents/testOneNote-fuzz2.one | Bin 0 -> 295501 bytes
.../resources/test-documents/testOneNote-fuzz3.one | Bin 0 -> 120720 bytes
.../test-documents/testOneNoteEmbeddedImage.one | Bin 0 -> 52223 bytes
34 files changed, 3918 insertions(+), 302 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 4d46e86962..64baa29f35 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -3,6 +3,17 @@ Release 3.3.3 - ???
* MagicDetector now compiles its regular expression once, in the
constructor, instead of recompiling it on every match (TIKA-4796).
+ * OneNote extraction now follows document order, omits superseded page
+ revisions, sorts author metadata, extracts embedded object BLOBs, and
+ bounds malformed-input recursion and file-derived allocations. Parse
+ warnings and embedded relationship IDs are exposed in metadata. Malformed
+ or truncated files that cannot be fully parsed, and files whose walk
+ yields no content, now fall back to the legacy string dump instead of
+ failing or returning empty output. The legacy MS-ONESTORE walker bounds
+ its recursion (depth caps plus file-node-list and fragment-chain cycle
+ guards) and now honors shouldParseEmbedded for embedded file data
+ (TIKA-4814).
+
* PDFs whose %PDF- header is preceded by a print-composition job ticket are
no longer detected as text/x-matlab. Up to 50 %% comment or blank lines of
up to 150 characters, and nothing else, may now precede the header; the
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/FileNodePtr.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/FileNodePtr.java
index 2646def38b..e82706a7ad 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/FileNodePtr.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/FileNodePtr.java
@@ -56,12 +56,21 @@ class FileNodePtr {
if (nodeListPositions.isEmpty()) {
return null;
}
- if (nodeListPositions.get(0) >= document.root.children.size()) {
+ int rootPosition = nodeListPositions.get(0);
+ if (rootPosition < 0 || rootPosition >= document.root.children.size()) {
throw new TikaException("Exceeded root child size");
}
- FileNode cur = document.root.children.get(nodeListPositions.get(0));
+ FileNode cur = document.root.children.get(rootPosition);
for (int i = 1, ie = nodeListPositions.size(); i < ie; ++i) {
- cur = cur.childFileNodeList.children.get(nodeListPositions.get(i));
+ if (cur == null || cur.childFileNodeList == null ||
+ cur.childFileNodeList.children == null) {
+ throw new TikaException("Missing child file node list");
+ }
+ int childPosition = nodeListPositions.get(i);
+ if (childPosition < 0 || childPosition >= cur.childFileNodeList.children.size()) {
+ throw new TikaException("Exceeded child file node size");
+ }
+ cur = cur.childFileNodeList.children.get(childPosition);
}
return cur;
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/GUID.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/GUID.java
index 670824eec7..55e788bb4a 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/GUID.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/GUID.java
@@ -23,6 +23,7 @@ import java.util.Arrays;
import java.util.List;
import java.util.Locale;
+import org.apache.tika.exception.TikaException;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.util.BitConverter;
import org.apache.tika.utils.StringUtils;
@@ -41,17 +42,44 @@ public class GUID implements Comparable<GUID> {
* -EEEEEEEEEEEE}
* @return GUID object parsed from guid bytes.
*/
- public static GUID fromCurlyBraceUTF16Bytes(byte[] guid) {
+ public static GUID fromCurlyBraceUTF16Bytes(byte[] guid) throws TikaException {
int[] intGuid = new int[16];
- String utf16Str = new String(guid, StandardCharsets.UTF_16LE).replaceAll("\\{", "")
- .replaceAll("-", "").replaceAll("}", "");
- for (int i = 0; i < utf16Str.length(); i += 2) {
- intGuid[i / 2] =
- Integer.parseUnsignedInt("" + utf16Str.charAt(i) + utf16Str.charAt(i + 1), 16);
+ String utf16Str = new String(guid, StandardCharsets.UTF_16LE);
+ if (utf16Str.length() != 38 || utf16Str.charAt(0) != '{' ||
+ utf16Str.charAt(9) != '-' || utf16Str.charAt(14) != '-' ||
+ utf16Str.charAt(19) != '-' || utf16Str.charAt(24) != '-' ||
+ utf16Str.charAt(37) != '}') {
+ throw new TikaException("Invalid GUID string");
+ }
+ String hex = utf16Str.substring(1, 37).replace("-", "");
+ if (hex.length() != 32) {
+ throw new TikaException("Invalid GUID string");
+ }
+ for (int i = 0; i < hex.length(); i += 2) {
+ int high = asciiHexDigit(hex.charAt(i));
+ int low = asciiHexDigit(hex.charAt(i + 1));
+ if (high < 0 || low < 0) {
+ throw new TikaException("Invalid GUID string");
+ }
+ intGuid[i / 2] = (high << 4) | low;
}
return new GUID(intGuid);
}
+ // Character.digit accepts non-ASCII Unicode digits; GUIDs are ASCII hex only
+ private static int asciiHexDigit(char c) {
+ if (c >= '0' && c <= '9') {
+ return c - '0';
+ }
+ if (c >= 'a' && c <= 'f') {
+ return c - 'a' + 10;
+ }
+ if (c >= 'A' && c <= 'F') {
+ return c - 'A' + 10;
+ }
+ return -1;
+ }
+
public static int memcmp(int[] b1, int[] b2, int sz) {
for (int i = 0; i < sz; i++) {
if (b1[i] != b2[i]) {
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteDocument.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteDocument.java
index 936228f25c..cd76a58a55 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteDocument.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteDocument.java
@@ -34,6 +34,9 @@ class OneNoteDocument {
Map<ExtendedGUID, Pair<Long, ExtendedGUID>> revisionRoleMap = new HashMap<>();
ExtendedGUID currentRevision = ExtendedGUID.nil();
FileNodeList root = new FileNodeList();
+ // set when the root file node list could not be fully parsed (e.g. a truncated file);
+ // the header and any structure parsed so far remain usable
+ Exception structureParseException;
public OneNoteDocument() {
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteParser.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteParser.java
index d4a265caee..f7e03378c0 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteParser.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteParser.java
@@ -19,6 +19,7 @@ package org.apache.tika.parser.microsoft.onenote;
import java.io.IOException;
import java.io.InputStream;
import java.time.Instant;
+import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
@@ -27,10 +28,13 @@ import java.util.Set;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.tuple.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import org.xml.sax.ContentHandler;
import org.xml.sax.SAXException;
import org.apache.tika.exception.TikaException;
+import org.apache.tika.exception.WriteLimitReachedException;
import org.apache.tika.io.TemporaryResources;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
@@ -52,6 +56,7 @@ import org.apache.tika.sax.XHTMLContentHandler;
public class OneNoteParser implements Parser {
public static final String ONE_NOTE_PREFIX = "onenote:";
+ private static final Logger LOG = LoggerFactory.getLogger(OneNoteParser.class);
private static final Map<MediaType, List<String>> TYPES_MAP = new HashMap<>();
/**
* Serial version UID
@@ -126,53 +131,76 @@ public class OneNoteParser implements Parser {
metadata.set(ONE_NOTE_PREFIX + "rgbPlaceholder",
"0x" + Long.toHexString(oneNoteDocument.header.rgbPlaceholder));
- Pair<Long, ExtendedGUID> roleAndContext = Pair.of(1L, ExtendedGUID.nil());
- OneNoteTreeWalker oneNoteTreeWalker =
- new OneNoteTreeWalker(options, oneNoteDocument, oneNoteDirectFileResource,
- xhtml, metadata, context, roleAndContext);
+ Exception structureFailure = oneNoteDocument.structureParseException;
+ boolean walked = false;
+ if (structureFailure == null) {
+ try {
+ Pair<Long, ExtendedGUID> roleAndContext = Pair.of(1L, ExtendedGUID.nil());
+ OneNoteTreeWalker oneNoteTreeWalker =
+ new OneNoteTreeWalker(options, oneNoteDocument,
+ oneNoteDirectFileResource, xhtml, metadata, context,
+ roleAndContext);
- oneNoteTreeWalker.walkTree();
+ oneNoteTreeWalker.walkTree();
- if (!oneNoteTreeWalker.getAuthors().isEmpty()) {
- metadata.set(TikaCoreProperties.CREATOR,
- oneNoteTreeWalker.getAuthors().toArray(new String[]{}));
+ if (!oneNoteTreeWalker.getAuthors().isEmpty()) {
+ metadata.set(TikaCoreProperties.CREATOR,
+ sortedValues(oneNoteTreeWalker.getAuthors()));
+ }
+ if (!oneNoteTreeWalker.getMostRecentAuthors().isEmpty()) {
+ metadata.set(
+ Property.externalTextBag(ONE_NOTE_PREFIX + "mostRecentAuthors"),
+ sortedValues(oneNoteTreeWalker.getMostRecentAuthors()));
+ }
+ if (!oneNoteTreeWalker.getOriginalAuthors().isEmpty()) {
+ metadata.set(
+ Property.externalTextBag(ONE_NOTE_PREFIX + "originalAuthors"),
+ sortedValues(oneNoteTreeWalker.getOriginalAuthors()));
+ }
+ if (!Instant.MAX.equals(
+ Instant.ofEpochMilli(oneNoteTreeWalker.getCreationTimestamp()))) {
+ metadata.set(ONE_NOTE_PREFIX + "creationTimestamp",
+ String.valueOf(oneNoteTreeWalker.getCreationTimestamp()));
+ }
+ if (!Instant.MIN.equals(oneNoteTreeWalker.getLastModifiedTimestamp())) {
+ metadata.set(ONE_NOTE_PREFIX + "lastModifiedTimestamp", String.valueOf(
+ oneNoteTreeWalker.getLastModifiedTimestamp().toEpochMilli()));
+ }
+ if (oneNoteTreeWalker.getLastModified() > Long.MIN_VALUE) {
+ metadata.set(TikaCoreProperties.MODIFIED,
+ String.valueOf(oneNoteTreeWalker.getLastModified()));
+ }
+ walked = true;
+ } catch (Exception e) {
+ rethrowIfLimitReached(e);
+ structureFailure = e;
+ }
}
- if (!oneNoteTreeWalker.getMostRecentAuthors().isEmpty()) {
- metadata.set(Property.externalTextBag(ONE_NOTE_PREFIX + "mostRecentAuthors"),
- oneNoteTreeWalker.getMostRecentAuthors().toArray(new String[]{}));
- }
- if (!oneNoteTreeWalker.getOriginalAuthors().isEmpty()) {
- metadata.set(Property.externalTextBag(ONE_NOTE_PREFIX + "originalAuthors"),
- oneNoteTreeWalker.getOriginalAuthors().toArray(new String[]{}));
- }
- if (!Instant.MAX.equals(
- Instant.ofEpochMilli(oneNoteTreeWalker.getCreationTimestamp()))) {
- metadata.set(ONE_NOTE_PREFIX + "creationTimestamp",
- String.valueOf(oneNoteTreeWalker.getCreationTimestamp()));
- }
- if (!Instant.MIN.equals(oneNoteTreeWalker.getLastModifiedTimestamp())) {
- metadata.set(ONE_NOTE_PREFIX + "lastModifiedTimestamp", String.valueOf(
- oneNoteTreeWalker.getLastModifiedTimestamp().toEpochMilli()));
- }
- if (oneNoteTreeWalker.getLastModified() > Long.MIN_VALUE) {
- metadata.set(TikaCoreProperties.MODIFIED,
- String.valueOf(oneNoteTreeWalker.getLastModified()));
+ if (!walked) {
+ legacyFallbackDump("OneNote parse failed; falling back to legacy text dump: " +
+ failureMessage(structureFailure), structureFailure, metadata,
+ xhtml, oneNoteDirectFileResource);
}
} else if (header.isLegacyOrAlternativePackaging()) {
+ MSOneStorePackage pkg = null;
try {
AlternativePackaging alternatePackageOneStoreFile = new AlternativePackaging();
+ //enable streaming deserialization
alternatePackageOneStoreFile.doDeserializeFromByteArray(oneStoreFileBytes, 0);
MSOneStoreParser onenoteParser = new MSOneStoreParser();
- MSOneStorePackage pkg =
- onenoteParser.parse(alternatePackageOneStoreFile.dataElementPackage);
+ pkg = onenoteParser.parse(alternatePackageOneStoreFile.dataElementPackage);
- pkg.walkTree(options, metadata, xhtml);
+ pkg.walkTree(options, metadata, xhtml, context);
} catch (Exception e) {
- OneNoteLegacyDumpStrings dumpStrings =
- new OneNoteLegacyDumpStrings(oneNoteDirectFileResource, xhtml);
- dumpStrings.dump();
+ rethrowIfLimitReached(e);
+ legacyFallbackDump(
+ "OneNote FSSHTTPB parse failed; falling back to legacy text dump: " +
+ failureMessage(e), e, metadata, xhtml,
+ oneNoteDirectFileResource);
+ pkg = null;
}
+ legacyFallbackIfNoContent(pkg, metadata, xhtml, oneNoteDirectFileResource);
} else {
throw new TikaException("Invalid OneStore document - could not parse headers");
}
@@ -182,6 +210,45 @@ public class OneNoteParser implements Parser {
}
+ private static String[] sortedValues(Set<String> values) {
+ String[] sorted = values.toArray(new String[0]);
+ Arrays.sort(sorted);
+ return sorted;
+ }
+
+ private static void rethrowIfLimitReached(Exception e) throws TikaException, SAXException {
+ WriteLimitReachedException.throwIfWriteLimitReached(e);
+ }
+
+ private static String failureMessage(Exception e) {
+ return e.getMessage() == null ? e.getClass().getSimpleName() : e.getMessage();
+ }
+
+ // the walk completed but every page dangled - without this a degraded
+ // file would yield empty output where the dump still finds its text
+ static void legacyFallbackIfNoContent(MSOneStorePackage pkg, Metadata metadata,
+ XHTMLContentHandler xhtml,
+ OneNoteDirectFileResource oneNoteDirectFileResource)
+ throws TikaException, SAXException {
+ if (pkg != null && !pkg.hasEmittedContent()) {
+ legacyFallbackDump("OneNote FSSHTTPB parse produced no content; " +
+ "falling back to legacy text dump", null, metadata, xhtml,
+ oneNoteDirectFileResource);
+ }
+ }
+
+ private static void legacyFallbackDump(String warning, Exception cause, Metadata metadata,
+ XHTMLContentHandler xhtml,
+ OneNoteDirectFileResource oneNoteDirectFileResource)
+ throws TikaException, SAXException {
+ LOG.warn(warning);
+ if (cause != null) {
+ LOG.debug("OneNote parse failure", cause);
+ }
+ metadata.add(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING, warning);
+ new OneNoteLegacyDumpStrings(oneNoteDirectFileResource, xhtml).dump();
+ }
+
/**
* Create a OneNoteDocument object.
* <p>
@@ -215,7 +282,8 @@ public class OneNoteParser implements Parser {
* content.
* @return A parsed one note document. This document does not contain any of the binary data,
* rather it just contains
- * the data pointers and metadata.
+ * the data pointers and metadata. A failure while parsing the root file node list is not
+ * thrown; it is recorded in the returned document's {@code structureParseException}.
* @throws IOException Will throw IOException in typical IO issue situations.
*/
public OneNoteDocument createOneNoteDocumentFromDirectFileResource(
@@ -227,9 +295,15 @@ public class OneNoteParser implements Parser {
if (oneNoteDocument.header.isMsOneStoreFormat()) {
// Now that we parsed the header, the "root file node list"
- oneNotePtr.reposition(oneNoteDocument.header.fcrFileNodeListRoot);
- FileNodePtr curPath = new FileNodePtr();
- oneNotePtr.deserializeFileNodeList(oneNoteDocument.root, curPath);
+ try {
+ oneNotePtr.reposition(oneNoteDocument.header.fcrFileNodeListRoot);
+ FileNodePtr curPath = new FileNodePtr();
+ oneNotePtr.deserializeFileNodeList(oneNoteDocument.root, curPath);
+ } catch (TikaException | IOException | RuntimeException e) {
+ // a truncated or malformed root list is recorded, not thrown, so the
+ // caller can fall back to the legacy string dump
+ oneNoteDocument.structureParseException = e;
+ }
}
return oneNoteDocument;
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNotePtr.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNotePtr.java
index 38a9f3d9f6..916b9bbb5a 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNotePtr.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNotePtr.java
@@ -19,11 +19,12 @@ package org.apache.tika.parser.microsoft.onenote;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
import java.util.Arrays;
+import java.util.HashSet;
import java.util.List;
import java.util.Map;
-import java.util.stream.Collectors;
-import java.util.stream.Stream;
+import java.util.Set;
import org.apache.commons.codec.binary.Hex;
import org.apache.commons.io.EndianUtils;
@@ -54,8 +55,28 @@ class OneNotePtr {
new byte[]{60, 0, 105, 0, 102, 0, 110, 0, 100, 0, 102, 0, 62, 0};
private static final String PACKAGE_STORAGE_FILE_FORMAT_GUID =
"{638DE92F-A6D4-4BC1-9A36-B3FC2511A5B7}";
+ private static final int MAX_PROPERTY_VALUES = 100_000;
+ private static final int MAX_PROPERTY_SET_DEPTH = 1000;
+ // the format nests file node lists only a handful of levels deep; 100 leaves wide
+ // margin while keeping the recursion far from any stack limit
+ private static final int MAX_FILE_NODE_LIST_DEPTH = 100;
+
+ private static final class PropertyValueBudget {
+ private long remaining = MAX_PROPERTY_VALUES;
+ }
+
+ /**
+ * Recursion state for the baseType-2 file-node-list nesting, shared by every pointer
+ * copied while parsing one document.
+ */
+ private static final class FileNodeListRecursion {
+ private int depth;
+ private final Set<Long> activeListOffsets = new HashSet<>();
+ }
int indentLevel = 0;
+ private PropertyValueBudget propertyValueBudget = new PropertyValueBudget();
+ private FileNodeListRecursion fileNodeListRecursion = new FileNodeListRecursion();
long offset;
long end;
@@ -77,6 +98,8 @@ class OneNotePtr {
this.offset = oneNotePtr.offset;
this.end = oneNotePtr.end;
this.indentLevel = oneNotePtr.indentLevel;
+ this.propertyValueBudget = oneNotePtr.propertyValueBudget;
+ this.fileNodeListRecursion = oneNotePtr.fileNodeListRecursion;
}
public OneNoteHeader deserializeHeader() throws IOException, TikaException {
@@ -260,8 +283,11 @@ class OneNotePtr {
public OneNotePtr internalDeserializeFileNodeList(OneNotePtr ptr, FileNodeList fileNodeList,
FileNodePtr curPath)
throws IOException, TikaException {
- OneNotePtr localPtr = new OneNotePtr(document, dif);
+ OneNotePtr localPtr = new OneNotePtr(ptr);
FileNodePtrBackPush bp = new FileNodePtrBackPush(curPath);
+ // a next-fragment reference pointing at an already-seen fragment would loop forever
+ Set<Long> seenFragmentOffsets = new HashSet<>();
+ seenFragmentOffsets.add(ptr.offset);
try {
while (true) {
FileChunkReference next = FileChunkReference.nil();
@@ -269,6 +295,11 @@ class OneNotePtr {
if (FileChunkReference.nil().equals(next)) {
break;
}
+ if (!seenFragmentOffsets.add(next.stp)) {
+ throw new TikaException(
+ "OneNote file node list fragment cycle detected at offset " +
+ next.stp);
+ }
localPtr.reposition(next);
ptr = localPtr;
}
@@ -279,9 +310,30 @@ class OneNotePtr {
}
+ /**
+ * Deserializes one file-node list and its fragments. The property-value budget is reset for
+ * each list; pointers copied while parsing that list share the budget.
+ */
public OneNotePtr deserializeFileNodeList(FileNodeList fileNodeList, FileNodePtr curPath)
throws IOException, TikaException {
- return internalDeserializeFileNodeList(this, fileNodeList, curPath);
+ if (fileNodeListRecursion.depth >= MAX_FILE_NODE_LIST_DEPTH) {
+ throw new TikaMemoryLimitException(
+ "OneNote file node list nesting exceeds depth limit " +
+ MAX_FILE_NODE_LIST_DEPTH);
+ }
+ long listOffset = offset;
+ if (!fileNodeListRecursion.activeListOffsets.add(listOffset)) {
+ throw new TikaException(
+ "OneNote file node list cycle detected at offset " + listOffset);
+ }
+ fileNodeListRecursion.depth++;
+ try {
+ propertyValueBudget = new PropertyValueBudget();
+ return internalDeserializeFileNodeList(this, fileNodeList, curPath);
+ } finally {
+ fileNodeListRecursion.depth--;
+ fileNodeListRecursion.activeListOffsets.remove(listOffset);
+ }
}
/**
@@ -453,23 +505,22 @@ class OneNotePtr {
data.subType.globalIdTableEntryFNDX.guid = deserializeGUID();
- document.revisionMap.get(document.currentRevision).globalId.put(
- data.subType.globalIdTableEntryFNDX.index,
+ Revision currentRevision = getCurrentRevision();
+ currentRevision.globalId.put(data.subType.globalIdTableEntryFNDX.index,
data.subType.globalIdTableEntryFNDX.guid);
} else if (data.id == FndStructureConstants.GlobalIdTableEntry2FNDX) {
data.subType.globalIdTableEntry2FNDX.indexMapFrom = deserializeLittleEndianInt();
data.subType.globalIdTableEntry2FNDX.indexMapTo = deserializeLittleEndianInt();
- ExtendedGUID dependentRevision =
- document.revisionMap.get(document.currentRevision).dependent;
- // Get the compactId from the revisionMap's globalId map.
- GUID compactId = document.revisionMap.get(dependentRevision).globalId.get(
+ Revision currentRevision = getCurrentRevision();
+ Revision dependentRevision = getRevision(currentRevision.dependent);
+ GUID compactId = dependentRevision.globalId.get(
data.subType.globalIdTableEntry2FNDX.indexMapFrom);
if (compactId == null) {
throw new TikaException("COMPACT_ID_MISSING");
}
- document.revisionMap.get(document.currentRevision).globalId.put(
- data.subType.globalIdTableEntry2FNDX.indexMapTo, compactId);
+ currentRevision.globalId.put(data.subType.globalIdTableEntry2FNDX.indexMapTo,
+ compactId);
} else if (data.id == FndStructureConstants.GlobalIdTableEntry3FNDX) {
data.subType.globalIdTableEntry3FNDX.indexCopyFromStart = deserializeLittleEndianInt();
@@ -477,16 +528,17 @@ class OneNotePtr {
data.subType.globalIdTableEntry3FNDX.indexCopyToStart = deserializeLittleEndianInt();
- ExtendedGUID dependent_revision =
- document.revisionMap.get(document.currentRevision).dependent;
- for (int i = 0; i < data.subType.globalIdTableEntry3FNDX.entriesToCopy; ++i) {
- Map<Long, GUID> globalIdMap = document.revisionMap.get(dependent_revision).globalId;
- GUID compactId = globalIdMap.get(
+ Revision currentRevision = getCurrentRevision();
+ Revision dependentRevision = getRevision(currentRevision.dependent);
+ int entriesToCopy = checkedCount(
+ data.subType.globalIdTableEntry3FNDX.entriesToCopy, "global ID entries");
+ for (int i = 0; i < entriesToCopy; ++i) {
+ GUID compactId = dependentRevision.globalId.get(
data.subType.globalIdTableEntry3FNDX.indexCopyFromStart + i);
if (compactId == null) {
throw new TikaException("COMPACT_ID_MISSING");
}
- document.revisionMap.get(document.currentRevision).globalId.put(
+ currentRevision.globalId.put(
data.subType.globalIdTableEntry3FNDX.indexCopyToStart + i, compactId);
}
} else if (data.id == FndStructureConstants.CanRevise.ObjectRevisionWithRefCountFNDX ||
@@ -623,9 +675,9 @@ class OneNotePtr {
"Data out of bounds - cch " + cch + " is > room left = " + roomLeftLong);
}
- if (cch > dif.size()) {
+ if (cch > dif.size() || cch > Integer.MAX_VALUE / 2L) {
throw new TikaMemoryLimitException(
- "CCH=" + cch + " was found that was greater" + " than file size " +
+ "CCH=" + cch + " cannot be safely allocated for file size " +
dif.size());
}
ByteBuffer dataSpaceBuffer = ByteBuffer.allocate((int) cch * 2);
@@ -816,7 +868,7 @@ class OneNotePtr {
compactID.guid = ExtendedGUID.nil();
compactID.guid.n = compactID.n;
long index = compactID.guidIndex;
- Map<Long, GUID> globalIdMap = document.revisionMap.get(document.currentRevision).globalId;
+ Map<Long, GUID> globalIdMap = getCurrentRevision().globalId;
GUID guid = globalIdMap.get(index);
if (guid != null) {
compactID.guid.guid = guid;
@@ -976,22 +1028,69 @@ class OneNotePtr {
}
}
+ private Revision getCurrentRevision() throws TikaException {
+ Revision revision = document.revisionMap.get(document.currentRevision);
+ if (revision == null) {
+ throw new TikaException("Missing current revision");
+ }
+ return revision;
+ }
+
+ private Revision getRevision(ExtendedGUID revisionId) throws TikaException {
+ Revision revision = document.revisionMap.get(revisionId);
+ if (revision == null) {
+ throw new TikaException("Missing dependent revision");
+ }
+ return revision;
+ }
+
+ private int checkedCount(long count, String description) throws TikaException {
+ if (count < 0 || count > MAX_PROPERTY_VALUES) {
+ throw new TikaMemoryLimitException(
+ description + " count " + count + " exceeds limit " + MAX_PROPERTY_VALUES);
+ }
+ return (int) count;
+ }
+
+ private int reservePropertyCount(long count, String description) throws TikaException {
+ int checked = checkedCount(count, description);
+ if (count > propertyValueBudget.remaining) {
+ throw new TikaMemoryLimitException(description + " count " + count +
+ " exceeds remaining property value budget " + propertyValueBudget.remaining);
+ }
+ propertyValueBudget.remaining -= count;
+ return checked;
+ }
+
private PropertySet deserializePropertySet(ObjectStreamCounters counters,
ObjectSpaceObjectPropSet streams)
throws IOException, TikaException {
+ return deserializePropertySet(counters, streams, 0);
+ }
+
+ private PropertySet deserializePropertySet(ObjectStreamCounters counters,
+ ObjectSpaceObjectPropSet streams,
+ int depth)
+ throws IOException, TikaException {
+ if (depth >= MAX_PROPERTY_SET_DEPTH) {
+ throw new TikaException("Property set nesting exceeds limit " +
+ MAX_PROPERTY_SET_DEPTH);
+ }
PropertySet data = new PropertySet();
- long count = deserializeLittleEndianShort();
- data.rgPridsData =
- Stream.generate(PropertyValue::new).limit((int) count).collect(Collectors.toList());
+ // Reserve the count before allocating any PropertyValue instances.
+ int count = reservePropertyCount(deserializeLittleEndianShort(), "property");
+ data.rgPridsData = new ArrayList<>();
for (int i = 0; i < count; ++i) {
- data.rgPridsData.get(i).propertyId = deserializePropertyID();
- LOG.debug("{}Property {}", getIndent(), data.rgPridsData.get(i).propertyId);
+ PropertyValue propertyValue = new PropertyValue();
+ propertyValue.propertyId = deserializePropertyID();
+ data.rgPridsData.add(propertyValue);
+ LOG.debug("{}Property {}", getIndent(), propertyValue.propertyId);
}
LOG.debug("{}{} elements in property set:", getIndent(), count);
for (int i = 0; i < count; ++i) {
data.rgPridsData.set(i,
deserializePropertyValueFromPropertyID(data.rgPridsData.get(i).propertyId,
- streams, counters));
+ streams, counters, depth));
}
LOG.debug("");
return data;
@@ -1000,8 +1099,13 @@ class OneNotePtr {
private PropertyValue deserializePropertyValueFromPropertyID(OneNotePropertyId propertyID,
ObjectSpaceObjectPropSet streams,
- ObjectStreamCounters counters)
+ ObjectStreamCounters counters,
+ int propertySetDepth)
throws IOException, TikaException {
+ if (propertySetDepth >= MAX_PROPERTY_SET_DEPTH) {
+ throw new TikaException("Property value nesting exceeds limit " +
+ MAX_PROPERTY_SET_DEPTH);
+ }
PropertyValue data = new PropertyValue();
data.propertyId = propertyID;
char val8;
@@ -1119,22 +1223,18 @@ class OneNotePtr {
val32 = deserializeLittleEndianInt();
OneNotePropertyId propId = deserializePropertyID();
LOG.debug(" UnifiedSubPropertySet {} {}", val32, propId);
- data.propertySet.rgPridsData =
- Stream.generate(PropertyValue::new).limit((int) val32)
- .collect(Collectors.toList());
- for (int i = 0; i < val32; ++i) {
- try {
- data.propertySet.rgPridsData.set(i,
- deserializePropertyValueFromPropertyID(propId, streams,
- counters));
- } catch (IOException e) {
- return data;
- }
+ int propertyCount = reservePropertyCount(val32, "unified property");
+ data.propertySet.rgPridsData = new ArrayList<>();
+ for (int i = 0; i < propertyCount; ++i) {
+ data.propertySet.rgPridsData.add(
+ deserializePropertyValueFromPropertyID(propId, streams, counters,
+ propertySetDepth + 1));
}
break;
case 0x11:
LOG.debug(" SubPropertySet");
- data.propertySet = deserializePropertySet(counters, streams);
+ data.propertySet = deserializePropertySet(counters, streams,
+ propertySetDepth + 1);
break;
default:
throw new TikaException("Invalid type: " + type);
@@ -1185,7 +1285,8 @@ class OneNotePtr {
getIndent(), data.count, data.osidsStreamNotPresent,
data.extendedStreamsPresent);
}
- for (int i = 0; i < data.count; ++i) {
+ int count = checkedCount(data.count, "object stream");
+ for (int i = 0; i < count; ++i) {
CompactID cid;
cid = deserializeCompactID();
data.data.add(cid);
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteTreeWalker.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteTreeWalker.java
index 452b48089e..939d36bbc0 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteTreeWalker.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteTreeWalker.java
@@ -27,6 +27,7 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.IdentityHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -35,6 +36,8 @@ import java.util.regex.Pattern;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.tuple.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import org.xml.sax.SAXException;
import org.xml.sax.helpers.AttributesImpl;
@@ -44,6 +47,7 @@ import org.apache.tika.extractor.EmbeddedDocumentExtractor;
import org.apache.tika.extractor.EmbeddedDocumentUtil;
import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.TikaCoreProperties;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.sax.EmbeddedContentHandler;
import org.apache.tika.sax.XHTMLContentHandler;
@@ -54,6 +58,10 @@ import org.apache.tika.sax.XHTMLContentHandler;
*/
class OneNoteTreeWalker {
+ private static final Logger LOG = LoggerFactory.getLogger(OneNoteTreeWalker.class);
+ private static final int MAX_CYCLE_WARNINGS = 100;
+ // low enough that the ~3 stack frames per level cannot overflow a default thread stack
+ private static final int MAX_WALK_DEPTH = 500;
private static final String P = "p";
/**
* See spec MS-ONE - 2.3.1 - TIME32 - epoch of jan 1 1980 UTC.
@@ -102,6 +110,14 @@ class OneNoteTreeWalker {
* Contains pairs of {Offset,Length} that we have added to the text stream already.
*/
private final Set<Pair<Long, Integer>> textAlreadyFetched = new HashSet<>();
+ private final Set<FileNode> activeFileNodes =
+ Collections.newSetFromMap(new IdentityHashMap<>());
+ private final Set<FileNode> reportedCycleNodes =
+ Collections.newSetFromMap(new IdentityHashMap<>());
+ private int cycleWarningCount;
+ private boolean cycleWarningsSuppressed;
+ private int walkDepth;
+ private boolean depthWarningRecorded;
/**
* Create a one tree walker.
@@ -290,6 +306,68 @@ class OneNoteTreeWalker {
public Map<String, Object> walkFileNode(FileNode fileNode,
OneNotePropertyId parentPropertyId)
throws IOException, TikaException, SAXException {
+ if (fileNode == null) {
+ return Collections.emptyMap();
+ }
+ if (walkDepth >= MAX_WALK_DEPTH) {
+ // the cycle guard cannot catch a long acyclic chain - cap the recursion depth
+ // so a crafted file cannot overflow the stack
+ recordDepthWarning();
+ return Collections.emptyMap();
+ }
+ if (!activeFileNodes.add(fileNode)) {
+ recordCycleWarning(fileNode);
+ return Collections.emptyMap();
+ }
+ walkDepth++;
+ try {
+ return walkFileNodeInternal(fileNode, parentPropertyId);
+ } finally {
+ walkDepth--;
+ activeFileNodes.remove(fileNode);
+ }
+ }
+
+ private void recordDepthWarning() {
+ if (depthWarningRecorded) {
+ return;
+ }
+ depthWarningRecorded = true;
+ String warning = "OneNote file-node traversal exceeded depth limit " + MAX_WALK_DEPTH;
+ LOG.warn(warning);
+ if (parentMetadata != null) {
+ parentMetadata.add(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING, warning);
+ }
+ }
+
+ private void recordCycleWarning(FileNode fileNode) {
+ if (reportedCycleNodes.contains(fileNode)) {
+ return;
+ }
+ if (cycleWarningCount >= MAX_CYCLE_WARNINGS) {
+ if (!cycleWarningsSuppressed) {
+ cycleWarningsSuppressed = true;
+ String warning = "Additional OneNote file-node cycle warnings were suppressed " +
+ "after " + MAX_CYCLE_WARNINGS + " distinct cycles";
+ LOG.warn(warning);
+ if (parentMetadata != null) {
+ parentMetadata.add(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING, warning);
+ }
+ }
+ return;
+ }
+ reportedCycleNodes.add(fileNode);
+ cycleWarningCount++;
+ String warning = "OneNote file-node cycle detected at " + fileNode.gosid;
+ LOG.warn(warning);
+ if (parentMetadata != null) {
+ parentMetadata.add(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING, warning);
+ }
+ }
+
+ private Map<String, Object> walkFileNodeInternal(FileNode fileNode,
+ OneNotePropertyId parentPropertyId)
+ throws IOException, TikaException, SAXException {
Map<String, Object> structure = new HashMap<>();
structure.put("oneNoteType", "FileNode");
structure.put("gosid", fileNode.gosid.toString());
@@ -312,7 +390,7 @@ class OneNoteTreeWalker {
if (fileNode.subType.fileDataStoreObjectReference.ref != null && !FileChunkReference.nil()
.equals(fileNode.subType.fileDataStoreObjectReference.ref.fileData)) {
structure.put("fileDataStoreObjectReference", walkFileDataStoreObjectReference(
- fileNode.subType.fileDataStoreObjectReference));
+ fileNode.subType.fileDataStoreObjectReference, fileNode.gosid));
}
return structure;
}
@@ -325,7 +403,7 @@ class OneNoteTreeWalker {
* @throws IOException Can throw these when manipulating the seekable byte channel.
*/
private Map<String, Object> walkFileDataStoreObjectReference(
- FileDataStoreObjectReference fileDataStoreObjectReference)
+ FileDataStoreObjectReference fileDataStoreObjectReference, ExtendedGUID gosid)
throws IOException, SAXException, TikaException {
Map<String, Object> structure = new HashMap<>();
OneNotePtr content = new OneNotePtr(oneNoteDocument, dif);
@@ -335,12 +413,13 @@ class OneNoteTreeWalker {
"File data store cb " + fileDataStoreObjectReference.ref.fileData.cb +
" exceeds document size: " + dif.size());
}
- handleEmbedded((int) fileDataStoreObjectReference.ref.fileData.cb);
+ handleEmbedded((int) fileDataStoreObjectReference.ref.fileData.cb, gosid);
structure.put("fileDataStoreObjectMetadata", fileDataStoreObjectReference);
return structure;
}
- private void handleEmbedded(int length) throws TikaException, IOException, SAXException {
+ private void handleEmbedded(int length, ExtendedGUID gosid)
+ throws TikaException, IOException, SAXException {
TikaInputStream stream = null;
ByteBuffer buf;
try {
@@ -352,14 +431,21 @@ class OneNoteTreeWalker {
return;
}
Metadata embeddedMetadata = new Metadata();
+ embeddedMetadata.set(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE,
+ TikaCoreProperties.EmbeddedResourceType.ATTACHMENT.toString());
+ if (gosid != null && !ExtendedGUID.nil().equals(gosid)) {
+ embeddedMetadata.set(TikaCoreProperties.EMBEDDED_RELATIONSHIP_ID, gosid.toString());
+ }
try {
AttributesImpl attributes = new AttributesImpl();
attributes.addAttribute("", "class", "class", "CDATA", "embedded");
xhtml.startElement("div", attributes);
xhtml.endElement("div");
- stream = TikaInputStream.get(buf.array());
- embeddedDocumentExtractor.parseEmbedded(stream, new EmbeddedContentHandler(xhtml),
- embeddedMetadata, false);
+ if (embeddedDocumentExtractor.shouldParseEmbedded(embeddedMetadata)) {
+ stream = TikaInputStream.get(buf.array());
+ embeddedDocumentExtractor.parseEmbedded(stream, new EmbeddedContentHandler(xhtml),
+ embeddedMetadata, false);
+ }
} finally {
IOUtils.closeQuietly(stream);
}
@@ -471,8 +557,11 @@ class OneNoteTreeWalker {
propMap.put("dataUnicode16LE", new String(buf.array(), StandardCharsets.UTF_16LE));
if (options.getUtf16PropertiesToPrint().contains(propertyValue.propertyId.propertyEnum)) {
xhtml.startElement(P);
- xhtml.characters((String) propMap.get("dataUnicode16LE"));
- xhtml.endElement(P);
+ try {
+ xhtml.characters((String) propMap.get("dataUnicode16LE"));
+ } finally {
+ xhtml.endElement(P);
+ }
}
} else if (propertyValue.propertyId.propertyEnum ==
OneNotePropertyEnum.TextExtendedAscii) {
@@ -485,8 +574,11 @@ class OneNoteTreeWalker {
dif.read(buf);
propMap.put("dataAscii", new String(buf.array(), StandardCharsets.US_ASCII));
xhtml.startElement(P);
- xhtml.characters((String) propMap.get("dataAscii"));
- xhtml.endElement(P);
+ try {
+ xhtml.characters((String) propMap.get("dataAscii"));
+ } finally {
+ xhtml.endElement(P);
+ }
} else if (!isBinary) {
if (content.size() > dif.size()) {
throw new TikaMemoryLimitException(
@@ -498,8 +590,11 @@ class OneNoteTreeWalker {
propMap.put("dataUnicode16LE", new String(buf.array(), StandardCharsets.UTF_16LE));
if (options.getUtf16PropertiesToPrint().contains(propertyValue.propertyId.propertyEnum)) {
xhtml.startElement(P);
- xhtml.characters((String) propMap.get("dataUnicode16LE"));
- xhtml.endElement(P);
+ try {
+ xhtml.characters((String) propMap.get("dataUnicode16LE"));
+ } finally {
+ xhtml.endElement(P);
+ }
}
} else {
if (content.size() > dif.size()) {
@@ -591,12 +686,18 @@ class OneNoteTreeWalker {
Matcher m = HYPERLINK_PATTERN.matcher(txt);
if (m.find()) {
xhtml.startElement("a", "href", m.group(1));
- xhtml.characters(m.group(2));
- xhtml.endElement("a");
+ try {
+ xhtml.characters(m.group(2));
+ } finally {
+ xhtml.endElement("a");
+ }
} else {
xhtml.startElement(P);
- xhtml.characters(txt);
- xhtml.endElement(P);
+ try {
+ xhtml.characters(txt);
+ } finally {
+ xhtml.endElement(P);
+ }
}
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackage.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackage.java
index 3b8ad38227..f459522ba9 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackage.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackage.java
@@ -25,27 +25,44 @@ import java.time.LocalDateTime;
import java.time.Month;
import java.time.ZoneOffset;
import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Map;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import org.xml.sax.SAXException;
+import org.xml.sax.helpers.AttributesImpl;
import org.apache.tika.exception.TikaException;
+import org.apache.tika.extractor.EmbeddedDocumentExtractor;
+import org.apache.tika.extractor.EmbeddedDocumentUtil;
+import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.Property;
import org.apache.tika.metadata.TikaCoreProperties;
+import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.microsoft.onenote.OneNotePropertyEnum;
import org.apache.tika.parser.microsoft.onenote.OneNoteTreeWalkerOptions;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.ArrayNumber;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.EightBytesOfData;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.FourBytesOfData;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.IProperty;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.PrtArrayOfPropertyValues;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.PrtFourBytesOfLengthFollowedByData;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.CellManifestDataElementData;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.PropertySet;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionManifestDataElementData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionManifestRootDeclare;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionStoreCell;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionStoreObject;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionStoreObjectGroup;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.StorageIndexCellMapping;
@@ -59,6 +76,7 @@ import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.Propert
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.PropertyType;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.unsigned.Unsigned;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.util.BitConverter;
+import org.apache.tika.sax.EmbeddedContentHandler;
import org.apache.tika.sax.XHTMLContentHandler;
public class MSOneStorePackage {
@@ -76,7 +94,11 @@ public class MSOneStorePackage {
private static final long DATETIME_EPOCH_DIFF_1601;
private static final Pattern HYPERLINK_PATTERN =
Pattern.compile("\uFDDFHYPERLINK\\s+\"([^\"]+)\"([^\"]+)$");
+ private static final Logger LOG = LoggerFactory.getLogger(MSOneStorePackage.class);
private static final String P = "p";
+ private static final int MAX_OBJECT_WALK_DEPTH = 1000;
+ private static final int MAX_REFERENCE_COUNT = 100000;
+ private static final int MAX_PARSE_WARNINGS = 100;
static {
LocalDateTime time32Epoch1980 = LocalDateTime.of(1980, Month.JANUARY, 1, 0, 0);
@@ -93,6 +115,10 @@ public class MSOneStorePackage {
private final Set<String> authors = new HashSet<>();
private final Set<String> mostRecentAuthors = new HashSet<>();
private final Set<String> originalAuthors = new HashSet<>();
+ /**
+ * The fully populated storage index. Set this before performing storage-index lookups; its
+ * mapping lists are indexed once and must not be mutated afterward.
+ */
public StorageIndexDataElementData storageIndex;
public StorageManifestDataElementData storageManifest;
public CellManifestDataElementData headerCellCellManifest;
@@ -100,52 +126,75 @@ public class MSOneStorePackage {
public List<RevisionManifestDataElementData> revisionManifests;
public List<CellManifestDataElementData> cellManifests;
public HeaderCell headerCell;
- public List<RevisionStoreObjectGroup> dataRoot;
+ public RevisionStoreCell dataRootCell;
public List<RevisionStoreObjectGroup> OtherFileNodeList;
- private boolean mostRecentAuthorProp = false;
- private boolean originalAuthorProp = false;
+ /**
+ * The content cells (object spaces, e.g. pages), each with its object groups and the
+ * root object declarations of its current revision.
+ */
+ public List<RevisionStoreCell> cells;
private Instant lastModifiedTimestamp = Instant.MIN;
private long creationTimestamp = Long.MAX_VALUE;
private long lastModified = Long.MIN_VALUE;
+ private ParseContext parseContext;
+ private EmbeddedDocumentExtractor embeddedDocumentExtractor;
+ private Metadata parentMetadata;
+ private final List<String> parseWarnings = new ArrayList<>();
+ // This state intentionally spans parser construction and tree walking for one package.
+ private final Set<String> recordedParseWarningKeys = new HashSet<>();
+ private boolean parseWarningsSuppressed;
+ private boolean storageMappingsIndexed;
+ private boolean contentEmitted;
+ // flattened property actions per object; objects can be re-flattened many times during
+ // picture/resource-name resolution, which is quadratic without this cache
+ private final Map<RevisionStoreObject, List<PropertyAction>> objectActionsCache =
+ new IdentityHashMap<>();
+ private final Map<CellID, StorageIndexCellMapping> storageIndexCellMappingsById =
+ new HashMap<>();
+ private final Map<ExGuid, StorageIndexRevisionMapping> storageIndexRevisionMappingsById =
+ new HashMap<>();
public MSOneStorePackage() {
this.revisionManifests = new ArrayList<>();
this.cellManifests = new ArrayList<>();
this.OtherFileNodeList = new ArrayList<>();
+ this.cells = new ArrayList<>();
}
/**
* This method is used to find the Storage Index Cell Mapping matches the Cell ID.
*
* @param cellID Specify the Cell ID.
- * @return Return the specific Storage Index Cell Mapping.
+ * @return the specific Storage Index Cell Mapping, or {@code null} if it is absent.
*/
public StorageIndexCellMapping findStorageIndexCellMapping(CellID cellID) {
- StorageIndexCellMapping storageIndexCellMapping = null;
- if (this.storageIndex != null) {
- storageIndexCellMapping = this.storageIndex.storageIndexCellMappingList.stream()
- .filter(s -> s.cellID.equals(cellID)).findFirst()
- .orElse(new StorageIndexCellMapping());
- }
- return storageIndexCellMapping;
+ indexStorageMappings();
+ return storageIndexCellMappingsById.get(cellID);
}
/**
* This method is used to find the Storage Index Revision Mapping that matches the Revision Mapping Extended GUID.
*
* @param revisionExtendedGUID Specify the Revision Mapping Extended GUID.
- * @return Return the instance of Storage Index Revision Mapping.
+ * @return the instance of Storage Index Revision Mapping, or {@code null} if it is absent.
*/
public StorageIndexRevisionMapping findStorageIndexRevisionMapping(
ExGuid revisionExtendedGUID) {
- StorageIndexRevisionMapping instance = null;
- if (this.storageIndex != null) {
- instance = this.storageIndex.storageIndexRevisionMappingList.stream()
- .filter(r -> r.revisionExGuid.equals(revisionExtendedGUID)).findFirst()
- .orElse(new StorageIndexRevisionMapping());
- }
+ indexStorageMappings();
+ return storageIndexRevisionMappingsById.get(revisionExtendedGUID);
+ }
- return instance;
+ private void indexStorageMappings() {
+ if (storageMappingsIndexed || storageIndex == null) {
+ return;
+ }
+ storageMappingsIndexed = true;
+ for (StorageIndexCellMapping mapping : storageIndex.storageIndexCellMappingList) {
+ storageIndexCellMappingsById.putIfAbsent(mapping.cellID, mapping);
+ }
+ for (StorageIndexRevisionMapping mapping : storageIndex.storageIndexRevisionMappingList) {
+ storageIndexRevisionMappingsById.putIfAbsent(mapping.revisionExGuid, mapping);
+ }
}
/**
@@ -160,112 +209,810 @@ public class MSOneStorePackage {
property == OneNotePropertyEnum.RichEditTextUnicode;
}
+ /**
+ * The attribution of an Author property, determined by the property through which the
+ * author object was referenced.
+ */
+ private enum AuthorRole {
+ NONE, MOST_RECENT, ORIGINAL
+ }
+
+ private enum ParseWarningKind {
+ DEFAULT,
+ OBJECT_REFERENCE_ARRAY_UNAVAILABLE,
+ OBJECT_REFERENCE_ARRAY_CAPPED,
+ OBJECT_SPACE_REFERENCE_ARRAY_UNAVAILABLE,
+ OBJECT_SPACE_REFERENCE_ARRAY_CAPPED
+ }
+
public void walkTree(OneNoteTreeWalkerOptions options, Metadata metadata,
- XHTMLContentHandler xhtml)
+ XHTMLContentHandler xhtml, ParseContext parseContext)
throws SAXException, TikaException, IOException {
- for (RevisionStoreObjectGroup revisionStoreObjectGroup : OtherFileNodeList) {
- for (RevisionStoreObject revisionStoreObject : revisionStoreObjectGroup.objects) {
- PropertySet propertySet =
- revisionStoreObject.propertySet.objectSpaceObjectPropSet.body;
- for (int i = 0; i < propertySet.rgData.size(); ++i) {
- IProperty property = propertySet.rgData.get(i);
- PropertyID propertyID = propertySet.rgPrids[i];
- PropertyType propertyType = PropertyType.fromIntVal(propertyID.type);
- OneNotePropertyEnum oneNotePropertyEnum =
- OneNotePropertyEnum.of(Unsigned.uint(propertyID.value).longValue());
- if (oneNotePropertyEnum == OneNotePropertyEnum.LastModifiedTimeStamp) {
- long fullval = getScalar(property);
- Instant instant = Instant.ofEpochSecond(
- fullval / 10000000 + DATETIME_EPOCH_DIFF_1601);
- if (instant.isAfter(lastModifiedTimestamp)) {
- lastModifiedTimestamp = instant;
- }
- metadata.set(ONE_NOTE_PREFIX + "lastModifiedTimestamp",
- String.valueOf(lastModifiedTimestamp.toEpochMilli()));
- } else if (oneNotePropertyEnum == OneNotePropertyEnum.CreationTimeStamp) {
- // add the TIME32_EPOCH_DIFF_1980 because OneNote TIME32 epoch time is per 1980, not
- // 1970
- long scalar = getScalar(property);
- long creationTs = scalar + TIME32_EPOCH_DIFF_1980;
- if (creationTs < creationTimestamp) {
- creationTimestamp = creationTs;
- }
- metadata.set(ONE_NOTE_PREFIX + "creationTimestamp", String.valueOf(creationTimestamp));
- } else if (oneNotePropertyEnum == OneNotePropertyEnum.LastModifiedTime) {
- // add the TIME32_EPOCH_DIFF_1980 because OneNote TIME32 epoch time is per 1980, not
- // 1970
- long scalar = getScalar(property);
- long lastMod = scalar + TIME32_EPOCH_DIFF_1980;
- if (lastMod > lastModified) {
- lastModified = lastMod;
- }
- metadata.set(TikaCoreProperties.MODIFIED, String.valueOf(lastModified));
- } else if (oneNotePropertyEnum == OneNotePropertyEnum.Author) {
- String author =
- new String(((PrtFourBytesOfLengthFollowedByData) property).data,
- StandardCharsets.UTF_8);
- if (mostRecentAuthorProp) {
- mostRecentAuthors.add(author);
- } else if (originalAuthorProp) {
- originalAuthors.add(author);
- } else {
- authors.add(author);
- }
- } else if (oneNotePropertyEnum == OneNotePropertyEnum.AuthorMostRecent) {
- mostRecentAuthorProp = true;
- } else if (oneNotePropertyEnum == OneNotePropertyEnum.AuthorOriginal) {
- originalAuthorProp = true;
- } else if (propertyType == PropertyType.FourBytesOfLengthFollowedByData) {
- boolean isBinary = propertyIsBinary(oneNotePropertyEnum);
- PrtFourBytesOfLengthFollowedByData dataProperty =
- (PrtFourBytesOfLengthFollowedByData) property;
- if ((dataProperty.data.length & 1) == 0 &&
- oneNotePropertyEnum != OneNotePropertyEnum.TextExtendedAscii &&
- !isBinary) {
- if (options.getUtf16PropertiesToPrint().contains(oneNotePropertyEnum)) {
- xhtml.startElement(P);
- xhtml.characters(
- new String(dataProperty.data, StandardCharsets.UTF_16LE));
- xhtml.endElement(P);
- }
- } else if (oneNotePropertyEnum == OneNotePropertyEnum.TextExtendedAscii) {
- xhtml.startElement(P);
- xhtml.characters(
- new String(dataProperty.data, StandardCharsets.US_ASCII));
- xhtml.endElement(P);
- } else if (!isBinary) {
- if (options.getUtf16PropertiesToPrint().contains(oneNotePropertyEnum)) {
- xhtml.startElement(P);
- xhtml.characters(
- new String(dataProperty.data, StandardCharsets.UTF_16LE));
- xhtml.endElement(P);
- }
- } else {
- if (oneNotePropertyEnum == OneNotePropertyEnum.RichEditTextUnicode) {
- handleRichEditTextUnicode(dataProperty.data, xhtml);
- } else {
- //TODO -- these seem to be somewhat broken font files and other
- //odds and ends...what are they and how should we process them?
- //handleEmbedded(content.size());
- }
- }
- }
+ this.parseContext = parseContext;
+ this.parentMetadata = metadata;
+ this.embeddedDocumentExtractor =
+ EmbeddedDocumentUtil.getEmbeddedDocumentExtractor(parseContext);
+ for (String warning : parseWarnings) {
+ metadata.add(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING, warning);
+ }
+ parseWarnings.clear();
+ if (!cells.isEmpty()) {
+ // Walk each page cell (object space) as a tree, starting from the root objects of
+ // its current revision and following the object references in property order. This
+ // emits the text in document order. The pages are walked in the order in which the
+ // section object space references them; cells that hold older versions of a page
+ // (the same object space in a different revision context) are skipped.
+ List<RevisionStoreCell> pageCells = new ArrayList<>();
+ List<RevisionStoreCell> otherCells = new ArrayList<>();
+ splitCells(pageCells, otherCells);
+ for (RevisionStoreCell cell : pageCells) {
+ xhtml.startElement("div", "class", "page");
+ try {
+ walkCell(cell, options, metadata, xhtml);
+ } finally {
+ xhtml.endElement("div");
+ }
+ }
+ for (RevisionStoreCell cell : otherCells) {
+ walkCell(cell, options, metadata, xhtml);
+ }
+ } else {
+ // no cell information available - walk the object groups in revision order
+ Map<ExGuid, RevisionStoreObject> objectsById = indexObjectsById(OtherFileNodeList);
+ Set<ExGuid> visited = new HashSet<>();
+ for (RevisionStoreObjectGroup objectGroup : OtherFileNodeList) {
+ for (RevisionStoreObject object : objectGroup.objects) {
+ walkObject(object, objectsById, visited, AuthorRole.NONE, options, metadata,
+ xhtml, 0);
}
}
}
if (!authors.isEmpty()) {
- metadata.set(TikaCoreProperties.CREATOR, authors.toArray(new String[]{}));
+ metadata.set(TikaCoreProperties.CREATOR, sortedValues(authors));
}
if (!mostRecentAuthors.isEmpty()) {
metadata.set(Property.externalTextBag(ONE_NOTE_PREFIX + "mostRecentAuthors"),
- mostRecentAuthors.toArray(new String[]{}));
+ sortedValues(mostRecentAuthors));
}
if (!originalAuthors.isEmpty()) {
metadata.set(Property.externalTextBag(ONE_NOTE_PREFIX + "originalAuthors"),
- originalAuthors.toArray(new String[]{}));
+ sortedValues(originalAuthors));
+ }
+ }
+
+ /**
+ * Splits the cells into page cells, ordered as the section object space references them,
+ * and the remaining cells. A cell that holds an older version of a page - the same object
+ * space referenced by a page cell, but in a different revision context - is dropped, so
+ * content is not emitted once per version snapshot.
+ */
+ private void splitCells(List<RevisionStoreCell> pageCells,
+ List<RevisionStoreCell> otherCells) {
+ if (dataRootCell == null) {
+ // Without a data root there is no reliable page ordering information.
+ otherCells.addAll(cells);
+ return;
+ }
+ List<CellID> orderedCellIds = collectSectionReferencedCells();
+ if (orderedCellIds.isEmpty()) {
+ // no page ordering information available - process the cells in storage order
+ otherCells.addAll(cells);
+ return;
+ }
+ Map<CellID, RevisionStoreCell> remainingCells = new LinkedHashMap<>();
+ for (RevisionStoreCell cell : cells) {
+ remainingCells.put(cell.cellID, cell);
+ }
+ Set<ExGuid> coveredObjectSpaces = new HashSet<>();
+ for (CellID cellId : orderedCellIds) {
+ RevisionStoreCell cell = remainingCells.remove(cellId);
+ if (cell != null) {
+ pageCells.add(cell);
+ if (cellId.extendGUID2 != null) {
+ coveredObjectSpaces.add(cellId.extendGUID2);
+ }
+ }
+ }
+ for (RevisionStoreCell cell : remainingCells.values()) {
+ if (!coveredObjectSpaces.contains(cell.cellID.extendGUID2)) {
+ // not an older version of one of the pages - keep it so no content is lost
+ otherCells.add(cell);
+ }
+ }
+ }
+
+ /**
+ * Walks the section object space (the data root cell) and collects the object space (cell)
+ * references in document order - this is the order of the pages in the section.
+ */
+ private List<CellID> collectSectionReferencedCells() {
+ List<CellID> orderedCellIds = new ArrayList<>();
+ if (dataRootCell == null) {
+ return orderedCellIds;
+ }
+ Map<ExGuid, RevisionStoreObject> objectsById = indexObjectsById(dataRootCell.objectGroups);
+ Set<ExGuid> visited = new HashSet<>();
+ for (RevisionManifestRootDeclare rootDeclare : dataRootCell.rootDeclares) {
+ RevisionStoreObject rootObject = objectsById.get(rootDeclare.objectExGuid);
+ if (rootObject == null) {
+ recordParseWarning("OneNote section root object " + rootDeclare.objectExGuid +
+ " could not be resolved");
+ }
+ collectReferencedCells(rootObject, objectsById, visited, orderedCellIds, 0);
+ }
+ return orderedCellIds;
+ }
+
+ private void collectReferencedCells(RevisionStoreObject object,
+ Map<ExGuid, RevisionStoreObject> objectsById,
+ Set<ExGuid> visited, List<CellID> out, int depth) {
+ if (object == null || object.propertySet == null ||
+ object.propertySet.objectSpaceObjectPropSet == null) {
+ return;
+ }
+ if (depth >= MAX_OBJECT_WALK_DEPTH) {
+ recordParseWarning("OneNote section reference traversal exceeded depth limit " +
+ MAX_OBJECT_WALK_DEPTH);
+ return;
+ }
+ if (object.objectID != null && !visited.add(object.objectID)) {
+ return;
+ }
+ List<PropertyAction> actions = collectObjectActions(object);
+ for (PropertyAction action : actions) {
+ if (action.spaceReference != null) {
+ out.add(action.spaceReference);
+ } else if (action.isChildReference && action.childReference != null) {
+ RevisionStoreObject child = objectsById.get(action.childReference);
+ if (child == null && !warningsSaturated()) {
+ recordParseWarning("OneNote section object " + action.childReference +
+ " could not be resolved");
+ }
+ collectReferencedCells(child, objectsById, visited, out, depth + 1);
+ }
+ }
+ }
+
+ private void walkCell(RevisionStoreCell cell, OneNoteTreeWalkerOptions options,
+ Metadata metadata, XHTMLContentHandler xhtml)
+ throws SAXException, TikaException, IOException {
+ Map<ExGuid, RevisionStoreObject> objectsById = indexObjectsById(cell.objectGroups);
+ Set<ExGuid> visited = new HashSet<>();
+ // Only objects reachable from the root objects of the current revision are part of
+ // the current content. The object groups may also contain older, superseded versions
+ // of objects (under a different object ID); those are intentionally not walked.
+ // A blob-only root (no property set) cannot reach the page body, so it does not
+ // count as a resolved content root: if the content root declare dangles next to it,
+ // the walk-everything fallback must still fire or the page body is silently lost.
+ boolean resolvedContentRoot = false;
+ boolean unresolvedRoot = false;
+ for (RevisionManifestRootDeclare rootDeclare : cell.rootDeclares) {
+ RevisionStoreObject rootObject = objectsById.get(rootDeclare.objectExGuid);
+ if (rootObject == null) {
+ recordParseWarning("OneNote cell root object " + rootDeclare.objectExGuid +
+ " could not be resolved");
+ unresolvedRoot = true;
+ } else {
+ if (rootObject.propertySet != null) {
+ resolvedContentRoot = true;
+ }
+ walkObject(rootObject, objectsById, visited, AuthorRole.NONE, options,
+ metadata, xhtml, 0);
+ }
+ }
+ if (cell.rootDeclares.isEmpty() || (unresolvedRoot && !resolvedContentRoot)) {
+ if (cell.rootDeclares.isEmpty()) {
+ recordParseWarning("OneNote cell has no declared root objects; walking all objects");
+ } else {
+ recordParseWarning("OneNote cell root objects could not be resolved; walking all objects");
+ }
+ for (RevisionStoreObjectGroup objectGroup : cell.objectGroups) {
+ for (RevisionStoreObject object : objectGroup.objects) {
+ walkObject(object, objectsById, visited, AuthorRole.NONE, options, metadata,
+ xhtml, 0);
+ }
+ }
+ }
+ }
+
+ private String[] sortedValues(Set<String> values) {
+ String[] sorted = values.toArray(new String[0]);
+ Arrays.sort(sorted);
+ return sorted;
+ }
+
+ private boolean warningsSaturated() {
+ return parseWarningsSuppressed;
+ }
+
+ void recordParseWarning(String warning) {
+ recordParseWarning(ParseWarningKind.DEFAULT, warning);
+ }
+
+ private void recordParseWarning(ParseWarningKind kind, String warning) {
+ String warningKey = kind == ParseWarningKind.DEFAULT ? warning : kind.name();
+ if (recordedParseWarningKeys.contains(warningKey)) {
+ return;
+ }
+ if (recordedParseWarningKeys.size() >= MAX_PARSE_WARNINGS) {
+ if (!parseWarningsSuppressed) {
+ parseWarningsSuppressed = true;
+ emitParseWarning("Additional OneNote parse warnings were suppressed after " +
+ MAX_PARSE_WARNINGS + " distinct warnings");
+ }
+ return;
+ }
+ recordedParseWarningKeys.add(warningKey);
+ emitParseWarning(warning);
+ }
+
+ private void emitParseWarning(String warning) {
+ LOG.warn(warning);
+ if (parentMetadata == null) {
+ parseWarnings.add(warning);
+ } else {
+ parentMetadata.add(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING, warning);
+ }
+ }
+
+ /**
+ * Builds a map of object ID to object. The object groups are ordered from the oldest
+ * revision to the newest, so a newer version of an object wins over an older one.
+ */
+ private Map<ExGuid, RevisionStoreObject> indexObjectsById(
+ List<RevisionStoreObjectGroup> objectGroups) {
+ Map<ExGuid, RevisionStoreObject> objectsById = new HashMap<>();
+ for (RevisionStoreObjectGroup objectGroup : objectGroups) {
+ for (RevisionStoreObject object : objectGroup.objects) {
+ if (object.objectID != null) {
+ objectsById.put(object.objectID, object);
+ }
+ }
+ }
+ return objectsById;
+ }
+
+ private void walkObject(RevisionStoreObject object,
+ Map<ExGuid, RevisionStoreObject> objectsById, Set<ExGuid> visited,
+ AuthorRole authorRole, OneNoteTreeWalkerOptions options,
+ Metadata metadata, XHTMLContentHandler xhtml, int depth)
+ throws SAXException, TikaException, IOException {
+ walkObject(object, objectsById, visited, authorRole, options, metadata, xhtml,
+ depth, null);
+ }
+
+ private void walkObject(RevisionStoreObject object,
+ Map<ExGuid, RevisionStoreObject> objectsById, Set<ExGuid> visited,
+ AuthorRole authorRole, OneNoteTreeWalkerOptions options,
+ Metadata metadata, XHTMLContentHandler xhtml, int depth,
+ EmbeddedResourceInfo inheritedResourceInfo)
+ throws SAXException, TikaException, IOException {
+ if (object == null) {
+ return;
+ }
+ if (depth >= MAX_OBJECT_WALK_DEPTH) {
+ recordParseWarning("OneNote object traversal exceeded depth limit " +
+ MAX_OBJECT_WALK_DEPTH);
+ return;
+ }
+ if (object.objectID != null && !visited.add(object.objectID)) {
+ // one object may be referenced under several author roles (e.g. the same author
+ // as both AuthorOriginal and AuthorMostRecent) - record the role even though
+ // the subtree is not walked again
+ recordAuthors(object, authorRole);
+ return;
+ }
+ List<PropertyAction> actions = object.propertySet != null &&
+ object.propertySet.objectSpaceObjectPropSet != null ?
+ collectObjectActions(object) : Collections.emptyList();
+ EmbeddedResourceInfo resourceInfo = embeddedResourceInfo(actions);
+ if (resourceInfo == null) {
+ resourceInfo = inheritedResourceInfo;
+ }
+ if (resourceInfo == null && object.fileDataObject != null && object.objectID != null) {
+ resourceInfo = resourceInfoFromReferencingObject(object, objectsById, depth);
+ }
+ if (object.fileDataObject != null) {
+ // the object carries opaque binary data, e.g. an embedded image or file
+ handleEmbedded(object.fileDataObject.getData(), xhtml, resourceInfo, object.objectID);
+ }
+ if (object.propertySet == null ||
+ object.propertySet.objectSpaceObjectPropSet == null) {
+ return;
+ }
+ // An image node can reference the same picture twice: PictureContainer holds the
+ // canonical image data and WebPictureContainer14 holds a rendition derived from it
+ // (e.g. re-rendered when the picture was resized). Only emit the derived rendition
+ // when the canonical container is missing, so the picture is not extracted twice.
+ boolean hasPrimaryPicture = false;
+ for (PropertyAction action : actions) {
+ if (action.isChildReference && action.childReference != null &&
+ action.oneNotePropertyEnum == OneNotePropertyEnum.PictureContainer &&
+ hasUsablePicture(action.childReference, objectsById, new HashSet<>(), depth)) {
+ hasPrimaryPicture = true;
+ break;
+ }
+ }
+ // The title structure of a page (StructureElementChildNodes) appears above the page
+ // body on screen, but is declared after the body child nodes. Emit it first so the
+ // text comes out in visual order.
+ for (PropertyAction action : actions) {
+ if (action.oneNotePropertyEnum == OneNotePropertyEnum.StructureElementChildNodes) {
+ processAction(action, objectsById, visited, authorRole, options, metadata, xhtml,
+ depth, resourceInfo);
+ }
+ }
+ for (PropertyAction action : actions) {
+ if (hasPrimaryPicture &&
+ action.oneNotePropertyEnum == OneNotePropertyEnum.WebPictureContainer14) {
+ continue;
+ }
+ if (action.oneNotePropertyEnum != OneNotePropertyEnum.StructureElementChildNodes) {
+ processAction(action, objectsById, visited, authorRole, options, metadata, xhtml,
+ depth, resourceInfo);
+ }
+ }
+ }
+
+ private boolean hasUsablePicture(ExGuid objectId,
+ Map<ExGuid, RevisionStoreObject> objectsById,
+ Set<ExGuid> visited, int depth) {
+ if (depth >= MAX_OBJECT_WALK_DEPTH || objectId == null || !visited.add(objectId)) {
+ return false;
+ }
+ RevisionStoreObject object = objectsById.get(objectId);
+ if (object == null) {
+ return false;
+ }
+ byte[] data = object.fileDataObject == null ? null : object.fileDataObject.getData();
+ if (data != null && data.length > 0) {
+ return true;
+ }
+ if (object.propertySet == null || object.propertySet.objectSpaceObjectPropSet == null) {
+ return false;
+ }
+ for (PropertyAction action : collectObjectActions(object)) {
+ if (action.isChildReference && action.childReference != null &&
+ hasUsablePicture(action.childReference, objectsById, visited, depth + 1)) {
+ return true;
+ }
}
+ return false;
}
+ private EmbeddedResourceInfo resourceInfoFromReferencingObject(
+ RevisionStoreObject target, Map<ExGuid, RevisionStoreObject> objectsById, int depth) {
+ if (depth >= MAX_OBJECT_WALK_DEPTH) {
+ return null;
+ }
+ EmbeddedResourceInfo typeOnlyFallback = null;
+ for (RevisionStoreObject candidate : objectsById.values()) {
+ if (candidate.propertySet == null ||
+ candidate.propertySet.objectSpaceObjectPropSet == null) {
+ continue;
+ }
+ List<PropertyAction> candidateActions = collectObjectActions(candidate);
+ for (PropertyAction action : candidateActions) {
+ if (action.isChildReference && target.objectID.equals(action.childReference) &&
+ (action.oneNotePropertyEnum == OneNotePropertyEnum.PictureContainer ||
+ action.oneNotePropertyEnum == OneNotePropertyEnum.EmbeddedFileContainer ||
+ action.oneNotePropertyEnum == OneNotePropertyEnum.WebPictureContainer14)) {
+ EmbeddedResourceInfo info = embeddedResourceInfo(candidateActions);
+ if (info != null) {
+ return info;
+ }
+ if (typeOnlyFallback == null) {
+ typeOnlyFallback = resourceInfoForChild(action, null);
+ }
+ }
+ }
+ }
+ return typeOnlyFallback;
+ }
+
+ private EmbeddedResourceInfo resourceInfoForChild(PropertyAction action,
+ EmbeddedResourceInfo parentInfo) {
+ if (action.oneNotePropertyEnum == OneNotePropertyEnum.PictureContainer ||
+ action.oneNotePropertyEnum == OneNotePropertyEnum.WebPictureContainer14) {
+ return parentInfo == null ? new EmbeddedResourceInfo(null,
+ TikaCoreProperties.EmbeddedResourceType.INLINE.toString()) : parentInfo;
+ }
+ if (action.oneNotePropertyEnum == OneNotePropertyEnum.EmbeddedFileContainer) {
+ return parentInfo == null ? new EmbeddedResourceInfo(null,
+ TikaCoreProperties.EmbeddedResourceType.ATTACHMENT.toString()) : parentInfo;
+ }
+ return parentInfo;
+ }
+
+ /**
+ * Metadata used when emitting an embedded resource.
+ */
+ private static final class EmbeddedResourceInfo {
+ private final String name;
+ private final String type;
+
+ private EmbeddedResourceInfo(String name, String type) {
+ this.name = name;
+ this.type = type;
+ }
+ }
+
+ /**
+ * A property of an object, together with the object reference assigned to it if it is
+ * an object reference property.
+ */
+ private static final class PropertyAction {
+ private final IProperty property;
+ private final PropertyType propertyType;
+ private final OneNotePropertyEnum oneNotePropertyEnum;
+ private final boolean isChildReference;
+ private final ExGuid childReference;
+ private final CellID spaceReference;
+
+ PropertyAction(IProperty property, PropertyType propertyType,
+ OneNotePropertyEnum oneNotePropertyEnum, boolean isChildReference,
+ ExGuid childReference, CellID spaceReference) {
+ this.property = property;
+ this.propertyType = propertyType;
+ this.oneNotePropertyEnum = oneNotePropertyEnum;
+ this.isChildReference = isChildReference;
+ this.childReference = childReference;
+ this.spaceReference = spaceReference;
+ }
+ }
+
+ /**
+ * Flattens the properties of an object, in order, into a list of actions. The result is
+ * cached per object; callers must not mutate the returned list.
+ */
+ private List<PropertyAction> collectObjectActions(RevisionStoreObject object) {
+ List<PropertyAction> actions = objectActionsCache.get(object);
+ if (actions != null) {
+ return actions;
+ }
+ List<ExGuid> referencedObjects =
+ object.referencedObjectID == null || object.referencedObjectID.content == null ?
+ Collections.emptyList() : object.referencedObjectID.content;
+ List<CellID> referencedSpaces = object.referencedObjectSpacesID == null ||
+ object.referencedObjectSpacesID.content == null ? Collections.emptyList() :
+ object.referencedObjectSpacesID.content;
+ actions = new ArrayList<>();
+ collectActions(object.propertySet.objectSpaceObjectPropSet.body, referencedObjects,
+ new int[]{0}, referencedSpaces, new int[]{0}, actions, 0);
+ objectActionsCache.put(object, actions);
+ return actions;
+ }
+
+ /**
+ * Flattens the properties of a property set, in order, into a list of actions.
+ * Properties of type ObjectID or ArrayOfObjectIDs consume, in property order, the object
+ * references of the containing object, and properties of type ObjectSpaceID or
+ * ArrayOfObjectSpaceIDs consume the object space (cell) references (see MS-ONESTORE
+ * section 2.7.8), so the references must be assigned here, in property order, no matter
+ * in which order the actions are processed later.
+ */
+ private void collectActions(PropertySet propertySet, List<ExGuid> referencedObjects,
+ int[] referenceCursor, List<CellID> referencedSpaces,
+ int[] spaceCursor, List<PropertyAction> actions, int depth) {
+ if (propertySet == null || propertySet.rgPrids == null || propertySet.rgData == null) {
+ return;
+ }
+ if (depth >= MAX_OBJECT_WALK_DEPTH) {
+ recordParseWarning("OneNote property traversal exceeded depth limit " +
+ MAX_OBJECT_WALK_DEPTH);
+ return;
+ }
+ for (int i = 0; i < propertySet.rgPrids.length && i < propertySet.rgData.size(); ++i) {
+ IProperty property = propertySet.rgData.get(i);
+ PropertyID propertyID = propertySet.rgPrids[i];
+ PropertyType propertyType = PropertyType.fromIntVal(propertyID.type);
+ OneNotePropertyEnum oneNotePropertyEnum =
+ OneNotePropertyEnum.of(Unsigned.uint(propertyID.value).longValue());
+ if (propertyType == PropertyType.ObjectID) {
+ ExGuid childReference = nextReference(referencedObjects, referenceCursor);
+ if (childReference == null) {
+ recordParseWarning("OneNote object reference slot was exhausted");
+ }
+ actions.add(new PropertyAction(property, propertyType, oneNotePropertyEnum, true,
+ childReference, null));
+ } else if (propertyType == PropertyType.ArrayOfObjectIDs) {
+ int available = referencedObjects.size() - referenceCursor[0];
+ int requestedCount = property instanceof ArrayNumber ?
+ Math.max(0, ((ArrayNumber) property).number) : 0;
+ int declaredCount = boundedAvailableReferenceCount(requestedCount, available);
+ int count = Math.min(declaredCount, MAX_REFERENCE_COUNT);
+ if (requestedCount > available && !warningsSaturated()) {
+ recordParseWarning(ParseWarningKind.OBJECT_REFERENCE_ARRAY_UNAVAILABLE,
+ "OneNote object reference array had unavailable entries " +
+ "(first occurrence: declared " + requestedCount +
+ " entries but only " + Math.max(available, 0) +
+ " were available)");
+ }
+ if (declaredCount > count && !warningsSaturated()) {
+ recordParseWarning(ParseWarningKind.OBJECT_REFERENCE_ARRAY_CAPPED,
+ "Capping OneNote object reference array at " +
+ MAX_REFERENCE_COUNT + " entries");
+ }
+ for (int j = 0; j < count; ++j) {
+ actions.add(new PropertyAction(property, propertyType, oneNotePropertyEnum,
+ true, nextReference(referencedObjects, referenceCursor), null));
+ }
+ referenceCursor[0] += declaredCount - count;
+ } else if (propertyType == PropertyType.ObjectSpaceID) {
+ CellID spaceReference = nextSpaceReference(referencedSpaces, spaceCursor);
+ if (spaceReference == null) {
+ recordParseWarning("OneNote object-space reference slot was exhausted");
+ }
+ actions.add(new PropertyAction(property, propertyType, oneNotePropertyEnum, false,
+ null, spaceReference));
+ } else if (propertyType == PropertyType.ArrayOfObjectSpaceIDs) {
+ int available = referencedSpaces.size() - spaceCursor[0];
+ int requestedCount = property instanceof ArrayNumber ?
+ Math.max(0, ((ArrayNumber) property).number) : 0;
+ int declaredCount = boundedAvailableReferenceCount(requestedCount, available);
+ int count = Math.min(declaredCount, MAX_REFERENCE_COUNT);
+ if (requestedCount > available && !warningsSaturated()) {
+ recordParseWarning(ParseWarningKind.OBJECT_SPACE_REFERENCE_ARRAY_UNAVAILABLE,
+ "OneNote object-space reference array had unavailable entries " +
+ "(first occurrence: declared " + requestedCount +
+ " entries but only " + Math.max(available, 0) +
+ " were available)");
+ }
+ if (declaredCount > count && !warningsSaturated()) {
+ recordParseWarning(ParseWarningKind.OBJECT_SPACE_REFERENCE_ARRAY_CAPPED,
+ "Capping OneNote object-space reference array at " +
+ MAX_REFERENCE_COUNT + " entries");
+ }
+ for (int j = 0; j < count; ++j) {
+ actions.add(new PropertyAction(property, propertyType, oneNotePropertyEnum,
+ false, null, nextSpaceReference(referencedSpaces, spaceCursor)));
+ }
+ spaceCursor[0] += declaredCount - count;
+ } else if (propertyType == PropertyType.PropertySet) {
+ if (property instanceof PropertySet) {
+ collectActions((PropertySet) property, referencedObjects, referenceCursor,
+ referencedSpaces, spaceCursor, actions, depth + 1);
+ }
+ } else if (propertyType == PropertyType.ArrayOfPropertyValues) {
+ if (property instanceof PrtArrayOfPropertyValues &&
+ ((PrtArrayOfPropertyValues) property).data != null) {
+ for (PropertySet nested : ((PrtArrayOfPropertyValues) property).data) {
+ collectActions(nested, referencedObjects, referenceCursor,
+ referencedSpaces, spaceCursor, actions, depth + 1);
+ }
+ }
+ } else {
+ actions.add(new PropertyAction(property, propertyType, oneNotePropertyEnum, false,
+ null, null));
+ }
+ }
+ }
+
+ private void processAction(PropertyAction action,
+ Map<ExGuid, RevisionStoreObject> objectsById, Set<ExGuid> visited,
+ AuthorRole authorRole, OneNoteTreeWalkerOptions options,
+ Metadata metadata, XHTMLContentHandler xhtml, int depth,
+ EmbeddedResourceInfo parentResourceInfo)
+ throws SAXException, TikaException, IOException {
+ if (action.spaceReference != null) {
+ // a reference to another object space (cell) - cells are walked separately
+ return;
+ }
+ if (action.isChildReference) {
+ AuthorRole childRole = AuthorRole.NONE;
+ if (action.oneNotePropertyEnum == OneNotePropertyEnum.AuthorMostRecent) {
+ childRole = AuthorRole.MOST_RECENT;
+ } else if (action.oneNotePropertyEnum == OneNotePropertyEnum.AuthorOriginal) {
+ childRole = AuthorRole.ORIGINAL;
+ }
+ EmbeddedResourceInfo childResourceInfo = resourceInfoForChild(action,
+ parentResourceInfo);
+ RevisionStoreObject child = action.childReference == null ? null :
+ objectsById.get(action.childReference);
+ if (child == null && action.childReference != null && !warningsSaturated()) {
+ recordParseWarning("OneNote object " + action.childReference +
+ " could not be resolved");
+ }
+ walkObject(child, objectsById, visited, childRole, options, metadata, xhtml,
+ depth + 1, childResourceInfo);
+ } else {
+ processPrimitiveProperty(action.property, action.propertyType,
+ action.oneNotePropertyEnum, authorRole, options, metadata, xhtml);
+ }
+ }
+
+ private int boundedAvailableReferenceCount(int declaredCount, int remainingReferences) {
+ if (declaredCount <= 0 || remainingReferences <= 0) {
+ return 0;
+ }
+ return Math.min(declaredCount, remainingReferences);
+ }
+
+ private ExGuid nextReference(List<ExGuid> referencedObjects, int[] referenceCursor) {
+ if (referenceCursor[0] < referencedObjects.size()) {
+ return referencedObjects.get(referenceCursor[0]++);
+ }
+ return null;
+ }
+
+ private CellID nextSpaceReference(List<CellID> referencedSpaces, int[] spaceCursor) {
+ if (spaceCursor[0] < referencedSpaces.size()) {
+ return referencedSpaces.get(spaceCursor[0]++);
+ }
+ return null;
+ }
+
+ private void processPrimitiveProperty(IProperty property, PropertyType propertyType,
+ OneNotePropertyEnum oneNotePropertyEnum,
+ AuthorRole authorRole,
+ OneNoteTreeWalkerOptions options, Metadata metadata,
+ XHTMLContentHandler xhtml)
+ throws SAXException, TikaException, IOException {
+ if (oneNotePropertyEnum == OneNotePropertyEnum.LastModifiedTimeStamp) {
+ long fullval = getScalar(property);
+ Instant instant = Instant.ofEpochSecond(
+ fullval / 10000000 + DATETIME_EPOCH_DIFF_1601);
+ if (instant.isAfter(lastModifiedTimestamp)) {
+ lastModifiedTimestamp = instant;
+ }
+ metadata.set(ONE_NOTE_PREFIX + "lastModifiedTimestamp",
+ String.valueOf(lastModifiedTimestamp.toEpochMilli()));
+ } else if (oneNotePropertyEnum == OneNotePropertyEnum.CreationTimeStamp) {
+ // add the TIME32_EPOCH_DIFF_1980 because OneNote TIME32 epoch time is per 1980, not
+ // 1970
+ long scalar = getScalar(property);
+ long creationTs = scalar + TIME32_EPOCH_DIFF_1980;
+ if (creationTs < creationTimestamp) {
+ creationTimestamp = creationTs;
+ }
+ metadata.set(ONE_NOTE_PREFIX + "creationTimestamp", String.valueOf(creationTimestamp));
+ } else if (oneNotePropertyEnum == OneNotePropertyEnum.LastModifiedTime) {
+ // add the TIME32_EPOCH_DIFF_1980 because OneNote TIME32 epoch time is per 1980, not
+ // 1970
+ long scalar = getScalar(property);
+ long lastMod = scalar + TIME32_EPOCH_DIFF_1980;
+ if (lastMod > lastModified) {
+ lastModified = lastMod;
+ }
+ metadata.set(TikaCoreProperties.MODIFIED, String.valueOf(lastModified));
+ } else if (oneNotePropertyEnum == OneNotePropertyEnum.Author) {
+ recordAuthor(decodeOneNoteText(
+ ((PrtFourBytesOfLengthFollowedByData) property).data), authorRole);
+ } else if (propertyType == PropertyType.FourBytesOfLengthFollowedByData) {
+ boolean isBinary = propertyIsBinary(oneNotePropertyEnum);
+ PrtFourBytesOfLengthFollowedByData dataProperty =
+ (PrtFourBytesOfLengthFollowedByData) property;
+ if ((dataProperty.data.length & 1) == 0 &&
+ oneNotePropertyEnum != OneNotePropertyEnum.TextExtendedAscii && !isBinary) {
+ if (options.getUtf16PropertiesToPrint().contains(oneNotePropertyEnum)) {
+ emitParagraph(xhtml, new String(dataProperty.data,
+ StandardCharsets.UTF_16LE));
+ }
+ } else if (oneNotePropertyEnum == OneNotePropertyEnum.TextExtendedAscii) {
+ emitParagraph(xhtml, new String(dataProperty.data, StandardCharsets.US_ASCII));
+ } else if (!isBinary) {
+ if (options.getUtf16PropertiesToPrint().contains(oneNotePropertyEnum)) {
+ emitParagraph(xhtml, new String(dataProperty.data,
+ StandardCharsets.UTF_16LE));
+ }
+ } else {
+ if (oneNotePropertyEnum == OneNotePropertyEnum.RichEditTextUnicode) {
+ handleRichEditTextUnicode(dataProperty.data, xhtml);
+ } else {
+ //TODO -- these seem to be somewhat broken font files and other
+ //odds and ends...what are they and how should we process them?
+ //handleEmbedded(content.size());
+ }
+ }
+ }
+ }
+
+
+ private String decodeOneNoteText(byte[] bytes) {
+ return new String(bytes, StandardCharsets.UTF_16LE).replace("\u0000", "");
+ }
+
+ private void recordAuthor(String author, AuthorRole role) {
+ if (role == AuthorRole.MOST_RECENT) {
+ mostRecentAuthors.add(author);
+ } else if (role == AuthorRole.ORIGINAL) {
+ originalAuthors.add(author);
+ // the original authors are the creators of the content
+ authors.add(author);
+ } else {
+ authors.add(author);
+ }
+ }
+
+ /**
+ * Records the Author properties of an already-visited object under the given role.
+ */
+ private void recordAuthors(RevisionStoreObject object, AuthorRole role) {
+ // NONE also records (into authors) so the result is visit-order independent
+ if (object.propertySet == null ||
+ object.propertySet.objectSpaceObjectPropSet == null) {
+ return;
+ }
+ for (PropertyAction action : collectObjectActions(object)) {
+ if (action.oneNotePropertyEnum == OneNotePropertyEnum.Author &&
+ action.property instanceof PrtFourBytesOfLengthFollowedByData) {
+ recordAuthor(decodeOneNoteText(
+ ((PrtFourBytesOfLengthFollowedByData) action.property).data), role);
+ }
+ }
+ }
+
+ private EmbeddedResourceInfo embeddedResourceInfo(List<PropertyAction> actions) {
+ for (PropertyAction action : actions) {
+ if (action.property instanceof PrtFourBytesOfLengthFollowedByData &&
+ (action.oneNotePropertyEnum == OneNotePropertyEnum.ImageFilename ||
+ action.oneNotePropertyEnum == OneNotePropertyEnum.EmbeddedFileName)) {
+ byte[] bytes = ((PrtFourBytesOfLengthFollowedByData) action.property).data;
+ String name = sanitizeResourceName(new String(bytes, StandardCharsets.UTF_16LE)
+ .replace("\u0000", ""));
+ if (!name.isEmpty()) {
+ String type = action.oneNotePropertyEnum == OneNotePropertyEnum.ImageFilename ?
+ TikaCoreProperties.EmbeddedResourceType.INLINE.toString() :
+ TikaCoreProperties.EmbeddedResourceType.ATTACHMENT.toString();
+ return new EmbeddedResourceInfo(name, type);
+ }
+ }
+ }
+ return null;
+ }
+
+ static String sanitizeResourceName(String name) {
+ name = name.replace('\\', '/');
+ // a drive-relative name like C:pic.png has no slash, so strip the prefix separately
+ if (name.length() > 1 && name.charAt(1) == ':') {
+ name = name.substring(2);
+ }
+ int slash = name.lastIndexOf('/');
+ name = slash >= 0 ? name.substring(slash + 1) : name;
+ return ".".equals(name) || "..".equals(name) ? "" : name;
+ }
+
+ /**
+ * Hands the binary data of an embedded object (e.g. an image or an attached file) to the
+ * embedded document extractor.
+ */
+ private void handleEmbedded(byte[] data, XHTMLContentHandler xhtml,
+ EmbeddedResourceInfo resourceInfo, ExGuid objectID)
+ throws SAXException, IOException {
+ if (data == null || data.length == 0 || embeddedDocumentExtractor == null) {
+ return;
+ }
+ Metadata embeddedMetadata = new Metadata();
+ embeddedMetadata.set(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE,
+ resourceInfo == null ? TikaCoreProperties.EmbeddedResourceType.ATTACHMENT.toString() :
+ resourceInfo.type);
+ if (resourceInfo != null && resourceInfo.name != null) {
+ embeddedMetadata.set(TikaCoreProperties.RESOURCE_NAME_KEY, resourceInfo.name);
+ }
+ String relationshipID = objectID == null ? null : "_" + objectID.value + "_" +
+ objectID.guid;
+ if (relationshipID != null) {
+ embeddedMetadata.set(TikaCoreProperties.EMBEDDED_RELATIONSHIP_ID, relationshipID);
+ }
+ AttributesImpl attributes = new AttributesImpl();
+ attributes.addAttribute("", "class", "class", "CDATA", "embedded");
+ if (relationshipID != null) {
+ attributes.addAttribute("", "id", "id", "CDATA", relationshipID);
+ }
+ xhtml.startElement("div", attributes);
+ xhtml.endElement("div");
+ try (TikaInputStream tis = TikaInputStream.get(data)) {
+ if (embeddedDocumentExtractor.shouldParseEmbedded(embeddedMetadata)) {
+ // only counts as content when the extractor accepts it, so a declined
+ // embedded object in an otherwise-empty file still triggers the fallback
+ contentEmitted = true;
+ embeddedDocumentExtractor.parseEmbedded(tis, new EmbeddedContentHandler(xhtml),
+ embeddedMetadata, false);
+ }
+ } catch (IOException e) {
+ EmbeddedDocumentUtil.recordEmbeddedStreamException(e, parentMetadata);
+ }
+ }
private void handleRichEditTextUnicode(byte[] arr, XHTMLContentHandler xhtml)
throws SAXException, IOException, TikaException {
@@ -285,13 +1032,37 @@ public class MSOneStorePackage {
Matcher m = HYPERLINK_PATTERN.matcher(txt);
if (m.find()) {
xhtml.startElement("a", "href", m.group(1));
- xhtml.characters(m.group(2));
- xhtml.endElement("a");
+ try {
+ xhtml.characters(m.group(2));
+ } finally {
+ xhtml.endElement("a");
+ }
+ contentEmitted = true;
} else {
- xhtml.startElement(P);
- xhtml.characters(txt);
+ emitParagraph(xhtml, txt);
+ }
+ }
+
+ private void emitParagraph(XHTMLContentHandler xhtml, String text) throws SAXException {
+ xhtml.startElement(P);
+ try {
+ xhtml.characters(text);
+ } finally {
xhtml.endElement(P);
}
+ // blank text does not count as content, so it cannot suppress the legacy fallback
+ if (!text.isBlank()) {
+ contentEmitted = true;
+ }
+ }
+
+ /**
+ * Whether the tree walk emitted any content - text or an embedded object. When it did
+ * not, the caller can fall back to the legacy string dump so a degraded file still
+ * yields its text.
+ */
+ public boolean hasEmittedContent() {
+ return contentEmitted;
}
private long getScalar(IProperty property) throws TikaException, IOException {
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreParser.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreParser.java
index 25790ead8b..d4a41b43d0 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreParser.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreParser.java
@@ -17,9 +17,15 @@
package org.apache.tika.parser.microsoft.onenote.fsshttpb;
import java.io.IOException;
+import java.util.ArrayDeque;
import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashMap;
import java.util.HashSet;
+import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Map;
+import java.util.Objects;
import java.util.Set;
import java.util.UUID;
import java.util.stream.Collectors;
@@ -31,6 +37,8 @@ import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.ObjectGroupDa
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionManifestDataElementData;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionManifestObjectGroupReferences;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionManifestRootDeclare;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionStoreCell;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionStoreObject;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionStoreObjectGroup;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.StorageIndexCellMapping;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.StorageIndexDataElementData;
@@ -40,8 +48,14 @@ import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.CellID;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.DataElementType;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.ExGuid;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.HeaderCell;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.util.GuidUtil;
public class MSOneStoreParser {
+ /**
+ * The root role declaration used for the encryption key of encrypted sections.
+ */
+ private static final ExGuid ENCRYPTION_KEY_ROOT_EXGUID =
+ new ExGuid(3, UUID.fromString("4A3717F8-1C14-49E7-9526-81D942DE1741"));
private final Set<CellID> storageIndexHashTab = new HashSet<>();
// The DataElements of Storage Index
private List<DataElement> storageIndexDataElements;
@@ -49,15 +63,47 @@ public class MSOneStoreParser {
private List<DataElement> storageManifestDataElements;
// The DataElements of Cell Manifest
private List<DataElement> cellManifestDataElements;
+ private Map<ExGuid, DataElement> cellManifestDataElementsById = new HashMap<>();
// The DataElements of Revision Manifest
private List<DataElement> revisionManifestDataElements;
+ private Map<ExGuid, DataElement> revisionManifestDataElementsById = new HashMap<>();
// The DataElements of Object Group Data
private List<DataElement> objectGroupDataElements;
- // The DataElements of Object BLOB
- private List<DataElement> objectBlOBElements;
+ // The DataElements of Object Group Data, keyed by their data element extended GUID
+ private Map<ExGuid, DataElement> objectGroupDataElementsById;
+ private final Map<ObjectGroupCacheKey, RevisionStoreObjectGroup> objectGroupsByDataElementId =
+ new HashMap<>();
+ // The DataElements of Object BLOB, keyed by their data element extended GUID
+ private Map<ExGuid, DataElement> objectBlobElementsById;
+
+ private static final class ObjectGroupCacheKey {
+ private final ExGuid dataElementId;
+ private final boolean encryption;
+
+ private ObjectGroupCacheKey(ExGuid dataElementId, boolean encryption) {
+ this.dataElementId = dataElementId;
+ this.encryption = encryption;
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ if (!(other instanceof ObjectGroupCacheKey)) {
+ return false;
+ }
+ ObjectGroupCacheKey that = (ObjectGroupCacheKey) other;
+ return encryption == that.encryption && Objects.equals(dataElementId, that.dataElementId);
+ }
+
+ @Override
+ public int hashCode() {
+ return 31 * Objects.hashCode(dataElementId) + Boolean.hashCode(encryption);
+ }
+ }
public MSOneStorePackage parse(DataElementPackage dataElementPackage) throws IOException {
+ storageIndexHashTab.clear();
MSOneStorePackage msOneStorePackage = new MSOneStorePackage();
+ objectGroupsByDataElementId.clear();
storageIndexDataElements = dataElementPackage.dataElements.stream()
.filter(d -> d.dataElementType == DataElementType.StorageIndexDataElementData)
@@ -68,20 +114,48 @@ public class MSOneStoreParser {
cellManifestDataElements = dataElementPackage.dataElements.stream()
.filter(d -> d.dataElementType == DataElementType.CellManifestDataElementData)
.collect(Collectors.toList());
+ cellManifestDataElementsById.clear();
+ for (DataElement element : cellManifestDataElements) {
+ cellManifestDataElementsById.putIfAbsent(element.dataElementExGuid, element);
+ }
revisionManifestDataElements = dataElementPackage.dataElements.stream()
.filter(d -> d.dataElementType == DataElementType.RevisionManifestDataElementData)
.collect(Collectors.toList());
+ revisionManifestDataElementsById.clear();
+ for (DataElement element : revisionManifestDataElements) {
+ revisionManifestDataElementsById.putIfAbsent(element.dataElementExGuid, element);
+ }
objectGroupDataElements = dataElementPackage.dataElements.stream()
.filter(d -> d.dataElementType == DataElementType.ObjectGroupDataElementData)
.collect(Collectors.toList());
- objectBlOBElements = dataElementPackage.dataElements.stream()
+ objectGroupDataElementsById = new HashMap<>();
+ for (DataElement objectGroupElement : objectGroupDataElements) {
+ objectGroupDataElementsById.putIfAbsent(objectGroupElement.dataElementExGuid,
+ objectGroupElement);
+ }
+ List<DataElement> objectBlobElements = dataElementPackage.dataElements.stream()
.filter(d -> d.dataElementType == DataElementType.ObjectDataBLOBDataElementData)
.collect(Collectors.toList());
+ objectBlobElementsById = new HashMap<>();
+ for (DataElement blobElement : objectBlobElements) {
+ objectBlobElementsById.putIfAbsent(blobElement.dataElementExGuid, blobElement);
+ }
+ if (storageIndexDataElements.isEmpty() || storageManifestDataElements.isEmpty()) {
+ throw new IOException("Missing OneNote storage index or storage manifest");
+ }
msOneStorePackage.storageIndex =
(StorageIndexDataElementData) storageIndexDataElements.get(0).data;
msOneStorePackage.storageManifest =
(StorageManifestDataElementData) storageManifestDataElements.get(0).data;
+ if (msOneStorePackage.storageIndex == null ||
+ msOneStorePackage.storageManifest == null) {
+ throw new IOException("Invalid OneNote storage index or storage manifest");
+ }
+ if (msOneStorePackage.storageManifest.storageManifestRootDeclareList == null ||
+ msOneStorePackage.storageManifest.storageManifestRootDeclareList.size() < 2) {
+ throw new IOException("Missing OneNote header or data-root declaration");
+ }
// Parse Header Cell
CellID headerCellID =
@@ -93,31 +167,50 @@ public class MSOneStoreParser {
if (headerCellStorageIndexCellMapping != null) {
msOneStorePackage.headerCellCellManifest =
this.findCellManifest(headerCellStorageIndexCellMapping.cellMappingExGuid);
- StorageIndexRevisionMapping headerCellRevisionManifestMapping =
- msOneStorePackage.findStorageIndexRevisionMapping(
- msOneStorePackage.headerCellCellManifest.cellManifestCurrentRevision
- .cellManifestCurrentRevisionExGuid);
- msOneStorePackage.headerCellRevisionManifest = this.findRevisionManifestDataElement(
- headerCellRevisionManifestMapping.revisionMappingExGuid);
- msOneStorePackage.headerCell =
- this.parseHeaderCell(msOneStorePackage.headerCellRevisionManifest);
-
- // Parse Data root
- CellID dataRootCellID =
- msOneStorePackage.storageManifest.storageManifestRootDeclareList.get(1).cellID;
- storageIndexHashTab.add(dataRootCellID);
- msOneStorePackage.dataRoot = this.parseObjectGroup(dataRootCellID, msOneStorePackage);
- // Parse other data
- for (StorageIndexCellMapping storageIndexCellMapping : msOneStorePackage.storageIndex
- .storageIndexCellMappingList) {
- if (!storageIndexHashTab.contains(storageIndexCellMapping.cellID)) {
- msOneStorePackage.OtherFileNodeList.addAll(
- this.parseObjectGroup(storageIndexCellMapping.cellID,
- msOneStorePackage));
- storageIndexHashTab.add(storageIndexCellMapping.cellID);
+ if (msOneStorePackage.headerCellCellManifest != null &&
+ msOneStorePackage.headerCellCellManifest.cellManifestCurrentRevision != null) {
+ StorageIndexRevisionMapping headerCellRevisionManifestMapping =
+ msOneStorePackage.findStorageIndexRevisionMapping(
+ msOneStorePackage.headerCellCellManifest.cellManifestCurrentRevision
+ .cellManifestCurrentRevisionExGuid);
+ if (headerCellRevisionManifestMapping != null) {
+ msOneStorePackage.headerCellRevisionManifest =
+ this.findRevisionManifestDataElement(
+ headerCellRevisionManifestMapping.revisionMappingExGuid);
+ if (msOneStorePackage.headerCellRevisionManifest != null) {
+ msOneStorePackage.headerCell =
+ this.parseHeaderCell(msOneStorePackage.headerCellRevisionManifest);
+ }
}
}
}
+
+ // Parse Data root independently of the header-cell metadata. A malformed header cell
+ // should not prevent valid section cells from being parsed.
+ CellID dataRootCellID =
+ msOneStorePackage.storageManifest.storageManifestRootDeclareList.get(1).cellID;
+ storageIndexHashTab.add(dataRootCellID);
+ RevisionStoreCell dataRootCell = this.parseCell(dataRootCellID, msOneStorePackage);
+ if (dataRootCell == null) {
+ throw new IOException("Could not parse the OneNote data root cell");
+ }
+ msOneStorePackage.dataRootCell = dataRootCell;
+ // Parse other data
+ for (StorageIndexCellMapping storageIndexCellMapping : msOneStorePackage.storageIndex
+ .storageIndexCellMappingList) {
+ if (!storageIndexHashTab.contains(storageIndexCellMapping.cellID)) {
+ RevisionStoreCell cell =
+ this.parseCell(storageIndexCellMapping.cellID, msOneStorePackage);
+ // The storage index can retain a mapping for a deleted version context.
+ // Such an entry has no CellManifestDataElementData (often its mapping GUID
+ // is all zero) and therefore cannot contain current document content.
+ if (cell != null) {
+ msOneStorePackage.OtherFileNodeList.addAll(cell.objectGroups);
+ msOneStorePackage.cells.add(cell);
+ }
+ storageIndexHashTab.add(storageIndexCellMapping.cellID);
+ }
+ }
return msOneStorePackage;
}
@@ -125,74 +218,187 @@ public class MSOneStoreParser {
* Find the CellManifestDataElementData
*
* @param cellMappingExtendedGUID The ExGuid of Cell Mapping Extended GUID.
- * @return The CellManifestDataElementData instance.
+ * @return The CellManifestDataElementData instance, or {@code null} if it is absent.
*/
private CellManifestDataElementData findCellManifest(ExGuid cellMappingExtendedGUID) {
- return (CellManifestDataElementData) this.cellManifestDataElements.stream()
- .filter(d -> d.dataElementExGuid.equals(cellMappingExtendedGUID)).findFirst()
- .orElse(new DataElement()).data;
+ DataElement element = cellManifestDataElementsById.get(cellMappingExtendedGUID);
+ return element == null ? null : (CellManifestDataElementData) element.data;
}
/**
* Find the Revision Manifest from Data Elements.
*
* @param revisionMappingExtendedGUID The Revision Mapping Extended GUID.
- * @return Returns the instance of RevisionManifestDataElementData
+ * @return The RevisionManifestDataElementData instance, or {@code null} if it is absent.
*/
private RevisionManifestDataElementData findRevisionManifestDataElement(
ExGuid revisionMappingExtendedGUID) {
- return (RevisionManifestDataElementData) this.revisionManifestDataElements.stream()
- .filter(d -> d.dataElementExGuid.equals(revisionMappingExtendedGUID)).findFirst()
- .orElse(new DataElement()).data;
+ DataElement element = revisionManifestDataElementsById.get(revisionMappingExtendedGUID);
+ return element == null ? null : (RevisionManifestDataElementData) element.data;
}
private HeaderCell parseHeaderCell(RevisionManifestDataElementData headerCellRevisionManifest)
throws IOException {
+ if (headerCellRevisionManifest == null ||
+ headerCellRevisionManifest.revisionManifestObjectGroupReferences == null ||
+ headerCellRevisionManifest.revisionManifestObjectGroupReferences.isEmpty()) {
+ return null;
+ }
ExGuid rootObjectId =
headerCellRevisionManifest.revisionManifestObjectGroupReferences.get(
0).objectGroupExtendedGUID;
- DataElement element = this.objectGroupDataElements.stream()
- .filter(d -> d.dataElementExGuid.equals(rootObjectId)).findFirst()
- .orElse(new DataElement());
-
+ DataElement element = objectGroupDataElementsById.get(rootObjectId);
+ if (element == null) {
+ return null;
+ }
return HeaderCell.createInstance((ObjectGroupDataElementData) element.data);
}
- private List<RevisionStoreObjectGroup> parseObjectGroup(CellID objectGroupCellID,
- MSOneStorePackage msOneStorePackage)
+ private RevisionStoreCell parseCell(CellID objectGroupCellID,
+ MSOneStorePackage msOneStorePackage)
throws IOException {
StorageIndexCellMapping storageIndexCellMapping =
msOneStorePackage.findStorageIndexCellMapping(objectGroupCellID);
+ if (storageIndexCellMapping == null) {
+ String warning = "Skipping OneNote cell " + objectGroupCellID +
+ ": no storage-index cell mapping";
+ msOneStorePackage.recordParseWarning(warning);
+ return null;
+ }
CellManifestDataElementData cellManifest =
this.findCellManifest(storageIndexCellMapping.cellMappingExGuid);
+ if (cellManifest == null || cellManifest.cellManifestCurrentRevision == null) {
+ String warning = "Skipping OneNote cell " + objectGroupCellID +
+ ": no current cell manifest";
+ msOneStorePackage.recordParseWarning(warning);
+ return null;
+ }
List<RevisionStoreObjectGroup> objectGroups = new ArrayList<>();
msOneStorePackage.cellManifests.add(cellManifest);
StorageIndexRevisionMapping revisionMapping =
msOneStorePackage.findStorageIndexRevisionMapping(
cellManifest.cellManifestCurrentRevision.cellManifestCurrentRevisionExGuid);
+ if (revisionMapping == null) {
+ String warning = "Skipping OneNote cell " + objectGroupCellID +
+ ": no revision mapping";
+ msOneStorePackage.recordParseWarning(warning);
+ return null;
+ }
RevisionManifestDataElementData revisionManifest =
findRevisionManifestDataElement(revisionMapping.revisionMappingExGuid);
- msOneStorePackage.revisionManifests.add(revisionManifest);
- RevisionManifestRootDeclare encryptionKeyRoot =
- revisionManifest.revisionManifestRootDeclareList.stream()
- .filter(r -> r.rootExGuid.equals(new ExGuid(3,
- UUID.fromString("4A3717F8-1C14-49E7-9526-81D942DE1741"))))
- .findFirst().orElse(null);
- boolean isEncryption = encryptionKeyRoot != null;
- for (RevisionManifestObjectGroupReferences objRef :
- revisionManifest.revisionManifestObjectGroupReferences) {
- ObjectGroupDataElementData dataObject =
- (ObjectGroupDataElementData) objectGroupDataElements.stream()
- .filter(d -> d.dataElementExGuid.equals(objRef.objectGroupExtendedGUID))
- .findFirst().get().data;
-
- RevisionStoreObjectGroup objectGroup =
- RevisionStoreObjectGroup.createInstance(objRef.objectGroupExtendedGUID,
- dataObject, isEncryption);
- objectGroups.add(objectGroup);
- }
-
- return objectGroups;
+ if (revisionManifest == null || revisionManifest.revisionManifest == null) {
+ String warning = "Skipping OneNote cell " + objectGroupCellID +
+ ": no revision manifest";
+ msOneStorePackage.recordParseWarning(warning);
+ return null;
+ }
+
+ // A revision manifest may only reference the object groups that were created or
+ // modified in that revision. The remaining object groups belong to the chain of
+ // base revisions (MS-FSSHTTPB "Base Revision ID"). Follow that chain and collect
+ // the object groups of every revision, oldest revision first, so no content is lost.
+ Deque<RevisionManifestDataElementData> revisionChain = new ArrayDeque<>();
+ Set<ExGuid> seenRevisionIds = new HashSet<>();
+ while (revisionManifest != null) {
+ ExGuid revisionId = revisionManifest.revisionManifest.revisionID;
+ if (revisionId != null && !seenRevisionIds.add(revisionId)) {
+ // cycle guard - stop if we have already visited this revision
+ break;
+ }
+ revisionChain.addFirst(revisionManifest);
+ ExGuid baseRevisionId = revisionManifest.revisionManifest.baseRevisionID;
+ if (baseRevisionId == null || baseRevisionId.guid == null ||
+ GuidUtil.emptyGuid().equals(baseRevisionId.guid)) {
+ break;
+ }
+ StorageIndexRevisionMapping baseRevisionMapping =
+ msOneStorePackage.findStorageIndexRevisionMapping(baseRevisionId);
+ revisionManifest = baseRevisionMapping == null ? null :
+ findRevisionManifestDataElement(baseRevisionMapping.revisionMappingExGuid);
+ }
+
+ Set<ExGuid> seenObjectGroupIds = new HashSet<>();
+ // for each root role, the declaration made by the most recent revision wins
+ Map<ExGuid, RevisionManifestRootDeclare> effectiveRootDeclares = new LinkedHashMap<>();
+ for (RevisionManifestDataElementData manifest : revisionChain) {
+ msOneStorePackage.revisionManifests.add(manifest);
+ RevisionManifestRootDeclare encryptionKeyRoot =
+ manifest.revisionManifestRootDeclareList.stream()
+ .filter(r -> r.rootExGuid.equals(ENCRYPTION_KEY_ROOT_EXGUID))
+ .findFirst().orElse(null);
+ boolean isEncryption = encryptionKeyRoot != null;
+ for (RevisionManifestRootDeclare rootDeclare :
+ manifest.revisionManifestRootDeclareList) {
+ if (!rootDeclare.rootExGuid.equals(ENCRYPTION_KEY_ROOT_EXGUID)) {
+ effectiveRootDeclares.put(rootDeclare.rootExGuid, rootDeclare);
+ }
+ }
+ for (RevisionManifestObjectGroupReferences objRef :
+ manifest.revisionManifestObjectGroupReferences) {
+ if (!seenObjectGroupIds.add(objRef.objectGroupExtendedGUID)) {
+ continue;
+ }
+ DataElement dataElement =
+ objectGroupDataElementsById.get(objRef.objectGroupExtendedGUID);
+ if (dataElement == null) {
+ msOneStorePackage.recordParseWarning(
+ "OneNote object group " + objRef.objectGroupExtendedGUID +
+ " could not be resolved");
+ continue;
+ }
+ ObjectGroupDataElementData dataObject =
+ (ObjectGroupDataElementData) dataElement.data;
+
+ ObjectGroupCacheKey cacheKey =
+ new ObjectGroupCacheKey(dataElement.dataElementExGuid, isEncryption);
+ RevisionStoreObjectGroup cachedObjectGroup = objectGroupsByDataElementId.get(cacheKey);
+ if (cachedObjectGroup == null) {
+ cachedObjectGroup = RevisionStoreObjectGroup.createInstance(
+ objRef.objectGroupExtendedGUID, dataObject, isEncryption,
+ objectBlobElementsById);
+ objectGroupsByDataElementId.put(cacheKey, cachedObjectGroup);
+ }
+ objectGroups.add(cachedObjectGroup.copy());
+ }
+ }
+
+ removeSupersededObjects(objectGroups);
+
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.cellID = objectGroupCellID;
+ cell.objectGroups = objectGroups;
+ cell.rootDeclares = new ArrayList<>(effectiveRootDeclares.values());
+ return cell;
+ }
+
+ /**
+ * An object that is modified in a later revision appears again, with the same object ID,
+ * in that revision's object group. Keep only the newest version of each object, replacing
+ * the older version in place so the original object ordering is preserved.
+ *
+ * @param objectGroups The object groups ordered from the oldest revision to the newest.
+ */
+ private void removeSupersededObjects(List<RevisionStoreObjectGroup> objectGroups) {
+ Map<ExGuid, List<RevisionStoreObject>> containingList = new HashMap<>();
+ Map<ExGuid, Integer> indexInList = new HashMap<>();
+ for (RevisionStoreObjectGroup objectGroup : objectGroups) {
+ List<RevisionStoreObject> objects = objectGroup.objects;
+ for (int i = 0; i < objects.size(); ) {
+ RevisionStoreObject object = objects.get(i);
+ ExGuid objectId = object.objectID;
+ if (objectId != null && containingList.containsKey(objectId)) {
+ // newer version of an already seen object - replace the older one in place
+ containingList.get(objectId).set(indexInList.get(objectId), object);
+ objects.remove(i);
+ } else {
+ if (objectId != null) {
+ containingList.put(objectId, objects);
+ indexInList.put(objectId, i);
+ }
+ ++i;
+ }
+ }
+ }
}
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtArrayOfPropertyValues.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtArrayOfPropertyValues.java
index 68b1dab169..7220f999bf 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtArrayOfPropertyValues.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtArrayOfPropertyValues.java
@@ -41,16 +41,27 @@ public class PrtArrayOfPropertyValues implements IProperty {
* @return
*/
public int doDeserializeFromByteArray(byte[] byteArray, int startIndex) throws IOException {
+ return doDeserializeFromByteArray(byteArray, startIndex, 0);
+ }
+
+ public int doDeserializeFromByteArray(byte[] byteArray, int startIndex, int depth)
+ throws IOException {
int index = startIndex;
this.cProperties = BitConverter.toInt32(byteArray, index);
index += 4;
this.propertyID = new PropertyID();
int len = this.propertyID.doDeserializeFromByteArray(byteArray, index);
index += len;
+ // each PropertySet consumes at least its 2-byte count, so a valid count cannot
+ // exceed the remaining bytes / 2; bounds the allocation on malformed counts
+ if (this.cProperties < 0 || this.cProperties > (byteArray.length - index) / 2) {
+ throw new IOException("prtArrayOfPropertyValues count " + this.cProperties +
+ " exceeds remaining data " + (byteArray.length - index));
+ }
this.data = new PropertySet[this.cProperties];
for (int i = 0; i < this.cProperties; i++) {
this.data[i] = new PropertySet();
- int length = this.data[i].doDeserializeFromByteArray(byteArray, index);
+ int length = this.data[i].doDeserializeFromByteArray(byteArray, index, depth + 1);
index += length;
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtFourBytesOfLengthFollowedByData.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtFourBytesOfLengthFollowedByData.java
index 8518be09b5..4eb0ca7b83 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtFourBytesOfLengthFollowedByData.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtFourBytesOfLengthFollowedByData.java
@@ -43,6 +43,12 @@ public class PrtFourBytesOfLengthFollowedByData implements IProperty {
int index = startIndex;
this.cb = (int) BitConverter.toUInt32(byteArray, startIndex);
index += 4;
+ // copyOfRange zero-fills past the source end, so cb must be bounded by
+ // the remaining bytes or a malformed length allocates cb bytes of heap
+ if (this.cb < 0 || this.cb > byteArray.length - index) {
+ throw new IOException("prtFourBytesOfLengthFollowedByData length " + this.cb +
+ " exceeds remaining data " + (byteArray.length - index));
+ }
this.data = Arrays.copyOfRange(byteArray, index, index + this.cb);
index += this.cb;
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/DataElement.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/DataElement.java
index 2ae203fb2b..96e94f0e59 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/DataElement.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/DataElement.java
@@ -50,7 +50,8 @@ public class DataElement extends StreamObject {
String className = DataElement.class.getPackage().getName() + "." + value.name();
try {
- DATA_ELEMENT_DATA_TYPE_MAPPING.put(value, Class.forName(className));
+ DATA_ELEMENT_DATA_TYPE_MAPPING.put(value,
+ Class.forName(className, false, DataElement.class.getClassLoader()));
} catch (ClassNotFoundException e) {
// This is OK, we are not pulling over every single class
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/FileDataObject.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/FileDataObject.java
index 1211d2bf9d..d3a70780a9 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/FileDataObject.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/FileDataObject.java
@@ -19,8 +19,24 @@ package org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj;
/**
* This class is used to represent the file data.
*/
-class FileDataObject {
+public class FileDataObject {
public ObjectGroupObjectBLOBDataDeclaration objectDataBLOBDeclaration;
public ObjectGroupObjectDataBLOBReference objectDataBLOBReference;
public DataElement objectDataBLOBDataElement;
+
+ /**
+ * @return the opaque binary data of this file data object, or null if it could not be
+ * resolved.
+ */
+ public byte[] getData() {
+ if (objectDataBLOBDataElement != null &&
+ objectDataBLOBDataElement.data instanceof ObjectDataBLOBDataElementData) {
+ ObjectDataBLOBDataElementData blobData =
+ (ObjectDataBLOBDataElementData) objectDataBLOBDataElement.data;
+ if (blobData.objectDataBLOB != null) {
+ return blobData.objectDataBLOB.getData();
+ }
+ }
+ return null;
+ }
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOB.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOB.java
new file mode 100644
index 0000000000..2873e1d013
--- /dev/null
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOB.java
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.BasicObject;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.BinaryItem;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.util.ByteUtil;
+
+/**
+ * Specifies an object data BLOB stream object - the opaque binary data of an object,
+ * e.g. an embedded image or file. See MS-FSSHTTPB section 2.2.1.12.8.
+ */
+public class ObjectDataBLOB extends StreamObject {
+ public BinaryItem data;
+
+ public ObjectDataBLOB() {
+ super(StreamObjectTypeHeaderStart.ObjectDataBLOB);
+ this.data = new BinaryItem();
+ }
+
+ /**
+ * @return the opaque binary data as a byte array, or null if not present.
+ */
+ public byte[] getData() {
+ if (this.data == null || this.data.content == null) {
+ return null;
+ }
+ return ByteUtil.toByteArray(this.data.content);
+ }
+
+ @Override
+ protected void deserializeItemsFromByteArray(byte[] byteArray, AtomicInteger currentIndex,
+ int lengthOfItems)
+ throws TikaException, IOException {
+ AtomicInteger index = new AtomicInteger(currentIndex.get());
+ this.data = BasicObject.parse(byteArray, index, BinaryItem.class);
+
+ if (index.get() - currentIndex.get() != lengthOfItems) {
+ throw new StreamObjectParseErrorException(currentIndex.get(), "ObjectDataBLOB",
+ "Stream object over-parse error", null);
+ }
+
+ currentIndex.set(index.get());
+ }
+
+ @Override
+ protected int serializeItemsToByteList(List<Byte> byteList) throws IOException {
+ int startPoint = byteList.size();
+ if (this.data != null) {
+ byteList.addAll(this.data.serializeToByteList());
+ }
+ return byteList.size() - startPoint;
+ }
+}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOBDataElementData.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOBDataElementData.java
new file mode 100644
index 0000000000..ff28e9d096
--- /dev/null
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOBDataElementData.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.tika.exception.TikaException;
+
+/**
+ * Object data BLOB data element - carries the opaque binary data of an object, e.g. an
+ * embedded image or file. See MS-FSSHTTPB section 2.2.1.12.8.
+ */
+public class ObjectDataBLOBDataElementData extends DataElementData {
+ public ObjectDataBLOB objectDataBLOB;
+
+ public ObjectDataBLOBDataElementData() {
+ this.objectDataBLOB = new ObjectDataBLOB();
+ }
+
+ @Override
+ public int deserializeDataElementDataFromByteArray(byte[] byteArray, int startIndex)
+ throws TikaException, IOException {
+ AtomicInteger index = new AtomicInteger(startIndex);
+ this.objectDataBLOB = StreamObject.getCurrent(byteArray, index, ObjectDataBLOB.class);
+ return index.get() - startIndex;
+ }
+
+ @Override
+ public List<Byte> serializeToByteList() throws TikaException, IOException {
+ return this.objectDataBLOB.serializeToByteList();
+ }
+}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/PropertySet.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/PropertySet.java
index 45e1278f42..64276e5aa5 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/PropertySet.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/PropertySet.java
@@ -37,6 +37,10 @@ import org.apache.tika.parser.microsoft.onenote.fsshttpb.util.BitConverter;
* This class is used to represent a PropertySet.
*/
public class PropertySet implements IProperty {
+ // a nesting level costs ~14 file bytes, so unbounded depth lets a small
+ // crafted file drive a StackOverflowError that escapes catch(Exception)
+ public static final int MAX_PROPERTY_NESTING = 100;
+
public int cProperties;
public PropertyID[] rgPrids;
@@ -72,6 +76,14 @@ public class PropertySet implements IProperty {
* @return Return the length in byte of the PropertySet.
*/
public int doDeserializeFromByteArray(byte[] byteArray, int startIndex) throws IOException {
+ return doDeserializeFromByteArray(byteArray, startIndex, 0);
+ }
+
+ public int doDeserializeFromByteArray(byte[] byteArray, int startIndex, int depth)
+ throws IOException {
+ if (depth > MAX_PROPERTY_NESTING) {
+ throw new IOException("PropertySet nesting exceeds " + MAX_PROPERTY_NESTING);
+ }
int index = startIndex;
this.cProperties = BitConverter.toInt16(byteArray, startIndex);
@@ -124,7 +136,16 @@ public class PropertySet implements IProperty {
break;
}
if (property != null) {
- int len = property.doDeserializeFromByteArray(byteArray, index);
+ int len;
+ if (property instanceof PropertySet) {
+ len = ((PropertySet) property)
+ .doDeserializeFromByteArray(byteArray, index, depth + 1);
+ } else if (property instanceof PrtArrayOfPropertyValues) {
+ len = ((PrtArrayOfPropertyValues) property)
+ .doDeserializeFromByteArray(byteArray, index, depth + 1);
+ } else {
+ len = property.doDeserializeFromByteArray(byteArray, index);
+ }
this.rgData.add(property);
index += len;
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/FileDataObject.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/RevisionStoreCell.java
similarity index 53%
copy from tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/FileDataObject.java
copy to tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/RevisionStoreCell.java
index 1211d2bf9d..188a2a45de 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/FileDataObject.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/RevisionStoreCell.java
@@ -16,11 +16,22 @@
*/
package org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.CellID;
+
/**
- * This class is used to represent the file data.
+ * The revision store content of a single cell (object space), together with the root object
+ * declarations of its current revision. The object groups are ordered from the oldest revision
+ * to the newest.
*/
-class FileDataObject {
- public ObjectGroupObjectBLOBDataDeclaration objectDataBLOBDeclaration;
- public ObjectGroupObjectDataBLOBReference objectDataBLOBReference;
- public DataElement objectDataBLOBDataElement;
+public class RevisionStoreCell {
+ public CellID cellID;
+ public List<RevisionStoreObjectGroup> objectGroups = new ArrayList<>();
+ /**
+ * The effective root object declarations of the cell's current revision, i.e. for each
+ * root role the declaration made by the most recent revision in the base revision chain.
+ */
+ public List<RevisionManifestRootDeclare> rootDeclares = new ArrayList<>();
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/RevisionStoreObjectGroup.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/RevisionStoreObjectGroup.java
index afd2f6c067..fb16a8402a 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/RevisionStoreObjectGroup.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/RevisionStoreObjectGroup.java
@@ -18,7 +18,7 @@ package org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj;
import java.io.IOException;
import java.util.ArrayList;
-import java.util.HashMap;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -38,13 +38,20 @@ public class RevisionStoreObjectGroup {
public static RevisionStoreObjectGroup createInstance(ExGuid objectGroupId,
ObjectGroupDataElementData dataObject,
- boolean isEncryption) throws IOException {
+ boolean isEncryption,
+ Map<ExGuid, DataElement> blobElements)
+ throws IOException {
RevisionStoreObjectGroup objectGroup = new RevisionStoreObjectGroup(objectGroupId);
- Map<ExGuid, RevisionStoreObject> objectDict = new HashMap<>();
+ // insertion-ordered so the object list (and any degraded-file fallback walk) is
+ // deterministic
+ Map<ExGuid, RevisionStoreObject> objectDict = new LinkedHashMap<>();
if (!isEncryption) {
RevisionStoreObject revisionObject = null;
for (int i = 0; i < dataObject.objectGroupDeclarations.objectDeclarationList.size();
i++) {
+ if (i >= dataObject.objectGroupData.objectGroupObjectDataList.size()) {
+ throw new IOException("Missing object data for object declaration " + i);
+ }
ObjectGroupObjectDeclare objectDeclaration =
dataObject.objectGroupDeclarations.objectDeclarationList.get(i);
ObjectGroupObjectData objectData =
@@ -63,16 +70,20 @@ public class RevisionStoreObjectGroup {
} else if (objectDeclaration.objectPartitionID.getDecodedValue() == 1) {
revisionObject.propertySet =
new PropertySetObject(objectDeclaration, objectData);
- if (revisionObject.jcid.jcid.isFileData != 0) {
- revisionObject.referencedObjectID = objectData.objectExGUIDArray;
- revisionObject.referencedObjectSpacesID = objectData.cellIDArray;
- }
+ // the object extended GUID array lists the objects referenced by this
+ // object, in the same order as the CompactIDs in the OID stream of the
+ // ObjectSpaceObjectPropSet - see MS-ONESTORE section 2.7.8
+ revisionObject.referencedObjectID = objectData.objectExGUIDArray;
+ revisionObject.referencedObjectSpacesID = objectData.cellIDArray;
}
}
for (int i = 0; i <
dataObject.objectGroupDeclarations.objectGroupObjectBLOBDataDeclarationList.size();
i++) {
+ if (i >= dataObject.objectGroupData.objectGroupObjectDataBLOBReferenceList.size()) {
+ throw new IOException("Missing BLOB reference for object declaration " + i);
+ }
ObjectGroupObjectBLOBDataDeclaration objectGroupObjectBLOBDataDeclaration =
dataObject.objectGroupDeclarations.objectGroupObjectBLOBDataDeclarationList.get(
i);
@@ -87,17 +98,24 @@ public class RevisionStoreObjectGroup {
objectDict.get(objectGroupObjectBLOBDataDeclaration.objectExGUID);
}
if (objectGroupObjectBLOBDataDeclaration.objectPartitionID.getDecodedValue() == 2) {
+ revisionObject.objectID = objectGroupObjectBLOBDataDeclaration.objectExGUID;
+ revisionObject.objectGroupID = objectGroupId;
revisionObject.fileDataObject = new FileDataObject();
revisionObject.fileDataObject.objectDataBLOBDeclaration =
objectGroupObjectBLOBDataDeclaration;
revisionObject.fileDataObject.objectDataBLOBReference =
objectGroupObjectDataBLOBReference;
+ revisionObject.fileDataObject.objectDataBLOBDataElement = blobElements.get(
+ objectGroupObjectDataBLOBReference.blobExtendedGUID);
}
}
objectGroup.objects.addAll(objectDict.values());
} else {
for (int i = 0; i < dataObject.objectGroupDeclarations.objectDeclarationList.size();
i++) {
+ if (i >= dataObject.objectGroupData.objectGroupObjectDataList.size()) {
+ throw new IOException("Missing object data for object declaration " + i);
+ }
ObjectGroupObjectDeclare objectDeclaration =
dataObject.objectGroupDeclarations.objectDeclarationList.get(i);
ObjectGroupObjectData objectData =
@@ -114,4 +132,17 @@ public class RevisionStoreObjectGroup {
return objectGroup;
}
+
+ /**
+ * Creates a cell-local wrapper around this parsed group. The object lists are copied so
+ * revision supersession filtering cannot mutate a cached group used by another cell.
+ *
+ * @return a shallow copy with independent mutable lists
+ */
+ public RevisionStoreObjectGroup copy() {
+ RevisionStoreObjectGroup copy = new RevisionStoreObjectGroup(objectGroupID);
+ copy.objects.addAll(objects);
+ copy.encryptionObjects.addAll(encryptionObjects);
+ return copy;
+ }
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/StreamObject.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/StreamObject.java
index 03bb1da92c..6a250ba124 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/StreamObject.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/StreamObject.java
@@ -73,7 +73,8 @@ public abstract class StreamObject implements IFSSHTTPBSerializable {
for (StreamObjectTypeHeaderStart value : StreamObjectTypeHeaderStart.values()) {
String className = StreamObject.class.getPackage().getName() + "." + value.name();
try {
- streamObjectTypeMapping.put(value, Class.forName(className));
+ streamObjectTypeMapping.put(value,
+ Class.forName(className, false, StreamObject.class.getClassLoader()));
} catch (ClassNotFoundException e) {
// This is OK, we are not pulling over every single class
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/basic/CellID.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/basic/CellID.java
index a352dc6315..7b6dbcf886 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/basic/CellID.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/basic/CellID.java
@@ -105,6 +105,14 @@ public class CellID extends BasicObject {
return this.extendGUID1.hashCode() + this.extendGUID2.hashCode();
}
+ @Override
+ public String toString() {
+ return "CellID{" +
+ "extendGUID1=" + extendGUID1 +
+ ", extendGUID2=" + extendGUID2 +
+ '}';
+ }
+
/**
* This method is used to deserialize the CellID basic object from the specified byte array and start index.
*
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/basic/ExGuid.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/basic/ExGuid.java
index 4a7e149de2..42ff354b0c 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/basic/ExGuid.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/basic/ExGuid.java
@@ -144,6 +144,15 @@ public class ExGuid extends BasicObject {
return this.guid.hashCode() + Integer.valueOf(this.value).hashCode();
}
+ @Override
+ public String toString() {
+ return "ExGuid{" +
+ "type=" + type +
+ ", value=" + value +
+ ", guid=" + guid +
+ '}';
+ }
+
/**
* This method is used to deserialize the ExGuid basic object from the specified byte array and start index.
*
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfContextIDs.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfContextIDs.java
index fe6d288330..7e1e348e7d 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfContextIDs.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfContextIDs.java
@@ -57,10 +57,16 @@ public class ObjectSpaceObjectStreamOfContextIDs {
int headerCount = this.header.doDeserializeFromByteArray(byteArray, index);
index += headerCount;
+ // each CompactID consumes 4 bytes, so a valid count cannot exceed the remaining
+ // bytes / 4; bounds the allocation on malformed counts
+ if (this.header.count > (byteArray.length - index) / 4L) {
+ throw new IOException("ObjectSpaceObjectStreamOfContextIDs count " +
+ this.header.count + " exceeds remaining data " + (byteArray.length - index));
+ }
this.body = new CompactID[(int) this.header.count];
for (int i = 0; i < this.header.count; i++) {
CompactID compactID = new CompactID();
- int count = compactID.doDeserializeFromByteArray(byteArray, startIndex);
+ int count = compactID.doDeserializeFromByteArray(byteArray, index);
this.body[i] = compactID;
index += count;
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOIDs.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOIDs.java
index 7901dd015e..53398663c3 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOIDs.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOIDs.java
@@ -58,10 +58,16 @@ public class ObjectSpaceObjectStreamOfOIDs {
int headerCount = this.header.doDeserializeFromByteArray(byteArray, index);
index += headerCount;
+ // each CompactID consumes 4 bytes, so a valid count cannot exceed the remaining
+ // bytes / 4; bounds the allocation on malformed counts
+ if (this.header.count > (byteArray.length - index) / 4L) {
+ throw new IOException("ObjectSpaceObjectStreamOfOIDs count " + this.header.count +
+ " exceeds remaining data " + (byteArray.length - index));
+ }
this.body = new CompactID[(int) this.header.count];
for (int i = 0; i < this.header.count; i++) {
CompactID compactID = new CompactID();
- int count = compactID.doDeserializeFromByteArray(byteArray, startIndex);
+ int count = compactID.doDeserializeFromByteArray(byteArray, index);
this.body[i] = compactID;
index += count;
}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOSIDs.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOSIDs.java
index 61f7deda90..f37de26f76 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOSIDs.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOSIDs.java
@@ -57,6 +57,12 @@ public class ObjectSpaceObjectStreamOfOSIDs {
int headerCount = this.header.doDeserializeFromByteArray(byteArray, index);
index += headerCount;
+ // each CompactID consumes 4 bytes, so a valid count cannot exceed the remaining
+ // bytes / 4; bounds the allocation on malformed counts
+ if (this.header.count > (byteArray.length - index) / 4L) {
+ throw new IOException("ObjectSpaceObjectStreamOfOSIDs count " + this.header.count +
+ " exceeds remaining data " + (byteArray.length - index));
+ }
this.body = new CompactID[(int) this.header.count];
for (int i = 0; i < this.header.count; i++) {
CompactID compactID = new CompactID();
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/GUIDTest.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/GUIDTest.java
new file mode 100644
index 0000000000..0f21f861db
--- /dev/null
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/GUIDTest.java
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.parser.microsoft.onenote;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import java.nio.charset.StandardCharsets;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.tika.exception.TikaException;
+
+public class GUIDTest {
+
+ @Test
+ public void testParsesValidGuidRoundTrip() throws Exception {
+ byte[] valid = "{638DE92F-a6d4-4BC1-9A36-4AFC2511A5B7}"
+ .getBytes(StandardCharsets.UTF_16LE);
+
+ assertEquals("{638DE92F-A6D4-4BC1-9A36-4AFC2511A5B7}",
+ GUID.fromCurlyBraceUTF16Bytes(valid).toString());
+ }
+
+ @Test
+ public void testRejectsNonAsciiUnicodeDigits() {
+ // fullwidth '6' is a Unicode digit that Character.digit(c, 16) would accept
+ byte[] malformed = "{６38DE92F-A6D4-4BC1-9A36-4AFC2511A5B7}"
+ .getBytes(StandardCharsets.UTF_16LE);
+
+ assertThrows(TikaException.class, () -> GUID.fromCurlyBraceUTF16Bytes(malformed));
+ }
+
+ @Test
+ public void testRejectsMalformedCurlyBraceGuid() {
+ byte[] malformed = "{638DE92F-A6D4-4BC1-9IJ36-4AFC2511A5B7}"
+ .getBytes(StandardCharsets.UTF_16LE);
+
+ assertThrows(TikaException.class, () -> GUID.fromCurlyBraceUTF16Bytes(malformed));
+ }
+
+ @Test
+ public void testRejectsInjectedDashes() {
+ String[] malformedGuids = {
+ "{-38DE92F-A6D4-4BC1-9A36-4AFC2511A5B7}",
+ "{--8DE92F-A6D4-4BC1-9A36-4AFC2511A5B7}",
+ "{---8DE92F-A6D4-4BC1-9A36-4AFC2511A5B7}",
+ "{----E92F-A6D4-4BC1-9A36-4AFC2511A5B7}"
+ };
+ for (String malformedGuid : malformedGuids) {
+ assertThrows(TikaException.class,
+ () -> GUID.fromCurlyBraceUTF16Bytes(
+ malformedGuid.getBytes(StandardCharsets.UTF_16LE)));
+ }
+ }
+}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/OneNoteParserTest.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/OneNoteParserTest.java
index bab0a6fbcb..e2c9e410dd 100644
--- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/OneNoteParserTest.java
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/OneNoteParserTest.java
@@ -19,23 +19,128 @@ package org.apache.tika.parser.microsoft.onenote;
import static org.apache.tika.parser.microsoft.onenote.OneNoteParser.ONE_NOTE_PREFIX;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.StringWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
import java.time.Instant;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
+import java.util.stream.Collectors;
import org.apache.commons.lang3.StringUtils;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.xml.sax.ContentHandler;
import org.apache.tika.TikaTest;
+import org.apache.tika.exception.TikaException;
+import org.apache.tika.exception.TikaMemoryLimitException;
+import org.apache.tika.extractor.EmbeddedDocumentExtractor;
+import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.TikaCoreProperties;
+import org.apache.tika.parser.ParseContext;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.MSOneStorePackage;
+import org.apache.tika.sax.ToTextContentHandler;
+import org.apache.tika.sax.XHTMLContentHandler;
public class OneNoteParserTest extends TikaTest {
//test recursive parser wrapper for image files
+ @Test
+ public void testFuzzerRegressionInputsFallBackToLegacyDump() throws Exception {
+ // a structural failure falls back to the legacy string dump; the failure is
+ // pinned in the parse-warning metadata
+ String[][] resources = {
+ {"testOneNote-fuzz1.one", "Missing dependent revision"},
+ {"testOneNote-fuzz2.one", "unified property count"},
+ {"testOneNote-fuzz3.one", "Invalid GUID string"}
+ };
+ for (String[] resource : resources) {
+ InputStream input = getClass().getResourceAsStream("/test-documents/" + resource[0]);
+ assertNotNull(input, resource[0]);
+ try (InputStream stream = input;
+ TikaInputStream tis = TikaInputStream.get(stream)) {
+ Metadata metadata = new Metadata();
+ new OneNoteParser().parse(tis, new ToTextContentHandler(), metadata,
+ new ParseContext());
+ assertTrue(Arrays.stream(
+ metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(warning -> warning.contains(resource[1])
+ && warning.contains("falling back to legacy text dump")),
+ () -> resource[0] + ": " + Arrays.toString(metadata.getValues(
+ TikaCoreProperties.TIKA_META_EXCEPTION_WARNING)));
+ }
+ }
+ }
+
+ @Test
+ public void testNoContentFsshttpbWalkFallsBackToLegacyDump(@TempDir Path tempDir)
+ throws Exception {
+ Path file = tempDir.resolve("empty-walk.one");
+ // trailing newline: the dump excludes the file's final byte from scanning
+ Files.write(file,
+ "recoverable legacy dump text\n".getBytes(StandardCharsets.US_ASCII));
+ Metadata metadata = new Metadata();
+ StringWriter writer = new StringWriter();
+ XHTMLContentHandler xhtml = new XHTMLContentHandler(new ToTextContentHandler(writer),
+ metadata);
+ xhtml.startDocument();
+ try (OneNoteDirectFileResource resource =
+ new OneNoteDirectFileResource(file.toFile())) {
+ // fresh package = the walk completed without emitting anything
+ OneNoteParser.legacyFallbackIfNoContent(new MSOneStorePackage(), metadata, xhtml,
+ resource);
+ assertTrue(writer.toString().contains("recoverable legacy dump text"));
+ assertTrue(metadata.get(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING)
+ .contains("produced no content"));
+
+ // pkg == null means the exception path already dumped - no second dump
+ StringWriter exceptionPath = new StringWriter();
+ XHTMLContentHandler xhtml2 = new XHTMLContentHandler(
+ new ToTextContentHandler(exceptionPath), metadata);
+ xhtml2.startDocument();
+ OneNoteParser.legacyFallbackIfNoContent(null, metadata, xhtml2, resource);
+ xhtml2.endDocument();
+ assertTrue(exceptionPath.toString().isBlank());
+ }
+ xhtml.endDocument();
+ }
+
+ @Test
+ public void testTruncatedFileFallsBackToLegacyDump(@TempDir Path tempDir) throws Exception {
+ byte[] full;
+ try (InputStream is = getClass()
+ .getResourceAsStream("/test-documents/testOneNote1.one")) {
+ full = is.readAllBytes();
+ }
+ // keep the 1024-byte header plus a sliver of content so the root file node list
+ // is unreachable
+ Path truncated = tempDir.resolve("truncated.one");
+ Files.write(truncated, Arrays.copyOf(full, 2048));
+
+ Metadata metadata = new Metadata();
+ try (TikaInputStream tis = TikaInputStream.get(truncated)) {
+ new OneNoteParser().parse(tis, new ToTextContentHandler(), metadata,
+ new ParseContext());
+ }
+ assertTrue(Arrays.stream(metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(warning -> warning.contains("falling back to legacy text dump")));
+ }
+
/**
* This is the sample document that is automatically created from onenote 2013.
*/
@@ -223,6 +328,217 @@ public class OneNoteParserTest extends TikaTest {
ml.get("Content-Type"))));
}
+ @Test
+ public void testOneNoteEmbeddedImage() throws Exception {
+ List<byte[]> embedded = new ArrayList<>();
+ List<String> embeddedTypes = new ArrayList<>();
+ ParseContext context = new ParseContext();
+ context.set(EmbeddedDocumentExtractor.class, new EmbeddedDocumentExtractor() {
+ @Override
+ public boolean shouldParseEmbedded(Metadata metadata) {
+ return true;
+ }
+
+ @Override
+ public void parseEmbedded(InputStream stream, ContentHandler handler,
+ Metadata metadata, boolean outputHtml) throws IOException {
+ embedded.add(stream.readAllBytes());
+ embeddedTypes.add(metadata.get(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE));
+ }
+ });
+ try (InputStream tis = getResourceAsStream("/test-documents/testOneNoteEmbeddedImage.one")) {
+ new OneNoteParser().parse(tis, new ToTextContentHandler(), new Metadata(), context);
+ }
+
+ assertEquals(1, embedded.size());
+ assertTrue(embedded.stream().anyMatch(bytes -> bytes.length > 1000),
+ () -> "embedded sizes: " + embedded.stream().map(bytes -> bytes.length).collect(Collectors.toList()));
+ assertTrue(embeddedTypes.contains("INLINE"));
+ }
+
+ @Test
+ public void testOneNoteEmbeddedImageRecursiveMetadata() throws Exception {
+ List<Metadata> metadataList = getRecursiveMetadata("testOneNoteEmbeddedImage.one");
+
+ assertEquals(2, metadataList.size());
+ Metadata embedded = metadataList.get(1);
+ assertEquals("INLINE", embedded.get(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE));
+ assertNotNull(embedded.get(TikaCoreProperties.EMBEDDED_RELATIONSHIP_ID));
+ }
+
+ @Test
+ public void testPropertyValueBudgetIsSharedAcrossCopiesAndResetsPerList(
+ @TempDir Path tempDir) throws Exception {
+ Path emptyFile = tempDir.resolve("empty");
+ Files.write(emptyFile, ByteBuffer.allocate(36).order(ByteOrder.LITTLE_ENDIAN)
+ .putLong(FileNodeListHeader.UNIT_MAGIC_CONSTANT).putInt(0x10).putInt(0)
+ .putLong(-1).putInt(0).putLong(OneNotePtr.FOOTER_CONST).array());
+ Method reservePropertyCount = OneNotePtr.class.getDeclaredMethod(
+ "reservePropertyCount", long.class, String.class);
+ reservePropertyCount.setAccessible(true);
+ try (OneNoteDirectFileResource dif = new OneNoteDirectFileResource(emptyFile.toFile())) {
+ OneNotePtr ptr = new OneNotePtr(new OneNoteDocument(), dif);
+ OneNotePtr copy = new OneNotePtr(ptr);
+ reservePropertyCount.invoke(ptr, 100_000L, "test");
+ InvocationTargetException exception = assertThrows(InvocationTargetException.class,
+ () -> reservePropertyCount.invoke(copy, 1L, "test"));
+ assertTrue(exception.getCause() instanceof TikaMemoryLimitException);
+
+ ptr.deserializeFileNodeList(new FileNodeList(), new FileNodePtr());
+ reservePropertyCount.invoke(ptr, 1L, "test");
+ }
+ }
+
+ @Test
+ public void testFileNodeCycleIsReportedAndStopsTraversal(@TempDir Path tempDir) throws Exception {
+ FileNode fileNode = new FileNode().setGosid(ExtendedGUID.nil());
+ fileNode.childFileNodeList.setFileNodeListHeader(new FileNodeListHeader(0,
+ FileNodeListHeader.UNIT_MAGIC_CONSTANT, 0x10, 0));
+ fileNode.childFileNodeList.children.add(fileNode);
+ Metadata metadata = new Metadata();
+ ParseContext parseContext = new ParseContext();
+ Path emptyFile = Files.createFile(tempDir.resolve("empty"));
+ try (OneNoteDirectFileResource dif = new OneNoteDirectFileResource(emptyFile.toFile())) {
+ OneNoteTreeWalker walker = new OneNoteTreeWalker(new OneNoteTreeWalkerOptions(),
+ new OneNoteDocument(), dif,
+ new XHTMLContentHandler(new ToTextContentHandler(), metadata),
+ metadata, parseContext, null);
+ walker.walkFileNode(fileNode, null);
+ }
+ assertEquals(1, Arrays.stream(
+ metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .filter(warning -> warning.contains("file-node cycle detected"))
+ .count());
+ }
+
+ @Test
+ public void testDeepFileNodeChainIsDepthCapped(@TempDir Path tempDir) throws Exception {
+ // a long acyclic chain slips past the cycle guard - the depth cap must stop it
+ // before the recursion can overflow the stack
+ FileNode root = new FileNode().setGosid(ExtendedGUID.nil());
+ FileNode current = root;
+ for (int i = 0; i < 600; i++) {
+ FileNode child = new FileNode().setGosid(ExtendedGUID.nil());
+ current.childFileNodeList.setFileNodeListHeader(new FileNodeListHeader(0,
+ FileNodeListHeader.UNIT_MAGIC_CONSTANT, 0x10, 0));
+ current.childFileNodeList.children.add(child);
+ current = child;
+ }
+ Metadata metadata = new Metadata();
+ ParseContext parseContext = new ParseContext();
+ Path emptyFile = Files.createFile(tempDir.resolve("empty"));
+ try (OneNoteDirectFileResource dif = new OneNoteDirectFileResource(emptyFile.toFile())) {
+ OneNoteTreeWalker walker = new OneNoteTreeWalker(new OneNoteTreeWalkerOptions(),
+ new OneNoteDocument(), dif,
+ new XHTMLContentHandler(new ToTextContentHandler(), metadata),
+ metadata, parseContext, null);
+ walker.walkFileNode(root, null);
+ }
+ assertEquals(1, Arrays.stream(
+ metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .filter(warning -> warning.contains("exceeded depth limit"))
+ .count());
+ }
+
+ /**
+ * Writes one minimal 56-byte file-node-list fragment: header, a single baseType-2 node
+ * whose child list is at {childStp, childCb}, a terminator, a nil next-fragment
+ * reference and the footer.
+ */
+ private static void writeFileNodeListBlock(ByteBuffer buf, long childStp, int childCb) {
+ // id=0x10, size=16 (node header + 8-byte stp + 4-byte cb), stpFormat=0, cbFormat=0,
+ // baseType=2, reserved=1
+ int fileNodeHeader = 0x10 | (16 << 10) | (2 << 27) | (1 << 31);
+ buf.putLong(FileNodeListHeader.UNIT_MAGIC_CONSTANT).putInt(0x10).putInt(0)
+ .putInt(fileNodeHeader).putLong(childStp).putInt(childCb)
+ .putInt(0)
+ .putLong(-1).putInt(0)
+ .putLong(OneNotePtr.FOOTER_CONST);
+ }
+
+ @Test
+ public void testFileNodeListCycleFailsCleanly(@TempDir Path tempDir) throws Exception {
+ // one fragment holding a baseType-2 node whose child list points back at itself
+ ByteBuffer buf = ByteBuffer.allocate(56).order(ByteOrder.LITTLE_ENDIAN);
+ writeFileNodeListBlock(buf, 0, 56);
+ Path cyclic = tempDir.resolve("cyclic");
+ Files.write(cyclic, buf.array());
+ try (OneNoteDirectFileResource dif = new OneNoteDirectFileResource(cyclic.toFile())) {
+ OneNotePtr ptr = new OneNotePtr(new OneNoteDocument(), dif);
+ TikaException e = assertThrows(TikaException.class,
+ () -> ptr.deserializeFileNodeList(new FileNodeList(), new FileNodePtr()));
+ assertTrue(e.getMessage().contains("cycle"), e.getMessage());
+ }
+ }
+
+ @Test
+ public void testFileNodeListNestingIsDepthCapped(@TempDir Path tempDir) throws Exception {
+ // 120 lists, each holding one baseType-2 node pointing at the next list - acyclic,
+ // so only the depth cap can stop the recursion
+ int lists = 120;
+ ByteBuffer buf = ByteBuffer.allocate(56 * lists).order(ByteOrder.LITTLE_ENDIAN);
+ for (int i = 0; i < lists; i++) {
+ long childStp = (i + 1 < lists ? i + 1 : i) * 56L;
+ writeFileNodeListBlock(buf, childStp, 56);
+ }
+ Path deep = tempDir.resolve("deep");
+ Files.write(deep, buf.array());
+ try (OneNoteDirectFileResource dif = new OneNoteDirectFileResource(deep.toFile())) {
+ OneNotePtr ptr = new OneNotePtr(new OneNoteDocument(), dif);
+ assertThrows(TikaMemoryLimitException.class,
+ () -> ptr.deserializeFileNodeList(new FileNodeList(), new FileNodePtr()));
+ }
+ }
+
+ @Test
+ public void testFragmentChainCycleFailsCleanly(@TempDir Path tempDir) throws Exception {
+ // an empty fragment whose next-fragment reference points back at itself would
+ // previously loop forever
+ ByteBuffer buf = ByteBuffer.allocate(36).order(ByteOrder.LITTLE_ENDIAN)
+ .putLong(FileNodeListHeader.UNIT_MAGIC_CONSTANT).putInt(0x10).putInt(0)
+ .putLong(0).putInt(36)
+ .putLong(OneNotePtr.FOOTER_CONST);
+ Path cyclic = tempDir.resolve("cyclic-fragment");
+ Files.write(cyclic, buf.array());
+ try (OneNoteDirectFileResource dif = new OneNoteDirectFileResource(cyclic.toFile())) {
+ OneNotePtr ptr = new OneNotePtr(new OneNoteDocument(), dif);
+ TikaException e = assertThrows(TikaException.class,
+ () -> ptr.deserializeFileNodeList(new FileNodeList(), new FileNodePtr()));
+ assertTrue(e.getMessage().contains("fragment cycle"), e.getMessage());
+ }
+ }
+
+ @Test
+ public void testLegacyEmbeddedExtractionHonorsShouldParseEmbedded() throws Exception {
+ List<Metadata> offered = new ArrayList<>();
+ ParseContext context = new ParseContext();
+ context.set(EmbeddedDocumentExtractor.class, new EmbeddedDocumentExtractor() {
+ @Override
+ public boolean shouldParseEmbedded(Metadata metadata) {
+ offered.add(metadata);
+ return false;
+ }
+
+ @Override
+ public void parseEmbedded(InputStream stream, ContentHandler handler,
+ Metadata metadata, boolean outputHtml) {
+ throw new AssertionError("must not parse embedded when shouldParseEmbedded" +
+ " returns false");
+ }
+ });
+ try (InputStream tis =
+ getResourceAsStream("/test-documents/testOneNoteEmbeddedWordDoc.one")) {
+ new OneNoteParser().parse(tis, new ToTextContentHandler(), new Metadata(), context);
+ }
+ assertTrue(offered.size() > 0);
+ // the legacy path offers every embedded object as an ATTACHMENT; the relationship
+ // id is set only when the object carries a non-nil gosid
+ for (Metadata offeredMetadata : offered) {
+ assertEquals(TikaCoreProperties.EmbeddedResourceType.ATTACHMENT.toString(),
+ offeredMetadata.get(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE));
+ }
+ }
+
/**
* Test a document pulled from Office 365 which stores the MS-ONESTORE document using the MS-FSSHTTPB
* protocol.
@@ -232,7 +548,14 @@ public class OneNoteParserTest extends TikaTest {
Metadata metadata = new Metadata();
String txt = getText("testOneNoteFromOffice365.one", metadata);
- assertEquals(1, metadata.getValues(ONE_NOTE_PREFIX + "mostRecentAuthors").length);
+ // only the authors of the current content count - authors that only appear in
+ // older page version snapshots are not reported
+ assertEquals(Arrays.asList("Chang Du", "Du Chang"),
+ Arrays.asList(metadata.getValues(TikaCoreProperties.CREATOR)));
+ // both authors are referenced as AuthorMostRecent by current content; one of them
+ // is first visited under another role, so its most-recent role must still register
+ assertEquals(Arrays.asList("Chang Du", "Du Chang"),
+ Arrays.asList(metadata.getValues(ONE_NOTE_PREFIX + "mostRecentAuthors")));
assertEquals(Instant.ofEpochSecond(1636621406),
Instant.ofEpochSecond(Long.parseLong(metadata.get(ONE_NOTE_PREFIX + "creationTimestamp"))));
@@ -241,6 +564,9 @@ public class OneNoteParserTest extends TikaTest {
assertEquals(Instant.ofEpochSecond(1636621448),
Instant.ofEpochSecond(Long.parseLong(metadata.get(TikaCoreProperties.MODIFIED))));
assertContains("Section1Page1Content", txt);
+ // content from revisions other than each cell's current revision manifest
+ assertContains("Section1Page2Content", txt);
+ assertTrue(txt.indexOf("Section1Page1Content") < txt.indexOf("Section1Page2Content"));
}
/**
@@ -252,20 +578,23 @@ public class OneNoteParserTest extends TikaTest {
Metadata metadata = new Metadata();
String txt = getText("testOneNoteFromOffice365-2.one", metadata);
+ assertEquals(List.of("Robert Lucarini"),
+ Arrays.asList(metadata.getValues(TikaCoreProperties.CREATOR)));
List<String> mostRecentAuthors = Arrays.asList(metadata.getValues(ONE_NOTE_PREFIX + "mostRecentAuthors"));
assertContains(
- "R\u0000o\u0000b\u0000e\u0000r\u0000t\u0000 \u0000L\u0000u\u0000c\u0000a" +
- "\u0000r\u0000i\u0000n\u0000i\u0000\u0000\u0000",
+ "Robert Lucarini",
mostRecentAuthors);
assertEquals(Instant.ofEpochSecond(1591712300),
Instant.ofEpochSecond(Long.parseLong(metadata.get(ONE_NOTE_PREFIX + "creationTimestamp"))));
- assertEquals(Instant.ofEpochMilli(1623252330000L),
+ assertEquals(Instant.ofEpochMilli(1623597638000L),
Instant.ofEpochMilli(Long.parseLong(metadata.get(ONE_NOTE_PREFIX + "lastModifiedTimestamp"))));
assertEquals(Instant.ofEpochSecond(1623597587),
Instant.ofEpochSecond(Long.parseLong(metadata.get(TikaCoreProperties.MODIFIED))));
assertContains("Section1Page1Content", txt);
+ // content from revisions other than each cell's current revision manifest
+ assertContains("Section1Page2Content", txt);
}
private void assertNoJunk(String txt) {
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreBlobTest.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreBlobTest.java
new file mode 100644
index 0000000000..ea5a4c986b
--- /dev/null
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreBlobTest.java
@@ -0,0 +1,343 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.parser.microsoft.onenote.fsshttpb;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.StringWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.junit.jupiter.api.Test;
+import org.xml.sax.ContentHandler;
+
+import org.apache.tika.extractor.EmbeddedDocumentExtractor;
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.TikaCoreProperties;
+import org.apache.tika.parser.ParseContext;
+import org.apache.tika.parser.microsoft.onenote.OneNoteTreeWalkerOptions;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.NoData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.PrtFourBytesOfLengthFollowedByData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.DataElement;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.FileDataObject;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.ObjectDataBLOB;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.ObjectDataBLOBDataElementData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.ObjectGroupDataElementData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.ObjectGroupObjectBLOBDataDeclaration;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.ObjectGroupObjectData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.ObjectGroupObjectDataBLOBReference;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.ObjectGroupObjectDeclare;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.PropertySet;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.PropertySetObject;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionStoreObject;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionStoreObjectGroup;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.StreamObject;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.StreamObjectParseErrorException;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.BinaryItem;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.DataElementType;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.ExGUIDArray;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.ExGuid;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.PropertyID;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.PropertyType;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.space.ObjectSpaceObjectPropSet;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.util.ByteUtil;
+import org.apache.tika.sax.ToTextContentHandler;
+import org.apache.tika.sax.XHTMLContentHandler;
+
+/**
+ * Tests for object data BLOBs (embedded images and files) in the MS-FSSHTTPB packaged
+ * revision store.
+ */
+public class MSOneStoreBlobTest {
+
+ private static final byte[] BLOB_BYTES =
+ "these are the bytes of an embedded image".getBytes(StandardCharsets.UTF_8);
+
+ /**
+ * An object data BLOB data element must survive a serialization round trip. Before
+ * ObjectDataBLOBDataElementData existed, deserializing a data element of this type threw,
+ * so any OneNote file with an embedded image or file lost its structure entirely.
+ */
+ @Test
+ public void testObjectDataBLOBDataElementRoundTrip() throws Exception {
+ DataElement reparsed = roundTripBlobElement();
+ assertEquals(DataElementType.ObjectDataBLOBDataElementData, reparsed.dataElementType);
+ assertArrayEquals(BLOB_BYTES,
+ ((ObjectDataBLOBDataElementData) reparsed.data).objectDataBLOB.getData());
+ }
+
+ /**
+ * A revision store object whose BLOB reference resolves to an object data BLOB element
+ * must expose the BLOB bytes and hand them to the embedded document extractor during the
+ * tree walk.
+ */
+ @Test
+ public void testBlobIsEmittedAsEmbeddedDocument() throws Exception {
+ DataElement blobElement = roundTripBlobElement();
+
+ ObjectGroupObjectBLOBDataDeclaration declaration =
+ new ObjectGroupObjectBLOBDataDeclaration();
+ declaration.objectExGUID = new ExGuid(1, UUID.randomUUID());
+ declaration.objectPartitionID.setDecodedValue(2);
+ ObjectGroupObjectDataBLOBReference blobReference =
+ new ObjectGroupObjectDataBLOBReference();
+ blobReference.blobExtendedGUID = blobElement.dataElementExGuid;
+
+ ObjectGroupDataElementData groupData = new ObjectGroupDataElementData();
+ groupData.objectGroupDeclarations.objectGroupObjectBLOBDataDeclarationList
+ .add(declaration);
+ groupData.objectGroupData.objectGroupObjectDataBLOBReferenceList.add(blobReference);
+
+ RevisionStoreObjectGroup objectGroup =
+ RevisionStoreObjectGroup.createInstance(new ExGuid(2, UUID.randomUUID()),
+ groupData, false,
+ Collections.singletonMap(blobElement.dataElementExGuid, blobElement));
+ assertEquals(1, objectGroup.objects.size());
+ assertArrayEquals(BLOB_BYTES, objectGroup.objects.get(0).fileDataObject.getData());
+ addImageContainer(objectGroup, objectGroup.objects.get(0), "picture.png");
+
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.OtherFileNodeList.add(objectGroup);
+
+ List<byte[]> embedded = new ArrayList<>();
+ List<String> embeddedNames = new ArrayList<>();
+ List<String> embeddedTypes = new ArrayList<>();
+ ParseContext context = new ParseContext();
+ context.set(EmbeddedDocumentExtractor.class, new EmbeddedDocumentExtractor() {
+ @Override
+ public boolean shouldParseEmbedded(Metadata metadata) {
+ return true;
+ }
+
+ @Override
+ public void parseEmbedded(InputStream stream, ContentHandler handler,
+ Metadata metadata, boolean outputHtml) throws IOException {
+ embedded.add(stream.readAllBytes());
+ embeddedNames.add(metadata.get(TikaCoreProperties.RESOURCE_NAME_KEY));
+ embeddedTypes.add(metadata.get(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE));
+ }
+ });
+
+ Metadata metadata = new Metadata();
+ XHTMLContentHandler xhtml =
+ new XHTMLContentHandler(new ToTextContentHandler(new StringWriter()), metadata);
+ xhtml.startDocument();
+ pkg.walkTree(new OneNoteTreeWalkerOptions(), metadata, xhtml, context);
+ xhtml.endDocument();
+
+ assertEquals(1, embedded.size());
+ assertArrayEquals(BLOB_BYTES, embedded.get(0));
+ assertEquals("picture.png", embeddedNames.get(0));
+ assertEquals("INLINE", embeddedTypes.get(0));
+ }
+
+ private static void addImageContainer(RevisionStoreObjectGroup objectGroup,
+ RevisionStoreObject blob, String name) throws IOException {
+ RevisionStoreObject parent = new RevisionStoreObject();
+ parent.objectID = new ExGuid(9, UUID.randomUUID());
+ parent.objectGroupID = objectGroup.objectGroupID;
+ parent.referencedObjectID = new ExGUIDArray();
+ parent.referencedObjectID.content = Collections.singletonList(blob.objectID);
+
+ PropertySet propertySet = new PropertySet();
+ propertySet.cProperties = 2;
+ PropertyID fileNameID = new PropertyID();
+ fileNameID.type = PropertyType.FourBytesOfLengthFollowedByData.getIntVal();
+ fileNameID.value = 0x1C001DD7;
+ PropertyID containerID = new PropertyID();
+ containerID.type = PropertyType.ObjectID.getIntVal();
+ containerID.value = 0x20001C3F;
+ propertySet.rgPrids = new PropertyID[]{fileNameID, containerID};
+ PrtFourBytesOfLengthFollowedByData fileName = new PrtFourBytesOfLengthFollowedByData();
+ fileName.data = (name + "\u0000").getBytes(StandardCharsets.UTF_16LE);
+ fileName.cb = fileName.data.length;
+ propertySet.rgData = Arrays.asList(fileName, new NoData());
+ ObjectSpaceObjectPropSet propSet = new ObjectSpaceObjectPropSet();
+ propSet.body = propertySet;
+ ObjectGroupObjectData objectData = new ObjectGroupObjectData();
+ objectData.data.content.addAll(ByteUtil.toListOfByte(new byte[]{0, 0, 0, (byte) 0x80,
+ 0, 0, 0, 0}));
+ PropertySetObject propertySetObject = new PropertySetObject(null, objectData);
+ propertySetObject.objectSpaceObjectPropSet = propSet;
+ parent.propertySet = propertySetObject;
+ objectGroup.objects.add(0, parent);
+ }
+
+ @Test
+ public void testMissingBlobDataAndMalformedLengthAreHandled() throws Exception {
+ FileDataObject fileData = new FileDataObject();
+ assertNull(fileData.getData());
+
+ DataElement wrongType = new DataElement();
+ fileData.objectDataBLOBDataElement = wrongType;
+ assertNull(fileData.getData());
+
+ ObjectDataBLOBDataElementData emptyBlobData = new ObjectDataBLOBDataElementData();
+ emptyBlobData.objectDataBLOB = null;
+ wrongType.data = emptyBlobData;
+ assertNull(fileData.getData());
+
+ ObjectDataBLOB blob = new ObjectDataBLOB();
+ blob.data = null;
+ assertNull(blob.getData());
+ blob.data = new BinaryItem();
+ blob.data.content = null;
+ assertNull(blob.getData());
+ blob.data = new BinaryItem();
+ blob.data.content.add((byte) 1);
+ assertFalse(blob.serializeToByteList().isEmpty());
+
+ ObjectDataBLOB validBlob = new ObjectDataBLOB();
+ validBlob.data.content.add((byte) 1);
+ byte[] itemBytes = ByteUtil.toByteArray(validBlob.data.serializeToByteList());
+ AtomicInteger index = new AtomicInteger(0);
+ java.lang.reflect.Method deserialize = ObjectDataBLOB.class.getDeclaredMethod(
+ "deserializeItemsFromByteArray", byte[].class, AtomicInteger.class, int.class);
+ deserialize.setAccessible(true);
+ InvocationTargetException exception = assertThrows(InvocationTargetException.class,
+ () -> deserialize.invoke(new ObjectDataBLOB(), itemBytes, index,
+ itemBytes.length + 1));
+ assertTrue(exception.getCause() instanceof StreamObjectParseErrorException);
+ }
+
+ @Test
+ public void testEmptyAndFailedEmbeddedDataAreIgnored() throws Exception {
+ MSOneStorePackage emptyPackage = packageWithFileData(new ObjectDataBLOBDataElementData());
+ walkWithExtractor(emptyPackage, new EmbeddedDocumentExtractor() {
+ @Override
+ public boolean shouldParseEmbedded(Metadata metadata) {
+ throw new AssertionError("empty data must not reach the extractor");
+ }
+
+ @Override
+ public void parseEmbedded(InputStream stream, ContentHandler handler,
+ Metadata metadata, boolean outputHtml) {
+ }
+ });
+
+ DataElement blobElement = roundTripBlobElement();
+ MSOneStorePackage failedPackage = packageWithFileData(
+ (ObjectDataBLOBDataElementData) blobElement.data);
+ walkWithExtractor(failedPackage, new EmbeddedDocumentExtractor() {
+ @Override
+ public boolean shouldParseEmbedded(Metadata metadata) {
+ return true;
+ }
+
+ @Override
+ public void parseEmbedded(InputStream stream, ContentHandler handler,
+ Metadata metadata, boolean outputHtml) throws IOException {
+ throw new IOException("synthetic embedded parse failure");
+ }
+ });
+ }
+
+ @Test
+ public void testDeclarationWithoutObjectDataFailsCleanly() {
+ ObjectGroupDataElementData mismatched = new ObjectGroupDataElementData();
+ ObjectGroupObjectDeclare declaration = new ObjectGroupObjectDeclare();
+ declaration.objectPartitionID.setDecodedValue(1);
+ mismatched.objectGroupDeclarations.objectDeclarationList.add(declaration);
+
+ IOException e = assertThrows(IOException.class,
+ () -> RevisionStoreObjectGroup.createInstance(new ExGuid(9, UUID.randomUUID()),
+ mismatched, false, Collections.emptyMap()));
+ assertTrue(e.getMessage().contains("Missing object data"));
+ }
+
+ @Test
+ public void testEncryptedObjectGroupAndMissingBlobReference() throws Exception {
+ ObjectGroupDataElementData encryptedData = new ObjectGroupDataElementData();
+ ObjectGroupObjectDeclare declaration = new ObjectGroupObjectDeclare();
+ declaration.objectPartitionID.setDecodedValue(1);
+ ObjectGroupObjectData objectData = new ObjectGroupObjectData();
+ objectData.data.content.add((byte) 42);
+ encryptedData.objectGroupDeclarations.objectDeclarationList.add(declaration);
+ encryptedData.objectGroupData.objectGroupObjectDataList.add(objectData);
+
+ RevisionStoreObjectGroup encrypted = RevisionStoreObjectGroup.createInstance(
+ new ExGuid(3, UUID.randomUUID()), encryptedData, true, Collections.emptyMap());
+ assertEquals(1, encrypted.encryptionObjects.size());
+
+ ObjectGroupDataElementData missingBlobData = new ObjectGroupDataElementData();
+ ObjectGroupObjectBLOBDataDeclaration blobDeclaration =
+ new ObjectGroupObjectBLOBDataDeclaration();
+ blobDeclaration.objectExGUID = new ExGuid(4, UUID.randomUUID());
+ blobDeclaration.objectPartitionID.setDecodedValue(2);
+ ObjectGroupObjectDataBLOBReference reference =
+ new ObjectGroupObjectDataBLOBReference();
+ reference.blobExtendedGUID = new ExGuid(5, UUID.randomUUID());
+ missingBlobData.objectGroupDeclarations.objectGroupObjectBLOBDataDeclarationList
+ .add(blobDeclaration);
+ missingBlobData.objectGroupData.objectGroupObjectDataBLOBReferenceList.add(reference);
+ RevisionStoreObjectGroup missing = RevisionStoreObjectGroup.createInstance(
+ new ExGuid(6, UUID.randomUUID()), missingBlobData, false, Collections.emptyMap());
+ assertEquals(1, missing.objects.size());
+ assertNull(missing.objects.get(0).fileDataObject.getData());
+ }
+
+ private MSOneStorePackage packageWithFileData(ObjectDataBLOBDataElementData blobData) {
+ DataElement element = new DataElement();
+ element.data = blobData;
+ FileDataObject fileData = new FileDataObject();
+ fileData.objectDataBLOBDataElement = element;
+ RevisionStoreObject object = new RevisionStoreObject();
+ object.objectID = new ExGuid(7, UUID.randomUUID());
+ object.fileDataObject = fileData;
+ RevisionStoreObjectGroup group = new RevisionStoreObjectGroup(
+ new ExGuid(8, UUID.randomUUID()));
+ group.objects.add(object);
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.OtherFileNodeList.add(group);
+ return pkg;
+ }
+
+ private void walkWithExtractor(MSOneStorePackage pkg, EmbeddedDocumentExtractor extractor)
+ throws Exception {
+ ParseContext context = new ParseContext();
+ context.set(EmbeddedDocumentExtractor.class, extractor);
+ Metadata metadata = new Metadata();
+ XHTMLContentHandler xhtml = new XHTMLContentHandler(new ToTextContentHandler(new StringWriter()), metadata);
+ xhtml.startDocument();
+ pkg.walkTree(new OneNoteTreeWalkerOptions(), metadata, xhtml, context);
+ xhtml.endDocument();
+ }
+
+ private DataElement roundTripBlobElement() throws Exception {
+ ObjectDataBLOBDataElementData blobData = new ObjectDataBLOBDataElementData();
+ for (byte b : BLOB_BYTES) {
+ blobData.objectDataBLOB.data.content.add(b);
+ }
+ DataElement blobElement =
+ new DataElement(DataElementType.ObjectDataBLOBDataElementData, blobData);
+ byte[] serialized = ByteUtil.toByteArray(blobElement.serializeToByteList());
+ return StreamObject.getCurrent(serialized, new AtomicInteger(0), DataElement.class);
+ }
+}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackageTest.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackageTest.java
new file mode 100644
index 0000000000..ab597063fc
--- /dev/null
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackageTest.java
@@ -0,0 +1,846 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.parser.microsoft.onenote.fsshttpb;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.StringWriter;
+import java.lang.reflect.Method;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+
+import org.junit.jupiter.api.Test;
+import org.xml.sax.SAXException;
+import org.xml.sax.helpers.DefaultHandler;
+
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.TikaCoreProperties;
+import org.apache.tika.parser.ParseContext;
+import org.apache.tika.parser.microsoft.onenote.OneNoteParser;
+import org.apache.tika.parser.microsoft.onenote.OneNoteTreeWalkerOptions;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.ArrayNumber;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.IProperty;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.NoData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.PrtArrayOfPropertyValues;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.PrtFourBytesOfLengthFollowedByData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.DataElement;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.FileDataObject;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.ObjectDataBLOB;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.ObjectDataBLOBDataElementData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.ObjectGroupObjectData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.PropertySet;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.PropertySetObject;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionManifestRootDeclare;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionStoreCell;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionStoreObject;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionStoreObjectGroup;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.CellID;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.DataElementType;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.ExGuid;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.PropertyID;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.PropertyType;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.space.ObjectSpaceObjectPropSet;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.util.ByteUtil;
+import org.apache.tika.sax.ToTextContentHandler;
+import org.apache.tika.sax.ToXMLContentHandler;
+import org.apache.tika.sax.XHTMLContentHandler;
+
+public class MSOneStorePackageTest {
+
+ @Test
+ public void testParseWarningsAreBoundedAcrossParserAndWalkPhases() throws Exception {
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ for (int i = 0; i < 99; i++) {
+ pkg.recordParseWarning("warning " + i);
+ }
+ pkg.recordParseWarning("duplicate warning");
+ pkg.recordParseWarning("duplicate warning");
+ RevisionStoreCell damagedCell = new RevisionStoreCell();
+ RevisionManifestRootDeclare missingRoot = new RevisionManifestRootDeclare();
+ missingRoot.objectExGuid = id(1006);
+ damagedCell.rootDeclares.add(missingRoot);
+ pkg.cells.add(damagedCell);
+
+ Metadata metadata = new Metadata();
+ pkg.walkTree(new OneNoteTreeWalkerOptions(), metadata,
+ new XHTMLContentHandler(new ToTextContentHandler(), metadata),
+ new ParseContext());
+
+ String[] warnings = metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING);
+ assertEquals(101, warnings.length);
+ assertEquals(1, Arrays.stream(warnings)
+ .filter(warning -> warning.equals("duplicate warning")).count());
+ assertTrue(Arrays.stream(warnings).anyMatch(warning -> warning.contains("suppressed")));
+ }
+
+ @Test
+ public void testReferenceArrayReportsUnavailableAndCappedWarnings() throws Exception {
+ ExGuid rootId = id(2000);
+ List<ExGuid> references = new ArrayList<>(100_001);
+ for (int i = 0; i < 100_001; i++) {
+ references.add(rootId);
+ }
+ RevisionStoreObject root = object(rootId,
+ propertySet(new PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001D5F,
+ arrayNumber(100_002))), references, Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(root));
+ RevisionManifestRootDeclare rootDeclare = new RevisionManifestRootDeclare();
+ rootDeclare.objectExGuid = rootId;
+ cell.rootDeclares.add(rootDeclare);
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+
+ Metadata metadata = new Metadata();
+ walk(pkg, metadata);
+ String[] warnings = metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING);
+ assertTrue(Arrays.stream(warnings)
+ .anyMatch(warning -> warning.contains("declared 100002 entries")));
+ assertTrue(Arrays.stream(warnings)
+ .anyMatch(warning -> warning.contains("Capping OneNote object reference array")));
+ }
+
+ @Test
+ public void testPagesFollowSectionOrderAndDropOlderCellVersions() throws Exception {
+ ExGuid sectionRootId = id(1);
+ CellID pageOne = cell(10, 100);
+ CellID pageTwo = cell(20, 200);
+ CellID oldPageOne = cell(11, 100);
+ RevisionStoreCell pageTwoCell = cellWithText(pageTwo, "page two");
+ RevisionStoreCell oldPageOneCell = cellWithText(oldPageOne, "old page one");
+ RevisionStoreCell pageOneCell = cellWithText(pageOne, "page one");
+ RevisionStoreCell unrelatedCell = cellWithText(cell(30, 300), "unrelated");
+
+ RevisionStoreObject sectionRoot = object(sectionRootId,
+ propertySet(new PropertySpec(PropertyType.ObjectSpaceID, 0x20001D78,
+ new NoData()),
+ new PropertySpec(PropertyType.ObjectSpaceID, 0x20001D79,
+ new NoData())),
+ Collections.emptyList(), Arrays.asList(pageOne, pageTwo));
+ RevisionStoreCell section = new RevisionStoreCell();
+ section.objectGroups.add(group(sectionRoot));
+ RevisionManifestRootDeclare rootDeclare = new RevisionManifestRootDeclare();
+ rootDeclare.objectExGuid = sectionRootId;
+ section.rootDeclares.add(rootDeclare);
+ RevisionManifestRootDeclare missingRoot = new RevisionManifestRootDeclare();
+ missingRoot.objectExGuid = id(1000);
+ section.rootDeclares.add(missingRoot);
+
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.dataRootCell = section;
+ pkg.cells.addAll(Arrays.asList(pageTwoCell, oldPageOneCell, pageOneCell, unrelatedCell));
+
+ Metadata metadata = new Metadata();
+ String text = walk(pkg, metadata);
+ assertTrue(text.indexOf("page one") >= 0);
+ assertTrue(text.indexOf("page two") >= 0);
+ assertTrue(text.indexOf("page one") < text.indexOf("page two"));
+ assertFalse(text.contains("old page one"));
+ assertTrue(text.contains("unrelated"));
+ assertTrue(Arrays.stream(metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(warning -> warning.contains("could not be resolved")));
+ }
+
+ @Test
+ public void testUnresolvedRootsFallBackToAllObjects() throws Exception {
+ RevisionStoreCell cell = cellWithText(cell(1, 1), "fallback content");
+ cell.rootDeclares.clear();
+ RevisionManifestRootDeclare missingRoot = new RevisionManifestRootDeclare();
+ missingRoot.objectExGuid = id(999);
+ cell.rootDeclares.add(missingRoot);
+
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+ Metadata metadata = new Metadata();
+
+ assertTrue(walk(pkg, metadata).contains("fallback content"));
+ assertTrue(Arrays.stream(metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(warning -> warning.contains(id(999).toString())));
+ }
+
+ @Test
+ public void testExhaustedObjectReferenceTriggersFallbackWarning() throws Exception {
+ RevisionStoreObject root = object(id(1004), propertySet(
+ new PropertySpec(PropertyType.ObjectID, 0x20001D78, new NoData())),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreObject fallback = object(id(1005), propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("fallback after exhausted reference"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(root, fallback));
+ RevisionManifestRootDeclare rootDeclare = new RevisionManifestRootDeclare();
+ rootDeclare.objectExGuid = root.objectID;
+ cell.rootDeclares.add(rootDeclare);
+
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+ Metadata metadata = new Metadata();
+ String text = walk(pkg, metadata);
+
+ assertFalse(text.contains("fallback after exhausted reference"));
+ assertTrue(Arrays.stream(metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(warning -> warning.contains("reference slot was exhausted")));
+ }
+
+ @Test
+ public void testExhaustedObjectSpaceReferenceWarns() throws Exception {
+ ExGuid sectionRootID = id(1007);
+ RevisionStoreCell section = new RevisionStoreCell();
+ section.objectGroups.add(group(object(sectionRootID,
+ propertySet(new PropertySpec(PropertyType.ObjectSpaceID, 0x20001D78,
+ new NoData())), Collections.emptyList(), Collections.emptyList())));
+ RevisionManifestRootDeclare sectionRoot = new RevisionManifestRootDeclare();
+ sectionRoot.objectExGuid = sectionRootID;
+ section.rootDeclares.add(sectionRoot);
+ RevisionStoreCell page = cellWithText(cell(1008, 1009), "page after missing space");
+
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.dataRootCell = section;
+ pkg.cells.add(page);
+ Metadata metadata = new Metadata();
+ String text = walk(pkg, metadata);
+
+ assertTrue(text.contains("page after missing space"));
+ assertTrue(Arrays.stream(metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(warning -> warning.contains("object-space reference slot was exhausted")));
+ }
+
+ @Test
+ public void testMixedRootsDoNotTriggerAllObjectFallback() throws Exception {
+ RevisionStoreObject root = object(id(1001), propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("resolved root"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreObject unrelated = object(id(1002), propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("unrelated object"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(root, unrelated));
+ RevisionManifestRootDeclare resolvedRoot = new RevisionManifestRootDeclare();
+ resolvedRoot.objectExGuid = root.objectID;
+ cell.rootDeclares.add(resolvedRoot);
+ RevisionManifestRootDeclare missingRoot = new RevisionManifestRootDeclare();
+ missingRoot.objectExGuid = id(1003);
+ cell.rootDeclares.add(missingRoot);
+
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+ Metadata metadata = new Metadata();
+ String text = walk(pkg, metadata);
+
+ assertTrue(text.contains("resolved root"));
+ assertFalse(text.contains("unrelated object"));
+ assertTrue(Arrays.stream(metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(warning -> warning.contains(id(1003).toString())));
+ }
+
+ @Test
+ public void testOriginalAuthorBecomesCreator() throws Exception {
+ ExGuid authorId = id(701);
+ RevisionStoreObject root = object(id(700), propertySet(
+ new PropertySpec(PropertyType.ObjectID, 0x20001D78, new NoData())),
+ Collections.singletonList(authorId), Collections.emptyList());
+ RevisionStoreObject author = object(authorId, propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C001D75, utf16Text("Иван Петров"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(root, author));
+ RevisionManifestRootDeclare rootDeclare = new RevisionManifestRootDeclare();
+ rootDeclare.objectExGuid = root.objectID;
+ cell.rootDeclares.add(rootDeclare);
+
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+ Metadata metadata = new Metadata();
+ walk(pkg, metadata);
+ assertEquals("Иван Петров", metadata.get(TikaCoreProperties.CREATOR));
+ assertEquals("Иван Петров", metadata.get(OneNoteParser.ONE_NOTE_PREFIX + "originalAuthors"));
+ }
+
+ @Test
+ public void testDualRoleAuthorRecordedForBothRoles() throws Exception {
+ ExGuid authorId = id(710);
+ // the same author object referenced as both AuthorOriginal and AuthorMostRecent
+ RevisionStoreObject root = object(id(711), propertySet(
+ new PropertySpec(PropertyType.ObjectID, 0x20001D78, new NoData()),
+ new PropertySpec(PropertyType.ObjectID, 0x20001D79, new NoData())),
+ Arrays.asList(authorId, authorId), Collections.emptyList());
+ RevisionStoreObject author = object(authorId, propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C001D75, utf16Text("Single Author"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(root, author));
+ RevisionManifestRootDeclare rootDeclare = new RevisionManifestRootDeclare();
+ rootDeclare.objectExGuid = root.objectID;
+ cell.rootDeclares.add(rootDeclare);
+
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+ Metadata metadata = new Metadata();
+ walk(pkg, metadata);
+
+ assertEquals("Single Author", metadata.get(TikaCoreProperties.CREATOR));
+ assertEquals("Single Author", metadata.get(OneNoteParser.ONE_NOTE_PREFIX + "originalAuthors"));
+ assertEquals("Single Author", metadata.get(OneNoteParser.ONE_NOTE_PREFIX + "mostRecentAuthors"));
+ }
+
+ @Test
+ public void testBlobOnlyRootWithDanglingContentRootWalksAllObjects() throws Exception {
+ RevisionStoreObject blobRoot = object(id(620), propertySet(),
+ Collections.emptyList(), Collections.emptyList());
+ blobRoot.propertySet = null;
+ blobRoot.fileDataObject = fileData("blob root");
+ RevisionStoreObject textObject = object(id(621), propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("page body text"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(blobRoot, textObject));
+ RevisionManifestRootDeclare blobDeclare = new RevisionManifestRootDeclare();
+ blobDeclare.objectExGuid = blobRoot.objectID;
+ cell.rootDeclares.add(blobDeclare);
+ RevisionManifestRootDeclare danglingContentRoot = new RevisionManifestRootDeclare();
+ danglingContentRoot.objectExGuid = id(622);
+ cell.rootDeclares.add(danglingContentRoot);
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+ Metadata metadata = new Metadata();
+
+ // the blob root alone cannot reach the page body; the dangling content root must
+ // trigger the walk-everything fallback so the body is not lost
+ assertTrue(walk(pkg, metadata).contains("page body text"));
+ assertTrue(Arrays.stream(metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(warning -> warning.contains("walking all objects")));
+ }
+
+ @Test
+ public void testHasEmittedContentTracksTextAndEmptyWalks() throws Exception {
+ MSOneStorePackage withText = new MSOneStorePackage();
+ withText.cells.add(cellWithText(cell(70, 71), "some text"));
+ walk(withText);
+ assertTrue(withText.hasEmittedContent());
+
+ MSOneStorePackage empty = new MSOneStorePackage();
+ RevisionStoreCell danglingCell = new RevisionStoreCell();
+ RevisionManifestRootDeclare missingRoot = new RevisionManifestRootDeclare();
+ missingRoot.objectExGuid = id(72);
+ danglingCell.rootDeclares.add(missingRoot);
+ empty.cells.add(danglingCell);
+ walk(empty);
+ assertFalse(empty.hasEmittedContent());
+ }
+
+ @Test
+ public void testSanitizeResourceNameKeepsBasenameOfPathShapedNames() {
+ assertEquals("pic 1.png",
+ MSOneStorePackage.sanitizeResourceName("D:\\images\\pic 1.png"));
+ assertEquals("pic.png", MSOneStorePackage.sanitizeResourceName("C:pic.png"));
+ assertEquals("a.png", MSOneStorePackage.sanitizeResourceName("/tmp/a.png"));
+ assertEquals("plain.png", MSOneStorePackage.sanitizeResourceName("plain.png"));
+ assertEquals("", MSOneStorePackage.sanitizeResourceName(".."));
+ assertEquals("", MSOneStorePackage.sanitizeResourceName("D:\\images\\.."));
+ }
+
+ @Test
+ public void testBlobOnlyRootDoesNotFallBackToOtherObjects() throws Exception {
+ RevisionStoreObject blobRoot = object(id(610), propertySet(),
+ Collections.emptyList(), Collections.emptyList());
+ blobRoot.propertySet = null;
+ blobRoot.fileDataObject = fileData("blob root");
+ RevisionStoreObject textObject = object(id(611), propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("fallback content"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(blobRoot, textObject));
+ RevisionManifestRootDeclare rootDeclare = new RevisionManifestRootDeclare();
+ rootDeclare.objectExGuid = blobRoot.objectID;
+ cell.rootDeclares.add(rootDeclare);
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+
+ assertFalse(walk(pkg).contains("fallback content"));
+ }
+
+ @Test
+ public void testPageMarkupIsBalancedAndPinned() throws Exception {
+ CellID pageID = cell(90, 91);
+ RevisionStoreCell page = cellWithText(pageID, "page text");
+ ExGuid sectionRootID = id(92);
+ RevisionStoreCell section = new RevisionStoreCell();
+ section.objectGroups.add(group(object(sectionRootID,
+ propertySet(new PropertySpec(PropertyType.ObjectSpaceID, 0x20001D78,
+ new NoData())), Collections.emptyList(),
+ Collections.singletonList(pageID))));
+ RevisionManifestRootDeclare sectionRoot = new RevisionManifestRootDeclare();
+ sectionRoot.objectExGuid = sectionRootID;
+ section.rootDeclares.add(sectionRoot);
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.dataRootCell = section;
+ pkg.cells.add(page);
+
+ String xml = walkXml(pkg);
+ assertEquals(1, count(xml, "<div class=\"page\">"));
+ assertEquals(1, count(xml, "</div>"));
+ assertTrue(xml.contains("page text"));
+ }
+
+ @Test
+ public void testPageMarkupClosesWhenWalkThrows() throws Exception {
+ CellID pageID = cell(95, 96);
+ RevisionStoreCell page = cellWithText(pageID, "page text");
+ ExGuid sectionRootID = id(97);
+ RevisionStoreCell section = new RevisionStoreCell();
+ section.objectGroups.add(group(object(sectionRootID,
+ propertySet(new PropertySpec(PropertyType.ObjectSpaceID, 0x20001D78,
+ new NoData())), Collections.emptyList(),
+ Collections.singletonList(pageID))));
+ RevisionManifestRootDeclare sectionRoot = new RevisionManifestRootDeclare();
+ sectionRoot.objectExGuid = sectionRootID;
+ section.rootDeclares.add(sectionRoot);
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.dataRootCell = section;
+ pkg.cells.add(page);
+ Metadata metadata = new Metadata();
+ List<String> elements = new ArrayList<>();
+ DefaultHandler recordingHandler = new DefaultHandler() {
+ private final StringBuilder text = new StringBuilder();
+
+ @Override
+ public void startElement(String uri, String localName, String qName,
+ org.xml.sax.Attributes atts) {
+ elements.add(qName);
+ }
+
+ @Override
+ public void endElement(String uri, String localName, String qName) {
+ elements.add("/" + qName);
+ }
+
+ @Override
+ public void characters(char[] ch, int start, int length) throws SAXException {
+ text.append(ch, start, length);
+ if (text.indexOf("page text") >= 0) {
+ throw new SAXException("intentional test failure");
+ }
+ }
+ };
+ XHTMLContentHandler xhtml = new XHTMLContentHandler(recordingHandler, metadata);
+ xhtml.startDocument();
+
+ assertThrows(SAXException.class, () -> pkg.walkTree(new OneNoteTreeWalkerOptions(),
+ metadata, xhtml, new ParseContext()));
+ assertTrue(elements.contains("p"));
+ assertTrue(elements.contains("/p"));
+ assertTrue(elements.contains("div"));
+ assertTrue(elements.contains("/div"));
+ assertTrue(elements.indexOf("p") < elements.indexOf("/p"));
+ assertTrue(elements.indexOf("div") < elements.indexOf("/div"));
+ assertTrue(elements.indexOf("/p") < elements.indexOf("/div"));
+ }
+
+ @Test
+ public void testRemoveSupersededObjectsKeepsNewestVersion() throws Exception {
+ ExGuid objectID = id(950);
+ RevisionStoreObject oldObject = object(objectID, propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("old"))), Collections.emptyList(),
+ Collections.emptyList());
+ RevisionStoreObject newObject = object(objectID, propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("new"))), Collections.emptyList(),
+ Collections.emptyList());
+ List<RevisionStoreObjectGroup> groups = new ArrayList<>(Arrays.asList(
+ group(oldObject), group(newObject)));
+ Method removeSupersededObjects = MSOneStoreParser.class.getDeclaredMethod(
+ "removeSupersededObjects", List.class);
+ removeSupersededObjects.setAccessible(true);
+
+ removeSupersededObjects.invoke(new MSOneStoreParser(), groups);
+
+ assertEquals(1, groups.get(0).objects.size());
+ assertSame(newObject, groups.get(0).objects.get(0));
+ assertTrue(groups.get(1).objects.isEmpty());
+ }
+
+ @Test
+ public void testArrayCountsAndRecursionDepthAreBounded() throws Exception {
+ RevisionStoreObject hugeArrayRoot = object(id(600), propertySet(
+ new PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001D5F,
+ arrayNumber(Integer.MAX_VALUE)),
+ new PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001D5F,
+ new NoData())),
+ Collections.emptyList(), Collections.emptyList());
+ Method collectActions = MSOneStorePackage.class.getDeclaredMethod("collectActions",
+ PropertySet.class, List.class, int[].class, List.class, int[].class, List.class,
+ int.class);
+ collectActions.setAccessible(true);
+ List<Object> actions = new ArrayList<>();
+ collectActions.invoke(new MSOneStorePackage(),
+ hugeArrayRoot.propertySet.objectSpaceObjectPropSet.body,
+ Collections.emptyList(), new int[]{0}, Collections.emptyList(), new int[]{0},
+ actions, 0);
+ assertTrue(actions.isEmpty());
+ actions.clear();
+ collectActions.invoke(new MSOneStorePackage(),
+ hugeArrayRoot.propertySet.objectSpaceObjectPropSet.body,
+ Collections.singletonList(id(601)), new int[]{0}, Collections.emptyList(),
+ new int[]{0}, actions, 1000);
+ assertTrue(actions.isEmpty());
+
+ Method collectReferencedCells = MSOneStorePackage.class.getDeclaredMethod(
+ "collectReferencedCells", RevisionStoreObject.class, Map.class, Set.class,
+ List.class, int.class);
+ collectReferencedCells.setAccessible(true);
+ RevisionStoreObject referencedRoot = object(id(602), propertySet(
+ new PropertySpec(PropertyType.ObjectSpaceID, 0x20001D78, new NoData())),
+ Collections.emptyList(), Collections.singletonList(cell(603, 604)));
+ List<CellID> orderedCellIds = new ArrayList<>();
+ collectReferencedCells.invoke(new MSOneStorePackage(), referencedRoot,
+ new java.util.HashMap<>(), new java.util.HashSet<>(), orderedCellIds, 1000);
+ assertTrue(orderedCellIds.isEmpty());
+
+ PropertySet alignedSet = propertySet(
+ new PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001D5F,
+ arrayNumber(100001)),
+ new PropertySpec(PropertyType.ObjectID, 0x24001D5F, new NoData()));
+ actions.clear();
+ collectActions.invoke(new MSOneStorePackage(), alignedSet,
+ Collections.nCopies(100001, id(601)), new int[]{0}, Collections.emptyList(),
+ new int[]{0}, actions, 0);
+ assertEquals(100001, actions.size());
+ java.lang.reflect.Field childReference = actions.get(actions.size() - 1).getClass()
+ .getDeclaredField("childReference");
+ childReference.setAccessible(true);
+ assertNull(childReference.get(actions.get(actions.size() - 1)));
+
+ Method walkObject = MSOneStorePackage.class.getDeclaredMethod("walkObject",
+ RevisionStoreObject.class, Map.class, Set.class,
+ Class.forName(MSOneStorePackage.class.getName() + "$AuthorRole"),
+ OneNoteTreeWalkerOptions.class, Metadata.class, XHTMLContentHandler.class,
+ int.class);
+ walkObject.setAccessible(true);
+ // an object that emits text when walked, so the depth-capped walk's blank
+ // output proves the cap fired rather than the setup having nothing to emit
+ RevisionStoreObject textRoot = object(id(605), propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("depth capped text"))),
+ Collections.emptyList(), Collections.emptyList());
+ Metadata metadata = new Metadata();
+ StringWriter writer = new StringWriter();
+ XHTMLContentHandler xhtml = new XHTMLContentHandler(new ToTextContentHandler(writer), metadata);
+ xhtml.startDocument();
+ walkObject.invoke(new MSOneStorePackage(), textRoot,
+ new java.util.HashMap<>(), new java.util.HashSet<>(), null,
+ new OneNoteTreeWalkerOptions(), metadata, xhtml, 1000);
+ assertTrue(writer.toString().isBlank());
+ walkObject.invoke(new MSOneStorePackage(), textRoot,
+ new java.util.HashMap<>(), new java.util.HashSet<>(), null,
+ new OneNoteTreeWalkerOptions(), metadata, xhtml, 0);
+ xhtml.endDocument();
+ assertTrue(writer.toString().contains("depth capped text"));
+ }
+
+ @Test
+ public void testDanglingPrimaryPictureDoesNotSuppressWebPicture() throws Exception {
+ ExGuid missingPictureID = id(50);
+ ExGuid webPictureID = id(51);
+ RevisionStoreObject root = object(id(52), propertySet(
+ new PropertySpec(PropertyType.ObjectID, 0x20001C3F, new NoData()),
+ new PropertySpec(PropertyType.ObjectID, 0x200034C8, new NoData())),
+ Arrays.asList(missingPictureID, webPictureID), Collections.emptyList());
+ RevisionStoreObject webPicture = object(webPictureID, propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("derived picture"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(root, webPicture));
+ RevisionManifestRootDeclare rootDeclare = new RevisionManifestRootDeclare();
+ rootDeclare.objectExGuid = root.objectID;
+ cell.rootDeclares.add(rootDeclare);
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+
+ String text = walk(pkg);
+ assertTrue(text.contains("derived picture"), text);
+ }
+
+ @Test
+ public void testContentlessPrimaryPictureDoesNotSuppressDerivedPicture() throws Exception {
+ ExGuid pictureID = id(40);
+ ExGuid webPictureID = id(41);
+ RevisionStoreObject root = object(id(42), propertySet(
+ new PropertySpec(PropertyType.ObjectID, 0x20001C3F, new NoData()),
+ new PropertySpec(PropertyType.ObjectID, 0x200034C8, new NoData())),
+ Arrays.asList(pictureID, webPictureID), Collections.emptyList());
+ RevisionStoreObject picture = object(pictureID, propertySet(),
+ Collections.emptyList(), Collections.emptyList());
+ picture.propertySet.objectSpaceObjectPropSet.body = null;
+ RevisionStoreObject webPicture = object(webPictureID, propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("derived picture"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(root, picture, webPicture));
+ RevisionManifestRootDeclare rootDeclare = new RevisionManifestRootDeclare();
+ rootDeclare.objectExGuid = root.objectID;
+ cell.rootDeclares.add(rootDeclare);
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+
+ assertTrue(walk(pkg).contains("derived picture"));
+ }
+
+ @Test
+ public void testUsablePrimaryPictureSuppressesDerivedPicture() throws Exception {
+ ExGuid pictureID = id(80);
+ ExGuid webPictureID = id(81);
+ RevisionStoreObject root = object(id(82), propertySet(
+ new PropertySpec(PropertyType.ObjectID, 0x20001C3F, new NoData()),
+ new PropertySpec(PropertyType.ObjectID, 0x200034C8, new NoData())),
+ Arrays.asList(pictureID, webPictureID), Collections.emptyList());
+ RevisionStoreObject picture = object(pictureID, propertySet(),
+ Collections.emptyList(), Collections.emptyList());
+ picture.fileDataObject = fileData("primary image");
+ RevisionStoreObject webPicture = object(webPictureID, propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("derived picture"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(root, picture, webPicture));
+ RevisionManifestRootDeclare rootDeclare = new RevisionManifestRootDeclare();
+ rootDeclare.objectExGuid = root.objectID;
+ cell.rootDeclares.add(rootDeclare);
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+
+ assertFalse(walk(pkg).contains("derived picture"));
+ }
+
+ @Test
+ public void testNestedPropertySetsAndMissingReferencesAreTraversedSafely() throws Exception {
+ ExGuid childId = id(2);
+ PropertySet nested = propertySet(new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("nested text")));
+ PrtArrayOfPropertyValues array = new PrtArrayOfPropertyValues();
+ array.data = new PropertySet[]{propertySet(new PropertySpec(
+ PropertyType.FourBytesOfLengthFollowedByData, 0x1C003498, text("array text")))};
+ RevisionStoreObject root = object(id(1), propertySet(
+ new PropertySpec(PropertyType.PropertySet, 0, nested),
+ new PropertySpec(PropertyType.ArrayOfPropertyValues, 0, array),
+ new PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001D5F,
+ arrayNumber(2)),
+ new PropertySpec(PropertyType.ObjectID, 0x24001D5F, new NoData()),
+ new PropertySpec(PropertyType.ObjectSpaceID, 0x20001D78, new NoData()),
+ new PropertySpec(PropertyType.ObjectSpaceID, 0x20001D79, new NoData()),
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C001DD7, bytes((byte) 'u', (byte) 0, (byte) 1)),
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C001C22, bytes((byte) 'h', (byte) 0, (byte) 'i', (byte) 0,
+ (byte) 0, (byte) 0))),
+ Collections.singletonList(childId), Collections.singletonList(cell(50, 51)));
+ RevisionStoreObject child = object(childId, propertySet(
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("child text"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(root, child));
+ RevisionManifestRootDeclare rootDeclare = new RevisionManifestRootDeclare();
+ rootDeclare.objectExGuid = root.objectID;
+ cell.rootDeclares.add(rootDeclare);
+
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+ String text = walk(pkg);
+ assertTrue(text.contains("nested text"));
+ assertTrue(text.contains("array text"));
+ assertTrue(text.contains("child text"));
+ assertTrue(text.contains("u"));
+ assertTrue(text.contains("hi"));
+ }
+
+ private static String walk(MSOneStorePackage pkg) throws Exception {
+ Metadata metadata = new Metadata();
+ return walk(pkg, metadata);
+ }
+
+ private static String walk(MSOneStorePackage pkg, Metadata metadata) throws Exception {
+ ParseContext context = new ParseContext();
+ StringWriter writer = new StringWriter();
+ XHTMLContentHandler xhtml = new XHTMLContentHandler(new ToTextContentHandler(writer), metadata);
+ xhtml.startDocument();
+ pkg.walkTree(new OneNoteTreeWalkerOptions(), metadata, xhtml, context);
+ xhtml.endDocument();
+ return writer.toString();
+ }
+
+ private static String walkXml(MSOneStorePackage pkg) throws Exception {
+ Metadata metadata = new Metadata();
+ ParseContext context = new ParseContext();
+ ToXMLContentHandler xml = new ToXMLContentHandler();
+ XHTMLContentHandler xhtml = new XHTMLContentHandler(xml, metadata);
+ xhtml.startDocument();
+ pkg.walkTree(new OneNoteTreeWalkerOptions(), metadata, xhtml, context);
+ xhtml.endDocument();
+ return xml.toString();
+ }
+
+ private static int count(String value, String needle) {
+ return value.split(java.util.regex.Pattern.quote(needle), -1).length - 1;
+ }
+
+ private static RevisionStoreCell cellWithText(CellID cellID, String value) throws Exception {
+ RevisionStoreObject object = object(id(cellID.extendGUID1.hashCode()),
+ propertySet(new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text(value))), Collections.emptyList(), Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.cellID = cellID;
+ cell.objectGroups.add(group(object));
+ RevisionManifestRootDeclare rootDeclare = new RevisionManifestRootDeclare();
+ rootDeclare.objectExGuid = object.objectID;
+ cell.rootDeclares.add(rootDeclare);
+ return cell;
+ }
+
+ private static RevisionStoreObject object(ExGuid objectID, PropertySet body,
+ List<ExGuid> references, List<CellID> spaces)
+ throws Exception {
+ RevisionStoreObject object = new RevisionStoreObject();
+ object.objectID = objectID;
+ PropertySetObject propertySetObject = new PropertySetObject(null, emptyObjectData());
+ ObjectSpaceObjectPropSet propSet = new ObjectSpaceObjectPropSet();
+ propSet.body = body;
+ propertySetObject.objectSpaceObjectPropSet = propSet;
+ object.propertySet = propertySetObject;
+ if (!references.isEmpty()) {
+ object.referencedObjectID = new org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.ExGUIDArray();
+ object.referencedObjectID.content = references;
+ }
+ if (!spaces.isEmpty()) {
+ object.referencedObjectSpacesID = new org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.CellIDArray();
+ object.referencedObjectSpacesID.content = spaces;
+ }
+ return object;
+ }
+
+ private static FileDataObject fileData(String value) {
+ ObjectDataBLOB blob = new ObjectDataBLOB();
+ blob.data.content.addAll(ByteUtil.toListOfByte(value.getBytes(StandardCharsets.UTF_8)));
+ ObjectDataBLOBDataElementData blobData = new ObjectDataBLOBDataElementData();
+ blobData.objectDataBLOB = blob;
+ DataElement element = new DataElement();
+ element.dataElementType = DataElementType.ObjectDataBLOBDataElementData;
+ element.data = blobData;
+ FileDataObject fileData = new FileDataObject();
+ fileData.objectDataBLOBDataElement = element;
+ return fileData;
+ }
+
+ private static RevisionStoreObjectGroup group(RevisionStoreObject... objects) {
+ RevisionStoreObjectGroup group = new RevisionStoreObjectGroup(id(500));
+ group.objects.addAll(Arrays.asList(objects));
+ return group;
+ }
+
+ private static PropertySet propertySet(PropertySpec... specs) {
+ PropertySet set = new PropertySet();
+ set.cProperties = specs.length;
+ set.rgPrids = new PropertyID[specs.length];
+ set.rgData = new ArrayList<>();
+ for (int i = 0; i < specs.length; i++) {
+ set.rgPrids[i] = propertyID(specs[i].type, specs[i].value);
+ set.rgData.add(specs[i].property);
+ }
+ return set;
+ }
+
+ private static PropertyID propertyID(PropertyType type, int value) {
+ PropertyID id = new PropertyID();
+ id.type = type.getIntVal();
+ id.value = value;
+ return id;
+ }
+
+ private static PrtFourBytesOfLengthFollowedByData bytes(byte... value) {
+ PrtFourBytesOfLengthFollowedByData data = new PrtFourBytesOfLengthFollowedByData();
+ data.data = value;
+ data.cb = data.data.length;
+ return data;
+ }
+
+ private static PrtFourBytesOfLengthFollowedByData text(String value) {
+ PrtFourBytesOfLengthFollowedByData data = new PrtFourBytesOfLengthFollowedByData();
+ data.data = value.getBytes(StandardCharsets.US_ASCII);
+ data.cb = data.data.length;
+ return data;
+ }
+
+ private static PrtFourBytesOfLengthFollowedByData utf16Text(String value) {
+ PrtFourBytesOfLengthFollowedByData data = new PrtFourBytesOfLengthFollowedByData();
+ data.data = (value + "\u0000").getBytes(StandardCharsets.UTF_16LE);
+ data.cb = data.data.length;
+ return data;
+ }
+
+ private static ArrayNumber arrayNumber(int number) {
+ ArrayNumber array = new ArrayNumber();
+ array.number = number;
+ return array;
+ }
+
+ private static ObjectGroupObjectData emptyObjectData() {
+ ObjectGroupObjectData data = new ObjectGroupObjectData();
+ data.data.content.addAll(ByteUtil.toListOfByte(new byte[]{0, 0, 0, (byte) 0x80,
+ 0, 0, 0, 0}));
+ return data;
+ }
+
+ private static CellID cell(int first, int second) {
+ return new CellID(id(first), id(second));
+ }
+
+ private static ExGuid id(int value) {
+ return new ExGuid(value, UUID.nameUUIDFromBytes(("id-" + value).getBytes(StandardCharsets.UTF_8)));
+ }
+
+ private static final class PropertySpec {
+ private final PropertyType type;
+ private final int value;
+ private final IProperty property;
+
+ private PropertySpec(PropertyType type, int value, IProperty property) {
+ this.type = type;
+ this.value = value;
+ this.property = property;
+ }
+ }
+}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreParserTest.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreParserTest.java
new file mode 100644
index 0000000000..bfbb34c8b3
--- /dev/null
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreParserTest.java
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.parser.microsoft.onenote.fsshttpb;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Method;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.UUID;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.tika.metadata.Metadata;
+import org.apache.tika.metadata.TikaCoreProperties;
+import org.apache.tika.parser.ParseContext;
+import org.apache.tika.parser.microsoft.onenote.OneNoteTreeWalkerOptions;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.CellManifestCurrentRevision;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.CellManifestDataElementData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.DataElement;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.ObjectGroupDataElementData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionManifestDataElementData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionManifestObjectGroupReferences;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.RevisionStoreCell;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.StorageIndexCellMapping;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.StorageIndexDataElementData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.StorageIndexRevisionMapping;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.CellID;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.ExGuid;
+import org.apache.tika.sax.ToTextContentHandler;
+import org.apache.tika.sax.XHTMLContentHandler;
+
+public class MSOneStoreParserTest {
+
+ @Test
+ public void testStorageMappingIndexes() {
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.storageIndex = new StorageIndexDataElementData();
+ CellID cellID = cell(700);
+ StorageIndexCellMapping cellMapping = new StorageIndexCellMapping();
+ cellMapping.cellID = cellID;
+ pkg.storageIndex.storageIndexCellMappingList.add(cellMapping);
+
+ ExGuid revisionID = id(701);
+ StorageIndexRevisionMapping revisionMapping = new StorageIndexRevisionMapping();
+ revisionMapping.revisionExGuid = revisionID;
+ pkg.storageIndex.storageIndexRevisionMappingList.add(revisionMapping);
+
+ assertSame(cellMapping, pkg.findStorageIndexCellMapping(cellID));
+ assertSame(revisionMapping, pkg.findStorageIndexRevisionMapping(revisionID));
+
+ StorageIndexCellMapping addedAfterIndexing = new StorageIndexCellMapping();
+ addedAfterIndexing.cellID = cell(702);
+ pkg.storageIndex.storageIndexCellMappingList.add(addedAfterIndexing);
+ assertNull(pkg.findStorageIndexCellMapping(addedAfterIndexing.cellID));
+ }
+
+ @Test
+ public void testMissingRootsAndRevisionMappingsReturnNoCell() throws Exception {
+ CellID cellID = cell(1);
+ ExGuid cellMappingID = id(2);
+ ExGuid missingRevisionID = id(3);
+ ExGuid revisionMappingID = id(4);
+
+ MSOneStoreParser noMappingParser = parserWithEmptyIndexes();
+ assertParseWarning(noMappingParser, new MSOneStorePackage(), cellID,
+ "no storage-index cell mapping");
+
+ MSOneStoreParser noManifestParser = parserWithEmptyIndexes();
+ MSOneStorePackage noManifestPackage = packageWithCellMapping(cellID, cellMappingID);
+ assertParseWarning(noManifestParser, noManifestPackage, cellID,
+ "no current cell manifest");
+
+ MSOneStoreParser noCurrentParser = parserWithEmptyIndexes();
+ DataElement currentlessManifest = cellManifest(cellMappingID, null);
+ set(noCurrentParser, "cellManifestDataElements", Arrays.asList(currentlessManifest));
+ set(noCurrentParser, "cellManifestDataElementsById",
+ Collections.singletonMap(cellMappingID, currentlessManifest));
+ assertParseWarning(noCurrentParser,
+ packageWithCellMapping(cellID, cellMappingID), cellID,
+ "no current cell manifest");
+
+ MSOneStoreParser noRevisionMappingParser = parserWithEmptyIndexes();
+ DataElement manifest = cellManifest(cellMappingID, missingRevisionID);
+ set(noRevisionMappingParser, "cellManifestDataElements", Arrays.asList(manifest));
+ set(noRevisionMappingParser, "cellManifestDataElementsById",
+ Collections.singletonMap(cellMappingID, manifest));
+ assertParseWarning(noRevisionMappingParser,
+ packageWithCellMapping(cellID, cellMappingID), cellID, "no revision mapping");
+
+ MSOneStoreParser noRevisionManifestParser = parserWithEmptyIndexes();
+ set(noRevisionManifestParser, "cellManifestDataElements", Arrays.asList(manifest));
+ set(noRevisionManifestParser, "cellManifestDataElementsById",
+ Collections.singletonMap(cellMappingID, manifest));
+ MSOneStorePackage noRevisionManifestPackage = packageWithCellMapping(cellID,
+ cellMappingID);
+ StorageIndexRevisionMapping revisionMapping = new StorageIndexRevisionMapping();
+ revisionMapping.revisionExGuid = missingRevisionID;
+ revisionMapping.revisionMappingExGuid = revisionMappingID;
+ noRevisionManifestPackage.storageIndex.storageIndexRevisionMappingList.add(revisionMapping);
+ assertParseWarning(noRevisionManifestParser, noRevisionManifestPackage, cellID,
+ "no revision manifest");
+ }
+
+ @Test
+ public void testRevisionChainStopsCyclesAndDeduplicatesObjectGroups() throws Exception {
+ MSOneStoreParser parser = new MSOneStoreParser();
+ set(parser, "cellManifestDataElements", new java.util.ArrayList<>());
+ set(parser, "revisionManifestDataElements", new java.util.ArrayList<>());
+ set(parser, "objectGroupDataElements", new java.util.ArrayList<>());
+ set(parser, "objectGroupDataElementsById", new java.util.HashMap<>());
+ set(parser, "objectBlobElementsById", new java.util.HashMap<>());
+
+ CellID cellID = cell(20);
+ ExGuid cellMappingID = id(21);
+ ExGuid currentMappingID = id(22);
+ ExGuid oldMappingID = id(23);
+ ExGuid currentRevisionID = id(24);
+ ExGuid oldRevisionID = id(25);
+ ExGuid currentGroupID = id(26);
+ ExGuid oldGroupID = id(27);
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.storageIndex = new StorageIndexDataElementData();
+ StorageIndexCellMapping cellMapping = new StorageIndexCellMapping();
+ cellMapping.cellID = cellID;
+ cellMapping.cellMappingExGuid = cellMappingID;
+ pkg.storageIndex.storageIndexCellMappingList.add(cellMapping);
+ StorageIndexRevisionMapping currentMapping = new StorageIndexRevisionMapping();
+ currentMapping.revisionMappingExGuid = currentMappingID;
+ currentMapping.revisionExGuid = currentRevisionID;
+ StorageIndexRevisionMapping oldMapping = new StorageIndexRevisionMapping();
+ oldMapping.revisionMappingExGuid = oldMappingID;
+ oldMapping.revisionExGuid = oldRevisionID;
+ pkg.storageIndex.storageIndexRevisionMappingList.add(currentMapping);
+ pkg.storageIndex.storageIndexRevisionMappingList.add(oldMapping);
+
+ DataElement cellManifestElement = new DataElement();
+ cellManifestElement.dataElementExGuid = cellMappingID;
+ CellManifestDataElementData cellManifest = new CellManifestDataElementData();
+ cellManifest.cellManifestCurrentRevision.cellManifestCurrentRevisionExGuid = currentRevisionID;
+ cellManifestElement.data = cellManifest;
+ set(parser, "cellManifestDataElements", Arrays.asList(cellManifestElement));
+ set(parser, "cellManifestDataElementsById",
+ Collections.singletonMap(cellManifestElement.dataElementExGuid,
+ cellManifestElement));
+
+ RevisionManifestDataElementData current = revision(currentRevisionID, oldRevisionID,
+ currentGroupID, currentGroupID, id(99));
+ RevisionManifestDataElementData old = revision(oldRevisionID, currentRevisionID,
+ oldGroupID);
+ DataElement currentElement = new DataElement();
+ currentElement.dataElementExGuid = currentMappingID;
+ currentElement.data = current;
+ DataElement oldElement = new DataElement();
+ oldElement.dataElementExGuid = oldMappingID;
+ oldElement.data = old;
+ set(parser, "revisionManifestDataElements", Arrays.asList(currentElement, oldElement));
+ Map<ExGuid, DataElement> revisionManifests = new HashMap<>();
+ revisionManifests.put(currentElement.dataElementExGuid, currentElement);
+ revisionManifests.put(oldElement.dataElementExGuid, oldElement);
+ set(parser, "revisionManifestDataElementsById", revisionManifests);
+
+ DataElement currentGroup = new DataElement();
+ currentGroup.dataElementExGuid = currentGroupID;
+ currentGroup.data = new ObjectGroupDataElementData();
+ DataElement oldGroup = new DataElement();
+ oldGroup.dataElementExGuid = oldGroupID;
+ oldGroup.data = new ObjectGroupDataElementData();
+ set(parser, "objectGroupDataElements", Arrays.asList(currentGroup, oldGroup));
+ java.util.Map<ExGuid, DataElement> objectGroups = new java.util.HashMap<>();
+ objectGroups.put(currentGroupID, currentGroup);
+ objectGroups.put(oldGroupID, oldGroup);
+ set(parser, "objectGroupDataElementsById", objectGroups);
+
+ RevisionStoreCell result = parseCell(parser, cellID, pkg);
+ assertEquals(2, result.objectGroups.size());
+
+ old.revisionManifest.baseRevisionID = id(1001);
+ RevisionStoreCell missingBaseResult = parseCell(parser, cellID, pkg);
+ assertEquals(2, missingBaseResult.objectGroups.size());
+ }
+
+ private static MSOneStoreParser parserWithEmptyIndexes() throws Exception {
+ MSOneStoreParser parser = new MSOneStoreParser();
+ set(parser, "cellManifestDataElements", new java.util.ArrayList<>());
+ set(parser, "revisionManifestDataElements", new java.util.ArrayList<>());
+ set(parser, "objectGroupDataElements", new java.util.ArrayList<>());
+ set(parser, "objectGroupDataElementsById", new java.util.HashMap<>());
+ set(parser, "objectBlobElementsById", new java.util.HashMap<>());
+ return parser;
+ }
+
+ private static MSOneStorePackage packageWithCellMapping(CellID cellID, ExGuid mappingID) {
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.storageIndex = new StorageIndexDataElementData();
+ StorageIndexCellMapping mapping = new StorageIndexCellMapping();
+ mapping.cellID = cellID;
+ mapping.cellMappingExGuid = mappingID;
+ pkg.storageIndex.storageIndexCellMappingList.add(mapping);
+ return pkg;
+ }
+
+ private static DataElement cellManifest(ExGuid mappingID, ExGuid revisionID) {
+ DataElement element = new DataElement();
+ element.dataElementExGuid = mappingID;
+ CellManifestDataElementData data = new CellManifestDataElementData();
+ data.cellManifestCurrentRevision = null;
+ if (revisionID != null) {
+ data.cellManifestCurrentRevision = new CellManifestCurrentRevision();
+ data.cellManifestCurrentRevision.cellManifestCurrentRevisionExGuid = revisionID;
+ }
+ element.data = data;
+ return element;
+ }
+
+ private static void assertParseWarning(MSOneStoreParser parser, MSOneStorePackage pkg,
+ CellID cellID, String expected) throws Exception {
+ assertNull(parseCell(parser, cellID, pkg));
+ Metadata metadata = new Metadata();
+ ParseContext context = new ParseContext();
+ XHTMLContentHandler xhtml = new XHTMLContentHandler(new ToTextContentHandler(), metadata);
+ xhtml.startDocument();
+ pkg.walkTree(new OneNoteTreeWalkerOptions(), metadata, xhtml, context);
+ xhtml.endDocument();
+ assertTrue(Arrays.stream(metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(warning -> warning.contains(expected)), expected);
+ }
+
+ private static RevisionManifestDataElementData revision(ExGuid revisionID,
+ ExGuid baseRevisionID,
+ ExGuid... groups) {
+ RevisionManifestDataElementData data = new RevisionManifestDataElementData();
+ data.revisionManifest.revisionID = revisionID;
+ data.revisionManifest.baseRevisionID = baseRevisionID;
+ for (ExGuid group : groups) {
+ RevisionManifestObjectGroupReferences reference =
+ new RevisionManifestObjectGroupReferences();
+ reference.objectGroupExtendedGUID = group;
+ data.revisionManifestObjectGroupReferences.add(reference);
+ }
+ return data;
+ }
+
+ private static RevisionStoreCell parseCell(MSOneStoreParser parser, CellID cellID,
+ MSOneStorePackage pkg) throws Exception {
+ Method method = MSOneStoreParser.class.getDeclaredMethod("parseCell", CellID.class,
+ MSOneStorePackage.class);
+ method.setAccessible(true);
+ return (RevisionStoreCell) method.invoke(parser, cellID, pkg);
+ }
+
+ private static void set(MSOneStoreParser parser, String field, Object value) throws Exception {
+ java.lang.reflect.Field declared = MSOneStoreParser.class.getDeclaredField(field);
+ declared.setAccessible(true);
+ declared.set(parser, value);
+ }
+
+ private static CellID cell(int value) {
+ return new CellID(id(value), id(value + 1));
+ }
+
+ private static ExGuid id(int value) {
+ return new ExGuid(value, UUID.nameUUIDFromBytes(
+ ("parser-" + value).getBytes(StandardCharsets.UTF_8)));
+ }
+}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/PropertyDeserializationBoundsTest.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/PropertyDeserializationBoundsTest.java
new file mode 100644
index 0000000000..73d5ecd395
--- /dev/null
+++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/PropertyDeserializationBoundsTest.java
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.parser.microsoft.onenote.fsshttpb;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.List;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.PrtArrayOfPropertyValues;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.PrtFourBytesOfLengthFollowedByData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.PropertySet;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.PropertyID;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.PropertyType;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.space.ObjectSpaceObjectStreamHeader;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.space.ObjectSpaceObjectStreamOfContextIDs;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.space.ObjectSpaceObjectStreamOfOIDs;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.space.ObjectSpaceObjectStreamOfOSIDs;
+
+/**
+ * File-derived counts and nesting in the fsshttpb property deserializers must be bounded -
+ * a malformed count would otherwise allocate up to gigabytes and a deeply nested property
+ * set would overflow the stack, both Errors that escape the parser's Exception fallback.
+ */
+public class PropertyDeserializationBoundsTest {
+
+ @Test
+ public void testFourBytesOfLengthRejectsLengthBeyondRemainingData() {
+ byte[] bytes = concat(int32(1000), new byte[2]);
+ IOException e = assertThrows(IOException.class,
+ () -> new PrtFourBytesOfLengthFollowedByData()
+ .doDeserializeFromByteArray(bytes, 0));
+ assertTrue(e.getMessage().contains("exceeds remaining data"), e.getMessage());
+ }
+
+ @Test
+ public void testFourBytesOfLengthRejectsNegativeLength() {
+ byte[] bytes = concat(int32(0xF0000000), new byte[8]);
+ assertThrows(IOException.class,
+ () -> new PrtFourBytesOfLengthFollowedByData()
+ .doDeserializeFromByteArray(bytes, 0));
+ }
+
+ @Test
+ public void testFourBytesOfLengthAcceptsExactFit() throws IOException {
+ byte[] bytes = concat(int32(3), new byte[]{1, 2, 3});
+ PrtFourBytesOfLengthFollowedByData property = new PrtFourBytesOfLengthFollowedByData();
+ assertEquals(7, property.doDeserializeFromByteArray(bytes, 0));
+ assertEquals(3, property.data.length);
+ }
+
+ @Test
+ public void testArrayOfPropertyValuesRejectsCountBeyondRemainingData() throws IOException {
+ byte[] bytes = concat(int32(Integer.MAX_VALUE),
+ propertyId(PropertyType.PropertySet.getIntVal()));
+ IOException e = assertThrows(IOException.class,
+ () -> new PrtArrayOfPropertyValues().doDeserializeFromByteArray(bytes, 0));
+ assertTrue(e.getMessage().contains("exceeds remaining data"), e.getMessage());
+ }
+
+ @Test
+ public void testArrayOfPropertyValuesAcceptsSmallCount() throws IOException {
+ // one element: an empty PropertySet (int16 count of 0)
+ byte[] bytes = concat(int32(1), propertyId(PropertyType.PropertySet.getIntVal()),
+ new byte[]{0, 0});
+ PrtArrayOfPropertyValues array = new PrtArrayOfPropertyValues();
+ array.doDeserializeFromByteArray(bytes, 0);
+ assertEquals(1, array.data.length);
+ }
+
+ @Test
+ public void testStreamOfIdsRejectCountBeyondRemainingData() throws IOException {
+ byte[] bytes = streamHeader(0xFFFFFF);
+ assertThrows(IOException.class,
+ () -> new ObjectSpaceObjectStreamOfOIDs().doDeserializeFromByteArray(bytes, 0));
+ assertThrows(IOException.class,
+ () -> new ObjectSpaceObjectStreamOfOSIDs().doDeserializeFromByteArray(bytes, 0));
+ assertThrows(IOException.class,
+ () -> new ObjectSpaceObjectStreamOfContextIDs()
+ .doDeserializeFromByteArray(bytes, 0));
+ }
+
+ @Test
+ public void testStreamOfOidsAcceptsSmallCount() throws IOException {
+ // one CompactID, 4 bytes
+ byte[] bytes = concat(streamHeader(1), new byte[4]);
+ ObjectSpaceObjectStreamOfOIDs stream = new ObjectSpaceObjectStreamOfOIDs();
+ stream.doDeserializeFromByteArray(bytes, 0);
+ assertEquals(1, stream.body.length);
+ }
+
+ @Test
+ public void testPropertySetNestingIsDepthCapped() throws IOException {
+ byte[] bytes = nestedPropertySet(PropertySet.MAX_PROPERTY_NESTING + 50);
+ IOException e = assertThrows(IOException.class,
+ () -> new PropertySet().doDeserializeFromByteArray(bytes, 0));
+ assertTrue(e.getMessage().contains("nesting exceeds"), e.getMessage());
+ }
+
+ @Test
+ public void testPropertySetShallowNestingParses() throws IOException {
+ PropertySet propertySet = new PropertySet();
+ propertySet.doDeserializeFromByteArray(nestedPropertySet(3), 0);
+ assertEquals(1, propertySet.rgData.size());
+ }
+
+ @Test
+ public void testNestingThroughArrayOfPropertyValuesIsDepthCapped() throws IOException {
+ // alternate PropertySet -> ArrayOfPropertyValues -> PropertySet -> ...
+ byte[] bytes = new byte[]{0, 0};
+ for (int i = 0; i < PropertySet.MAX_PROPERTY_NESTING + 50; i++) {
+ bytes = concat(new byte[]{1, 0},
+ propertyId(PropertyType.ArrayOfPropertyValues.getIntVal()), int32(1),
+ propertyId(PropertyType.PropertySet.getIntVal()), bytes);
+ }
+ byte[] finalBytes = bytes;
+ IOException e = assertThrows(IOException.class,
+ () -> new PropertySet().doDeserializeFromByteArray(finalBytes, 0));
+ assertTrue(e.getMessage().contains("nesting exceeds"), e.getMessage());
+ }
+
+ private static byte[] nestedPropertySet(int depth) throws IOException {
+ // innermost: empty PropertySet; each wrapper declares one PropertySet-typed property
+ byte[] bytes = new byte[]{0, 0};
+ for (int i = 0; i < depth; i++) {
+ bytes = concat(new byte[]{1, 0},
+ propertyId(PropertyType.PropertySet.getIntVal()), bytes);
+ }
+ return bytes;
+ }
+
+ private static byte[] propertyId(int type) throws IOException {
+ PropertyID propertyID = new PropertyID();
+ propertyID.id = 1;
+ propertyID.type = type;
+ return toArray(propertyID.serializeToByteList());
+ }
+
+ private static byte[] streamHeader(int count) throws IOException {
+ ObjectSpaceObjectStreamHeader header = new ObjectSpaceObjectStreamHeader();
+ header.count = count;
+ return toArray(header.serializeToByteList());
+ }
+
+ private static byte[] int32(int value) {
+ return new byte[]{(byte) value, (byte) (value >> 8), (byte) (value >> 16),
+ (byte) (value >> 24)};
+ }
+
+ private static byte[] toArray(List<Byte> bytes) {
+ byte[] result = new byte[bytes.size()];
+ for (int i = 0; i < result.length; i++) {
+ result[i] = bytes.get(i);
+ }
+ return result;
+ }
+
+ private static byte[] concat(byte[]... parts) {
+ ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ for (byte[] part : parts) {
+ bos.writeBytes(part);
+ }
+ return bos.toByteArray();
+ }
+}
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNote-fuzz1.one b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNote-fuzz1.one
new file mode 100644
index 0000000000..857ac4f7d3
Binary files /dev/null and b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNote-fuzz1.one differ
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNote-fuzz2.one b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNote-fuzz2.one
new file mode 100644
index 0000000000..a038f43984
Binary files /dev/null and b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNote-fuzz2.one differ
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNote-fuzz3.one b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNote-fuzz3.one
new file mode 100644
index 0000000000..35539efef8
Binary files /dev/null and b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNote-fuzz3.one differ
diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNoteEmbeddedImage.one b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNoteEmbeddedImage.one
new file mode 100644
index 0000000000..2e2b122952
Binary files /dev/null and b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNoteEmbeddedImage.one differ