You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by kr...@apache.org on 2026/08/14 03:17:30 UTC
(opennlp) 12/16: util: Bound timeouts, schemes, promotion, and tar headers
This is an automated email from the ASF dual-hosted git repository.
krickert pushed a commit to branch OPENNLP-1909-resource-installer
in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit 65d1154ecac568c03bba2a2ba66cd73ef1657edf
Author: Kristian Rickert <kr...@gmail.com>
AuthorDate: Wed Aug 12 15:04:09 2026 -0400
util: Bound timeouts, schemes, promotion, and tar headers
ResourceInstaller
Timeouts: a positive duration shorter than a millisecond rounded to zero,
which HttpURLConnection reads as no timeout at all, and Duration.toMillis
raises ArithmeticException on a duration too large for the long range,
before the old Math.min could cap it. Conversion now clamps into
[1, Integer.MAX_VALUE] and catches the overflow.
Schemes: only http, https, and file are accepted, checked at the public
boundary before the target directory is created. Anything else went to
whichever URL handler the runtime had installed; those carry no connection
or read timeout, so an unresponsive server blocked the caller forever. The
byte ceilings did apply on that path, since they are charged as bytes
arrive. file locations now read through Files.newInputStream.
Download file: created on the target filesystem rather than in the system
temporary directory, where the 1 GiB default ceiling could exhaust a small
/tmp while the target had room. It is hidden, so a leaked one shows up as
staging residue rather than as an installed file.
Promotion: refuses to descend through a symbolic link that already exists
below the target. Every entry name can be inside the staging directory and
the content still land outside the target, because createDirectories
followed such a link. This covers links present when the installation runs,
not a tree modified while it runs.
Limits: adds a builder seeded from DEFAULT, so a caller states only what
differs instead of five positional arguments, two of them Duration and two
of them long. The record stays immutable and the builder validates through
the canonical constructor.
TarStream
Detection now verifies the header checksum instead of accepting the ustar
magic outright, so arbitrary content carrying that magic at offset 257 is
no longer read as an archive with a trusted size field. Both the unsigned
and the signed sum are accepted, as historical writers differ.
Long names are read from the extension header that carries them. A pax
extended header supplies path and size for the entry after it; a GNU
long-name header supplies its name. Both are everyday formats: bsdtar and
tar --format=posix write an extended header ahead of every entry, and the
entry header that follows holds only a truncated name, so neither ignoring
nor refusing the extension header is workable.
Keywords other than path and size are ignored, because this reader exposes
only an entry's name, size, type, and content, and no other pax keyword
changes those. Sparse entries are the exception and are refused, by type
flag S and by the GNU.sparse.* records, because their archived bytes encode
holes rather than the content. A global header is refused if it carries
path or size, which would change every entry after it.
The POSIX ustar name prefix is honored, but only for POSIX ustar. GNU
writes "ustar" and two blanks where ustar writes "ustar" and a NUL, and
puts atime at the offset ustar gives to the prefix, so reading the two
alike delivered every entry of a GNU incremental archive under a directory
named after an octal timestamp.
A size field in the base-256 encoding, which GNU writes for entries beyond
the eleven octal digits the field holds, is read, so an entry of 8 GiB or
more states its length correctly. The sign sits in the bit below the
encoding marker, not in the marker itself, so a negative value is refused
rather than wrapped into an enormous positive length, and a value past the
long range is refused rather than truncated into a short one, which would
stop the reader inside the entry and leave it reading content as headers.
A header with a valid checksum but no name is refused, matching what
detection already required of the first byte.
entryStream's read(byte[], int, int) validates its range before answering,
so an invalid range is reported even when the entry is exhausted or the
length is zero, and a zero-length read returns 0 rather than -1. This
override raises the exceptions InputStream specifies rather than the
IllegalArgumentException used elsewhere in the package.
---
.../java/opennlp/tools/util/ResourceInstaller.java | 267 ++++++++++++--
.../java/opennlp/tools/util/archive/TarStream.java | 386 +++++++++++++++++++--
2 files changed, 591 insertions(+), 62 deletions(-)
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/ResourceInstaller.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/ResourceInstaller.java
index bcf90ef18..ce87ba0c0 100644
--- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/ResourceInstaller.java
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/ResourceInstaller.java
@@ -24,6 +24,7 @@ import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URI;
import java.nio.file.AtomicMoveNotSupportedException;
+import java.nio.file.FileSystemNotFoundException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
@@ -43,7 +44,8 @@ import opennlp.tools.util.archive.TarStream;
/**
* Fetches a third-party resource, such as a training corpus, a dictionary archive, or a
* lexicon, into a local directory. The caller supplies the location and thereby accepts
- * that resource's license; no locations are built in and no data is bundled.
+ * that resource's license; no locations are built in and no data is bundled. Only
+ * {@code http}, {@code https}, and {@code file} locations are accepted.
*
* <p>An optional checksum is verified against the downloaded bytes before anything is
* unpacked: a 64-character hex digest selects SHA-256, a 128-character one SHA-512.
@@ -59,12 +61,13 @@ import opennlp.tools.util.archive.TarStream;
* connection and read timeouts, follow at most a fixed number of redirects, refuse
* redirects that leave the http and https schemes or downgrade https to http, and
* abort once the download or the expanded content crosses its size ceiling. The
- * defaults in {@link Limits#DEFAULT} apply when no limits are given.</p>
+ * defaults in {@link Limits#DEFAULT} apply when no limits are given, and
+ * {@link Limits#builder()} starts from them.</p>
*
* <p>Installation is staged: content is unpacked into a hidden staging directory on
* the same filesystem and moved into the target only after the download was verified
- * and every entry unpacked cleanly. A failed installation therefore leaves the target
- * directory as it was, without partially written files.</p>
+ * and every entry unpacked cleanly. A fetch, verification, or unpacking failure
+ * therefore leaves the target directory as it was, without partially written files.</p>
*
* @see DownloadUtil
* @since 3.0.0
@@ -81,6 +84,8 @@ public final class ResourceInstaller {
private static final String MODEL_SUFFIX = ".bin";
private static final String DEFAULT_RESOURCE_NAME = "resource";
private static final String STAGING_PREFIX = ".opennlp-staging";
+ private static final String DOWNLOAD_PREFIX = ".opennlp-download";
+ private static final String DOWNLOAD_SUFFIX = ".part";
private static final int BUFFER_SIZE = 8192;
private static final int MAGIC_LENGTH = 2;
private static final int GZIP_MAGIC_FIRST = 0x1F;
@@ -91,6 +96,7 @@ public final class ResourceInstaller {
private static final int HTTP_PERMANENT_REDIRECT = 308;
private static final String SCHEME_HTTP = "http";
private static final String SCHEME_HTTPS = "https";
+ private static final String SCHEME_FILE = "file";
/**
* Safety ceilings and network behavior for one installation.
@@ -133,6 +139,104 @@ public final class ResourceInstaller {
throw new IllegalArgumentException("maxExpandedBytes must be positive");
}
}
+
+ /**
+ * Starts from {@link #DEFAULT} so a caller can state only the limits that differ
+ * from it, instead of repeating all five in the canonical constructor.
+ *
+ * @return A builder holding the default limits. Never {@code null}.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Collects limit values and validates them on {@link #build()}. Each setter returns
+ * this builder. Not thread safe; the {@link Limits} it builds is immutable.
+ */
+ public static final class Builder {
+
+ private Duration connectTimeout = DEFAULT.connectTimeout();
+ private Duration readTimeout = DEFAULT.readTimeout();
+ private int maxRedirects = DEFAULT.maxRedirects();
+ private long maxDownloadBytes = DEFAULT.maxDownloadBytes();
+ private long maxExpandedBytes = DEFAULT.maxExpandedBytes();
+
+ private Builder() {
+ }
+
+ /**
+ * Sets how long to wait for a connection to be established.
+ *
+ * @param connectTimeout How long to wait for a connection to be established.
+ * Must be positive.
+ * @return This builder. Never {@code null}.
+ */
+ public Builder connectTimeout(Duration connectTimeout) {
+ this.connectTimeout = connectTimeout;
+ return this;
+ }
+
+ /**
+ * Sets how long to wait for data on an established connection.
+ *
+ * @param readTimeout How long to wait for data on an established connection.
+ * Must be positive.
+ * @return This builder. Never {@code null}.
+ */
+ public Builder readTimeout(Duration readTimeout) {
+ this.readTimeout = readTimeout;
+ return this;
+ }
+
+ /**
+ * Sets how many http redirects to follow before failing.
+ *
+ * @param maxRedirects How many http redirects to follow before failing. Must not
+ * be negative; zero refuses all redirects.
+ * @return This builder. Never {@code null}.
+ */
+ public Builder maxRedirects(int maxRedirects) {
+ this.maxRedirects = maxRedirects;
+ return this;
+ }
+
+ /**
+ * Sets the largest download accepted.
+ *
+ * @param maxDownloadBytes The largest download accepted, in bytes. Must be
+ * positive.
+ * @return This builder. Never {@code null}.
+ */
+ public Builder maxDownloadBytes(long maxDownloadBytes) {
+ this.maxDownloadBytes = maxDownloadBytes;
+ return this;
+ }
+
+ /**
+ * Sets the largest total expanded content accepted.
+ *
+ * @param maxExpandedBytes The largest total expanded content accepted, in bytes,
+ * summed over all archive entries. Must be positive.
+ * @return This builder. Never {@code null}.
+ */
+ public Builder maxExpandedBytes(long maxExpandedBytes) {
+ this.maxExpandedBytes = maxExpandedBytes;
+ return this;
+ }
+
+ /**
+ * Builds the limits.
+ *
+ * @return The limits collected so far. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if any value is outside its documented
+ * range, exactly as the canonical constructor rejects it.
+ */
+ public Limits build() {
+ return new Limits(connectTimeout, readTimeout, maxRedirects, maxDownloadBytes,
+ maxExpandedBytes);
+ }
+ }
}
private ResourceInstaller() {
@@ -142,13 +246,15 @@ public final class ResourceInstaller {
* Fetches and unpacks a resource without checksum verification, under
* {@link Limits#DEFAULT}.
*
- * @param source The resource location. Must not be {@code null}.
+ * @param source The resource location, an {@code http}, {@code https}, or
+ * {@code file} URI. Must not be {@code null}.
* @param targetDirectory The directory to install into; created when absent. Must
* not be {@code null}.
* @return The target directory. Never {@code null}.
* @throws IOException Thrown if fetching or unpacking fails.
* @throws IllegalArgumentException Thrown if {@code source} or
- * {@code targetDirectory} is {@code null}.
+ * {@code targetDirectory} is {@code null}, or {@code source} carries a scheme
+ * other than {@code http}, {@code https}, or {@code file}.
*/
public static Path install(URI source, Path targetDirectory) throws IOException {
return install(source, targetDirectory, null);
@@ -157,7 +263,8 @@ public final class ResourceInstaller {
/**
* Fetches, verifies, and unpacks a resource under {@link Limits#DEFAULT}.
*
- * @param source The resource location. Must not be {@code null}.
+ * @param source The resource location, an {@code http}, {@code https}, or
+ * {@code file} URI. Must not be {@code null}.
* @param targetDirectory The directory to install into; created when absent. Must
* not be {@code null}.
* @param checksum The expected digest of the downloaded bytes as a hex string,
@@ -168,8 +275,9 @@ public final class ResourceInstaller {
* @throws IOException Thrown if fetching fails, the checksum does not match, or
* unpacking fails.
* @throws IllegalArgumentException Thrown if {@code source} or
- * {@code targetDirectory} is {@code null}, or {@code checksum} is neither a
- * 64-character nor a 128-character hex string.
+ * {@code targetDirectory} is {@code null}, {@code source} carries a scheme
+ * other than {@code http}, {@code https}, or {@code file}, or {@code checksum}
+ * is neither a 64-character nor a 128-character hex string.
*/
public static Path install(URI source, Path targetDirectory, String checksum)
throws IOException {
@@ -179,7 +287,8 @@ public final class ResourceInstaller {
/**
* Fetches, verifies, and unpacks a resource under the given {@link Limits}.
*
- * @param source The resource location. Must not be {@code null}.
+ * @param source The resource location, an {@code http}, {@code https}, or
+ * {@code file} URI. Must not be {@code null}.
* @param targetDirectory The directory to install into; created when absent. Must
* not be {@code null}.
* @param checksum The expected digest of the downloaded bytes as a hex string,
@@ -192,8 +301,9 @@ public final class ResourceInstaller {
* @throws IOException Thrown if fetching fails, a limit is exceeded, the checksum
* does not match, or unpacking fails.
* @throws IllegalArgumentException Thrown if {@code source}, {@code targetDirectory},
- * or {@code limits} is {@code null}, or {@code checksum} is neither a
- * 64-character nor a 128-character hex string.
+ * or {@code limits} is {@code null}, {@code source} carries a scheme other
+ * than {@code http}, {@code https}, or {@code file}, or {@code checksum} is
+ * neither a 64-character nor a 128-character hex string.
*/
public static Path install(URI source, Path targetDirectory, String checksum,
Limits limits) throws IOException {
@@ -206,9 +316,10 @@ public final class ResourceInstaller {
if (limits == null) {
throw new IllegalArgumentException("limits must not be null");
}
+ validateSource(source);
final String expected = validateChecksum(checksum);
Files.createDirectories(targetDirectory);
- final Path downloaded = Files.createTempFile("opennlp-resource", ".download");
+ final Path downloaded = createDownloadFile(targetDirectory);
try {
download(source, downloaded, limits);
if (expected != null) {
@@ -221,6 +332,20 @@ public final class ResourceInstaller {
}
}
+ /**
+ * Creates the file the download is written to. It is placed on the target's
+ * filesystem, not in the system temporary directory, so a large download cannot
+ * exhaust the system temporary directory while the target has room. It is hidden, so
+ * a leaked one is visible as residue rather than as an installed file.
+ *
+ * @param targetDirectory The directory to install into. Must already exist.
+ * @return The newly created, empty download file. Never {@code null}.
+ * @throws IOException Thrown if the file cannot be created.
+ */
+ static Path createDownloadFile(Path targetDirectory) throws IOException {
+ return Files.createTempFile(targetDirectory, DOWNLOAD_PREFIX, DOWNLOAD_SUFFIX);
+ }
+
/**
* Validates the checksum argument and normalizes it for comparison.
*
@@ -243,6 +368,8 @@ public final class ResourceInstaller {
}
/**
+ * Checks whether a string is made up entirely of hexadecimal digits.
+ *
* @param value The string to inspect.
* @return {@code true} if every character is a hexadecimal digit.
*/
@@ -261,8 +388,9 @@ public final class ResourceInstaller {
/**
* Fetches the source into the given file, bounded by the download ceiling. Http and
- * https locations are fetched with timeouts and the redirect policy; other schemes,
- * such as {@code file}, are opened directly.
+ * https locations are fetched with timeouts and the redirect policy; a {@code file}
+ * location is read directly. The scheme was accepted by {@link #validateSource(URI)}
+ * at the public boundary.
*
* @param source The resource location.
* @param file The file receiving the downloaded bytes.
@@ -272,16 +400,58 @@ public final class ResourceInstaller {
private static void download(URI source, Path file, Limits limits) throws IOException {
final Budget budget = new Budget(limits.maxDownloadBytes(),
"download exceeds the ceiling of " + limits.maxDownloadBytes() + " bytes");
- final String scheme = source.getScheme();
- if (SCHEME_HTTP.equalsIgnoreCase(scheme) || SCHEME_HTTPS.equalsIgnoreCase(scheme)) {
+ if (isHttp(source.getScheme())) {
downloadHttp(source, file, limits, budget);
} else {
- try (InputStream in = source.toURL().openStream()) {
+ try (InputStream in = Files.newInputStream(localFile(source))) {
copyBounded(in, file, budget);
}
}
}
+ /**
+ * Rejects a source the installer will not fetch. Only {@code http}, {@code https}, and
+ * {@code file} are accepted: any other scheme would be handed to whichever URL handler
+ * the runtime happens to have installed, none of which honor the timeouts and ceilings
+ * this class enforces.
+ *
+ * @param source The resource location as given by the caller.
+ * @throws IllegalArgumentException Thrown if the scheme is absent or unsupported.
+ */
+ private static void validateSource(URI source) {
+ final String scheme = source.getScheme();
+ if (!isHttp(scheme) && !SCHEME_FILE.equalsIgnoreCase(scheme)) {
+ throw new IllegalArgumentException(
+ "source scheme must be http, https, or file, but was: " + source);
+ }
+ }
+
+ /**
+ * Classifies a scheme as one the http fetch path handles.
+ *
+ * @param scheme The URI scheme, or {@code null} when the location has none.
+ * @return {@code true} for {@code http} and {@code https}, ignoring case.
+ */
+ private static boolean isHttp(String scheme) {
+ return SCHEME_HTTP.equalsIgnoreCase(scheme) || SCHEME_HTTPS.equalsIgnoreCase(scheme);
+ }
+
+ /**
+ * Resolves a {@code file} location to a path on the default filesystem.
+ *
+ * @param source The {@code file} location, already validated as such.
+ * @return The local path. Never {@code null}.
+ * @throws IOException Thrown if the location does not name a file this runtime can
+ * open, such as a {@code file} URI naming a remote host.
+ */
+ private static Path localFile(URI source) throws IOException {
+ try {
+ return Path.of(source);
+ } catch (IllegalArgumentException | FileSystemNotFoundException e) {
+ throw new IOException("not a readable local file location: " + source, e);
+ }
+ }
+
/**
* Fetches an http or https source with connection and read timeouts, following at
* most the allowed number of redirects under the redirect policy, checking any
@@ -337,6 +507,8 @@ public final class ResourceInstaller {
}
/**
+ * Classifies a response status as a redirect the installer follows.
+ *
* @param status The HTTP response status.
* @return {@code true} if the status is one of the redirect statuses 301, 302, 303,
* 307, or 308.
@@ -385,11 +557,23 @@ public final class ResourceInstaller {
}
/**
- * @param timeout The timeout as a duration.
- * @return The timeout in milliseconds, clamped to the int range.
+ * Converts a timeout to the millisecond form the connection setters take. A positive
+ * timeout shorter than a millisecond becomes one millisecond rather than zero, because
+ * {@link HttpURLConnection#setReadTimeout(int) zero means no timeout at all}, and a
+ * timeout too large for the int range is capped instead of overflowing.
+ *
+ * @param timeout The timeout as a duration. Must be positive.
+ * @return The timeout in milliseconds, at least {@code 1} and at most
+ * {@link Integer#MAX_VALUE}.
*/
private static int timeoutMillis(Duration timeout) {
- return (int) Math.min(timeout.toMillis(), Integer.MAX_VALUE);
+ final long millis;
+ try {
+ millis = timeout.toMillis();
+ } catch (ArithmeticException e) {
+ return Integer.MAX_VALUE;
+ }
+ return Math.clamp(millis, 1, Integer.MAX_VALUE);
}
/**
@@ -461,7 +645,8 @@ public final class ResourceInstaller {
*
* @param staging The staging directory holding the fully unpacked content.
* @param target The directory to install into.
- * @throws IOException Thrown if a move fails.
+ * @throws IOException Thrown if a move fails or a directory on the way to a
+ * destination is an existing symbolic link.
*/
private static void promote(Path staging, Path target) throws IOException {
final List<Path> files;
@@ -469,8 +654,7 @@ public final class ResourceInstaller {
files = walk.filter(Files::isRegularFile).toList();
}
for (final Path file : files) {
- final Path destination = target.resolve(staging.relativize(file).toString());
- Files.createDirectories(destination.getParent());
+ final Path destination = destination(target, staging.relativize(file));
try {
Files.move(file, destination, StandardCopyOption.REPLACE_EXISTING,
StandardCopyOption.ATOMIC_MOVE);
@@ -480,6 +664,39 @@ public final class ResourceInstaller {
}
}
+ /**
+ * Resolves one staged file's destination beneath the target, creating the directories
+ * leading to it one at a time and refusing to descend through a symbolic link that is
+ * already there. An entry name that stays inside the staging directory can still land
+ * outside the target if a directory below the target is a link to somewhere else.
+ *
+ * <p>This covers links present when the installation runs. It is not a defense against
+ * a link created concurrently, between the check here and the move that follows.</p>
+ *
+ * @param target The directory to install into.
+ * @param relative The staged file's path relative to the staging directory.
+ * @return The destination path beneath the target. Never {@code null}.
+ * @throws IOException Thrown if a directory on the way is a symbolic link or exists as
+ * something other than a directory, or if a directory cannot be created.
+ */
+ private static Path destination(Path target, Path relative) throws IOException {
+ Path directory = target;
+ for (int i = 0; i < relative.getNameCount() - 1; i++) {
+ directory = directory.resolve(relative.getName(i));
+ if (Files.isSymbolicLink(directory)) {
+ throw new IOException(
+ "installation path crosses a symbolic link: " + directory);
+ }
+ if (!Files.exists(directory)) {
+ Files.createDirectory(directory);
+ } else if (!Files.isDirectory(directory)) {
+ throw new IOException(
+ "installation path crosses an existing file: " + directory);
+ }
+ }
+ return directory.resolve(relative.getFileName());
+ }
+
/**
* Removes the given directory tree, deepest entries first.
*
@@ -669,6 +886,8 @@ public final class ResourceInstaller {
private long used;
/**
+ * Creates a budget that has spent nothing yet.
+ *
* @param ceiling The largest total number of bytes accepted.
* @param message The failure message raised when the ceiling is crossed.
*/
diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/archive/TarStream.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/archive/TarStream.java
index 309b6d2ef..a46db984f 100644
--- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/archive/TarStream.java
+++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/archive/TarStream.java
@@ -21,6 +21,7 @@ import java.io.EOFException;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
+import java.util.Objects;
import opennlp.tools.commons.Internal;
@@ -29,23 +30,61 @@ import opennlp.tools.commons.Internal;
* entry, skipping whatever remains of the current one, and {@link #entryStream()}
* exposes exactly the current entry's bytes.
*
- * <p>Reads classic and ustar headers: entry name, octal size, and type flag. The ustar
- * name prefix field is not consulted, so an entry's name is always the content of the
- * 100-byte name field alone. Long-name extension entries are not interpreted either;
- * they appear under their extension header names and are skippable like any other.</p>
+ * <p>Reads the classic v7, POSIX ustar, GNU, and pax formats: entry name, octal size, and
+ * type flag, with a valid header checksum required in every case. On a POSIX ustar header
+ * the 155-byte name prefix field is honored, so a name may span the prefix and the
+ * 100-byte name field, joined by {@code /}. A GNU header is not read that way, because GNU
+ * stores {@code atime} at the offset ustar gives to the prefix; the two are told apart by
+ * the magic field, {@code "ustar\0"} plus version {@code "00"} for ustar against
+ * {@code "ustar \0"} for GNU.</p>
+ *
+ * <p>A name longer than the header fields is read from the extension header that carries
+ * it, because the entry header following one holds only a truncated name. A pax extended
+ * header ({@code x}) supplies {@code path} and {@code size} for the next entry; a GNU
+ * long-name header ({@code L}) supplies its name. Both are in everyday use:
+ * {@code bsdtar} and {@code tar --format=posix} write an {@code x} header ahead of every
+ * entry, including entries that need no override.</p>
+ *
+ * <p>A size field too large for eleven octal digits is read in the base-256 encoding GNU
+ * writes for it, so an entry of 8 GiB or more states its length correctly.</p>
+ *
+ * <p>Keywords other than {@code path} and {@code size} are ignored, because this reader
+ * exposes only an entry's name, size, type, and content, and no other pax keyword changes
+ * those. The exception is the {@code GNU.sparse.*} family, along with the GNU sparse type
+ * flag {@code S}: there the archived bytes are a sparse-file encoding rather than the
+ * content, so those entries are refused instead of being unpacked as though they were
+ * literal. A pax global header ({@code g}) is refused if it carries {@code path} or
+ * {@code size}, which would change every entry after it.</p>
*/
@Internal
public final class TarStream {
private static final int BLOCK = 512;
private static final int NAME_LENGTH = 100;
+ private static final int CHECKSUM_OFFSET = 148;
+ private static final int CHECKSUM_LENGTH = 8;
private static final int SIZE_OFFSET = 124;
private static final int SIZE_LENGTH = 12;
private static final int TYPE_OFFSET = 156;
private static final int MAGIC_OFFSET = 257;
+ private static final int PREFIX_OFFSET = 345;
+ private static final int PREFIX_LENGTH = 155;
private static final String USTAR_MAGIC = "ustar";
private static final char TYPE_REGULAR_FILE = '0';
- private static final char TYPE_REGULAR_FILE_CLASSIC = 0;
+ private static final char TYPE_REGULAR_FILE_CLASSIC = '\0';
+ private static final char TYPE_GNU_LONG_NAME = 'L';
+ private static final char TYPE_GNU_LONG_LINK = 'K';
+ private static final char TYPE_GNU_SPARSE = 'S';
+ private static final char TYPE_PAX_EXTENDED = 'x';
+ private static final char TYPE_PAX_GLOBAL = 'g';
+ private static final int MAX_EXTENSION_BYTES = 1 << 20;
+ private static final String MALFORMED_RECORD = "malformed pax extended header record";
+ private static final String KEYWORD_PATH = "path";
+ private static final String KEYWORD_SIZE = "size";
+ private static final String SPARSE_PREFIX = "GNU.sparse.";
+ private static final int BASE_256_MARKER = 0x80;
+ private static final int BASE_256_NEGATIVE = 0x40;
+ private static final int BASE_256_FIRST_BYTE_BITS = 0x7F;
private final InputStream in;
private final byte[] header = new byte[BLOCK];
@@ -55,6 +94,12 @@ public final class TarStream {
private char type;
private long remaining;
+ /** The name an extension header supplied for the entry that follows it, else null. */
+ private String pendingPath;
+
+ /** The size a pax header supplied for the entry that follows it, else {@code -1}. */
+ private long pendingSize = -1;
+
/**
* Initializes the reader.
*
@@ -105,21 +150,201 @@ public final class TarStream {
*/
public boolean next() throws IOException {
skip(remaining + padding(size));
- if (!readBlock()) {
- return false;
+ while (true) {
+ if (!readBlock() || isEndBlock()) {
+ return false;
+ }
+ if (!hasValidChecksum(header)) {
+ throw new IOException("malformed tar header checksum");
+ }
+ type = (char) header[TYPE_OFFSET];
+ size = parseSize();
+ remaining = size;
+ if (type == TYPE_GNU_SPARSE) {
+ throw new IOException("sparse tar entries are not supported: "
+ + "the archived bytes encode holes rather than the file content");
+ }
+ if (type == TYPE_PAX_EXTENDED || type == TYPE_PAX_GLOBAL) {
+ readRecords(readExtensionPayload(), type == TYPE_PAX_GLOBAL);
+ continue;
+ }
+ if (type == TYPE_GNU_LONG_NAME) {
+ pendingPath = trimNul(new String(readExtensionPayload(), StandardCharsets.UTF_8));
+ continue;
+ }
+ if (type == TYPE_GNU_LONG_LINK) {
+ // The link target of an entry this reader does not expose.
+ readExtensionPayload();
+ continue;
+ }
+ name = pendingPath != null ? pendingPath : readName();
+ pendingPath = null;
+ if (pendingSize >= 0) {
+ size = pendingSize;
+ remaining = size;
+ pendingSize = -1;
+ }
+ if (name.isEmpty()) {
+ throw new IOException("tar entry header carries an empty name");
+ }
+ return true;
}
- if (isEndBlock()) {
- return false;
+ }
+
+ /**
+ * Reads the payload of the extension header just read, leaving the stream positioned on
+ * the header that follows it.
+ *
+ * @return The payload bytes. Never {@code null}.
+ * @throws IOException Thrown if the payload is larger than
+ * {@link #MAX_EXTENSION_BYTES} or the archive ends inside it.
+ */
+ private byte[] readExtensionPayload() throws IOException {
+ if (size > MAX_EXTENSION_BYTES) {
+ throw new IOException("tar extension header of " + size + " bytes exceeds the "
+ + MAX_EXTENSION_BYTES + " byte limit");
}
- int nameEnd = 0;
- while (nameEnd < NAME_LENGTH && header[nameEnd] != 0) {
- nameEnd++;
+ final byte[] payload = new byte[(int) size];
+ if (in.readNBytes(payload, 0, payload.length) < payload.length) {
+ throw new IOException("truncated tar archive");
+ }
+ skip(padding(size));
+ size = 0;
+ remaining = 0;
+ return payload;
+ }
+
+ /**
+ * Reads a pax extended header payload, which is a sequence of
+ * {@code "<length> <keyword>=<value>\n"} records whose length counts the whole record
+ * including its own digits, the blank, and the newline. Records are parsed on the raw
+ * bytes rather than on decoded text, because the length is a byte count and a multibyte
+ * value would otherwise shift every record after it.
+ *
+ * @param payload The raw extended header payload.
+ * @param global Whether this is a global header, which applies to every entry that
+ * follows rather than to just the next one.
+ * @throws IOException Thrown if a record is malformed, if a global header carries a
+ * keyword that would change the entries after it, or if the entry is sparse.
+ */
+ private void readRecords(byte[] payload, boolean global) throws IOException {
+ int offset = 0;
+ while (offset < payload.length) {
+ int blank = offset;
+ while (blank < payload.length && payload[blank] != ' ') {
+ blank++;
+ }
+ if (blank == payload.length || blank == offset) {
+ throw new IOException(MALFORMED_RECORD);
+ }
+ int length = 0;
+ for (int i = offset; i < blank; i++) {
+ final byte b = payload[i];
+ if (b < '0' || b > '9') {
+ throw new IOException(MALFORMED_RECORD);
+ }
+ length = length * 10 + (b - '0');
+ if (length > payload.length) {
+ throw new IOException(MALFORMED_RECORD);
+ }
+ }
+ final int end = offset + length;
+ if (length <= blank - offset || end > payload.length || payload[end - 1] != '\n') {
+ throw new IOException(MALFORMED_RECORD);
+ }
+ int equals = blank + 1;
+ while (equals < end - 1 && payload[equals] != '=') {
+ equals++;
+ }
+ if (equals >= end - 1 || equals == blank + 1) {
+ throw new IOException(MALFORMED_RECORD);
+ }
+ apply(new String(payload, blank + 1, equals - blank - 1, StandardCharsets.UTF_8),
+ new String(payload, equals + 1, end - equals - 2, StandardCharsets.UTF_8),
+ global);
+ offset = end;
+ }
+ }
+
+ /**
+ * Applies one pax record. Only {@code path} and {@code size} change what this reader
+ * reports, so every other keyword is ignored, except the sparse family, whose entries
+ * cannot be unpacked at all.
+ *
+ * @param keyword The record's keyword.
+ * @param value The record's value.
+ * @param global Whether the record came from a global header.
+ * @throws IOException Thrown if the entry is sparse, if a global header carries a
+ * keyword that would change the entries after it, or if {@code size} is not a
+ * number.
+ */
+ private void apply(String keyword, String value, boolean global) throws IOException {
+ if (keyword.startsWith(SPARSE_PREFIX)) {
+ throw new IOException("sparse tar entries are not supported: "
+ + "the archived bytes encode holes rather than the file content");
+ }
+ if (!KEYWORD_PATH.equals(keyword) && !KEYWORD_SIZE.equals(keyword)) {
+ return;
+ }
+ if (global) {
+ throw new IOException("pax global header carries " + keyword
+ + ", which would change every entry after it");
+ }
+ if (KEYWORD_PATH.equals(keyword)) {
+ pendingPath = value;
+ return;
+ }
+ try {
+ pendingSize = Long.parseLong(value);
+ } catch (NumberFormatException e) {
+ throw new IOException("pax size record is not a number: " + value, e);
+ }
+ if (pendingSize < 0) {
+ throw new IOException("pax size record is negative: " + value);
}
- name = new String(header, 0, nameEnd, StandardCharsets.UTF_8);
- size = parseOctal();
- remaining = size;
- type = (char) header[TYPE_OFFSET];
- return true;
+ }
+
+ /**
+ * Drops everything from the first NUL onward, which is how a GNU long-name header
+ * terminates the name it carries.
+ *
+ * @param value The decoded payload.
+ * @return The name without its terminator. Never {@code null}.
+ */
+ private static String trimNul(String value) {
+ final int nul = value.indexOf('\0');
+ return nul < 0 ? value : value.substring(0, nul);
+ }
+
+ /**
+ * Reads the current header's entry name. On a POSIX ustar header a non-empty name
+ * prefix is joined to the name field with {@code /}, which is how a name longer than
+ * the 100-byte name field is stored when no extension header carries it.
+ *
+ * @return The entry name. Never {@code null}.
+ */
+ private String readName() {
+ final String stored = field(0, NAME_LENGTH);
+ if (!hasPosixUstarMagic(header)) {
+ return stored;
+ }
+ final String prefix = field(PREFIX_OFFSET, PREFIX_LENGTH);
+ return prefix.isEmpty() ? stored : prefix + "/" + stored;
+ }
+
+ /**
+ * Reads a NUL-terminated text field of the current header.
+ *
+ * @param offset The field's offset in the header block.
+ * @param length The field's length in bytes.
+ * @return The field content up to its first NUL, decoded as UTF-8. Never {@code null}.
+ */
+ private String field(int offset, int length) {
+ int end = 0;
+ while (end < length && header[offset + end] != 0) {
+ end++;
+ }
+ return new String(header, offset, end, StandardCharsets.UTF_8);
}
/**
@@ -148,7 +373,9 @@ public final class TarStream {
* Opens the current entry's content.
*
* @return A stream over exactly this entry's bytes; reading past the end returns end
- * of stream. Never {@code null}. Closing it is not required.
+ * of stream, and a zero-length read returns {@code 0} as
+ * {@link InputStream#read(byte[], int, int)} requires. Never {@code null}.
+ * Closing it is not required.
*/
public InputStream entryStream() {
return new InputStream() {
@@ -165,8 +392,25 @@ public final class TarStream {
return b;
}
+ /**
+ * {@inheritDoc}
+ *
+ * <p>The range is checked before anything else, so an invalid one is reported even
+ * when the entry is exhausted or the length is zero. This override raises the
+ * exceptions {@link InputStream} specifies rather than the
+ * {@code IllegalArgumentException} used elsewhere in this package.</p>
+ *
+ * @throws NullPointerException Thrown if {@code buffer} is {@code null}.
+ * @throws IndexOutOfBoundsException Thrown if {@code offset} or {@code length} is
+ * negative, or {@code length} is greater than
+ * {@code buffer.length - offset}.
+ */
@Override
public int read(byte[] buffer, int offset, int length) throws IOException {
+ Objects.checkFromIndexSize(offset, length, buffer.length);
+ if (length == 0) {
+ return 0;
+ }
if (remaining <= 0) {
return -1;
}
@@ -181,41 +425,71 @@ public final class TarStream {
}
/**
- * Checks whether a full 512-byte block reads as a tar entry header. A block carrying
- * the ustar magic is accepted outright; without it, the block is accepted as a classic
- * header when it starts with a name and its size field holds only octal digits, blanks,
- * or NUL padding.
+ * Checks whether a full 512-byte block reads as a tar entry header, which it does when
+ * it starts with a name and its stored checksum matches the block. Both classic and
+ * ustar headers carry that checksum, so it identifies either without relying on the
+ * ustar magic, which arbitrary content can also carry.
*
* @param block The block to inspect. Must be 512 bytes long.
* @return {@code true} if the block reads as a tar header.
*/
private static boolean isHeader(byte[] block) {
- if (hasUstarMagic(block)) {
- return true;
- }
- if (block[0] == 0) {
- return false;
- }
- for (int i = SIZE_OFFSET; i < SIZE_OFFSET + SIZE_LENGTH; i++) {
+ return block[0] != 0 && hasValidChecksum(block);
+ }
+
+ /**
+ * Verifies a header block against the checksum stored in it. The checksum is the sum
+ * of every header byte with the checksum field itself read as eight blanks. Historical
+ * writers summed the bytes as signed rather than unsigned, so either total is accepted.
+ *
+ * @param block The header block to verify. Must be 512 bytes long.
+ * @return {@code true} if the stored checksum is well formed and matches the block.
+ */
+ private static boolean hasValidChecksum(byte[] block) {
+ long stored = 0;
+ boolean digits = false;
+ for (int i = CHECKSUM_OFFSET; i < CHECKSUM_OFFSET + CHECKSUM_LENGTH; i++) {
final byte b = block[i];
- if (b != 0 && b != ' ' && !isOctalDigit(b)) {
+ if (b == 0 || b == ' ') {
+ continue;
+ }
+ if (!isOctalDigit(b)) {
return false;
}
+ stored = stored * 8 + (b - '0');
+ digits = true;
}
- return true;
+ if (!digits) {
+ return false;
+ }
+ int unsigned = 0;
+ int signed = 0;
+ for (int i = 0; i < BLOCK; i++) {
+ final byte b = i >= CHECKSUM_OFFSET && i < CHECKSUM_OFFSET + CHECKSUM_LENGTH
+ ? (byte) ' ' : block[i];
+ unsigned += b & 0xFF;
+ signed += b;
+ }
+ return stored == unsigned || stored == signed;
}
/**
+ * Checks for the POSIX ustar magic specifically, which is {@code "ustar"} followed by a
+ * NUL. GNU writes {@code "ustar"} followed by two blanks and a NUL in the same field,
+ * and its headers must not be read as ustar: GNU stores {@code atime} at the offset
+ * ustar gives to the name prefix, so a GNU incremental archive would otherwise deliver
+ * every entry under a directory named after an octal timestamp.
+ *
* @param block The block to inspect. Must be 512 bytes long.
- * @return {@code true} if the block carries the ustar magic in its magic field.
+ * @return {@code true} if the block carries the POSIX ustar magic.
*/
- private static boolean hasUstarMagic(byte[] block) {
+ private static boolean hasPosixUstarMagic(byte[] block) {
for (int i = 0; i < USTAR_MAGIC.length(); i++) {
if (block[MAGIC_OFFSET + i] != USTAR_MAGIC.charAt(i)) {
return false;
}
}
- return true;
+ return block[MAGIC_OFFSET + USTAR_MAGIC.length()] == 0;
}
/**
@@ -258,14 +532,19 @@ public final class TarStream {
}
/**
- * Parses the octal size field of the current header, tolerating NUL and blank
- * padding around the digits.
+ * Parses the size field of the current header, tolerating NUL and blank padding around
+ * the octal digits. A field whose leading bit is set is in the base-256 encoding
+ * instead, and is read by {@link #parseBase256Size()}.
*
* @return The entry size in bytes.
* @throws IOException Thrown if the field contains a character that is not an octal
- * digit, a blank, or NUL padding.
+ * digit, a blank, or NUL padding, or if the base-256 form is negative or too
+ * large for a {@code long}.
*/
- private long parseOctal() throws IOException {
+ private long parseSize() throws IOException {
+ if ((header[SIZE_OFFSET] & BASE_256_MARKER) != 0) {
+ return parseBase256Size();
+ }
long value = 0;
for (int i = SIZE_OFFSET; i < SIZE_OFFSET + SIZE_LENGTH; i++) {
final byte b = header[i];
@@ -280,6 +559,37 @@ public final class TarStream {
return value;
}
+ /**
+ * Reads a size field in the base-256 encoding, which GNU writes when a value does not
+ * fit the eleven octal digits the field otherwise holds, so entries of 8 GiB and above
+ * can state their length.
+ *
+ * <p>The leading bit marks the encoding, the next bit is the sign, and the remaining
+ * bits of that byte followed by every later byte form a big-endian two's complement
+ * number. A size is never negative, so the negative form is refused rather than wrapped
+ * into a huge positive length.</p>
+ *
+ * @return The entry size in bytes.
+ * @throws IOException Thrown if the encoded value is negative or does not fit a
+ * {@code long}.
+ */
+ private long parseBase256Size() throws IOException {
+ if ((header[SIZE_OFFSET] & BASE_256_NEGATIVE) != 0) {
+ throw new IOException("tar size field is negative");
+ }
+ long value = 0;
+ for (int i = SIZE_OFFSET; i < SIZE_OFFSET + SIZE_LENGTH; i++) {
+ final int b = i == SIZE_OFFSET
+ ? header[i] & BASE_256_FIRST_BYTE_BITS : header[i] & 0xFF;
+ if (value >>> (Long.SIZE - Byte.SIZE - 1) != 0) {
+ throw new IOException(
+ "tar size field exceeds the largest length this reader can represent");
+ }
+ value = value << Byte.SIZE | b;
+ }
+ return value;
+ }
+
/**
* @param entrySize The size of an entry's content in bytes.
* @return The number of padding bytes that align the entry to the next 512-byte