You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by kr...@apache.org on 2026/08/14 03:17:29 UTC
(opennlp) 11/16: util: Pin installer and tar hardening (failing tests)
This is an automated email from the ASF dual-hosted git repository.
krickert pushed a commit to branch OPENNLP-1909-resource-installer
in repository https://gitbox.apache.org/repos/asf/opennlp.git
commit 6929207f6d44c054861aa8f284943d0f0d1d0844
Author: Kristian Rickert <kr...@gmail.com>
AuthorDate: Wed Aug 12 15:04:09 2026 -0400
util: Pin installer and tar hardening (failing tests)
Does not compile on its own: the tests call Limits.builder() and
createDownloadFile, both added in the following commit. Against the
current implementation, with those two added as stubs, the run is:
ResourceInstallerHttpTest
testSubMillisecondReadTimeoutStillTimesOut
timed out after 15 seconds
testTimeoutBeyondTheMillisecondRangeIsCapped
Arithmetic long overflow
ResourceInstallerTest
testPromotionRefusesToFollowASymlinkedDirectory
Expected java.io.IOException to be thrown, but nothing was thrown
testUnsupportedSourceSchemeIsRejected
expected IllegalArgumentException but was java.net.UnknownHostException
expected IllegalArgumentException but was java.net.UnknownServiceException
expected IllegalArgumentException but was java.nio.file.NoSuchFileException
expected "source scheme must be ..." but was "URI is not absolute"
testUnsupportedSourceSchemeIsRejectedBeforeCreatingTheTarget
expected IllegalArgumentException but was java.net.UnknownHostException
TarStreamTest
testHeaderWithWrongChecksumIsRejected
testRejectedPaxGlobalHeader (8 cases)
testSparseEntriesAreRejected (2 cases)
testBase256SizeFieldIsRead
testBase256SizeFieldCarriesLengthsBeyondTheOctalRange
testBase256SizeFieldAcceptsTheLargestRepresentableLength
testBase256SizeFieldBeyondTheLongRangeIsRejected
testNegativeBase256SizeFieldIsRejected
the base-256 size encoding is not read at all
testStartsWithHeaderRejectsUstarMagicWithoutAChecksum
expected false but was true
testPaxExtendedHeaderSuppliesTheEntryName
testGnuLongNameHeaderSuppliesTheEntryName
expected the full path, but was its first 100 bytes
testPaxExtendedHeaderSuppliesTheEntrySize
expected 10 but was 0
testGnuHeaderDoesNotReadItsAtimeAsANamePrefix
expected "./short.txt" but was "15237132225/./short.txt"
testUstarPrefixIsJoinedToTheName
testHeaderWithAnEmptyNameIsRejected
testEntryStreamRejectsInvalidReadRanges
testZeroLengthReadReturnsZero
expected 0 but was -1
The sub-millisecond read timeout hangs rather than failing an assertion:
zero milliseconds means no timeout to HttpURLConnection, so the tightest
setting a caller can express becomes the loosest. That is why the test
carries an explicit @Timeout.
The tar fixtures follow archives written by GNU tar 1.35 rather than a
minimal shape, because the shape is the point. A pax archive carries an
extended header ahead of every entry, including entries needing no
override, so the metadata-only case is the common one. The entry header
after an extension header holds a truncated name, so the extension header
is the only place the real one appears. Verified against tar --format=pax,
--format=posix, --format=gnu, and --format=gnu --incremental.
The base-256 size fixture is synthetic, since producing a real one needs an
entry of 8 GiB or more. It was cross-checked the other way instead: GNU tar
1.35 lists a header written by TarArchives.base256Header as 8589934592
bytes, so the encoding the tests assert against is the one tar writes.
Two timeout tests replace an earlier pair that computed the expected
milliseconds with their own copy of the conversion. Those asserted the
test's arithmetic, not the installer's, and would stay green through any
regression. These drive HttpURLConnection instead.
testClassicHeaderWithoutUstarMagicIsRead passes before the change as well.
It guards the new checksum-based detection against dropping classic v7
archives, which the previous ustar-magic shortcut happened to accept.
Also consolidates the tar fixture that existed once per test package into
TarArchives, which now writes real ustar magic and header checksums, and
builds classic, GNU, and prefix headers plus pax records with correct
length prefixes.
---
.../opennlp/tools/util/InstallerTestSupport.java | 59 +-
.../tools/util/ResourceInstallerHttpTest.java | 100 +++-
.../opennlp/tools/util/ResourceInstallerTest.java | 369 ++++++++----
.../opennlp/tools/util/archive/TarArchives.java | 311 ++++++++++
.../opennlp/tools/util/archive/TarStreamTest.java | 632 ++++++++++++++++++---
5 files changed, 1253 insertions(+), 218 deletions(-)
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/InstallerTestSupport.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/InstallerTestSupport.java
index 5848934bc..ffb3c936c 100644
--- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/InstallerTestSupport.java
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/InstallerTestSupport.java
@@ -29,28 +29,31 @@ import java.util.List;
import java.util.stream.Stream;
import java.util.zip.GZIPOutputStream;
+import opennlp.tools.util.archive.TarArchives;
+
/**
- * Shared fixtures for the {@code ResourceInstaller} test classes: hand-rolled tar
- * archive building, gzip compression, digest computation, and installed-file listing.
+ * Shared fixtures for the {@code ResourceInstaller} test classes: tar archive building,
+ * gzip compression, digest computation, and installed-file listing.
*/
final class InstallerTestSupport {
- static final int BLOCK = 512;
- static final int TERMINATOR_SIZE = 2 * BLOCK;
+ static final int BLOCK = TarArchives.BLOCK;
+ static final int TERMINATOR_SIZE = TarArchives.TERMINATOR_SIZE;
+
+ /** One kibibyte, a convenient small ceiling for limit tests. */
+ static final long KIBIBYTE = 1024;
+
+ /** One mebibyte, a convenient generous ceiling for tests that do not exercise it. */
+ static final long MEBIBYTE = 1024 * KIBIBYTE;
- private static final int NAME_LENGTH = 100;
- private static final int SIZE_OFFSET = 124;
- private static final int TYPE_OFFSET = 156;
- private static final String SIZE_FORMAT = "%011o";
- private static final char TYPE_REGULAR_FILE = '0';
+ private static final String SHA_256 = "SHA-256";
+ private static final String SHA_512 = "SHA-512";
private InstallerTestSupport() {
}
/**
- * Writes one tar entry into the given buffer: a 512-byte header block carrying the
- * name, the octal content size, and the regular-file type flag, followed by the
- * content padded to the next 512-byte block boundary.
+ * Writes one regular-file tar entry into the given buffer.
*
* @param tar The buffer receiving the entry bytes. Must not be {@code null}.
* @param name The entry name; at most 100 bytes when encoded as UTF-8.
@@ -60,20 +63,7 @@ final class InstallerTestSupport {
*/
static void tarEntry(ByteArrayOutputStream tar, String name, byte[] content)
throws IOException {
- final byte[] header = new byte[BLOCK];
- final byte[] nameBytes = name.getBytes(StandardCharsets.UTF_8);
- if (nameBytes.length > NAME_LENGTH) {
- throw new IllegalArgumentException(
- "entry name exceeds " + NAME_LENGTH + " bytes: " + name);
- }
- System.arraycopy(nameBytes, 0, header, 0, nameBytes.length);
- final byte[] size = String.format(SIZE_FORMAT, content.length)
- .getBytes(StandardCharsets.US_ASCII);
- System.arraycopy(size, 0, header, SIZE_OFFSET, size.length);
- header[TYPE_OFFSET] = TYPE_REGULAR_FILE;
- tar.write(header);
- tar.write(content);
- tar.write(new byte[(BLOCK - content.length % BLOCK) % BLOCK]);
+ TarArchives.entry(tar, name, content);
}
/**
@@ -117,7 +107,7 @@ final class InstallerTestSupport {
* @throws NoSuchAlgorithmException Thrown if the digest algorithm is unavailable.
*/
static String sha256(byte[] content) throws NoSuchAlgorithmException {
- return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(content));
+ return digest(SHA_256, content);
}
/**
@@ -128,7 +118,20 @@ final class InstallerTestSupport {
* @throws NoSuchAlgorithmException Thrown if the digest algorithm is unavailable.
*/
static String sha512(byte[] content) throws NoSuchAlgorithmException {
- return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-512").digest(content));
+ return digest(SHA_512, content);
+ }
+
+ /**
+ * Computes a digest of the given bytes as a lowercase hex string.
+ *
+ * @param algorithm The digest algorithm name.
+ * @param content The bytes to digest. Must not be {@code null}.
+ * @return The lowercase hex digest. Never {@code null}.
+ * @throws NoSuchAlgorithmException Thrown if the digest algorithm is unavailable.
+ */
+ private static String digest(String algorithm, byte[] content)
+ throws NoSuchAlgorithmException {
+ return HexFormat.of().formatHex(MessageDigest.getInstance(algorithm).digest(content));
}
/**
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/ResourceInstallerHttpTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/ResourceInstallerHttpTest.java
index 7e9ed9d4b..414e2a35f 100644
--- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/ResourceInstallerHttpTest.java
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/ResourceInstallerHttpTest.java
@@ -39,10 +39,13 @@ import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
import org.junit.jupiter.api.io.TempDir;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
+import static opennlp.tools.util.InstallerTestSupport.KIBIBYTE;
+import static opennlp.tools.util.InstallerTestSupport.MEBIBYTE;
import static opennlp.tools.util.InstallerTestSupport.installedFiles;
import static opennlp.tools.util.InstallerTestSupport.sha256;
import static opennlp.tools.util.InstallerTestSupport.tarGz;
@@ -55,7 +58,9 @@ import static opennlp.tools.util.InstallerTestSupport.tarGz;
public class ResourceInstallerHttpTest {
private static final Duration GENEROUS = Duration.ofSeconds(10);
- private static final long MEBIBYTE = 1024 * 1024;
+
+ /** Long enough that a stalled route outlives any timeout a test configures. */
+ private static final Duration STALL = Duration.ofSeconds(30);
private StubServer server;
@@ -249,18 +254,21 @@ public class ResourceInstallerHttpTest {
void testHttpsToHttpDowngradeIsRejected() {
final URI https = URI.create("https://example.invalid/archive.tar.gz");
- final IOException thrown = Assertions.assertThrows(IOException.class,
- () -> ResourceInstaller.resolveRedirect(https,
- "http://example.invalid/archive.tar.gz"));
- Assertions.assertEquals(
- "redirect downgrades https to http: http://example.invalid/archive.tar.gz",
- thrown.getMessage());
-
- Assertions.assertDoesNotThrow(() -> ResourceInstaller.resolveRedirect(https,
- "https://mirror.invalid/archive.tar.gz"));
- Assertions.assertDoesNotThrow(() -> ResourceInstaller.resolveRedirect(
- URI.create("http://example.invalid/archive.tar.gz"),
- "https://mirror.invalid/archive.tar.gz"));
+ // assertAll so a wrongly rejected upgrade is not hidden by the rejection case.
+ Assertions.assertAll(
+ () -> {
+ final IOException thrown = Assertions.assertThrows(IOException.class,
+ () -> ResourceInstaller.resolveRedirect(https,
+ "http://example.invalid/archive.tar.gz"));
+ Assertions.assertEquals(
+ "redirect downgrades https to http: http://example.invalid/archive.tar.gz",
+ thrown.getMessage());
+ },
+ () -> Assertions.assertDoesNotThrow(() -> ResourceInstaller.resolveRedirect(https,
+ "https://mirror.invalid/archive.tar.gz")),
+ () -> Assertions.assertDoesNotThrow(() -> ResourceInstaller.resolveRedirect(
+ URI.create("http://example.invalid/archive.tar.gz"),
+ "https://mirror.invalid/archive.tar.gz")));
}
@Test
@@ -282,7 +290,7 @@ public class ResourceInstallerHttpTest {
*/
@Test
void testStalledResponseHitsReadTimeout(@TempDir Path target) {
- server.route("/stall", out -> StubServer.sleep(Duration.ofSeconds(30)));
+ server.route("/stall", out -> StubServer.sleep(STALL));
final IOException thrown = Assertions.assertThrows(IOException.class,
() -> ResourceInstaller.install(server.uri("/stall"), target, null,
@@ -300,7 +308,7 @@ public class ResourceInstallerHttpTest {
StubServer.head(out, "200 OK", "Content-Length: 100000", "");
out.write("just a few bytes".getBytes(StandardCharsets.UTF_8));
out.flush();
- StubServer.sleep(Duration.ofSeconds(30));
+ StubServer.sleep(STALL);
});
final IOException thrown = Assertions.assertThrows(IOException.class,
@@ -322,7 +330,7 @@ public class ResourceInstallerHttpTest {
final IOException thrown = Assertions.assertThrows(IOException.class,
() -> ResourceInstaller.install(server.uri("/liar.tar.gz"), target, null,
- withDownloadCeiling(1024)));
+ withDownloadCeiling(KIBIBYTE)));
Assertions.assertEquals(
"declared content length 10000000 exceeds the download ceiling of 1024 bytes",
thrown.getMessage());
@@ -343,12 +351,56 @@ public class ResourceInstallerHttpTest {
final IOException thrown = Assertions.assertThrows(IOException.class,
() -> ResourceInstaller.install(server.uri("/endless"), target, null,
- withDownloadCeiling(1024)));
+ withDownloadCeiling(KIBIBYTE)));
Assertions.assertEquals("download exceeds the ceiling of 1024 bytes",
thrown.getMessage());
Assertions.assertEquals(List.of(), installedFiles(target));
}
+ /**
+ * Proves that a positive timeout shorter than a millisecond is applied as a real
+ * timeout. Rounded down to zero it would mean no timeout at all to
+ * {@code HttpURLConnection}, so the tightest configuration a caller can express would
+ * become the loosest, and this install would never return.
+ */
+ @Test
+ @Timeout(value = 15, threadMode = Timeout.ThreadMode.SEPARATE_THREAD)
+ void testSubMillisecondReadTimeoutStillTimesOut(@TempDir Path target) {
+ server.route("/stall", out -> StubServer.sleep(STALL));
+ final ResourceInstaller.Limits limits = ResourceInstaller.Limits.builder()
+ .readTimeout(Duration.ofNanos(1))
+ .build();
+
+ final IOException thrown = Assertions.assertThrows(IOException.class,
+ () -> ResourceInstaller.install(server.uri("/stall"), target, null, limits));
+
+ Assertions.assertInstanceOf(SocketTimeoutException.class, thrown);
+ }
+
+
+ /**
+ * Proves that a timeout too large to express in milliseconds is capped rather than
+ * overflowing. {@link Duration#toMillis()} raises an {@code ArithmeticException} on
+ * such a value, which would abort the install before a single byte was fetched.
+ */
+ @Test
+ void testTimeoutBeyondTheMillisecondRangeIsCapped(@TempDir Path target)
+ throws Exception {
+ final byte[] archive = tarGz(new String[][] {{"corpus/data.txt", "patient"}});
+ server.route("/corpus.tar.gz", out -> StubServer.ok(out, archive));
+ final Duration beyondMillis = Duration.ofSeconds(Long.MAX_VALUE / 1000 + 1);
+ final ResourceInstaller.Limits limits = ResourceInstaller.Limits.builder()
+ .connectTimeout(beyondMillis)
+ .readTimeout(beyondMillis)
+ .build();
+
+ ResourceInstaller.install(server.uri("/corpus.tar.gz"), target, sha256(archive),
+ limits);
+
+ Assertions.assertEquals("patient",
+ Files.readString(target.resolve("corpus/data.txt")));
+ }
+
/**
* A minimal scripted HTTP server on a loopback socket. Each registered route is a
* function from the response output stream to the raw bytes it wants on the wire,
@@ -359,6 +411,13 @@ public class ResourceInstallerHttpTest {
/** Writes a raw HTTP response to the connected client. */
@FunctionalInterface
interface Responder {
+
+ /**
+ * Writes the raw response bytes for one request.
+ *
+ * @param out The response stream.
+ * @throws IOException Thrown if writing fails.
+ */
void respond(OutputStream out) throws IOException;
}
@@ -394,13 +453,14 @@ public class ResourceInstallerHttpTest {
}
/**
- * Writes a response head: the HTTP/1.0 status line, the given header lines, and
- * the blank separator line, followed by any further lines verbatim.
+ * Writes the HTTP/1.0 status line followed by the given lines, each terminated by
+ * CRLF. The caller supplies the header lines, then an empty string for the blank
+ * line separating head from body, then optional body text.
*
* @param out The response stream.
* @param status The status line content after the protocol, such as {@code 200 OK}.
* @param lines Header lines, then an empty string separator, then optional body
- * text written verbatim.
+ * text, each written with a trailing CRLF.
* @throws IOException Thrown if writing fails.
*/
static void head(OutputStream out, String status, String... lines)
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/ResourceInstallerTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/ResourceInstallerTest.java
index 74a6d836c..8477ae697 100644
--- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/ResourceInstallerTest.java
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/ResourceInstallerTest.java
@@ -19,6 +19,7 @@ package opennlp.tools.util;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
+import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
@@ -34,6 +35,8 @@ import java.util.zip.ZipOutputStream;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.condition.DisabledOnOs;
+import org.junit.jupiter.api.condition.OS;
import org.junit.jupiter.api.function.Executable;
import org.junit.jupiter.api.io.TempDir;
import org.junit.jupiter.params.ParameterizedTest;
@@ -41,6 +44,8 @@ import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;
import static opennlp.tools.util.InstallerTestSupport.BLOCK;
+import static opennlp.tools.util.InstallerTestSupport.KIBIBYTE;
+import static opennlp.tools.util.InstallerTestSupport.MEBIBYTE;
import static opennlp.tools.util.InstallerTestSupport.TERMINATOR_SIZE;
import static opennlp.tools.util.InstallerTestSupport.gzip;
import static opennlp.tools.util.InstallerTestSupport.installedFiles;
@@ -51,6 +56,48 @@ import static opennlp.tools.util.InstallerTestSupport.tarGz;
public class ResourceInstallerTest {
+ private static final String CHECKSUM_ARGUMENT_ERROR =
+ "checksum must be 64 (SHA-256) or 128 (SHA-512) hex characters; pass null to skip";
+ private static final String ESCAPE_ERROR =
+ "archive entry escapes the target directory: ";
+ private static final String EXPANSION_CEILING_ERROR =
+ "expanded content exceeds the ceiling of " + KIBIBYTE + " bytes";
+
+ /**
+ * Installs the given file under the default limits and asserts that it fails with the
+ * expected message, leaving the target directory without a single installed file.
+ *
+ * @param file The source file to install.
+ * @param target The target directory, which must have been empty before the attempt.
+ * @param message The exact failure message expected.
+ * @throws IOException Thrown if listing the target directory fails.
+ */
+ private static void assertInstallFails(Path file, Path target, String message)
+ throws IOException {
+ final IOException thrown = Assertions.assertThrows(IOException.class,
+ () -> ResourceInstaller.install(file.toUri(), target));
+ Assertions.assertEquals(message, thrown.getMessage());
+ Assertions.assertEquals(List.of(), installedFiles(target));
+ }
+
+ /**
+ * Installs the given file under the given limits and asserts that it fails with the
+ * expected message, leaving the target directory without a single installed file.
+ *
+ * @param file The source file to install.
+ * @param target The target directory, which must have been empty before the attempt.
+ * @param limits The limits to install under.
+ * @param message The exact failure message expected.
+ * @throws IOException Thrown if listing the target directory fails.
+ */
+ private static void assertInstallFails(Path file, Path target,
+ ResourceInstaller.Limits limits, String message) throws IOException {
+ final IOException thrown = Assertions.assertThrows(IOException.class,
+ () -> ResourceInstaller.install(file.toUri(), target, null, limits));
+ Assertions.assertEquals(message, thrown.getMessage());
+ Assertions.assertEquals(List.of(), installedFiles(target));
+ }
+
/**
* Computes the SHA-256 of the given bytes as an uppercase hex string, so tests can
* prove that checksum comparison does not depend on the hex letter case.
@@ -134,7 +181,6 @@ public class ResourceInstallerTest {
final String wrong = "0".repeat(64);
final IOException thrown = Assertions.assertThrows(IOException.class,
() -> ResourceInstaller.install(file.toUri(), target, wrong));
- Assertions.assertEquals(IOException.class, thrown.getClass());
Assertions.assertEquals("checksum mismatch: expected " + wrong
+ " but downloaded " + sha256(archive), thrown.getMessage());
Assertions.assertEquals(List.of(), installedFiles(target));
@@ -167,11 +213,7 @@ public class ResourceInstallerTest {
final Path file = source.resolve("evil.tar.gz");
Files.write(file, archive);
- final IOException thrown = Assertions.assertThrows(IOException.class,
- () -> ResourceInstaller.install(file.toUri(), target));
- Assertions.assertEquals(
- "archive entry escapes the target directory: ../escape.txt",
- thrown.getMessage());
+ assertInstallFails(file, target, ESCAPE_ERROR + "../escape.txt");
Assertions.assertTrue(Files.notExists(target.getParent().resolve("escape.txt")));
}
@@ -187,12 +229,8 @@ public class ResourceInstallerTest {
final Path file = source.resolve("absolute.tar.gz");
Files.write(file, archive);
- final IOException thrown = Assertions.assertThrows(IOException.class,
- () -> ResourceInstaller.install(file.toUri(), target));
- Assertions.assertEquals(
- "archive entry escapes the target directory: /absolute-escape-attempt/evil.txt",
- thrown.getMessage());
- Assertions.assertEquals(List.of(), installedFiles(target));
+ assertInstallFails(file, target,
+ ESCAPE_ERROR + "/absolute-escape-attempt/evil.txt");
Assertions.assertTrue(Files.notExists(Path.of("/absolute-escape-attempt")));
}
@@ -212,13 +250,8 @@ public class ResourceInstallerTest {
final Path file = source.resolve("evil.zip");
Files.write(file, out.toByteArray());
- final IOException thrown = Assertions.assertThrows(IOException.class,
- () -> ResourceInstaller.install(file.toUri(), target));
- Assertions.assertEquals(
- "archive entry escapes the target directory: ../zip-escape.txt",
- thrown.getMessage());
+ assertInstallFails(file, target, ESCAPE_ERROR + "../zip-escape.txt");
Assertions.assertTrue(Files.notExists(target.getParent().resolve("zip-escape.txt")));
- Assertions.assertEquals(List.of(), installedFiles(target));
}
@Test
@@ -290,25 +323,54 @@ public class ResourceInstallerTest {
Assertions.assertEquals("cat 100", Files.readString(target.resolve("frequencies.txt")));
}
+ /**
+ * Proves that every argument the public API rejects is rejected with its own message,
+ * before anything is fetched.
+ */
@Test
void testInvalidArguments(@TempDir Path target) {
- IllegalArgumentException thrown = Assertions.assertThrows(
- IllegalArgumentException.class, () -> ResourceInstaller.install(null, target));
- Assertions.assertEquals("source must not be null", thrown.getMessage());
+ // assertAll so an unvalidated argument does not hide the others.
+ Assertions.assertAll(
+ () -> assertArgumentError("source must not be null",
+ () -> ResourceInstaller.install(null, target)),
+ () -> assertArgumentError("targetDirectory must not be null",
+ () -> ResourceInstaller.install(target.toUri(), null)),
+ () -> assertArgumentError("limits must not be null",
+ () -> ResourceInstaller.install(target.toUri(), target, null, null)));
+ }
- thrown = Assertions.assertThrows(IllegalArgumentException.class,
- () -> ResourceInstaller.install(target.toUri(), null));
- Assertions.assertEquals("targetDirectory must not be null", thrown.getMessage());
+ /**
+ * Enumerates checksum arguments that are neither a valid SHA-256 nor a valid SHA-512
+ * digest, so a typo cannot masquerade as a checksum mismatch.
+ *
+ * @return One case per rejected digest. Never {@code null}.
+ */
+ static Stream<Arguments> rejectedChecksums() {
+ return Stream.of(
+ Arguments.of("blank", " "),
+ Arguments.of("too short", "abc123"),
+ Arguments.of("right length, non-hex characters", "g".repeat(64)),
+ Arguments.of("between the two supported lengths", "a".repeat(96)),
+ Arguments.of("longer than SHA-512", "a".repeat(129)));
+ }
- thrown = Assertions.assertThrows(IllegalArgumentException.class,
- () -> ResourceInstaller.install(target.toUri(), target, " "));
- Assertions.assertEquals(
- "checksum must be 64 (SHA-256) or 128 (SHA-512) hex characters; pass null to skip",
- thrown.getMessage());
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("rejectedChecksums")
+ void testInvalidChecksumIsRejected(String label, String checksum, @TempDir Path target) {
+ assertArgumentError(CHECKSUM_ARGUMENT_ERROR,
+ () -> ResourceInstaller.install(target.toUri(), target, checksum));
+ }
- thrown = Assertions.assertThrows(IllegalArgumentException.class,
- () -> ResourceInstaller.install(target.toUri(), target, null, null));
- Assertions.assertEquals("limits must not be null", thrown.getMessage());
+ /**
+ * Asserts that the given call fails as an argument error carrying the exact message.
+ *
+ * @param message The exact failure message expected.
+ * @param call The call under test.
+ */
+ private static void assertArgumentError(String message, Executable call) {
+ final IllegalArgumentException thrown =
+ Assertions.assertThrows(IllegalArgumentException.class, call);
+ Assertions.assertEquals(message, thrown.getMessage());
}
/**
@@ -343,34 +405,6 @@ public class ResourceInstallerTest {
Assertions.assertEquals(List.of(), installedFiles(target));
}
- /**
- * Proves that a digest whose length matches neither SHA-256 nor SHA-512 is rejected
- * as an argument error before anything is fetched.
- */
- @Test
- void testChecksumOfUnsupportedLengthIsRejected(@TempDir Path target) {
- final IllegalArgumentException thrown = Assertions.assertThrows(
- IllegalArgumentException.class,
- () -> ResourceInstaller.install(target.toUri(), target, "abc123"));
- Assertions.assertEquals(
- "checksum must be 64 (SHA-256) or 128 (SHA-512) hex characters; pass null to skip",
- thrown.getMessage());
- }
-
- /**
- * Proves that a digest of the right length but with non-hex characters is rejected
- * as an argument error, so a typo cannot masquerade as a checksum mismatch.
- */
- @Test
- void testChecksumWithNonHexCharactersIsRejected(@TempDir Path target) {
- final IllegalArgumentException thrown = Assertions.assertThrows(
- IllegalArgumentException.class,
- () -> ResourceInstaller.install(target.toUri(), target, "g".repeat(64)));
- Assertions.assertEquals(
- "checksum must be 64 (SHA-256) or 128 (SHA-512) hex characters; pass null to skip",
- thrown.getMessage());
- }
-
/**
* Proves staged installation for tar content: when a later entry escapes the target
* directory, the earlier entry that already unpacked cleanly must not appear either.
@@ -384,12 +418,7 @@ public class ResourceInstallerTest {
final Path file = source.resolve("partial.tar.gz");
Files.write(file, archive);
- final IOException thrown = Assertions.assertThrows(IOException.class,
- () -> ResourceInstaller.install(file.toUri(), target));
- Assertions.assertEquals(
- "archive entry escapes the target directory: ../escape.txt",
- thrown.getMessage());
- Assertions.assertEquals(List.of(), installedFiles(target));
+ assertInstallFails(file, target, ESCAPE_ERROR + "../escape.txt");
}
/**
@@ -411,12 +440,7 @@ public class ResourceInstallerTest {
final Path file = source.resolve("partial.zip");
Files.write(file, out.toByteArray());
- final IOException thrown = Assertions.assertThrows(IOException.class,
- () -> ResourceInstaller.install(file.toUri(), target));
- Assertions.assertEquals(
- "archive entry escapes the target directory: ../zip-escape.txt",
- thrown.getMessage());
- Assertions.assertEquals(List.of(), installedFiles(target));
+ assertInstallFails(file, target, ESCAPE_ERROR + "../zip-escape.txt");
}
/**
@@ -536,12 +560,8 @@ public class ResourceInstallerTest {
final Path file = source.resolve("large.txt");
Files.write(file, new byte[8192]);
- final IOException thrown = Assertions.assertThrows(IOException.class,
- () -> ResourceInstaller.install(file.toUri(), target, null,
- ceilings(1024, 1024 * 1024)));
- Assertions.assertEquals("download exceeds the ceiling of 1024 bytes",
- thrown.getMessage());
- Assertions.assertEquals(List.of(), installedFiles(target));
+ assertInstallFails(file, target, ceilings(KIBIBYTE, MEBIBYTE),
+ "download exceeds the ceiling of " + KIBIBYTE + " bytes");
}
/**
@@ -557,12 +577,8 @@ public class ResourceInstallerTest {
final Path file = source.resolve("bomb.tar.gz");
Files.write(file, gzip(tar.toByteArray()));
- final IOException thrown = Assertions.assertThrows(IOException.class,
- () -> ResourceInstaller.install(file.toUri(), target, null,
- ceilings(1024 * 1024, 1024)));
- Assertions.assertEquals("expanded content exceeds the ceiling of 1024 bytes",
- thrown.getMessage());
- Assertions.assertEquals(List.of(), installedFiles(target));
+ assertInstallFails(file, target, ceilings(MEBIBYTE, KIBIBYTE),
+ EXPANSION_CEILING_ERROR);
}
/**
@@ -581,12 +597,8 @@ public class ResourceInstallerTest {
final Path file = source.resolve("bomb.zip");
Files.write(file, out.toByteArray());
- final IOException thrown = Assertions.assertThrows(IOException.class,
- () -> ResourceInstaller.install(file.toUri(), target, null,
- ceilings(1024 * 1024, 1024)));
- Assertions.assertEquals("expanded content exceeds the ceiling of 1024 bytes",
- thrown.getMessage());
- Assertions.assertEquals(List.of(), installedFiles(target));
+ assertInstallFails(file, target, ceilings(MEBIBYTE, KIBIBYTE),
+ EXPANSION_CEILING_ERROR);
}
/**
@@ -599,12 +611,8 @@ public class ResourceInstallerTest {
final Path file = source.resolve("zeros.bin.gz");
Files.write(file, gzip(new byte[64 * 1024]));
- final IOException thrown = Assertions.assertThrows(IOException.class,
- () -> ResourceInstaller.install(file.toUri(), target, null,
- ceilings(1024 * 1024, 1024)));
- Assertions.assertEquals("expanded content exceeds the ceiling of 1024 bytes",
- thrown.getMessage());
- Assertions.assertEquals(List.of(), installedFiles(target));
+ assertInstallFails(file, target, ceilings(MEBIBYTE, KIBIBYTE),
+ EXPANSION_CEILING_ERROR);
}
/**
@@ -619,7 +627,7 @@ public class ResourceInstallerTest {
Files.write(file, archive);
ResourceInstaller.install(file.toUri(), target, sha256(archive),
- ceilings(1024 * 1024, 1024 * 1024));
+ ceilings(MEBIBYTE, MEBIBYTE));
Assertions.assertEquals("small", Files.readString(target.resolve("corpus/data.txt")));
}
@@ -667,7 +675,7 @@ public class ResourceInstallerTest {
final Path file = source.resolve("exact.dat");
Files.write(file, new byte[1024]);
- ResourceInstaller.install(file.toUri(), target, null, ceilings(1024, 1024 * 1024));
+ ResourceInstaller.install(file.toUri(), target, null, ceilings(KIBIBYTE, MEBIBYTE));
Assertions.assertEquals(List.of("exact.dat"), installedFiles(target));
Assertions.assertEquals(1024, Files.size(target.resolve("exact.dat")));
@@ -686,7 +694,7 @@ public class ResourceInstallerTest {
final Path file = source.resolve("exact.tar.gz");
Files.write(file, gzip(tar.toByteArray()));
- ResourceInstaller.install(file.toUri(), target, null, ceilings(1024 * 1024, 1024));
+ ResourceInstaller.install(file.toUri(), target, null, ceilings(MEBIBYTE, KIBIBYTE));
Assertions.assertEquals(List.of("corpus/exact.bin"), installedFiles(target));
Assertions.assertEquals(1024, Files.size(target.resolve("corpus/exact.bin")));
@@ -707,12 +715,8 @@ public class ResourceInstallerTest {
final Path file = source.resolve("cumulative.tar.gz");
Files.write(file, gzip(tar.toByteArray()));
- final IOException thrown = Assertions.assertThrows(IOException.class,
- () -> ResourceInstaller.install(file.toUri(), target, null,
- ceilings(1024 * 1024, 1024)));
- Assertions.assertEquals("expanded content exceeds the ceiling of 1024 bytes",
- thrown.getMessage());
- Assertions.assertEquals(List.of(), installedFiles(target));
+ assertInstallFails(file, target, ceilings(MEBIBYTE, KIBIBYTE),
+ EXPANSION_CEILING_ERROR);
}
/**
@@ -739,4 +743,159 @@ public class ResourceInstallerTest {
Files.readString(target.resolve("corpus/data.txt")));
Assertions.assertEquals(List.of("corpus/data.txt"), installedFiles(target));
}
+
+ /**
+ * Proves that promotion refuses to follow a symbolic link that already exists below
+ * the target. Every archive entry here stays inside the staging directory, so the
+ * entry-name guard never fires; without a check at promotion time the content lands
+ * wherever the link points.
+ */
+ @Test
+ @DisabledOnOs(OS.WINDOWS)
+ void testPromotionRefusesToFollowASymlinkedDirectory(@TempDir Path source,
+ @TempDir Path target, @TempDir Path outside) throws Exception {
+ final Path link = target.resolve("link");
+ Files.createSymbolicLink(link, outside);
+ final byte[] archive = tarGz(new String[][] {{"link/planted.txt", "escaped"}});
+ final Path file = source.resolve("symlink.tar.gz");
+ Files.write(file, archive);
+
+ final IOException thrown = Assertions.assertThrows(IOException.class,
+ () -> ResourceInstaller.install(file.toUri(), target));
+
+ Assertions.assertEquals("installation path crosses a symbolic link: " + link,
+ thrown.getMessage());
+ Assertions.assertEquals(List.of(), installedFiles(outside));
+ Assertions.assertTrue(Files.notExists(outside.resolve("planted.txt")));
+ }
+
+ /**
+ * Proves that an ordinary directory of the same shape still installs, so the symlink
+ * guard rejects the link rather than nested paths in general.
+ */
+ @Test
+ void testNestedDirectoryThatIsNotASymlinkStillInstalls(@TempDir Path source,
+ @TempDir Path target) throws Exception {
+ Files.createDirectory(target.resolve("link"));
+ final byte[] archive = tarGz(new String[][] {{"link/planted.txt", "fine"}});
+ final Path file = source.resolve("nested.tar.gz");
+ Files.write(file, archive);
+
+ ResourceInstaller.install(file.toUri(), target);
+
+ Assertions.assertEquals("fine",
+ Files.readString(target.resolve("link/planted.txt")));
+ }
+
+ /**
+ * Enumerates source schemes the installer refuses, each of which would otherwise be
+ * handed to whatever URL handler the runtime has installed, outside the timeouts and
+ * ceilings this class enforces.
+ *
+ * @return One case per rejected scheme. Never {@code null}.
+ */
+ static Stream<Arguments> rejectedSourceSchemes() {
+ return Stream.of(
+ Arguments.of("ftp", URI.create("ftp://example.invalid/corpus.tar.gz")),
+ Arguments.of("jar", URI.create("jar:file:/tmp/a.jar!/corpus.tar.gz")),
+ Arguments.of("mailto", URI.create("mailto:someone@example.invalid")),
+ Arguments.of("no scheme at all", URI.create("corpus.tar.gz")));
+ }
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("rejectedSourceSchemes")
+ void testUnsupportedSourceSchemeIsRejected(String label, URI source,
+ @TempDir Path target) throws IOException {
+ assertArgumentError("source scheme must be http, https, or file, but was: " + source,
+ () -> ResourceInstaller.install(source, target));
+ Assertions.assertEquals(List.of(), installedFiles(target));
+ }
+
+ /**
+ * Proves that the scheme is rejected before the target directory is touched, so an
+ * unusable location cannot leave an empty directory behind.
+ */
+ @Test
+ void testUnsupportedSourceSchemeIsRejectedBeforeCreatingTheTarget(@TempDir Path parent) {
+ final Path target = parent.resolve("not-created-yet");
+
+ assertArgumentError(
+ "source scheme must be http, https, or file, but was: ftp://example.invalid/c.gz",
+ () -> ResourceInstaller.install(URI.create("ftp://example.invalid/c.gz"), target));
+ Assertions.assertTrue(Files.notExists(target));
+ }
+
+
+
+
+ /**
+ * Proves that a builder with nothing set produces exactly the default limits, so
+ * {@code builder()} is a safe starting point rather than a second set of defaults.
+ */
+ @Test
+ void testBuilderWithoutOverridesEqualsTheDefaults() {
+ Assertions.assertEquals(ResourceInstaller.Limits.DEFAULT,
+ ResourceInstaller.Limits.builder().build());
+ }
+
+ /**
+ * Proves that each builder setter changes its own value and leaves the other four at
+ * their defaults.
+ */
+ @Test
+ void testBuilderOverridesOnlyWhatIsSet() {
+ final ResourceInstaller.Limits limits = ResourceInstaller.Limits.builder()
+ .maxDownloadBytes(MEBIBYTE)
+ .maxExpandedBytes(2 * MEBIBYTE)
+ .build();
+
+ Assertions.assertEquals(MEBIBYTE, limits.maxDownloadBytes());
+ Assertions.assertEquals(2 * MEBIBYTE, limits.maxExpandedBytes());
+ Assertions.assertEquals(ResourceInstaller.Limits.DEFAULT.connectTimeout(),
+ limits.connectTimeout());
+ Assertions.assertEquals(ResourceInstaller.Limits.DEFAULT.readTimeout(),
+ limits.readTimeout());
+ Assertions.assertEquals(ResourceInstaller.Limits.DEFAULT.maxRedirects(),
+ limits.maxRedirects());
+ }
+
+ /**
+ * Proves that the builder validates exactly as the canonical constructor does, so it
+ * cannot be used to sneak past a limit check.
+ */
+ @Test
+ void testBuilderRejectsInvalidValues() {
+ Assertions.assertAll(
+ () -> assertArgumentError("connectTimeout must be positive",
+ () -> ResourceInstaller.Limits.builder().connectTimeout(Duration.ZERO).build()),
+ () -> assertArgumentError("readTimeout must be positive",
+ () -> ResourceInstaller.Limits.builder().readTimeout(null).build()),
+ () -> assertArgumentError("maxRedirects must not be negative",
+ () -> ResourceInstaller.Limits.builder().maxRedirects(-1).build()),
+ () -> assertArgumentError("maxDownloadBytes must be positive",
+ () -> ResourceInstaller.Limits.builder().maxDownloadBytes(0).build()),
+ () -> assertArgumentError("maxExpandedBytes must be positive",
+ () -> ResourceInstaller.Limits.builder().maxExpandedBytes(-1).build()));
+ }
+
+ /**
+ * Proves that the download is written on the target's filesystem rather than in the
+ * system temporary directory, where a 1 GiB default ceiling could exhaust a small
+ * {@code /tmp} while the target has room. The file is hidden and deleted before
+ * {@code install} returns, so a leak surfaces in
+ * {@link #testInstallationLeavesNoStagingResidue} instead.
+ */
+ @Test
+ void testDownloadFileIsCreatedInTheTargetDirectory(@TempDir Path target)
+ throws Exception {
+ final Path downloaded = ResourceInstaller.createDownloadFile(target);
+ try {
+ Assertions.assertEquals(target, downloaded.getParent());
+ Assertions.assertTrue(
+ downloaded.getFileName().toString().startsWith(".opennlp-download"),
+ downloaded.getFileName().toString());
+ } finally {
+ Files.deleteIfExists(downloaded);
+ }
+ }
}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/archive/TarArchives.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/archive/TarArchives.java
new file mode 100644
index 000000000..f0de6c931
--- /dev/null
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/archive/TarArchives.java
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.util.archive;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Builds tar archives byte by byte, so tests can assemble well-formed, boundary, and
+ * deliberately corrupt archives without a third-party library. Headers carry the ustar
+ * magic and a valid checksum, as a real archive does; {@link #reseal(byte[])} restores
+ * the checksum after a test has corrupted some other field on purpose.
+ */
+public final class TarArchives {
+
+ /** The tar block size; headers, contents, and terminators are multiples of it. */
+ public static final int BLOCK = 512;
+
+ /** The size of the two all-zero blocks that terminate a tar archive. */
+ public static final int TERMINATOR_SIZE = 2 * BLOCK;
+
+ /** The type flag of a regular file. */
+ public static final char TYPE_REGULAR_FILE = '0';
+
+ /** The type flag an old-style archive leaves NUL for a regular file. */
+ public static final char TYPE_REGULAR_FILE_CLASSIC = '\0';
+
+ /** The type flag of a directory. */
+ public static final char TYPE_DIRECTORY = '5';
+
+ /** The offset of the 155-byte ustar name prefix field. */
+ public static final int PREFIX_OFFSET = 345;
+
+ private static final int NAME_LENGTH = 100;
+ private static final int PREFIX_LENGTH = 155;
+ private static final int MODE_OFFSET = 100;
+ private static final int SIZE_OFFSET = 124;
+ private static final int SIZE_LENGTH = 12;
+ private static final int BASE_256_MARKER = 0x80;
+ private static final int BASE_256_NEGATIVE = 0x40;
+ private static final int CHECKSUM_OFFSET = 148;
+ private static final int CHECKSUM_LENGTH = 8;
+ private static final int TYPE_OFFSET = 156;
+ private static final int MAGIC_OFFSET = 257;
+ private static final int BLANK = ' ';
+ private static final String SIZE_FORMAT = "%011o";
+ private static final String CHECKSUM_FORMAT = "%06o";
+ private static final String MODE = "0000644 ";
+
+ /** The ustar magic and version fields: {@code "ustar"}, NUL, then {@code "00"}. */
+ private static final byte[] USTAR_MAGIC = {'u', 's', 't', 'a', 'r', 0, '0', '0'};
+
+ /** The GNU magic and version fields: {@code "ustar"}, two blanks, then NUL. */
+ private static final byte[] GNU_MAGIC = {'u', 's', 't', 'a', 'r', ' ', ' ', 0};
+
+ private TarArchives() {
+ }
+
+ /**
+ * Builds one 512-byte tar header block with the given name, declared content size,
+ * and type flag. The mode field is populated with a non-zero octal value so that a
+ * name filling the whole 100-byte name field is followed by non-zero bytes, which
+ * makes name-boundary tests meaningful.
+ *
+ * @param name The entry name; at most 100 bytes when encoded as UTF-8.
+ * @param size The content size to declare in the octal size field; must not be
+ * negative.
+ * @param typeFlag The tar type flag, for example {@link #TYPE_REGULAR_FILE}.
+ * @return The header block. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if the name does not fit the name field or
+ * the size is negative.
+ */
+ public static byte[] header(String name, long size, char typeFlag) {
+ return header(name, "", size, typeFlag, true);
+ }
+
+ /**
+ * Builds one 512-byte classic v7 header block, which carries no ustar magic and no
+ * name prefix field. Real archives in this format still exist, and a reader that keys
+ * off the magic rather than the checksum will not see them.
+ *
+ * @param name The entry name; at most 100 bytes when encoded as UTF-8.
+ * @param size The content size to declare in the octal size field; must not be
+ * negative.
+ * @param typeFlag The tar type flag, for example {@link #TYPE_REGULAR_FILE}.
+ * @return The header block. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if the name does not fit the name field or
+ * the size is negative.
+ */
+ public static byte[] classicHeader(String name, long size, char typeFlag) {
+ return header(name, "", size, typeFlag, false);
+ }
+
+ /**
+ * Builds one 512-byte GNU header block. GNU writes {@code "ustar"} followed by two
+ * blanks and a NUL where ustar writes {@code "ustar"}, a NUL, and the version, and it
+ * uses the offset ustar gives to the name prefix for {@code atime} instead.
+ *
+ * @param name The entry name; at most 100 bytes when encoded as UTF-8.
+ * @param size The content size to declare in the octal size field; must not be
+ * negative.
+ * @param typeFlag The tar type flag, for example {@link #TYPE_REGULAR_FILE}.
+ * @param atime The octal {@code atime} to write at {@link #PREFIX_OFFSET}, as a GNU
+ * incremental archive does, or empty to leave it NUL.
+ * @return The header block. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if the name does not fit the name field or
+ * the size is negative.
+ */
+ public static byte[] gnuHeader(String name, long size, char typeFlag, String atime) {
+ final byte[] block = header(name, "", size, typeFlag, false);
+ System.arraycopy(GNU_MAGIC, 0, block, MAGIC_OFFSET, GNU_MAGIC.length);
+ write(block, PREFIX_OFFSET, atime);
+ return reseal(block);
+ }
+
+ /**
+ * Builds one 512-byte header block whose size field uses the base-256 encoding, which
+ * GNU writes when a length does not fit the eleven octal digits the field holds.
+ *
+ * <p>The leading bit marks the encoding, the next bit carries the sign, and the
+ * remaining bits of that byte followed by every later byte form a big-endian two's
+ * complement number.</p>
+ *
+ * @param name The entry name; at most 100 bytes when encoded as UTF-8.
+ * @param size The content size to encode. May be negative, which no real writer emits
+ * for a size, so that the reader's rejection of it can be exercised.
+ * @param typeFlag The tar type flag, for example {@link #TYPE_REGULAR_FILE}.
+ * @return The header block. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if the name does not fit the name field.
+ */
+ public static byte[] base256Header(String name, long size, char typeFlag) {
+ final byte[] block = header(name, 0, typeFlag);
+ long remaining = size;
+ for (int i = SIZE_OFFSET + SIZE_LENGTH - 1; i >= SIZE_OFFSET; i--) {
+ block[i] = (byte) (remaining & 0xFF);
+ remaining >>= Byte.SIZE;
+ }
+ block[SIZE_OFFSET] = (byte) (size < 0
+ ? block[SIZE_OFFSET] | BASE_256_MARKER | BASE_256_NEGATIVE
+ : block[SIZE_OFFSET] & ~BASE_256_NEGATIVE | BASE_256_MARKER);
+ return reseal(block);
+ }
+
+ /**
+ * Encodes one pax extended header record, {@code "<length> <keyword>=<value>\n"}. The
+ * length counts the whole record including its own digits, the blank, and the newline,
+ * so it is solved for rather than guessed.
+ *
+ * @param keyword The pax keyword.
+ * @param value The keyword's value.
+ * @return The encoded record. Never {@code null}.
+ */
+ public static byte[] paxRecord(String keyword, String value) {
+ final byte[] body = (keyword + "=" + value + "\n").getBytes(StandardCharsets.UTF_8);
+ int digits = 1;
+ while (Integer.toString(body.length + 1 + digits).length() > digits) {
+ digits++;
+ }
+ final byte[] prefix = ((body.length + 1 + digits) + " ")
+ .getBytes(StandardCharsets.US_ASCII);
+ final byte[] record = new byte[prefix.length + body.length];
+ System.arraycopy(prefix, 0, record, 0, prefix.length);
+ System.arraycopy(body, 0, record, prefix.length, body.length);
+ return record;
+ }
+
+ /**
+ * Builds one 512-byte ustar header block whose entry name is split across the name
+ * prefix field and the name field, which is how a real archive stores a name longer
+ * than 100 bytes.
+ *
+ * @param name The name field content; at most 100 bytes when encoded as UTF-8.
+ * @param prefix The name prefix field content; at most 155 bytes when encoded as
+ * UTF-8. Empty for a header without a prefix.
+ * @param size The content size to declare in the octal size field; must not be
+ * negative.
+ * @param typeFlag The tar type flag, for example {@link #TYPE_REGULAR_FILE}.
+ * @return The header block. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if either name part does not fit its field
+ * or the size is negative.
+ */
+ public static byte[] header(String name, String prefix, long size, char typeFlag) {
+ return header(name, prefix, size, typeFlag, true);
+ }
+
+ /**
+ * Builds one 512-byte header block, in either the ustar or the classic v7 format.
+ *
+ * @param name The name field content; at most 100 bytes when encoded as UTF-8.
+ * @param prefix The name prefix field content; at most 155 bytes when encoded as
+ * UTF-8. Empty for a header without a prefix, and required to be empty
+ * for a classic header, which has no prefix field.
+ * @param size The content size to declare in the octal size field; must not be
+ * negative.
+ * @param typeFlag The tar type flag, for example {@link #TYPE_REGULAR_FILE}.
+ * @param ustar Whether to write the ustar magic and version fields.
+ * @return The header block. Never {@code null}.
+ * @throws IllegalArgumentException Thrown if either name part does not fit its field,
+ * the size is negative, or a classic header is asked for with a prefix.
+ */
+ private static byte[] header(String name, String prefix, long size, char typeFlag,
+ boolean ustar) {
+ final byte[] nameBytes = name.getBytes(StandardCharsets.UTF_8);
+ if (nameBytes.length > NAME_LENGTH) {
+ throw new IllegalArgumentException(
+ "entry name exceeds " + NAME_LENGTH + " bytes: " + name);
+ }
+ final byte[] prefixBytes = prefix.getBytes(StandardCharsets.UTF_8);
+ if (prefixBytes.length > PREFIX_LENGTH) {
+ throw new IllegalArgumentException(
+ "entry name prefix exceeds " + PREFIX_LENGTH + " bytes: " + prefix);
+ }
+ if (size < 0) {
+ throw new IllegalArgumentException("size must not be negative: " + size);
+ }
+ if (!ustar && prefixBytes.length > 0) {
+ throw new IllegalArgumentException("a classic header has no name prefix field");
+ }
+ final byte[] block = new byte[BLOCK];
+ System.arraycopy(nameBytes, 0, block, 0, nameBytes.length);
+ System.arraycopy(prefixBytes, 0, block, PREFIX_OFFSET, prefixBytes.length);
+ write(block, MODE_OFFSET, MODE);
+ write(block, SIZE_OFFSET, String.format(SIZE_FORMAT, size));
+ if (ustar) {
+ System.arraycopy(USTAR_MAGIC, 0, block, MAGIC_OFFSET, USTAR_MAGIC.length);
+ }
+ block[TYPE_OFFSET] = (byte) typeFlag;
+ return reseal(block);
+ }
+
+ /**
+ * Recomputes and writes the header checksum of the given block, so a test can corrupt
+ * a field on purpose and still hand the reader an otherwise well-formed header.
+ *
+ * @param block The header block to seal. Must be 512 bytes long.
+ * @return The same block, with its checksum field filled in. Never {@code null}.
+ */
+ public static byte[] reseal(byte[] block) {
+ for (int i = CHECKSUM_OFFSET; i < CHECKSUM_OFFSET + CHECKSUM_LENGTH; i++) {
+ block[i] = BLANK;
+ }
+ int sum = 0;
+ for (final byte b : block) {
+ sum += b & 0xFF;
+ }
+ write(block, CHECKSUM_OFFSET, String.format(CHECKSUM_FORMAT, sum));
+ block[CHECKSUM_OFFSET + CHECKSUM_LENGTH - 1] = BLANK;
+ return block;
+ }
+
+ /**
+ * Writes US-ASCII text into a header field.
+ *
+ * @param block The header block to write into.
+ * @param offset The field's offset in the block.
+ * @param text The text to write.
+ */
+ private static void write(byte[] block, int offset, String text) {
+ final byte[] bytes = text.getBytes(StandardCharsets.US_ASCII);
+ System.arraycopy(bytes, 0, block, offset, bytes.length);
+ }
+
+ /**
+ * Writes one complete tar entry into the given buffer: the header block declaring the
+ * content's actual length, the content itself, and zero padding up to the next
+ * 512-byte block boundary.
+ *
+ * @param tar The buffer receiving the entry bytes. Must not be {@code null}.
+ * @param name The entry name; at most 100 bytes when encoded as UTF-8.
+ * @param content The entry content. Must not be {@code null}.
+ * @param typeFlag The tar type flag for the header.
+ * @throws IOException Thrown if writing to the buffer fails.
+ * @throws IllegalArgumentException Thrown if the name does not fit the name field.
+ */
+ public static void entry(ByteArrayOutputStream tar, String name, byte[] content,
+ char typeFlag) throws IOException {
+ tar.write(header(name, content.length, typeFlag));
+ tar.write(content);
+ tar.write(new byte[(BLOCK - content.length % BLOCK) % BLOCK]);
+ }
+
+ /**
+ * Writes one complete regular-file tar entry into the given buffer.
+ *
+ * @param tar The buffer receiving the entry bytes. Must not be {@code null}.
+ * @param name The entry name; at most 100 bytes when encoded as UTF-8.
+ * @param content The entry content. Must not be {@code null}.
+ * @throws IOException Thrown if writing to the buffer fails.
+ * @throws IllegalArgumentException Thrown if the name does not fit the name field.
+ */
+ public static void entry(ByteArrayOutputStream tar, String name, byte[] content)
+ throws IOException {
+ entry(tar, name, content, TYPE_REGULAR_FILE);
+ }
+}
diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/archive/TarStreamTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/archive/TarStreamTest.java
index 91cd10195..a7e2aa0d8 100644
--- a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/archive/TarStreamTest.java
+++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/util/archive/TarStreamTest.java
@@ -24,6 +24,7 @@ import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
+import java.util.function.Supplier;
import java.util.stream.Stream;
import org.junit.jupiter.api.Assertions;
@@ -32,76 +33,40 @@ import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;
+import static opennlp.tools.util.archive.TarArchives.BLOCK;
+import static opennlp.tools.util.archive.TarArchives.TERMINATOR_SIZE;
+import static opennlp.tools.util.archive.TarArchives.TYPE_DIRECTORY;
+import static opennlp.tools.util.archive.TarArchives.TYPE_REGULAR_FILE;
+import static opennlp.tools.util.archive.TarArchives.TYPE_REGULAR_FILE_CLASSIC;
+import static opennlp.tools.util.archive.TarArchives.entry;
+import static opennlp.tools.util.archive.TarArchives.header;
+
/**
* Exercises {@link TarStream} against archives built byte by byte in the test, covering
* regular traversal as well as boundary and corruption cases.
*/
public class TarStreamTest {
- private static final int BLOCK = 512;
- private static final int TERMINATOR_SIZE = 2 * BLOCK;
private static final int NAME_LENGTH = 100;
private static final int SIZE_OFFSET = 124;
- private static final int TYPE_OFFSET = 156;
- private static final String SIZE_FORMAT = "%011o";
- private static final String MODE = "0000644 ";
- private static final char TYPE_REGULAR_FILE = '0';
- private static final char TYPE_REGULAR_FILE_CLASSIC = '\0';
- private static final char TYPE_DIRECTORY = '5';
-
- /**
- * Builds one 512-byte tar header block with the given name, declared content size,
- * and type flag. The mode field is populated with a non-zero octal value so that a
- * name filling the whole 100-byte name field is followed by non-zero bytes, which
- * makes name-boundary tests meaningful.
- *
- * @param name The entry name; at most 100 bytes when encoded as UTF-8.
- * @param size The content size to declare in the octal size field; must not be
- * negative.
- * @param typeFlag The tar type flag, for example {@code '0'} for a regular file,
- * {@code '5'} for a directory, or {@code '\0'} for an old-style
- * regular file.
- * @return The header block. Never {@code null}.
- * @throws IllegalArgumentException Thrown if the name does not fit the name field or
- * the size is negative.
- */
- private static byte[] header(String name, long size, char typeFlag) {
- final byte[] nameBytes = name.getBytes(StandardCharsets.UTF_8);
- if (nameBytes.length > NAME_LENGTH) {
- throw new IllegalArgumentException(
- "entry name exceeds " + NAME_LENGTH + " bytes: " + name);
- }
- if (size < 0) {
- throw new IllegalArgumentException("size must not be negative: " + size);
- }
- final byte[] block = new byte[BLOCK];
- System.arraycopy(nameBytes, 0, block, 0, nameBytes.length);
- final byte[] mode = MODE.getBytes(StandardCharsets.US_ASCII);
- System.arraycopy(mode, 0, block, NAME_LENGTH, mode.length);
- final byte[] sizeOctal = String.format(SIZE_FORMAT, size)
- .getBytes(StandardCharsets.US_ASCII);
- System.arraycopy(sizeOctal, 0, block, SIZE_OFFSET, sizeOctal.length);
- block[TYPE_OFFSET] = (byte) typeFlag;
- return block;
- }
+ private static final int CHECKSUM_OFFSET = 148;
- /**
- * Writes one complete tar entry into the given buffer: the header block declaring
- * the content's actual length, the content itself, and zero padding up to the next
- * 512-byte block boundary.
- *
- * @param tar The buffer receiving the entry bytes. Must not be {@code null}.
- * @param name The entry name; at most 100 bytes when encoded as UTF-8.
- * @param content The entry content. Must not be {@code null}.
- * @param typeFlag The tar type flag for the header.
- * @throws IOException Thrown if writing to the buffer fails.
- */
- private static void entry(ByteArrayOutputStream tar, String name, byte[] content,
- char typeFlag) throws IOException {
- tar.write(header(name, content.length, typeFlag));
- tar.write(content);
- tar.write(new byte[(BLOCK - content.length % BLOCK) % BLOCK]);
- }
+ /** The largest length eleven octal digits can hold, one byte short of 8 GiB. */
+ private static final long LARGEST_OCTAL_SIZE = (1L << 33) - 1;
+
+ private static final String MALFORMED_RECORD = "malformed pax extended header record";
+ private static final String SPARSE_REJECTED = "sparse tar entries are not supported: "
+ + "the archived bytes encode holes rather than the file content";
+
+ /** The metadata records GNU tar and bsdtar write ahead of an ordinary entry. */
+ private static final String[][] METADATA = {
+ {"mtime", "1786557589.505896519"},
+ {"atime", "1786557589.496852337"},
+ {"ctime", "1786557589.505896519"},
+ {"uid", "1000"},
+ {"gname", "krickert"},
+ {"SCHILY.dev", "66306"},
+ {"hdrcharset", "BINARY"}};
/**
* Supplies blocks that must not be mistaken for a tar header, each with a description
@@ -266,6 +231,8 @@ public class TarStreamTest {
void testMalformedSizeFieldFailsLoud() {
final byte[] block = header("bad.bin", 0, TYPE_REGULAR_FILE);
block[SIZE_OFFSET] = '9';
+ // Reseal, so the reader reaches the size field instead of stopping at the checksum.
+ TarArchives.reseal(block);
final TarStream stream = new TarStream(new ByteArrayInputStream(block));
final IOException thrown = Assertions.assertThrows(IOException.class, stream::next);
@@ -314,16 +281,18 @@ public class TarStreamTest {
*/
@Test
void testStartsWithHeaderRejectsUnusableStreams() {
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> TarStream.startsWithHeader(null));
final InputStream notMarkable = new InputStream() {
@Override
public int read() {
return -1;
}
};
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> TarStream.startsWithHeader(notMarkable));
+ // assertAll so a missing check on one argument does not hide the other.
+ Assertions.assertAll(
+ () -> Assertions.assertThrows(IllegalArgumentException.class,
+ () -> TarStream.startsWithHeader(null)),
+ () -> Assertions.assertThrows(IllegalArgumentException.class,
+ () -> TarStream.startsWithHeader(notMarkable)));
}
/**
@@ -333,4 +302,537 @@ public class TarStreamTest {
void testNullStreamIsRejected() {
Assertions.assertThrows(IllegalArgumentException.class, () -> new TarStream(null));
}
+
+ /**
+ * Proves that a header whose stored checksum does not match its bytes is rejected.
+ * Without the checksum, arbitrary content carrying the ustar magic would be read as an
+ * archive and its size field trusted.
+ */
+ @Test
+ void testHeaderWithWrongChecksumIsRejected() {
+ final byte[] block = header("tampered.bin", 10, TYPE_REGULAR_FILE);
+ block[0] = 'X';
+ final TarStream stream = new TarStream(new ByteArrayInputStream(block));
+
+ final IOException thrown = Assertions.assertThrows(IOException.class, stream::next);
+ Assertions.assertEquals("malformed tar header checksum", thrown.getMessage());
+ }
+
+ /**
+ * Proves that detection is not fooled by content that merely carries the ustar magic
+ * at the right offset without a matching checksum.
+ */
+ @Test
+ void testStartsWithHeaderRejectsUstarMagicWithoutAChecksum() throws IOException {
+ final byte[] block = header("looks-real.bin", 0, TYPE_REGULAR_FILE);
+ Arrays.fill(block, CHECKSUM_OFFSET, CHECKSUM_OFFSET + 8, (byte) '0');
+
+ Assertions.assertFalse(TarStream.startsWithHeader(new ByteArrayInputStream(block)));
+ }
+
+ /**
+ * Proves that a ustar name prefix is joined to the name field, which is how a path
+ * longer than the 100-byte name field is stored. Without it the entry would surface
+ * under the truncated tail of its own path.
+ */
+ @Test
+ void testUstarPrefixIsJoinedToTheName() throws IOException {
+ final String prefix = "corpus-1.0/" + "d".repeat(120);
+ final String tail = "annotations/train.conllu";
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ final byte[] content = "# sent_id = 1\n".getBytes(StandardCharsets.UTF_8);
+ tar.write(TarArchives.header(tail, prefix, content.length, TYPE_REGULAR_FILE));
+ tar.write(content);
+ tar.write(new byte[BLOCK - content.length]);
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals(prefix + "/" + tail, stream.name());
+ Assertions.assertTrue(stream.name().length() > 100);
+ Assertions.assertArrayEquals(content, stream.entryStream().readAllBytes());
+ }
+
+ /**
+ * Proves that an empty ustar prefix leaves the name field alone rather than producing
+ * a leading slash.
+ */
+ @Test
+ void testEmptyUstarPrefixLeavesTheNameAlone() throws IOException {
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ entry(tar, "plain.txt", "x".getBytes(StandardCharsets.UTF_8));
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals("plain.txt", stream.name());
+ }
+
+
+
+ /**
+ * Proves that a pax global header carrying only a {@code comment} record is consumed
+ * and the entry after it is delivered normally. A tar written by {@code git archive}
+ * starts with exactly this header.
+ */
+ @Test
+ void testPaxGlobalHeaderWithOnlyACommentIsConsumed() throws IOException {
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ entry(tar, "pax_global_header",
+ concat(TarArchives.paxRecord("comment", "0123456789abcdef"),
+ metadataRecords()), 'g');
+ entry(tar, "data.txt", "content".getBytes(StandardCharsets.UTF_8));
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals("data.txt", stream.name());
+ Assertions.assertTrue(stream.isFile());
+ Assertions.assertArrayEquals("content".getBytes(StandardCharsets.UTF_8),
+ stream.entryStream().readAllBytes());
+ Assertions.assertFalse(stream.next());
+ }
+
+ /**
+ * Supplies pax global headers that must be rejected: the two keywords that would change
+ * every entry after them, the sparse family whose content cannot be unpacked, and
+ * payloads that are not pax records at all.
+ *
+ * @return One case per rejected global header. Never {@code null}.
+ */
+ private static Stream<Arguments> rejectedGlobalHeaders() {
+ return Stream.of(
+ Arguments.of("path redirects every following entry",
+ TarArchives.paxRecord("path", "../escape.txt"),
+ "pax global header carries path, which would change every entry after it"),
+ Arguments.of("size resizes every following entry",
+ TarArchives.paxRecord("size", "999999"),
+ "pax global header carries size, which would change every entry after it"),
+ Arguments.of("sparse map cannot be unpacked",
+ TarArchives.paxRecord("GNU.sparse.name", "holes.bin"),
+ SPARSE_REJECTED),
+ Arguments.of("no length prefix",
+ "comment=0\n".getBytes(StandardCharsets.UTF_8), MALFORMED_RECORD),
+ Arguments.of("length longer than the payload",
+ "99 comment=0\n".getBytes(StandardCharsets.UTF_8), MALFORMED_RECORD),
+ Arguments.of("length shorter than its own prefix",
+ "1 comment=0\n".getBytes(StandardCharsets.UTF_8), MALFORMED_RECORD),
+ Arguments.of("no keyword before the equals sign",
+ "12 =value\n".getBytes(StandardCharsets.UTF_8), MALFORMED_RECORD),
+ Arguments.of("no equals sign at all",
+ "12 comment0\n".getBytes(StandardCharsets.UTF_8), MALFORMED_RECORD));
+ }
+ /**
+ * Proves that a pax global header is checked rather than skipped. A global header
+ * applies to every entry that follows it, so one carrying {@code path} or {@code size}
+ * would silently change what those entries are.
+ */
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("rejectedGlobalHeaders")
+ void testRejectedPaxGlobalHeader(String description, byte[] payload, String message)
+ throws IOException {
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ entry(tar, "pax_global_header", payload, 'g');
+ entry(tar, "data.txt", "content".getBytes(StandardCharsets.UTF_8));
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+
+ final IOException thrown = Assertions.assertThrows(IOException.class, stream::next);
+ Assertions.assertEquals(message, thrown.getMessage());
+ }
+
+ /**
+ * Proves that a classic v7 archive, which carries no ustar magic at all, is read.
+ * Keying detection off the magic rather than the header checksum would miss it.
+ */
+ @Test
+ void testClassicHeaderWithoutUstarMagicIsRead() throws IOException {
+ final byte[] content = "classic\n".getBytes(StandardCharsets.UTF_8);
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ tar.write(TarArchives.classicHeader("v7/data.txt", content.length,
+ TYPE_REGULAR_FILE));
+ tar.write(content);
+ tar.write(new byte[BLOCK - content.length]);
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final byte[] archive = tar.toByteArray();
+
+ Assertions.assertTrue(TarStream.startsWithHeader(
+ new BufferedInputStream(new ByteArrayInputStream(archive))));
+ final TarStream stream = new TarStream(new ByteArrayInputStream(archive));
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals("v7/data.txt", stream.name());
+ Assertions.assertTrue(stream.isFile());
+ Assertions.assertArrayEquals(content, stream.entryStream().readAllBytes());
+ Assertions.assertFalse(stream.next());
+ }
+
+ /**
+ * Proves that a header with a valid checksum but no name is refused rather than
+ * yielding a nameless entry, and that detection refuses the same block, so the two
+ * agree.
+ */
+ @Test
+ void testHeaderWithAnEmptyNameIsRejected() throws IOException {
+ final byte[] block = TarArchives.reseal(header("", 0, TYPE_REGULAR_FILE));
+ final TarStream stream = new TarStream(new ByteArrayInputStream(block));
+
+ Assertions.assertAll(
+ () -> {
+ final IOException thrown =
+ Assertions.assertThrows(IOException.class, stream::next);
+ Assertions.assertEquals("tar entry header carries an empty name",
+ thrown.getMessage());
+ },
+ () -> Assertions.assertFalse(
+ TarStream.startsWithHeader(new ByteArrayInputStream(block))));
+ }
+
+ /**
+ * Pins the {@link InputStream#read(byte[], int, int)} range contract: the arguments are
+ * checked before anything else, so an invalid range is reported even when the entry is
+ * exhausted or the length is zero, rather than being masked by an early return.
+ */
+ @Test
+ void testEntryStreamRejectsInvalidReadRanges() throws IOException {
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ entry(tar, "data.txt", "content".getBytes(StandardCharsets.UTF_8));
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+ Assertions.assertTrue(stream.next());
+ final InputStream content = stream.entryStream();
+ final byte[] buffer = new byte[8];
+
+ Assertions.assertAll(
+ () -> Assertions.assertThrows(NullPointerException.class,
+ () -> content.read(null, 0, 1)),
+ () -> Assertions.assertThrows(NullPointerException.class,
+ () -> content.read(null, 0, 0)),
+ () -> Assertions.assertThrows(IndexOutOfBoundsException.class,
+ () -> content.read(buffer, -1, 1)),
+ () -> Assertions.assertThrows(IndexOutOfBoundsException.class,
+ () -> content.read(buffer, 0, -1)),
+ () -> Assertions.assertThrows(IndexOutOfBoundsException.class,
+ () -> content.read(buffer, 0, buffer.length + 1)),
+ () -> Assertions.assertThrows(IndexOutOfBoundsException.class,
+ () -> content.read(buffer, buffer.length, 1)),
+ // Zero length at a valid offset is legal and must not be mistaken for an error.
+ () -> Assertions.assertEquals(0, content.read(buffer, buffer.length, 0)));
+ }
+ /**
+ * Pins the {@link java.io.InputStream#read(byte[], int, int)} contract for a
+ * zero-length read: it must report zero bytes read, both mid-entry and at the end of
+ * the entry, and must never be mistaken for end of stream.
+ */
+ @Test
+ void testZeroLengthReadReturnsZero() throws IOException {
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ entry(tar, "data.txt", "content".getBytes(StandardCharsets.UTF_8));
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+ Assertions.assertTrue(stream.next());
+ final InputStream content = stream.entryStream();
+ final byte[] buffer = new byte[8];
+
+ Assertions.assertEquals(0, content.read(buffer, 0, 0));
+ Assertions.assertArrayEquals("content".getBytes(StandardCharsets.UTF_8),
+ content.readAllBytes());
+ // Exhausted: a zero-length read still reports zero, not end of stream.
+ Assertions.assertEquals(0, content.read(buffer, 0, 0));
+ Assertions.assertEquals(-1, content.read(buffer, 0, buffer.length));
+ // Exhausted, and the range is still checked before the end-of-stream answer.
+ Assertions.assertThrows(IndexOutOfBoundsException.class,
+ () -> content.read(buffer, 0, buffer.length + 1));
+ }
+
+ /**
+ * Concatenates byte arrays, so a test can assemble one extension header payload from
+ * several records.
+ *
+ * @param parts The pieces to join. Must not be {@code null}.
+ * @return The concatenation. Never {@code null}.
+ */
+ private static byte[] concat(byte[]... parts) {
+ final ByteArrayOutputStream joined = new ByteArrayOutputStream();
+ for (final byte[] part : parts) {
+ joined.writeBytes(part);
+ }
+ return joined.toByteArray();
+ }
+
+ /**
+ * Encodes the metadata records a real writer emits alongside the ones that matter, so
+ * tests exercise the shape an archive actually has rather than a minimal one.
+ *
+ * @return The encoded records. Never {@code null}.
+ */
+ private static byte[] metadataRecords() {
+ final ByteArrayOutputStream records = new ByteArrayOutputStream();
+ for (final String[] record : METADATA) {
+ records.writeBytes(TarArchives.paxRecord(record[0], record[1]));
+ }
+ return records.toByteArray();
+ }
+
+ /**
+ * Proves that a pax extended header supplies the name of the entry after it. The entry
+ * header carries only a truncated name, so ignoring the extended header delivers the
+ * wrong path, and refusing it rejects every archive {@code bsdtar} or
+ * {@code tar --format=posix} writes.
+ */
+ @Test
+ void testPaxExtendedHeaderSuppliesTheEntryName() throws IOException {
+ final String path = "./corpus-1.0/" + "d".repeat(120) + "/annotations/train.conllu";
+ final byte[] content = "content\n".getBytes(StandardCharsets.UTF_8);
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ entry(tar, "./PaxHeaders/train.conllu",
+ concat(TarArchives.paxRecord("path", path), metadataRecords()), 'x');
+ // The entry header holds the truncated name, exactly as GNU tar writes it.
+ entry(tar, path.substring(0, 100), content);
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals(path, stream.name());
+ Assertions.assertTrue(stream.isFile());
+ Assertions.assertArrayEquals(content, stream.entryStream().readAllBytes());
+ Assertions.assertFalse(stream.next());
+ }
+
+ /**
+ * Proves that a pax header carrying nothing but metadata leaves the entry alone. A pax
+ * archive has one of these ahead of every entry, including entries needing no override,
+ * so this is the common case rather than an edge one.
+ */
+ @Test
+ void testPaxMetadataOnlyHeaderLeavesTheEntryAlone() throws IOException {
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ entry(tar, "./PaxHeaders/short.txt", metadataRecords(), 'x');
+ entry(tar, "./short.txt", "short\n".getBytes(StandardCharsets.UTF_8));
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals("./short.txt", stream.name());
+ Assertions.assertEquals(6, stream.size());
+ Assertions.assertFalse(stream.next());
+ }
+
+ /**
+ * Proves that a pax extended header applies to the entry immediately after it and to no
+ * other, so a name override cannot leak onto later entries.
+ */
+ @Test
+ void testPaxExtendedHeaderAppliesOnlyToTheNextEntry() throws IOException {
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ entry(tar, "./PaxHeaders/first",
+ TarArchives.paxRecord("path", "overridden/first.txt"), 'x');
+ entry(tar, "truncated-first", "one".getBytes(StandardCharsets.UTF_8));
+ entry(tar, "second.txt", "two".getBytes(StandardCharsets.UTF_8));
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals("overridden/first.txt", stream.name());
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals("second.txt", stream.name());
+ Assertions.assertFalse(stream.next());
+ }
+
+ /**
+ * Proves that a pax {@code size} record overrides the header's octal size field, which
+ * is how an entry too large for eleven octal digits states its length.
+ */
+ @Test
+ void testPaxExtendedHeaderSuppliesTheEntrySize() throws IOException {
+ final byte[] content = "0123456789".getBytes(StandardCharsets.US_ASCII);
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ entry(tar, "./PaxHeaders/data.bin", TarArchives.paxRecord("size", "10"), 'x');
+ tar.write(header("data.bin", 0, TYPE_REGULAR_FILE));
+ tar.write(content);
+ tar.write(new byte[BLOCK - content.length]);
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals(10, stream.size());
+ Assertions.assertArrayEquals(content, stream.entryStream().readAllBytes());
+ }
+
+ /**
+ * Proves that a GNU long-name header supplies the name of the entry after it. Its
+ * payload is the name terminated by a NUL, which must not survive into the name.
+ */
+ @Test
+ void testGnuLongNameHeaderSuppliesTheEntryName() throws IOException {
+ final String path = "./corpus-1.0/" + "d".repeat(120) + "/annotations/train.conllu";
+ final byte[] content = "content\n".getBytes(StandardCharsets.UTF_8);
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ entry(tar, "././@LongLink",
+ (path + "\0").getBytes(StandardCharsets.UTF_8), 'L');
+ entry(tar, path.substring(0, 100), content);
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals(path, stream.name());
+ Assertions.assertArrayEquals(content, stream.entryStream().readAllBytes());
+ Assertions.assertFalse(stream.next());
+ }
+
+ /**
+ * Proves that a GNU long-link header is consumed without disturbing the entry after it.
+ * It names a link target, which this reader does not expose.
+ */
+ @Test
+ void testGnuLongLinkHeaderIsConsumed() throws IOException {
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ entry(tar, "././@LongLink",
+ ("../" + "t".repeat(120) + "/target\0").getBytes(StandardCharsets.UTF_8), 'K');
+ entry(tar, "data.txt", "content".getBytes(StandardCharsets.UTF_8));
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals("data.txt", stream.name());
+ Assertions.assertArrayEquals("content".getBytes(StandardCharsets.UTF_8),
+ stream.entryStream().readAllBytes());
+ }
+
+ /**
+ * Proves that a GNU header is not read as a ustar one. GNU puts {@code atime} at the
+ * offset ustar gives to the name prefix, so consulting it there would deliver every
+ * entry of a GNU incremental archive under a directory named after an octal timestamp.
+ */
+ @Test
+ void testGnuHeaderDoesNotReadItsAtimeAsANamePrefix() throws IOException {
+ final byte[] content = "short\n".getBytes(StandardCharsets.UTF_8);
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ tar.write(TarArchives.gnuHeader("./short.txt", content.length, TYPE_REGULAR_FILE,
+ "15237132225"));
+ tar.write(content);
+ tar.write(new byte[BLOCK - content.length]);
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals("./short.txt", stream.name());
+ Assertions.assertArrayEquals(content, stream.entryStream().readAllBytes());
+ }
+
+ /**
+ * Supplies the two ways an archive declares a sparse entry, whose archived bytes encode
+ * holes rather than the content, so unpacking them literally would write a wrong file.
+ *
+ * @return One case per sparse declaration. Never {@code null}.
+ */
+ private static Stream<Arguments> sparseEntries() {
+ return Stream.of(
+ Arguments.of("GNU sparse type flag", (Supplier<byte[]>) () -> {
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ tar.writeBytes(header("holes.bin", 0, 'S'));
+ return tar.toByteArray();
+ }),
+ Arguments.of("pax sparse records", (Supplier<byte[]>) () -> {
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ try {
+ entry(tar, "./PaxHeaders/holes.bin",
+ concat(TarArchives.paxRecord("GNU.sparse.major", "1"),
+ TarArchives.paxRecord("GNU.sparse.name", "holes.bin")), 'x');
+ entry(tar, "holes.bin", "not the content".getBytes(StandardCharsets.UTF_8));
+ } catch (IOException e) {
+ throw new IllegalStateException(e);
+ }
+ return tar.toByteArray();
+ }));
+ }
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("sparseEntries")
+ void testSparseEntriesAreRejected(String description, Supplier<byte[]> archive) {
+ final TarStream stream = new TarStream(new ByteArrayInputStream(archive.get()));
+
+ final IOException thrown = Assertions.assertThrows(IOException.class, stream::next);
+ Assertions.assertEquals(SPARSE_REJECTED, thrown.getMessage());
+ }
+
+ /**
+ * Proves that a size field in the base-256 encoding is read, content and all. GNU
+ * writes this form when a length does not fit the eleven octal digits the field holds.
+ */
+ @Test
+ void testBase256SizeFieldIsRead() throws IOException {
+ final byte[] content = "0123456789".getBytes(StandardCharsets.US_ASCII);
+ final ByteArrayOutputStream tar = new ByteArrayOutputStream();
+ tar.write(TarArchives.base256Header("big.bin", content.length, TYPE_REGULAR_FILE));
+ tar.write(content);
+ tar.write(new byte[BLOCK - content.length]);
+ tar.write(new byte[TERMINATOR_SIZE]);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(tar.toByteArray()));
+
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals("big.bin", stream.name());
+ Assertions.assertEquals(content.length, stream.size());
+ Assertions.assertArrayEquals(content, stream.entryStream().readAllBytes());
+ }
+
+ /**
+ * Proves that a length beyond what eleven octal digits can hold is read correctly,
+ * which is the only reason the encoding exists. Eleven octal digits stop one byte short
+ * of 8 GiB, so the octal path cannot express this size at all.
+ */
+ @Test
+ void testBase256SizeFieldCarriesLengthsBeyondTheOctalRange() throws IOException {
+ final long beyondOctal = 8L * 1024 * 1024 * 1024;
+ Assertions.assertTrue(beyondOctal > LARGEST_OCTAL_SIZE);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(
+ TarArchives.base256Header("huge.bin", beyondOctal, TYPE_REGULAR_FILE)));
+
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals(beyondOctal, stream.size());
+ }
+
+ /**
+ * Proves that the largest length a {@code long} can hold round-trips, so the overflow
+ * guard rejects only what genuinely does not fit.
+ */
+ @Test
+ void testBase256SizeFieldAcceptsTheLargestRepresentableLength() throws IOException {
+ final TarStream stream = new TarStream(new ByteArrayInputStream(
+ TarArchives.base256Header("max.bin", Long.MAX_VALUE, TYPE_REGULAR_FILE)));
+
+ Assertions.assertTrue(stream.next());
+ Assertions.assertEquals(Long.MAX_VALUE, stream.size());
+ }
+
+ /**
+ * Proves that a base-256 length too large for a {@code long} is refused rather than
+ * silently truncated to a small one, which would make the reader stop early inside the
+ * entry and read the rest of it as headers.
+ */
+ @Test
+ void testBase256SizeFieldBeyondTheLongRangeIsRejected() {
+ final byte[] block = TarArchives.base256Header("huge.bin", 0, TYPE_REGULAR_FILE);
+ // One bit above the 63 a long can hold, left of everything the encoder can write.
+ block[SIZE_OFFSET + 1] = 1;
+ TarArchives.reseal(block);
+ final TarStream stream = new TarStream(new ByteArrayInputStream(block));
+
+ final IOException thrown = Assertions.assertThrows(IOException.class, stream::next);
+ Assertions.assertEquals(
+ "tar size field exceeds the largest length this reader can represent",
+ thrown.getMessage());
+ }
+
+ /**
+ * Proves that a negative base-256 length is refused. The sign lives in the bit below
+ * the marker, so ignoring it would turn a negative value into an enormous positive one.
+ */
+ @Test
+ void testNegativeBase256SizeFieldIsRejected() {
+ final TarStream stream = new TarStream(new ByteArrayInputStream(
+ TarArchives.base256Header("negative.bin", -1, TYPE_REGULAR_FILE)));
+
+ final IOException thrown = Assertions.assertThrows(IOException.class, stream::next);
+ Assertions.assertEquals("tar size field is negative", thrown.getMessage());
+ }
}