You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2026/07/26 21:31:39 UTC
[I] Audit third-party code bundled in published JARs [parquet-java]
kevinjqliu opened a new issue, #3690:
URL: https://github.com/apache/parquet-java/issues/3690
## Background
We need to identify which published JARs bundle third-party code. ASF release policy requires JARs to place [LICENSE and NOTICE in `META-INF`](https://www.apache.org/legal/release-policy.html#what-are-the-requirements-to-distribute-other-artifacts-in-addition-to-the-source-package) and requires the [LICENSE file to cover every bundled non-Apache component](https://www.apache.org/legal/release-policy.html#the-license-file). Any [required third-party notices](https://www.apache.org/legal/resolved.html#what-are-required-third-party-notices) also need to be included.
This issue documents the findings and provides a script to verify them.
## Findings
Cross-checking the final JAR contents against the Maven Shade configuration found:
- 7 binary JARs with bundled third-party classes
- 13 Javadoc JARs with JDK-generated assets
- 39 JARs with no bundled third-party code
| Artifact | Maven configuration | Final JAR contents |
|---|---|---|
| `parquet-avro-*.jar` | Explicitly shades FastUtil | FastUtil |
| `parquet-column-*.jar` | Inherits FastUtil/OpenHFT Shade rules | FastUtil, OpenHFT |
| `parquet-format-structures-*.jar` | Explicitly includes `libthrift` | Apache Thrift |
| `parquet-hadoop-*.jar` | Shade processes Jackson and FastUtil; filters/minimization remove Jackson | FastUtil only |
| `parquet-jackson-*.jar` | Explicitly includes Jackson artifacts | Jackson |
| `parquet-hadoop-bundle-*.jar` | Includes other `org.apache.parquet:parquet-*` JARs | Carries their already-shaded third-party classes |
| `parquet-cli-*-runtime.jar` | `shadedArtifactAttached=true`, classifier `runtime`, includes all dependencies | Full runtime dependency set |
A few edge cases:
- `parquet-variant`, `parquet-protobuf`, and `parquet-thrift` run Shade, but filtering and minimization leave no third-party classes in the final JARs.
- `parquet-benchmarks.jar` bundles the full dependency tree, but it is a local-only uber-JAR. The published versioned `parquet-benchmarks-*.jar` contains only project classes.
- The 15 source and 15 test JARs contain project output rather than dependency code.
- The 13 Javadoc JARs contain JDK-generated JavaScript and CSS. [ASF policy explicitly permits these assets in Maven Javadoc JARs](https://www.apache.org/legal/resolved.html#from-java-9-onwards-javadoc-can-include-search-functionality-that-includes-javascript-under-other-open-source-licenses-can-apache-projects-include-this-javadoc).
<details>
<summary>Setup and verification commands</summary>
### Setup
Run the setup commands from the repository root with Java 17. Maven output is left visible so build progress and failures are easy to follow.
```bash
java -version
./mvnw clean package --batch-mode -DskipTests
./mvnw --batch-mode process-resources source:jar-no-fork javadoc:jar
```
### Verification
After setup completes, run the verification separately:
```bash
bash <<'EOF'
set -euo pipefail
export LC_ALL=C
tmp_dir=$(mktemp -d)
trap 'rm -rf "$tmp_dir"' EXIT
candidate_jars="$tmp_dir/candidates"
third_party_jars="$tmp_dir/third-party"
javadoc_jars="$tmp_dir/javadocs"
clean_jars="$tmp_dir/clean"
: >"$third_party_jars"
: >"$javadoc_jars"
: >"$clean_jars"
find . -mindepth 3 -maxdepth 3 -type f \
-path './*/target/*.jar' \
! -name 'original-*.jar' \
! -name 'parquet-benchmarks.jar' \
-print0 >"$candidate_jars"
count_non_parquet_classes() {
jar tf "$1" | awk '
/\.class$/ {
path=$0
sub(/^META-INF\/versions\/[0-9]+\//, "", path)
if (path !~ /^org\/apache\/parquet\//) count++
}
END { print count+0 }
'
}
count_external_sources() {
jar tf "$1" | awk '
/\.java$/ && $0 !~ /^org\/apache\/parquet\// { count++ }
END { print count+0 }
'
}
count_extra_test_classes() {
local jar_file=$1
local module_dir=$2
local jar_classes="$tmp_dir/jar-classes"
local project_classes="$tmp_dir/project-classes"
jar tf "$jar_file" | awk '/\.class$/' | sort -u >"$jar_classes"
: >"$project_classes"
if [[ -d "$module_dir/target/test-classes" ]]; then
(
cd "$module_dir/target/test-classes"
find . -type f -name '*.class' -print |
sed 's#^\./##' |
sort -u
) >"$project_classes"
fi
comm -13 "$project_classes" "$jar_classes" | awk 'END { print NR+0 }'
}
printf 'Inspecting existing JARs...\n'
jar_count=0
while IFS= read -r -d '' jar_file; do
jar_count=$((jar_count + 1))
jar_name=$(basename "$jar_file")
module_dir=${jar_file%/target/*}
case "$jar_name" in
*-javadoc.jar)
unzip -p "$jar_file" script.js >"$tmp_dir/script.js"
if grep -Fq 'Oracle and/or its affiliates' "$tmp_dir/script.js"; then
printf '%s\tJDK-generated JavaScript/CSS\n' "$jar_file" >>"$javadoc_jars"
else
printf '%s\n' "$jar_file" >>"$clean_jars"
fi
continue
;;
*-sources.jar)
bundled_count=$(count_external_sources "$jar_file")
;;
*-tests.jar)
bundled_count=$(count_extra_test_classes "$jar_file" "$module_dir")
;;
*)
bundled_count=$(count_non_parquet_classes "$jar_file")
;;
esac
if [[ $bundled_count -gt 0 ]]; then
printf '%s\t%d bundled classes or sources\n' \
"$jar_file" "$bundled_count" >>"$third_party_jars"
else
printf '%s\n' "$jar_file" >>"$clean_jars"
fi
done <"$candidate_jars"
if [[ $jar_count -eq 0 ]]; then
printf 'No JARs found. Run the setup commands first.\n' >&2
exit 1
fi
printf '\n=== JARs containing bundled third-party code ===\n\n'
sort "$third_party_jars"
printf '\n=== Javadoc JARs containing JDK-provided assets ===\n\n'
sort "$javadoc_jars"
printf '\n=== JARs with no bundled third-party code found ===\n\n'
sort "$clean_jars"
printf '\nInspected %d published JARs.\n' "$jar_count"
EOF
```
</details>
The Java 17 build completed successfully, and the legal-file check verified all 59 JARs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Audit third-party code bundled in published JARs [parquet-java]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3690:
URL: https://github.com/apache/parquet-java/issues/3690#issuecomment-5097687226
Thanks for checking! Compatibility looks good. We still need to review each bundled dependency for any required LICENSE or NOTICE updates. [ASF release policy](https://www.apache.org/legal/release-policy.html#licensing-documentation) requires those files to account for the exact contents of each artifact, including non-Apache license terms and relevant upstream notices.
For example, we were pretty meticulous about this in iceberg: https://github.com/apache/iceberg/pull/16196
Good news is, we only need to check the 3rd party dependencies listed above for each jar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Audit third-party code bundled in published JARs [parquet-java]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #3690:
URL: https://github.com/apache/parquet-java/issues/3690#issuecomment-5096138026
Thanks for generating the list. I just checked, and all the included binaries are [ASF compatible đź‘Ť](https://www.apache.org/legal/resolved.html).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Audit third-party code bundled in published JARs [parquet-java]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3690:
URL: https://github.com/apache/parquet-java/issues/3690#issuecomment-5085686385
I checked the final binary JARs on `master`. These are the third-party dependencies whose classes are physically present:
| JAR | Bundled third-party dependencies |
|---|---|
| `parquet-avro-*.jar` | `it.unimi.dsi:fastutil:8.5.18` |
| `parquet-column-*.jar` | `it.unimi.dsi:fastutil:8.5.18`<br>`net.openhft:zero-allocation-hashing:0.27ea1` |
| `parquet-format-structures-*.jar` | `org.apache.thrift:libthrift:0.23.0` |
| `parquet-hadoop-*.jar` | `it.unimi.dsi:fastutil:8.5.18` |
| `parquet-jackson-*.jar` | `com.fasterxml.jackson.core:jackson-annotations:2.22`<br>`com.fasterxml.jackson.core:jackson-core:2.22.1`<br>`com.fasterxml.jackson.core:jackson-databind:2.22.1` |
| `parquet-hadoop-bundle-*.jar` | `it.unimi.dsi:fastutil:8.5.18`<br>`net.openhft:zero-allocation-hashing:0.27ea1`<br>`org.apache.thrift:libthrift:0.23.0`<br>`com.fasterxml.jackson.core:jackson-annotations:2.22`<br>`com.fasterxml.jackson.core:jackson-core:2.22.1`<br>`com.fasterxml.jackson.core:jackson-databind:2.22.1` |
| `parquet-cli-*-runtime.jar` | `com.beust:jcommander:1.82`<br>`com.fasterxml.jackson.core:jackson-annotations:2.22`<br>`com.fasterxml.jackson.core:jackson-core:2.22.1`<br>`com.fasterxml.jackson.core:jackson-databind:2.22.1`<br>`com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.22.1`<br>`com.github.luben:zstd-jni:1.5.7-11`<br>`commons-codec:commons-codec:1.11`<br>`commons-pool:commons-pool:1.6`<br>`io.airlift:aircompressor:2.0.3`<br>`it.unimi.dsi:fastutil:8.5.18`<br>`javax.annotation:javax.annotation-api:1.3.2`<br>`net.openhft:zero-allocation-hashing:0.27ea1`<br>`net.sf.opencsv:opencsv:2.3`<br>`org.apache.avro:avro:1.11.5`<br>`org.apache.commons:commons-compress:1.26.2`<br>`org.apache.commons:commons-lang3:3.20.0`<br>`org.apache.commons:commons-text:1.15.0`<br>`org.apache.thrift:libthrift:0.23.0`<br>`org.locationtech.jts:jts-core:1.20.0`<br>`org.slf4j:slf4j-api:1.7.33`<br>`org.tukaani:xz:1.12`<br>`org.xerial.snappy:snappy-java:1.1.10.8` |
The check removes Parquet’s own classes, matches and removes every expected dependency, and fails if any expected dependency is missing or unexplained classes remain. All seven JARs ended with zero unmatched classes.
<details>
<summary>Verification command</summary>
Run from the repository root after building the JARs:
```bash
bash <<'EOF'
set -euo pipefail
export LC_ALL=C
tmp_dir=$(mktemp -d)
trap 'rm -rf "$tmp_dir"' EXIT
catalog="$tmp_dir/dependencies"
failures=0
verified_jars=0
cat >"$catalog" <<'CATALOG'
fastutil;it.unimi.dsi:fastutil:8.5.18;^(META-INF/versions/[0-9]+/)?(shaded/parquet/)?it/unimi/dsi/fastutil/
openhft;net.openhft:zero-allocation-hashing:0.27ea1;^(META-INF/versions/[0-9]+/)?(shaded/parquet/)?net/openhft/hashing/
thrift;org.apache.thrift:libthrift:0.23.0;^(META-INF/versions/[0-9]+/)?shaded/parquet/org/apache/thrift/
jackson-annotations;com.fasterxml.jackson.core:jackson-annotations:2.22;^(META-INF/versions/[0-9]+/)?(shaded/parquet/)?com/fasterxml/jackson/annotation/
jackson-core;com.fasterxml.jackson.core:jackson-core:2.22.1;^(META-INF/versions/[0-9]+/)?(shaded/parquet/)?com/fasterxml/jackson/core/|^META-INF/versions/9/module-info\.class$
jackson-databind;com.fasterxml.jackson.core:jackson-databind:2.22.1;^(META-INF/versions/[0-9]+/)?(shaded/parquet/)?com/fasterxml/jackson/databind/
jackson-jsr310;com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.22.1;^(META-INF/versions/[0-9]+/)?com/fasterxml/jackson/datatype/jsr310/
javax-annotation;javax.annotation:javax.annotation-api:1.3.2;^(META-INF/versions/[0-9]+/)?javax/annotation/
jts;org.locationtech.jts:jts-core:1.20.0;^(META-INF/versions/[0-9]+/)?org/locationtech/jts/
snappy;org.xerial.snappy:snappy-java:1.1.10.8;^(META-INF/versions/[0-9]+/)?org/xerial/snappy/
aircompressor;io.airlift:aircompressor:2.0.3;^(META-INF/versions/[0-9]+/)?io/airlift/compress/
commons-pool;commons-pool:commons-pool:1.6;^(META-INF/versions/[0-9]+/)?org/apache/commons/pool/
avro;org.apache.avro:avro:1.11.5;^(META-INF/versions/[0-9]+/)?shaded/parquet/org/apache/avro/
commons-compress;org.apache.commons:commons-compress:1.26.2;^(META-INF/versions/[0-9]+/)?org/apache/commons/compress/
xz;org.tukaani:xz:1.12;^(META-INF/versions/[0-9]+/)?org/tukaani/xz/
zstd;com.github.luben:zstd-jni:1.5.7-11;^(META-INF/versions/[0-9]+/)?com/github/luben/zstd/
slf4j;org.slf4j:slf4j-api:1.7.33;^(META-INF/versions/[0-9]+/)?org/slf4j/
opencsv;net.sf.opencsv:opencsv:2.3;^(META-INF/versions/[0-9]+/)?au/com/bytecode/opencsv/
commons-text;org.apache.commons:commons-text:1.15.0;^(META-INF/versions/[0-9]+/)?org/apache/commons/text/
commons-lang3;org.apache.commons:commons-lang3:3.20.0;^(META-INF/versions/[0-9]+/)?org/apache/commons/lang3/
jcommander;com.beust:jcommander:1.82;^(META-INF/versions/[0-9]+/)?com/beust/(jcommander|ah)/
commons-codec;commons-codec:commons-codec:1.11;^(META-INF/versions/[0-9]+/)?org/apache/commons/codec/
CATALOG
check_jar() {
local jar_file=$1
local jar_name
local expected
local all_classes="$tmp_dir/all-classes"
local remaining="$tmp_dir/remaining"
local next="$tmp_dir/next"
local total_classes
local project_classes
local matched_classes
local remaining_classes
local processed_dependencies=0
local expected_dependencies
jar_name=$(basename "$jar_file")
case "$jar_name" in
parquet-avro-*.jar)
expected="fastutil"
;;
parquet-column-*.jar)
expected="fastutil openhft"
;;
parquet-format-structures-*.jar)
expected="thrift"
;;
parquet-hadoop-bundle-*.jar)
expected="fastutil openhft thrift jackson-annotations jackson-core jackson-databind"
;;
parquet-hadoop-*.jar)
expected="fastutil"
;;
parquet-jackson-*.jar)
expected="jackson-annotations jackson-core jackson-databind"
;;
parquet-cli-*-runtime.jar)
expected="fastutil openhft thrift jackson-annotations jackson-core
jackson-databind jackson-jsr310 javax-annotation jts snappy
aircompressor commons-pool avro commons-compress xz zstd slf4j
opencsv commons-text commons-lang3 jcommander commons-codec"
;;
*)
return
;;
esac
expected=$(printf '%s' "$expected" | tr '\n' ' ' | tr -s ' ')
verified_jars=$((verified_jars + 1))
if ! jar tf "$jar_file" |
awk '/\.class$/ { print }' >"$all_classes"; then
printf 'FAIL %s: unreadable JAR\n' "$jar_file" >&2
failures=$((failures + 1))
return
fi
total_classes=$(wc -l <"$all_classes" | tr -d ' ')
grep -Ev \
'^(META-INF/versions/[0-9]+/)?org/apache/parquet/' \
"$all_classes" >"$remaining" || true
remaining_classes=$(wc -l <"$remaining" | tr -d ' ')
project_classes=$((total_classes - remaining_classes))
printf '\n%s\n' "$jar_file"
printf ' %-68s %6d classes\n' \
'Parquet project classes' "$project_classes"
while IFS=';' read -r dependency_id coordinate pattern; do
case " $expected " in
*" $dependency_id "*)
;;
*)
continue
;;
esac
processed_dependencies=$((processed_dependencies + 1))
matched_classes=$(grep -Ec "$pattern" "$remaining" || true)
printf ' %-68s %6d classes\n' "$coordinate" "$matched_classes"
if [[ $matched_classes -eq 0 ]]; then
printf ' FAIL: expected dependency matched no classes\n' >&2
failures=$((failures + 1))
continue
fi
grep -Ev "$pattern" "$remaining" >"$next" || true
mv "$next" "$remaining"
done <"$catalog"
expected_dependencies=$(printf '%s\n' "$expected" | wc -w | tr -d ' ')
if [[ $processed_dependencies -ne $expected_dependencies ]]; then
printf ' FAIL: dependency catalog is incomplete\n' >&2
failures=$((failures + 1))
fi
remaining_classes=$(wc -l <"$remaining" | tr -d ' ')
printf ' %-68s %6d classes\n' \
'Unmatched after filtering' "$remaining_classes"
if [[ $remaining_classes -ne 0 ]]; then
printf ' FAIL: unexplained classes remain:\n' >&2
sed 's/^/ /' "$remaining" >&2
failures=$((failures + 1))
fi
}
while IFS= read -r -d '' jar_file; do
check_jar "$jar_file"
done < <(
find parquet-avro parquet-column parquet-format-structures \
parquet-hadoop parquet-jackson parquet-hadoop-bundle parquet-cli \
-type f -path '*/target/*.jar' \
! -name 'original-*.jar' \
! -name '*-tests.jar' \
! -name '*-sources.jar' \
! -name '*-javadoc.jar' \
-print0
)
if [[ $verified_jars -ne 7 ]]; then
printf '\nFAIL: expected 7 JARs, found %d. Run a clean package build first.\n' \
"$verified_jars" >&2
failures=$((failures + 1))
fi
if [[ $failures -ne 0 ]]; then
printf '\nVerification failed with %d error(s).\n' "$failures" >&2
exit 1
fi
printf '\nVerified %d JARs; every dependency matched and 0 classes remain.\n' \
"$verified_jars"
EOF
```
</details>
<details>
<summary>Output</summary>
```
parquet-avro/target/parquet-avro-1.18.0-SNAPSHOT.jar
Parquet project classes 106 classes
it.unimi.dsi:fastutil:8.5.18 943 classes
Unmatched after filtering 0 classes
parquet-column/target/parquet-column-1.18.0-SNAPSHOT.jar
Parquet project classes 653 classes
it.unimi.dsi:fastutil:8.5.18 1074 classes
net.openhft:zero-allocation-hashing:0.27ea1 60 classes
Unmatched after filtering 0 classes
parquet-format-structures/target/parquet-format-structures-1.18.0-SNAPSHOT.jar
Parquet project classes 431 classes
org.apache.thrift:libthrift:0.23.0 73 classes
Unmatched after filtering 0 classes
parquet-hadoop/target/parquet-hadoop-1.18.0-SNAPSHOT.jar
Parquet project classes 304 classes
it.unimi.dsi:fastutil:8.5.18 371 classes
Unmatched after filtering 0 classes
parquet-jackson/target/parquet-jackson-1.18.0-SNAPSHOT.jar
Parquet project classes 0 classes
com.fasterxml.jackson.core:jackson-annotations:2.22 77 classes
com.fasterxml.jackson.core:jackson-core:2.22.1 221 classes
com.fasterxml.jackson.core:jackson-databind:2.22.1 812 classes
Unmatched after filtering 0 classes
parquet-hadoop-bundle/target/parquet-hadoop-bundle-1.18.0-SNAPSHOT.jar
Parquet project classes 1795 classes
it.unimi.dsi:fastutil:8.5.18 1074 classes
net.openhft:zero-allocation-hashing:0.27ea1 60 classes
org.apache.thrift:libthrift:0.23.0 73 classes
com.fasterxml.jackson.core:jackson-annotations:2.22 77 classes
com.fasterxml.jackson.core:jackson-core:2.22.1 221 classes
com.fasterxml.jackson.core:jackson-databind:2.22.1 812 classes
Unmatched after filtering 0 classes
parquet-cli/target/parquet-cli-1.18.0-SNAPSHOT-runtime.jar
Parquet project classes 2032 classes
it.unimi.dsi:fastutil:8.5.18 14280 classes
net.openhft:zero-allocation-hashing:0.27ea1 120 classes
org.apache.thrift:libthrift:0.23.0 73 classes
com.fasterxml.jackson.core:jackson-annotations:2.22 154 classes
com.fasterxml.jackson.core:jackson-core:2.22.1 441 classes
com.fasterxml.jackson.core:jackson-databind:2.22.1 1624 classes
com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.22.1 64 classes
javax.annotation:javax.annotation-api:1.3.2 15 classes
org.locationtech.jts:jts-core:1.20.0 732 classes
org.xerial.snappy:snappy-java:1.1.10.8 40 classes
io.airlift:aircompressor:2.0.3 124 classes
commons-pool:commons-pool:1.6 55 classes
org.apache.avro:avro:1.11.5 450 classes
org.apache.commons:commons-compress:1.26.2 571 classes
org.tukaani:xz:1.12 129 classes
com.github.luben:zstd-jni:1.5.7-11 37 classes
org.slf4j:slf4j-api:1.7.33 34 classes
net.sf.opencsv:opencsv:2.3 10 classes
org.apache.commons:commons-text:1.15.0 168 classes
org.apache.commons:commons-lang3:3.20.0 421 classes
com.beust:jcommander:1.82 73 classes
commons-codec:commons-codec:1.11 96 classes
Unmatched after filtering 0 classes
Verified 7 JARs; every dependency matched and 0 classes remain.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org