You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mixermt (via GitHub)" <gi...@apache.org> on 2026/07/08 07:01:12 UTC
[I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
mixermt opened a new issue, #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859
### Describe the bug
Comet's native Parquet scan appears to ignore nested schema pruning for complex columns. In a production Spark 3.5.6 comparison job, plain Spark and Comet produced the same Spark physical-plan `ReadSchema`, same input row count, same selected files, and same downstream shuffle output, but Comet read/decompressed far more data during the scan.
Spark 3.5.6, Comet 0.17.0
The query reads Parquet from HDFS and writes Parquet back to HDFS. Column names below are anonymized. The relevant scan reads four top-level fields:
```text
is_flagged
source_type
events
event_hour
```
The Spark UI shows the same pruned nested schema under `events` for both plans:
```text
ReadSchema: struct<
is_flagged:boolean,
source_type:string,
events:array<struct<
is_available:boolean,
event_time_ms:bigint,
is_active:boolean,
items:array<struct<
group_id:bigint,
entity_id:bigint,
has_amount:boolean,
item_type:string,
metric_value:double,
actor_id:bigint,
is_skipped:boolean
>>
>>,
event_hour:timestamp
>
```
The physical input schema is much wider than the requested schema. A representative anonymized shape is:
```text
InputSchema: struct<
is_flagged:boolean,
source_type:string,
event_hour:timestamp,
dimension_id:bigint,
region_code:string,
client_type:string,
ingestion_time:timestamp,
events:array<struct<
is_available:boolean,
event_time_ms:bigint,
is_active:boolean,
event_token:string,
source_name:string,
secondary_id:bigint,
debug_flags:array<string>,
latency_parts:struct<
queue_time_ms:bigint,
fetch_time_ms:bigint,
render_time_ms:bigint,
retry_count:int
>,
items:array<struct<
group_id:bigint,
entity_id:bigint,
has_amount:boolean,
item_type:string,
metric_value:double,
actor_id:bigint,
is_skipped:boolean,
auxiliary_id:bigint,
class_code:string,
mode:string,
raw_score:double,
normalized_score:double,
reason_codes:array<string>,
audit_blob:binary,
feature_map:map<string,double>,
diagnostics:struct<
module_id:string,
module_version:string,
trace_id:string,
extra_payload:binary
>
>>
>>,
raw_payload:binary,
metadata_json:string
>
```
The expected behavior is to read only the requested nested leaves, not all child fields under `events` / `items`.
However, the observed scan metrics were very different:
| Metric | Plain Spark scan | Comet native scan |
| --- | ---: | ---: |
| Spark version | 3.5.6 | 3.5.6 |
| SQL duration | 120,642 ms | 1,688,780 ms |
| Scan tasks | 20,012 | 20,012 |
| Files selected | 10,300 | 10,300 |
| Input records | 310,897,758 | 310,897,758 |
| Stage input bytes | 30,945,027,031 | 1,351,791,550,588 |
| Shuffle records written | 157,270,190 | 157,270,190 |
| Shuffle bytes written | 3,216,674,929 | 3,216,674,929 |
| Comet `bytes_scanned` metric | n/a | 1259.0 GiB |
`size of files read = 2.2 TiB` appeared in both UIs, but that is the selected file length metric, not physical bytes fetched. The physical read evidence is Spark stage `inputBytes` and Comet's native `bytes_scanned` metric.
This strongly suggests the logical Spark-side pruning is present, but Comet's native reader still reads a much wider nested Parquet schema for the projected complex column.
### Steps to reproduce
1. Use Spark 3.5.x with Comet native Parquet scans enabled.
2. Read a Parquet dataset with a wide nested column, for example an `array<struct<...>>` with many child fields.
3. Run a query that only needs a subset of nested fields from that complex column.
4. Compare against plain Spark using the same query and selected files.
5. In Spark UI, verify both plans show the same pruned `ReadSchema`.
6. Compare stage input bytes and Comet native scan `Number of bytes scanned`.
Minimal shape of the reproducer:
```scala
val df = spark.read.schema(prunedSchema).parquet(path)
df
.filter(!$"is_flagged")
.selectExpr(
"event_hour",
"posexplode_outer(events) as (event_idx, event)"
)
.filter("event.is_available and (event.is_active is null or event.is_active)")
.selectExpr(
"event_hour",
"posexplode_outer(event.items) as (item_idx, item)",
"event.event_time_ms"
)
.filter("item.entity_id is not null")
.groupBy(
$"event_hour",
$"item.actor_id",
$"item.group_id",
$"item.entity_id"
)
.count()
.write.mode("overwrite").parquet(outputPath)
```
The important part is that the query projects only selected child fields from a nested Parquet column while the original physical Parquet files contain many more children under that same top-level column.
### Expected behavior
Comet native Parquet scan should honor Spark's pruned nested `ReadSchema` and avoid reading/decompressing unrequested nested child fields, matching Spark's Parquet scan behavior as closely as possible.
For this workload, Comet should not read ~1.35 TB of stage input when plain Spark reads ~30.9 GB for the same files, same rows, and same displayed `ReadSchema`.
### Additional context
Local code inspection points to the native V1 Parquet scan path using full `data_schema` plus a top-level projection vector, instead of giving DataFusion a nested-pruned Parquet read schema.
Relevant code path:
- Spark writes the pruned `requiredSchema` into Parquet read config:
- `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala`
- `ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA`
- Spark clips nested Parquet schemas in:
- `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala`
- `clipParquetSchema`, `clipParquetType`, `clipParquetListType`, `clipParquetMapType`
- Comet serializes both the pruned schema and the full data schema:
- `spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala`
- `requiredSchema = schema2Proto(scan.requiredSchema.fields)`
- `dataSchema = schema2Proto(scan.relation.dataSchema.fields)`
- `projectionVector` is built with `scan.relation.dataSchema.fieldIndex(field.name)`, so it is top-level only.
- Native planner passes both schemas into the native scan:
- `native/core/src/execution/planner.rs`
- `init_datasource_exec(required_schema, Some(data_schema), ..., Some(projection_vector), ...)`
- Native Parquet setup chooses the full `data_schema` as the base schema when present:
- `native/core/src/parquet/parquet_exec.rs`
- `(Some(schema), Some(proj)) => (Arc::clone(schema), Some(proj.clone()))`
- then applies `with_projection_indices(Some(projection))`
That can prune top-level columns, but it does not appear to prune nested leaves inside a projected complex column such as `events`.
Possible fixes / workarounds:
1. Workaround: set `spark.comet.scan.enabled=false` for this workload. That keeps Spark's Parquet scan path and avoids the native scan behavior.
2. Optional workaround to test: combine `spark.comet.scan.enabled=false` with `spark.comet.convert.parquet.enabled=true`, so Spark performs the Parquet read and Comet can still convert the resulting batches after the scan.
3. Code fix direction: for native V1 Parquet scan, initialize DataFusion's Parquet reader with the nested-pruned `required_schema` when Spark has already produced the read schema, instead of using full `data_schema` with only a top-level projection vector.
4. Add a regression test with `array<struct<...many fields...>>`, select only a subset of nested fields, verify `CometNativeScanExec` is used, and assert the native scan does not read unrequested nested leaves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5551401772
Thanks @andygrove for adding benchmarks π
Tell me if you need anything else from my side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5225737110
I would be happy if someone will verify build logs to know if I build correctly
```
# git branch
* (HEAD detached at origin/df55)
# make release-nogit PROFILES="-Pspark-3.5 -Pscala-2.13"
cd native && RUSTFLAGS="-Ctarget-cpu=x86-64-v3" cargo build --release
Compiling proc-macro2 v1.0.107
Compiling unicode-ident v1.0.24
Compiling quote v1.0.47
Compiling libc v0.2.189
Compiling cfg-if v1.0.4
Compiling bytes v1.12.1
Compiling memchr v2.8.3
Compiling serde_core v1.0.229
Compiling autocfg v1.5.1
Compiling libm v0.2.16
Compiling itoa v1.0.18
Compiling version_check v0.9.5
Compiling pin-project-lite v0.2.17
Compiling typenum v1.20.1
Compiling futures-core v0.3.33
Compiling serde v1.0.229
Compiling equivalent v1.0.2
Compiling foldhash v0.2.0
Compiling allocator-api2 v0.2.21
Compiling futures-sink v0.3.33
Compiling zeroize v1.9.0
Compiling shlex v2.0.1
Compiling find-msvc-tools v0.1.9
Compiling slab v0.4.12
Compiling subtle v2.6.1
Compiling futures-io v0.3.33
Compiling futures-task v0.3.33
Compiling once_cell v1.21.4
Compiling zerocopy v0.8.56
Compiling smallvec v1.15.2
Compiling zmij v1.0.23
Compiling serde_json v1.0.151
Compiling cpufeatures v0.3.0
Compiling pkg-config v0.3.33
Compiling const-oid v0.9.6
Compiling percent-encoding v2.3.2
Compiling ryu v1.0.23
Compiling rand_core v0.10.1
Compiling getrandom v0.4.3
Compiling stable_deref_trait v1.2.1
Compiling fnv v1.0.7
Compiling bitflags v2.13.1
Compiling hex v0.4.3
Compiling writeable v0.6.3
Compiling litemap v0.8.2
Compiling icu_normalizer_data v2.2.0
Compiling utf8_iter v1.0.4
Compiling icu_properties_data v2.2.0
Compiling base64 v0.22.1
Compiling either v1.17.0
Compiling semver v1.0.28
Compiling dunce v1.0.5
Compiling fs_extra v1.3.0
Compiling getrandom v0.3.4
Compiling cmov v0.5.4
Compiling iana-time-zone v0.1.65
Compiling atomic-waker v1.1.2
Compiling aws-lc-rs v1.17.3
Compiling const-oid v0.10.2
Compiling untrusted v0.9.0
Compiling httparse v1.10.1
Compiling rustls v0.23.43
Compiling siphasher v1.0.3
Compiling tower-service v0.3.3
Compiling try-lock v0.2.5
Compiling chrono-tz v0.10.4
Compiling openssl-probe v0.2.1
Compiling tower-layer v0.3.3
Compiling ipnet v2.12.1
Compiling lexical-util v1.0.7
Compiling zstd-safe v7.2.4
Compiling base64 v0.23.1
Compiling unicode-segmentation v1.13.3
Compiling unicode-width v0.2.2
Compiling regex-syntax v0.8.11
Compiling parking_lot_core v0.9.12
Compiling simdutf8 v0.1.5
Compiling thiserror v2.0.19
Compiling scopeguard v1.2.0
Compiling crc32fast v1.5.0
Compiling adler2 v2.0.1
Compiling simd-adler32 v0.3.10
Compiling fastrand v2.5.0
Compiling zlib-rs v0.6.7
Compiling alloc-no-stdlib v2.0.4
Compiling anyhow v1.0.104
Compiling snap v1.1.2
Compiling crossbeam-utils v0.8.22
Compiling humantime v2.4.0
Compiling same-file v1.0.6
Compiling paste v1.0.15
Compiling hashbrown v0.14.5
Compiling seq-macro v0.3.6
Compiling cpufeatures v0.2.17
Compiling rustix v1.1.4
Compiling linux-raw-sys v0.12.1
Compiling base64ct v1.8.3
Compiling spin v0.9.9
Compiling datafusion-doc v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling foldhash v0.1.5
Compiling fixedbitset v0.5.7
Compiling num-bigint-dig v0.8.6
Compiling arrayvec v0.7.8
Compiling arrayref v0.3.9
Compiling constant_time_eq v0.4.2
Compiling jiff-core v0.1.0
Compiling libbz2-rs-sys v0.2.5
Compiling vsimd v0.8.0
Compiling outref v0.5.2
Compiling powerfmt v0.2.0
Compiling compression-core v0.4.32
Compiling time-core v0.1.9
Compiling deranged v0.5.8
Compiling num-conv v0.2.2
Compiling pin-utils v0.1.0
Compiling crunchy v0.2.4
Compiling glob v0.3.4
Compiling tiny-keccak v2.0.2
Compiling strsim v0.11.1
Compiling ident_case v1.0.1
Compiling rustversion v1.0.23
Compiling parking v2.2.1
Compiling prettyplease v0.2.37
Compiling heck v0.5.0
Compiling regex-lite v0.1.9
Compiling typeid v1.0.3
Compiling async-task v4.7.1
Compiling hashbrown v0.17.1
Compiling spin v0.10.1
Compiling crossbeam-epoch v0.9.20
Compiling opaque-debug v0.3.1
Compiling erased-serde v0.4.10
Compiling portable-atomic v1.14.0
Compiling log v0.4.33
Compiling thiserror v1.0.69
Compiling jni-sys v0.3.1
Compiling multimap v0.10.1
Compiling fastnum v0.7.5
Compiling link-section v0.19.2
Compiling xmlparser v0.13.6
Compiling typetag v0.2.23
Compiling linktime-proc-macro v0.2.2
Compiling byteorder v1.5.0
Compiling unicode-xid v0.2.6
Compiling destructure_traitobject v0.2.0
Compiling urlencoding v2.1.3
Compiling bytemuck v1.25.2
Compiling dissimilar v1.0.11
Compiling inventory v0.3.24
Compiling tagptr v0.2.0
Compiling array-init v2.1.0
Compiling bimap v0.6.3
Compiling unsafe-libyaml v0.2.11
Compiling cesu8 v1.1.0
Compiling as-any v0.3.2
Compiling procfs v0.18.0
Compiling murmur3 v0.5.2
Compiling log-mdc v0.1.0
Compiling mock_instant v0.6.1
Compiling datafusion-comet v1.1.0 (/home/michael.taranov/datafusion-comet/native/core)
Compiling futures-channel v0.3.33
Compiling sync_wrapper v1.0.2
Compiling futures-lite v2.6.1
Compiling generic-array v0.14.7
Compiling ahash v0.8.12
Compiling num-traits v0.2.19
Compiling bigdecimal v0.4.10
Compiling rustls-pki-types v1.15.1
Compiling http v1.5.0
Compiling aho-corasick v1.1.5
Compiling csv-core v0.1.13
Compiling event-listener v5.4.2
Compiling combine v4.6.7
Compiling simd_cesu8 v1.2.0
Compiling hashbrown v0.15.5
Compiling unsafe-any-ors v1.0.0
Compiling mea v0.6.5
Compiling walkdir v2.5.0
Compiling roaring v0.11.4
Compiling hybrid-array v0.4.14
Compiling rustc_version v0.4.1
Compiling indexmap v2.14.0
Compiling form_urlencoded v1.2.2
Compiling miniz_oxide v0.8.9
Compiling http v0.2.12
Compiling itertools v0.14.0
Compiling java-locator v0.1.9
Compiling tracing-core v0.1.36
Compiling lexical-write-integer v1.0.6
Compiling lexical-parse-integer v1.0.6
Compiling lock_api v0.4.14
Compiling itertools v0.15.0
Compiling lazy_static v1.5.0
Compiling bytes-utils v0.1.4
Compiling expect-test v1.5.1
Compiling itertools v0.13.0
Compiling ctutils v0.4.2
Compiling phf_shared v0.12.1
Compiling base64-simd v0.8.0
Compiling alloc-stdlib v0.2.4
Compiling bzip2 v0.6.1
Compiling want v0.3.1
Compiling pem-rfc7468 v0.7.0
Compiling regex-automata v0.4.18
Compiling chacha20 v0.10.1
Compiling piper v0.2.5
Compiling pem v3.0.6
Compiling comfy-table v7.2.2
Compiling ctor v1.0.12
Compiling typemap-ors v1.0.0
Compiling rustls-native-certs v0.8.4
Compiling time v0.3.55
Compiling jni v0.22.4
Compiling flatbuffers v25.12.19
Compiling aws-types v1.5.0
Compiling jni-macros v0.22.4
Compiling crc32c v0.6.8
Compiling lexical-parse-float v1.0.6
Compiling phf v0.12.1
Compiling brotli-decompressor v5.0.3
Compiling lexical-write-float v1.0.6
Compiling der v0.7.10
Compiling http-body v1.1.0
Compiling crypto-common v0.2.2
Compiling block-buffer v0.12.1
Compiling petgraph v0.8.3
Compiling event-listener-strategy v0.5.4
Compiling http-body v0.4.6
Compiling arc-swap v1.9.2
Compiling regex v1.13.1
Compiling http-body-util v0.1.4
Compiling concurrent-queue v2.5.0
Compiling crossbeam-channel v0.5.16
Compiling async-lock v3.4.2
Compiling lexical-core v1.0.6
Compiling syn v2.0.119
Compiling syn v3.0.3
Compiling async-channel v2.5.0
Compiling getrandom v0.2.17
Compiling errno v0.3.14
Compiling mio v1.2.2
Compiling socket2 v0.6.5
Compiling thread-id v5.1.0
Compiling blocking v1.6.2
Compiling signal-hook-registry v1.4.8
Compiling block-buffer v0.10.4
Compiling block-padding v0.3.3
Compiling rand_core v0.6.4
Compiling rand_core v0.9.5
Compiling rand v0.10.2
Compiling inout v0.1.4
Compiling parking_lot v0.12.5
Compiling dashmap v6.2.1
Compiling digest v0.11.3
Compiling crypto-common v0.1.7
Compiling brotli v8.0.4
Compiling digest v0.10.7
Compiling cipher v0.4.4
Compiling universal-hash v0.5.1
Compiling aead v0.5.2
Compiling flate2 v1.1.9
Compiling jobserver v0.1.35
Compiling sha2 v0.11.0
Compiling md-5 v0.11.0
Compiling hmac v0.13.0
Compiling sha1 v0.11.0
Compiling polyval v0.6.2
Compiling spki v0.7.3
Compiling md-5 v0.10.6
Compiling hmac v0.12.1
Compiling sha2 v0.10.9
Compiling signature v2.2.0
Compiling blake2 v0.10.6
Compiling sha1 v0.10.7
Compiling aes v0.8.4
Compiling cc v1.4.0
Compiling salsa20 v0.10.2
Compiling cbc v0.1.2
Compiling const-random-macro v0.1.16
Compiling crc-fast v1.10.0
Compiling ghash v0.5.1
Compiling ctr v0.9.2
Compiling pbkdf2 v0.12.2
Compiling twox-hash v2.1.3
Compiling const-random v0.1.18
Compiling scrypt v0.11.0
Compiling dlv-list v0.5.2
Compiling tempfile v3.27.0
Compiling pkcs5 v0.7.1
Compiling aes-gcm v0.10.3
Compiling ordered-multimap v0.7.3
Compiling num-integer v0.1.46
Compiling num-complex v0.4.6
Compiling atoi v2.0.0
Compiling ordered-float v4.6.0
Compiling ordered-float v2.10.1
Compiling lz4_flex v0.13.1
Compiling lz4_flex v0.14.0
Compiling rust-ini v0.21.3
Compiling uuid v1.24.0
Compiling csv v1.4.0
Compiling jiff v0.2.35
Compiling serde_bytes v0.11.19
Compiling num-iter v0.1.46
Compiling aws-smithy-types v1.6.1
Compiling cmake v0.1.58
Compiling pkcs8 v0.10.2
Compiling ring v0.17.14
Compiling zstd-sys v2.0.16+zstd.1.5.7
Compiling blake3 v1.8.6
Compiling liblzma-sys v0.4.7
Compiling hdfs-sys v0.3.0
Compiling aws-lc-sys v0.43.0
Compiling pkcs1 v0.7.5
Compiling arrow-schema v59.1.0
Compiling serde_derive v1.0.229
Compiling tokio-macros v2.7.2
Compiling displaydoc v0.2.7
Compiling thiserror-impl v2.0.19
Compiling async-trait v0.1.91
Compiling datafusion-macros v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling typetag-impl v0.2.23
Compiling serde_repr v0.1.21
Compiling xattr v1.6.1
Compiling tokio v1.53.1
Compiling liblzma v0.4.7
Compiling synstructure v0.13.2
Compiling darling_core v0.23.0
Compiling darling_core v0.20.11
Compiling futures-macro v0.3.33
Compiling zerocopy-derive v0.8.56
Compiling zerofrom-derive v0.1.7
Compiling yoke-derive v0.8.2
Compiling zerovec-derive v0.11.3
Compiling tracing-attributes v0.1.31
Compiling sqlparser_derive v0.5.0
Compiling pin-project-internal v1.1.13
Compiling aws-smithy-runtime-api-macros v1.1.0
Compiling prost-derive v0.14.4
Compiling jni-sys-macros v0.4.1
Compiling strum_macros v0.27.2
Compiling thiserror-impl v1.0.69
Compiling typed-builder-macro v0.20.1
Compiling derive_more-impl v2.1.1
Compiling jni-sys v0.4.1
Compiling sqlparser v0.62.0
Compiling futures-util v0.3.33
Compiling pin-project v1.1.13
Compiling zerofrom v0.1.8
Compiling tracing v0.1.44
Compiling yoke v0.8.3
Compiling jni v0.21.1
Compiling typed-builder v0.20.1
Compiling derive_more v2.1.1
Compiling zerovec v0.11.6
Compiling zerotrie v0.2.4
Compiling prost v0.14.4
Compiling darling_macro v0.20.11
Compiling darling_macro v0.23.0
Compiling strum v0.27.2
Compiling prost-types v0.14.4
Compiling darling v0.20.11
Compiling derive_builder_core v0.20.2
Compiling num-bigint v0.4.8
Compiling chrono v0.4.45
Compiling serde_urlencoded v0.7.1
Compiling quick-xml v0.39.4
Compiling quick-xml v0.41.0
Compiling serde-big-array v0.5.1
Compiling serde_yaml v0.9.34+deprecated
Compiling serde-value v0.7.0
Compiling darling v0.23.0
Compiling tinystr v0.8.3
Compiling potential_utf v0.1.5
Compiling bnum v0.12.1
Compiling bon-macros v3.9.3
Compiling serde_with_macros v3.21.0
Compiling icu_collections v2.2.0
Compiling icu_locale_core v2.2.0
Compiling derive_builder_macro v0.20.2
Compiling serde_with v3.21.0
Compiling derive_builder v0.20.2
Compiling prost-build v0.14.4
Compiling icu_provider v2.2.0
Compiling procfs-core v0.18.0
Compiling num-rational v0.4.2
Compiling icu_normalizer v2.2.0
Compiling icu_properties v2.2.0
Compiling datafusion-comet-proto v1.1.0 (/home/michael.taranov/datafusion-comet/native/proto)
Compiling futures-executor v0.3.33
Compiling moka v0.12.15
Compiling futures v0.3.33
Compiling num v0.4.3
Compiling hdrs v0.3.2
Compiling tokio-util v0.7.19
Compiling tower v0.5.3
Compiling datafusion-common-runtime v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling aws-smithy-async v1.3.0
Compiling backon v1.6.0
Compiling idna_adapter v1.2.2
Compiling aws-smithy-runtime-api v1.14.0
Compiling idna v1.1.0
Compiling bon v3.9.3
Compiling h2 v0.4.15
Compiling tokio-stream v0.1.19
Compiling url v2.5.8
Compiling ppv-lite86 v0.2.21
Compiling half v2.7.1
Compiling tower-http v0.6.11
Compiling aws-smithy-schema v0.2.0
Compiling aws-credential-types v1.3.0
Compiling aws-smithy-http v0.64.0
Compiling aws-smithy-observability v0.3.0
Compiling rand_chacha v0.9.0
Compiling rand_chacha v0.3.1
Compiling arrow-buffer v59.1.0
Compiling aws-sigv4 v1.5.1
Compiling rand v0.9.5
Compiling rand v0.8.7
Compiling aws-smithy-json v0.63.0
Compiling aws-smithy-xml v0.62.0
Compiling log4rs v1.4.0
Compiling arrow-data v59.1.0
Compiling aws-smithy-query v0.62.0
Compiling rsa v0.9.10
Compiling hyper v1.11.0
Compiling reqsign-core v3.2.1
Compiling opendal-core v0.58.1
Compiling reqsign-file-read-tokio v3.0.4
Compiling reqsign-aws-core v3.0.3
Compiling reqsign-aliyun-oss v3.1.3
Compiling reqsign-azure-storage v3.1.2
Compiling arrow-array v59.1.0
Compiling hyper-util v0.1.20
Compiling reqsign-aws-v4 v3.1.0
Compiling reqsign-google v3.0.4
Compiling arrow-select v59.1.0
Compiling arrow-arith v59.1.0
Compiling arrow-row v59.1.0
Compiling opendal-service-azure-common v0.58.1
Compiling opendal-layer-logging v0.58.1
Compiling opendal-service-hdfs v0.58.1
Compiling opendal-service-gcs v0.58.1
Compiling opendal-service-s3 v0.58.1
Compiling opendal-service-oss v0.58.1
Compiling opendal-layer-concurrent-limit v0.58.1
Compiling opendal-layer-timeout v0.58.1
Compiling opendal-service-fs v0.58.1
Compiling opendal-layer-retry v0.58.1
Compiling zstd v0.13.3
Compiling opendal-service-azdls v0.58.1
Compiling compression-codecs v0.4.38
Compiling apache-avro v0.21.0
Compiling async-compression v0.4.43
Compiling arrow-ord v59.1.0
Compiling arrow-string v59.1.0
Compiling arrow-ipc v59.1.0
Compiling arrow-cast v59.1.0
Compiling arrow-json v59.1.0
Compiling arrow-csv v59.1.0
Compiling arrow v59.1.0
Compiling parquet-variant v59.1.0
Compiling parquet-variant-json v59.1.0
Compiling parquet-variant-compute v59.1.0
Compiling rustls-webpki v0.103.13
Compiling tokio-rustls v0.26.4
Compiling rustls-platform-verifier v0.7.0
Compiling hyper-rustls v0.27.9
Compiling reqwest v0.12.28
Compiling aws-smithy-http-client v1.2.0
Compiling reqwest v0.13.4
Compiling aws-smithy-runtime v1.12.1
Compiling opendal-http-transport-reqwest v0.58.1
Compiling object_store v0.13.2
Compiling opendal v0.58.1
Compiling aws-runtime v1.9.1
Compiling aws-sdk-ssooidc v1.107.0
Compiling aws-sdk-sts v1.110.0
Compiling aws-sdk-sso v1.105.0
Compiling aws-config v1.10.1
Compiling parquet v59.1.0
Compiling object_store_opendal v0.58.0
Compiling datafusion-common v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling iceberg v0.10.0 (https://github.com/mbutrovich/iceberg-rust?rev=07de8fc8d650f7439b251d57a55613f7de693eb8#07de8fc8)
Compiling datafusion-expr-common v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-physical-expr-common v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-functions-aggregate-common v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-functions-window-common v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-expr v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling iceberg-storage-opendal v0.10.0 (https://github.com/mbutrovich/iceberg-rust?rev=07de8fc8d650f7439b251d57a55613f7de693eb8#07de8fc8)
Compiling datafusion-physical-expr v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-execution v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-functions v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-functions-aggregate v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-optimizer v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-functions-window v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-physical-plan v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-physical-expr-adapter v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-functions-nested v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-sql v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-session v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-datasource v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-catalog v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-pruning v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-datasource-arrow v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-datasource-json v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-datasource-csv v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-datasource-parquet v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-physical-optimizer v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-catalog-listing v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-functions-table v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-comet-common v1.1.0 (/home/michael.taranov/datafusion-comet/native/common)
Compiling datafusion-spark v54.1.0 (https://github.com/apache/datafusion.git?rev=70c26a06716eb552f0fc1d2115957a4089504f11#70c26a06)
Compiling datafusion-comet-jni-bridge v1.1.0 (/home/michael.taranov/datafusion-comet/native/jni-bridge)
Compiling datafusion-comet-spark-expr v1.1.0 (/home/michael.taranov/datafusion-comet/native/spark-expr)
Compiling datafusion-comet-shuffle v1.1.0 (/home/michael.taranov/datafusion-comet/native/shuffle)
Finished `release` profile [optimized + debuginfo] target(s) in 8m 21s
./mvnw install -Prelease -DskipTests -Pspark-3.5 -Pscala-2.13 -Dmaven.gitcommitid.skip=true
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.datafusion:comet-common-spark3.5_2.13:jar:1.1.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.datafusion:comet-common-spark${spark.version.short}_${scala.binary.version}:1.1.0-SNAPSHOT, /home/michael.taranov/datafusion-comet/common/pom.xml, line 33, column 15
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.datafusion:comet-parent-spark${spark.version.short}_${scala.binary.version}:1.1.0-SNAPSHOT, /home/michael.taranov/datafusion-comet/pom.xml, line 32, column 15
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.datafusion:comet-spark-spark3.5_2.13:jar:1.1.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.datafusion:comet-spark-spark${spark.version.short}_${scala.binary.version}:1.1.0-SNAPSHOT, /home/michael.taranov/datafusion-comet/spark/pom.xml, line 33, column 15
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.datafusion:comet-spark-integration-spark3.5_2.13:pom:1.1.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.datafusion:comet-spark-integration-spark${spark.version.short}_${scala.binary.version}:1.1.0-SNAPSHOT, /home/michael.taranov/datafusion-comet/spark-integration/pom.xml, line 33, column 17
[WARNING] 'build.plugins.plugin.version' for org.codehaus.mojo:exec-maven-plugin is missing. @ org.apache.datafusion:comet-spark-integration-spark${spark.version.short}_${scala.binary.version}:1.1.0-SNAPSHOT, /home/michael.taranov/datafusion-comet/spark-integration/pom.xml, line 89, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.datafusion:comet-parent-spark3.5_2.13:pom:1.1.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.datafusion:comet-parent-spark${spark.version.short}_${scala.binary.version}:1.1.0-SNAPSHOT, /home/michael.taranov/datafusion-comet/pom.xml, line 32, column 15
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Comet Project Parent POM [pom]
[INFO] comet-common [jar]
[INFO] comet-spark [jar]
[INFO] comet-spark-integration [pom]
[INFO]
[INFO] ----------< org.apache.datafusion:comet-parent-spark3.5_2.13 >----------
[INFO] Building Comet Project Parent POM 1.1.0-SNAPSHOT [1/4]
[INFO] from pom.xml
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- enforcer:3.3.0:enforce (enforce-maven-version) @ comet-parent-spark3.5_2.13 ---
[INFO] Rule 0: org.apache.maven.enforcer.rules.version.RequireMavenVersion passed
[INFO]
[INFO] --- enforcer:3.3.0:enforce (no-duplicate-declared-dependencies) @ comet-parent-spark3.5_2.13 ---
[INFO] Rule 0: org.codehaus.mojo.extraenforcer.dependencies.BanCircularDependencies passed
[INFO] Rule 1: org.apache.maven.enforcer.rules.BanDuplicatePomDependencyVersions passed
[INFO] Adding ignore: module-info
[INFO] Adding ignore: META-INF/versions/*/module-info
[INFO] Adding ignore: org.apache.spark.unused.UnusedStubClass
[INFO] Adding ignorable dependency: org.apache.spark:spark-sql_2.13:null
[INFO] Adding ignore: javax.annotation.meta.TypeQualifier
[INFO] Adding ignore: javax.annotation.Nonnull
[INFO] Adding ignore: javax.annotation.meta.When
[INFO] Adding ignore: javax.annotation.Nonnull$Checker
[INFO] Adding ignore: javax.annotation.meta.TypeQualifierValidator
[INFO] Adding ignore: org.apache.parquet.filter2.predicate.SparkFilterApi
[INFO] Adding ignore: org.apache.spark.sql.ExtendedExplainGenerator
[INFO] Adding ignorable dependency: com.google.code.findbugs:jsr305:null
[INFO] Adding ignore: javax.annotation.meta.TypeQualifier
[INFO] Adding ignore: javax.annotation.Nonnull
[INFO] Adding ignore: javax.annotation.meta.When
[INFO] Adding ignore: javax.annotation.Nonnull$Checker
[INFO] Adding ignore: javax.annotation.meta.TypeQualifierValidator
[INFO] Adding ignore: javax.annotation.Nullable
[INFO] Adding ignore: javax.annotation.meta.TypeQualifierNickname
[INFO] Adding ignorable dependency: com.google.guava:guava:null
[INFO] Adding ignore: com.google.thirdparty.publicsuffix.TrieParser
[INFO] Adding ignore: com.google.thirdparty.publicsuffix.PublicSuffixPatterns
[INFO] Adding ignore: com.google.thirdparty.publicsuffix.PublicSuffixType
[INFO] Adding ignorable dependency: org.apache.datasketches:datasketches-memory:null
[INFO] Adding ignore: org.apache.datasketches.memory.internal.ResourceImpl
[INFO] Rule 2: org.codehaus.mojo.extraenforcer.dependencies.BanDuplicateClasses passed
[INFO]
[INFO] --- jacoco:0.8.11:prepare-agent (default-prepare-agent) @ comet-parent-spark3.5_2.13 ---
[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.11/org.jacoco.agent-0.8.11-runtime.jar=destfile=/home/michael.taranov/datafusion-comet/target/jacoco.exec -ea -Xmx4g -Xss4m -XX:+IgnoreUnrecognizedVMOptions
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
-Djdk.reflect.useDirectMethodHandle=false
[INFO]
[INFO] --- remote-resources:1.7.0:process (process-resource-bundles) @ comet-parent-spark3.5_2.13 ---
[INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4
[INFO] Copying 3 resources from 1 bundle.
[INFO]
[INFO] --- flatten:1.3.0:flatten (flatten) @ comet-parent-spark3.5_2.13 ---
[INFO] Generating flattened POM of project org.apache.datafusion:comet-parent-spark3.5_2.13:pom:1.1.0-SNAPSHOT...
[INFO]
[INFO] --- scalastyle:1.0.0:check (default) @ comet-parent-spark3.5_2.13 ---
[WARNING] sourceDirectory is not specified or does not exist value=/home/michael.taranov/datafusion-comet/src/main/scala
Saving to outputFile=/home/michael.taranov/datafusion-comet/target/scalastyle-output.xml
Processed 0 file(s)
Found 0 errors
Found 0 warnings
Found 0 infos
Finished in 83 ms
[INFO]
[INFO] --- spotless:2.43.0:check (default) @ comet-parent-spark3.5_2.13 ---
[INFO] Index file does not exist. Fallback to an empty index
[INFO]
[INFO] --- jacoco:0.8.11:report (report) @ comet-parent-spark3.5_2.13 ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO]
[INFO] --- source:3.3.0:jar-no-fork (create-source-jar) @ comet-parent-spark3.5_2.13 ---
[INFO]
[INFO] --- source:3.3.0:test-jar-no-fork (create-source-jar) @ comet-parent-spark3.5_2.13 ---
[INFO]
[INFO] --- site:3.7.1:attach-descriptor (attach-descriptor) @ comet-parent-spark3.5_2.13 ---
[INFO] No site descriptor found: nothing to attach.
[INFO]
[INFO] --- failsafe:3.1.0:integration-test (default) @ comet-parent-spark3.5_2.13 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- failsafe:3.1.0:verify (default) @ comet-parent-spark3.5_2.13 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- apache-rat:0.16.1:check (default) @ comet-parent-spark3.5_2.13 ---
[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 1, approved: 500 licenses.
[INFO]
[INFO] --- install:2.5.2:install (default-install) @ comet-parent-spark3.5_2.13 ---
[INFO] Installing /home/michael.taranov/datafusion-comet/pom.xml to /root/.m2/repository/org/apache/datafusion/comet-parent-spark3.5_2.13/1.1.0-SNAPSHOT/comet-parent-spark3.5_2.13-1.1.0-SNAPSHOT.pom
[INFO]
[INFO] ----------< org.apache.datafusion:comet-common-spark3.5_2.13 >----------
[INFO] Building comet-common 1.1.0-SNAPSHOT [2/4]
[INFO] from common/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- enforcer:3.3.0:enforce (enforce-maven-version) @ comet-common-spark3.5_2.13 ---
[INFO]
[INFO] --- enforcer:3.3.0:enforce (no-duplicate-declared-dependencies) @ comet-common-spark3.5_2.13 ---
[INFO] Rule 0: org.codehaus.mojo.extraenforcer.dependencies.BanCircularDependencies passed
[INFO] Rule 1: org.apache.maven.enforcer.rules.BanDuplicatePomDependencyVersions passed
[INFO] Adding ignore: module-info
[INFO] Adding ignore: META-INF/versions/*/module-info
[INFO] Adding ignore: org.apache.spark.unused.UnusedStubClass
[INFO] Adding ignorable dependency: org.apache.spark:spark-sql_2.13:null
[INFO] Adding ignore: javax.annotation.meta.TypeQualifier
[INFO] Adding ignore: javax.annotation.Nonnull
[INFO] Adding ignore: javax.annotation.meta.When
[INFO] Adding ignore: javax.annotation.Nonnull$Checker
[INFO] Adding ignore: javax.annotation.meta.TypeQualifierValidator
[INFO] Adding ignore: org.apache.parquet.filter2.predicate.SparkFilterApi
[INFO] Adding ignore: org.apache.spark.sql.ExtendedExplainGenerator
[INFO] Adding ignorable dependency: com.google.code.findbugs:jsr305:null
[INFO] Adding ignore: javax.annotation.meta.TypeQualifier
[INFO] Adding ignore: javax.annotation.Nonnull
[INFO] Adding ignore: javax.annotation.meta.When
[INFO] Adding ignore: javax.annotation.Nonnull$Checker
[INFO] Adding ignore: javax.annotation.meta.TypeQualifierValidator
[INFO] Adding ignore: javax.annotation.Nullable
[INFO] Adding ignore: javax.annotation.meta.TypeQualifierNickname
[INFO] Adding ignorable dependency: com.google.guava:guava:null
[INFO] Adding ignore: com.google.thirdparty.publicsuffix.TrieParser
[INFO] Adding ignore: com.google.thirdparty.publicsuffix.PublicSuffixPatterns
[INFO] Adding ignore: com.google.thirdparty.publicsuffix.PublicSuffixType
[INFO] Adding ignorable dependency: org.apache.datasketches:datasketches-memory:null
[INFO] Adding ignore: org.apache.datasketches.memory.internal.ResourceImpl
[INFO] Rule 2: org.codehaus.mojo.extraenforcer.dependencies.BanDuplicateClasses passed
[INFO]
[INFO] --- jacoco:0.8.11:prepare-agent (default-prepare-agent) @ comet-common-spark3.5_2.13 ---
[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.11/org.jacoco.agent-0.8.11-runtime.jar=destfile=/home/michael.taranov/datafusion-comet/common/target/jacoco.exec -ea -Xmx4g -Xss4m -XX:+IgnoreUnrecognizedVMOptions
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
-Djdk.reflect.useDirectMethodHandle=false
[INFO]
[INFO] --- scala:4.9.6:add-source (eclipse-add-source) @ comet-common-spark3.5_2.13 ---
[INFO] Add Source directory: /home/michael.taranov/datafusion-comet/common/src/main/scala
[INFO] Add Test Source directory: /home/michael.taranov/datafusion-comet/common/src/test/scala
[INFO]
[INFO] --- remote-resources:1.7.0:process (process-resource-bundles) @ comet-common-spark3.5_2.13 ---
[INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4
[INFO] Copying 3 resources from 1 bundle.
[INFO]
[INFO] --- resources:3.1.0:resources (default-resources) @ comet-common-spark3.5_2.13 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/michael.taranov/datafusion-comet/common/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- flatten:1.3.0:flatten (flatten) @ comet-common-spark3.5_2.13 ---
[INFO] Generating flattened POM of project org.apache.datafusion:comet-common-spark3.5_2.13:jar:1.1.0-SNAPSHOT...
[INFO]
[INFO] --- scala:4.9.6:compile (scala-compile-first) @ comet-common-spark3.5_2.13 ---
[INFO] Compiler bridge file: /root/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.10.8-bin_2.13.16__61.0-1.10.8_20250303T164041.jar
[INFO] compiling 5 Java sources to /home/michael.taranov/datafusion-comet/common/target/classes ...
[INFO] compile in 2.1 s
[INFO]
[INFO] --- scala:4.9.6:add-source (scala-compile-first) @ comet-common-spark3.5_2.13 ---
[INFO]
[INFO] --- compiler:3.11.0:compile (default-compile) @ comet-common-spark3.5_2.13 ---
[INFO] Not compiling main sources
[INFO]
[INFO] --- scalastyle:1.0.0:check (default) @ comet-common-spark3.5_2.13 ---
[WARNING] sourceDirectory is not specified or does not exist value=/home/michael.taranov/datafusion-comet/common/src/main/scala
Saving to outputFile=/home/michael.taranov/datafusion-comet/common/target/scalastyle-output.xml
Processed 0 file(s)
Found 0 errors
Found 0 warnings
Found 0 infos
Finished in 1 ms
[INFO]
[INFO] --- spotless:2.43.0:check (default) @ comet-common-spark3.5_2.13 ---
[INFO] Index file does not exist. Fallback to an empty index
[INFO] Spotless.Java is keeping 5 files clean - 0 needs changes to be clean, 5 were already clean, 0 were skipped because caching determined they were already clean
[INFO]
[INFO] --- resources:3.1.0:testResources (default-testResources) @ comet-common-spark3.5_2.13 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/michael.taranov/datafusion-comet/common/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala:4.9.6:testCompile (scala-test-compile-first) @ comet-common-spark3.5_2.13 ---
[INFO] compile in 0.0 s
[INFO] No sources to compile
[INFO]
[INFO] --- compiler:3.11.0:testCompile (default-testCompile) @ comet-common-spark3.5_2.13 ---
[INFO] Not compiling test sources
[INFO]
[INFO] --- surefire:3.5.4:test (default-test) @ comet-common-spark3.5_2.13 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- jacoco:0.8.11:report (report) @ comet-common-spark3.5_2.13 ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO]
[INFO] --- jar:3.2.0:jar (default-jar) @ comet-common-spark3.5_2.13 ---
[INFO] Building jar: /home/michael.taranov/datafusion-comet/common/target/comet-common-spark3.5_2.13-1.1.0-SNAPSHOT.jar
[INFO]
[INFO] --- source:3.3.0:jar-no-fork (create-source-jar) @ comet-common-spark3.5_2.13 ---
[INFO] Building jar: /home/michael.taranov/datafusion-comet/common/target/comet-common-spark3.5_2.13-1.1.0-SNAPSHOT-sources.jar
[INFO]
[INFO] --- source:3.3.0:test-jar-no-fork (create-source-jar) @ comet-common-spark3.5_2.13 ---
[INFO] Building jar: /home/michael.taranov/datafusion-comet/common/target/comet-common-spark3.5_2.13-1.1.0-SNAPSHOT-test-sources.jar
[INFO]
[INFO] --- site:3.7.1:attach-descriptor (attach-descriptor) @ comet-common-spark3.5_2.13 ---
[INFO] Skipping because packaging 'jar' is not pom.
[INFO]
[INFO] --- failsafe:3.1.0:integration-test (default) @ comet-common-spark3.5_2.13 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- failsafe:3.1.0:verify (default) @ comet-common-spark3.5_2.13 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- apache-rat:0.16.1:check (default) @ comet-common-spark3.5_2.13 ---
[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 7 licenses.
[INFO]
[INFO] --- install:2.5.2:install (default-install) @ comet-common-spark3.5_2.13 ---
[INFO] Installing /home/michael.taranov/datafusion-comet/common/target/comet-common-spark3.5_2.13-1.1.0-SNAPSHOT.jar to /root/.m2/repository/org/apache/datafusion/comet-common-spark3.5_2.13/1.1.0-SNAPSHOT/comet-common-spark3.5_2.13-1.1.0-SNAPSHOT.jar
[INFO] Installing /home/michael.taranov/datafusion-comet/common/.flattened-pom.xml to /root/.m2/repository/org/apache/datafusion/comet-common-spark3.5_2.13/1.1.0-SNAPSHOT/comet-common-spark3.5_2.13-1.1.0-SNAPSHOT.pom
[INFO] Installing /home/michael.taranov/datafusion-comet/common/target/comet-common-spark3.5_2.13-1.1.0-SNAPSHOT-sources.jar to /root/.m2/repository/org/apache/datafusion/comet-common-spark3.5_2.13/1.1.0-SNAPSHOT/comet-common-spark3.5_2.13-1.1.0-SNAPSHOT-sources.jar
[INFO] Installing /home/michael.taranov/datafusion-comet/common/target/comet-common-spark3.5_2.13-1.1.0-SNAPSHOT-test-sources.jar to /root/.m2/repository/org/apache/datafusion/comet-common-spark3.5_2.13/1.1.0-SNAPSHOT/comet-common-spark3.5_2.13-1.1.0-SNAPSHOT-test-sources.jar
[INFO]
[INFO] ----------< org.apache.datafusion:comet-spark-spark3.5_2.13 >-----------
[INFO] Building comet-spark 1.1.0-SNAPSHOT [3/4]
[INFO] from spark/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- enforcer:3.3.0:enforce (enforce-maven-version) @ comet-spark-spark3.5_2.13 ---
[INFO]
[INFO] --- enforcer:3.3.0:enforce (no-duplicate-declared-dependencies) @ comet-spark-spark3.5_2.13 ---
[INFO] Rule 0: org.codehaus.mojo.extraenforcer.dependencies.BanCircularDependencies passed
[INFO] Rule 1: org.apache.maven.enforcer.rules.BanDuplicatePomDependencyVersions passed
[INFO] Adding ignore: module-info
[INFO] Adding ignore: META-INF/versions/*/module-info
[INFO] Adding ignore: org.apache.spark.unused.UnusedStubClass
[INFO] Adding ignorable dependency: org.apache.spark:spark-sql_2.13:null
[INFO] Adding ignore: javax.annotation.meta.TypeQualifier
[INFO] Adding ignore: javax.annotation.Nonnull
[INFO] Adding ignore: javax.annotation.meta.When
[INFO] Adding ignore: javax.annotation.Nonnull$Checker
[INFO] Adding ignore: javax.annotation.meta.TypeQualifierValidator
[INFO] Adding ignore: org.apache.parquet.filter2.predicate.SparkFilterApi
[INFO] Adding ignore: org.apache.spark.sql.ExtendedExplainGenerator
[INFO] Adding ignorable dependency: com.google.code.findbugs:jsr305:null
[INFO] Adding ignore: javax.annotation.meta.TypeQualifier
[INFO] Adding ignore: javax.annotation.Nonnull
[INFO] Adding ignore: javax.annotation.meta.When
[INFO] Adding ignore: javax.annotation.Nonnull$Checker
[INFO] Adding ignore: javax.annotation.meta.TypeQualifierValidator
[INFO] Adding ignore: javax.annotation.Nullable
[INFO] Adding ignore: javax.annotation.meta.TypeQualifierNickname
[INFO] Adding ignorable dependency: com.google.guava:guava:null
[INFO] Adding ignore: com.google.thirdparty.publicsuffix.TrieParser
[INFO] Adding ignore: com.google.thirdparty.publicsuffix.PublicSuffixPatterns
[INFO] Adding ignore: com.google.thirdparty.publicsuffix.PublicSuffixType
[INFO] Adding ignorable dependency: org.apache.datasketches:datasketches-memory:null
[INFO] Adding ignore: org.apache.datasketches.memory.internal.ResourceImpl
[INFO] Rule 2: org.codehaus.mojo.extraenforcer.dependencies.BanDuplicateClasses passed
[INFO]
[INFO] --- git-commit-id:9.0.2:revision (get-the-git-infos) @ comet-spark-spark3.5_2.13 ---
[INFO]
[INFO] --- jacoco:0.8.11:prepare-agent (default-prepare-agent) @ comet-spark-spark3.5_2.13 ---
[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.11/org.jacoco.agent-0.8.11-runtime.jar=destfile=/home/michael.taranov/datafusion-comet/spark/target/jacoco.exec -ea -Xmx4g -Xss4m -XX:+IgnoreUnrecognizedVMOptions
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
-Djdk.reflect.useDirectMethodHandle=false
[INFO]
[INFO] --- scala:4.9.6:add-source (eclipse-add-source) @ comet-spark-spark3.5_2.13 ---
[INFO] Add Source directory: /home/michael.taranov/datafusion-comet/spark/src/main/scala
[INFO] Add Test Source directory: /home/michael.taranov/datafusion-comet/spark/src/test/scala
[INFO]
[INFO] --- protoc-jar:3.11.4:run (default) @ comet-spark-spark3.5_2.13 ---
[INFO] Resolving artifact: com.google.protobuf:protoc:3.25.5, platform: linux-x86_64
protoc-jar: executing: [/tmp/protoc778087774272784973.exe, --version]
libprotoc 25.5
[INFO] Protoc command: /tmp/protoc778087774272784973.exe
[INFO] Input directories:
[INFO] /home/michael.taranov/datafusion-comet/native/proto/src/proto
[INFO] Output targets:
[INFO] java: /home/michael.taranov/datafusion-comet/spark/target/generated-sources (add: main, clean: false, plugin: null, outputOptions: null)
[INFO] /home/michael.taranov/datafusion-comet/spark/target/generated-sources does not exist. Creating...
[INFO] Processing (java): types.proto
protoc-jar: executing: [/tmp/protoc778087774272784973.exe, -I/home/michael.taranov/datafusion-comet/native/proto/src/proto, --java_out=/home/michael.taranov/datafusion-comet/spark/target/generated-sources, /home/michael.taranov/datafusion-comet/native/proto/src/proto/types.proto]
[INFO] Processing (java): metric.proto
protoc-jar: executing: [/tmp/protoc778087774272784973.exe, -I/home/michael.taranov/datafusion-comet/native/proto/src/proto, --java_out=/home/michael.taranov/datafusion-comet/spark/target/generated-sources, /home/michael.taranov/datafusion-comet/native/proto/src/proto/metric.proto]
[INFO] Processing (java): operator.proto
protoc-jar: executing: [/tmp/protoc778087774272784973.exe, -I/home/michael.taranov/datafusion-comet/native/proto/src/proto, --java_out=/home/michael.taranov/datafusion-comet/spark/target/generated-sources, /home/michael.taranov/datafusion-comet/native/proto/src/proto/operator.proto]
[INFO] Processing (java): expr.proto
protoc-jar: executing: [/tmp/protoc778087774272784973.exe, -I/home/michael.taranov/datafusion-comet/native/proto/src/proto, --java_out=/home/michael.taranov/datafusion-comet/spark/target/generated-sources, /home/michael.taranov/datafusion-comet/native/proto/src/proto/expr.proto]
[INFO] Processing (java): config.proto
protoc-jar: executing: [/tmp/protoc778087774272784973.exe, -I/home/michael.taranov/datafusion-comet/native/proto/src/proto, --java_out=/home/michael.taranov/datafusion-comet/spark/target/generated-sources, /home/michael.taranov/datafusion-comet/native/proto/src/proto/config.proto]
[INFO] Processing (java): partitioning.proto
protoc-jar: executing: [/tmp/protoc778087774272784973.exe, -I/home/michael.taranov/datafusion-comet/native/proto/src/proto, --java_out=/home/michael.taranov/datafusion-comet/spark/target/generated-sources, /home/michael.taranov/datafusion-comet/native/proto/src/proto/partitioning.proto]
[INFO] Processing (java): literal.proto
protoc-jar: executing: [/tmp/protoc778087774272784973.exe, -I/home/michael.taranov/datafusion-comet/native/proto/src/proto, --java_out=/home/michael.taranov/datafusion-comet/spark/target/generated-sources, /home/michael.taranov/datafusion-comet/native/proto/src/proto/literal.proto]
[INFO] Adding generated sources (java): /home/michael.taranov/datafusion-comet/spark/target/generated-sources
[INFO]
[INFO] --- build-helper:3.4.0:add-source (add-shim-source) @ comet-spark-spark3.5_2.13 ---
[INFO] Source directory: /home/michael.taranov/datafusion-comet/spark/src/main/spark-3.x added.
[INFO] Source directory: /home/michael.taranov/datafusion-comet/spark/src/main/spark-none added.
[INFO] Source directory: /home/michael.taranov/datafusion-comet/spark/src/main/spark-3.5 added.
[INFO]
[INFO] --- remote-resources:1.7.0:process (process-resource-bundles) @ comet-spark-spark3.5_2.13 ---
[INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4
[INFO] Copying 3 resources from 1 bundle.
[INFO]
[INFO] --- resources:3.1.0:resources (default-resources) @ comet-spark-spark3.5_2.13 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] skip non existing resourceDirectory /home/michael.taranov/datafusion-comet/spark/../native/target/x86_64-apple-darwin/release
[INFO] skip non existing resourceDirectory /home/michael.taranov/datafusion-comet/spark/../native/target/aarch64-apple-darwin/release
[INFO] Copying 1 resource to org/apache/comet/linux/amd64
[INFO] Copying 3 resources
[INFO]
[INFO] --- flatten:1.3.0:flatten (flatten) @ comet-spark-spark3.5_2.13 ---
[INFO] Generating flattened POM of project org.apache.datafusion:comet-spark-spark3.5_2.13:jar:1.1.0-SNAPSHOT...
[INFO]
[INFO] --- scala:4.9.6:compile (scala-compile-first) @ comet-spark-spark3.5_2.13 ---
[INFO] Compiler bridge file: /root/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.10.8-bin_2.13.16__61.0-1.10.8_20250303T164041.jar
[INFO] compiling 159 Scala sources and 45 Java sources to /home/michael.taranov/datafusion-comet/spark/target/classes ...
[WARNING] [Warn] /home/michael.taranov/datafusion-comet/spark/src/main/scala/org/apache/spark/sql/comet/plans/AliasAwareOutputExpression.scala:74: type Stream in package scala is deprecated (since 2.13.0): Use LazyList instead of Stream
[WARNING] [Warn] /home/michael.taranov/datafusion-comet/spark/src/main/scala/org/apache/spark/sql/comet/plans/AliasAwareOutputExpression.scala:85: method toStream in trait IterableOnceOps is deprecated (since 2.13.0): Use .to(LazyList) instead of .toStream
[WARNING] [Warn] /home/michael.taranov/datafusion-comet/spark/src/main/scala/org/apache/spark/sql/comet/plans/AliasAwareOutputExpression.scala:88: type Stream in package scala is deprecated (since 2.13.0): Use LazyList instead of Stream
[WARNING] [Warn] /home/michael.taranov/datafusion-comet/spark/src/main/scala/org/apache/spark/sql/comet/plans/AliasAwareOutputExpression.scala:89: value Stream in package scala is deprecated (since 2.13.0): Use LazyList instead of Stream
[WARNING] four warnings found
[WARNING] Unexpected javac output: /root/.m2/repository/org/apache/arrow/arrow-memory-core/18.3.0/arrow-memory-core-18.3.0.jar(/org/apache/arrow/memory/BaseAllocator.class): warning: Cannot find annotation method 'value()' in type 'KeyFor': class file for org.checkerframework.checker.nullness.qual.KeyFor not found
/root/.m2/repository/org/apache/arrow/arrow-memory-core/18.3.0/arrow-memory-core-18.3.0.jar(/org/apache/arrow/memory/BaseAllocator.class): warning: Cannot find annotation method 'value()' in type 'KeyFor'
/root/.m2/repository/org/apache/arrow/arrow-memory-core/18.3.0/arrow-memory-core-18.3.0.jar(/org/apache/arrow/memory/BaseAllocator.class): warning: Cannot find annotation method 'value()' in type 'KeyFor'
/root/.m2/repository/org/apache/arrow/arrow-memory-core/18.3.0/arrow-memory-core-18.3.0.jar(/org/apache/arrow/memory/BaseAllocator.class): warning: Cannot find annotation method 'value()' in type 'KeyFor'
/home/michael.taranov/datafusion-comet/spark/src/main/java/org/apache/comet/vector/CometVector.java:67: warning: Unsafe is internal proprietary API and may be removed in a future release
java.lang.reflect.Field unsafeField = sun.misc.Unsafe.class.getDeclaredField("theUnsafe");
^
/home/michael.taranov/datafusion-comet/spark/src/main/java/org/apache/comet/vector/CometVector.java:69: warning: Unsafe is internal proprietary API and may be removed in a future release
final sun.misc.Unsafe unsafe = (sun.misc.Unsafe) unsafeField.get(null);
^
/home/michael.taranov/datafusion-comet/spark/src/main/java/org/apache/comet/vector/CometVector.java:69: warning: Unsafe is internal proprietary API and may be removed in a future release
final sun.misc.Unsafe unsafe = (sun.misc.Unsafe) unsafeField.get(null);
^
7 warnings.
[INFO] compile in 41.5 s
[INFO]
[INFO] --- scala:4.9.6:add-source (scala-compile-first) @ comet-spark-spark3.5_2.13 ---
[INFO]
[INFO] --- compiler:3.11.0:compile (default-compile) @ comet-spark-spark3.5_2.13 ---
[INFO] Not compiling main sources
[INFO]
[INFO] --- scalastyle:1.0.0:check (default) @ comet-spark-spark3.5_2.13 ---
Saving to outputFile=/home/michael.taranov/datafusion-comet/spark/target/scalastyle-output.xml
Processed 130 file(s)
Found 0 errors
Found 0 warnings
Found 0 infos
Finished in 5676 ms
[INFO]
[INFO] --- spotless:2.43.0:check (default) @ comet-spark-spark3.5_2.13 ---
[INFO] Index file does not exist. Fallback to an empty index
[INFO] Spotless.Java is keeping 49 files clean - 0 needs changes to be clean, 49 were already clean, 0 were skipped because caching determined they were already clean
[INFO] Spotless.Scala is keeping 388 files clean - 0 needs changes to be clean, 388 were already clean, 0 were skipped because caching determined they were already clean
[INFO]
[INFO] --- build-helper:3.4.0:add-test-source (add-test-source) @ comet-spark-spark3.5_2.13 ---
[INFO] Test Source directory: /home/michael.taranov/datafusion-comet/spark/src/test/spark-3.x added.
[INFO] Test Source directory: /home/michael.taranov/datafusion-comet/spark/src/test/spark-none added.
[INFO] Test Source directory: /home/michael.taranov/datafusion-comet/spark/src/test/spark-3.5 added.
[INFO]
[INFO] --- resources:3.1.0:testResources (default-testResources) @ comet-spark-spark3.5_2.13 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 746 resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala:4.9.6:testCompile (scala-test-compile-first) @ comet-spark-spark3.5_2.13 ---
[INFO] Compiler bridge file: /root/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.10.8-bin_2.13.16__61.0-1.10.8_20250303T164041.jar
[INFO] compiling 148 Scala sources and 12 Java sources to /home/michael.taranov/datafusion-comet/spark/target/test-classes ...
[WARNING] [Warn] /home/michael.taranov/datafusion-comet/spark/src/test/scala/org/apache/spark/sql/CometTPCDSQuerySuite.scala:29: early initializers are deprecated; they will be replaced by trait parameters in 3.0, see the migration guide on avoiding var/val in traits.
[WARNING] [Warn] /home/michael.taranov/datafusion-comet/spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala:162: method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is deprecated (since 2.13.0): implicit conversions from Array to immutable.IndexedSeq are implemented by copying; use `toIndexedSeq` explicitly if you want to copy, or use the more efficient non-copying ArraySeq.unsafeWrapArray
[WARNING] [Warn] /home/michael.taranov/datafusion-comet/spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala:185: method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is deprecated (since 2.13.0): implicit conversions from Array to immutable.IndexedSeq are implemented by copying; use `toIndexedSeq` explicitly if you want to copy, or use the more efficient non-copying ArraySeq.unsafeWrapArray
[WARNING] [Warn] /home/michael.taranov/datafusion-comet/spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala:214: method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is deprecated (since 2.13.0): implicit conversions from Array to immutable.IndexedSeq are implemented by copying; use `toIndexedSeq` explicitly if you want to copy, or use the more efficient non-copying ArraySeq.unsafeWrapArray
[WARNING] [Warn] /home/michael.taranov/datafusion-comet/spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala:235: method copyArrayToImmutableIndexedSeq in class LowPriorityImplicits2 is deprecated (since 2.13.0): implicit conversions from Array to immutable.IndexedSeq are implemented by copying; use `toIndexedSeq` explicitly if you want to copy, or use the more efficient non-copying ArraySeq.unsafeWrapArray
^[[B[WARNING] [Warn] /home/michael.taranov/datafusion-comet/spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala:689: method withRowGroupSize in class Builder is deprecated
[WARNING] 6 warnings found
[WARNING] Unexpected javac output: /root/.m2/repository/org/apache/arrow/arrow-memory-core/18.3.0/arrow-memory-core-18.3.0.jar(/org/apache/arrow/memory/BaseAllocator.class): warning: Cannot find annotation method 'value()' in type 'KeyFor': class file for org.checkerframework.checker.nullness.qual.KeyFor not found
/root/.m2/repository/org/apache/arrow/arrow-memory-core/18.3.0/arrow-memory-core-18.3.0.jar(/org/apache/arrow/memory/BaseAllocator.class): warning: Cannot find annotation method 'value()' in type 'KeyFor'
/root/.m2/repository/org/apache/arrow/arrow-memory-core/18.3.0/arrow-memory-core-18.3.0.jar(/org/apache/arrow/memory/BaseAllocator.class): warning: Cannot find annotation method 'value()' in type 'KeyFor'
/root/.m2/repository/org/apache/arrow/arrow-memory-core/18.3.0/arrow-memory-core-18.3.0.jar(/org/apache/arrow/memory/BaseAllocator.class): warning: Cannot find annotation method 'value()' in type 'KeyFor'
/home/michael.taranov/datafusion-comet/spark/src/test/java/org/apache/iceberg/rest/RESTCatalogAdapter.java:261: warning: [unchecked] unchecked cast
Map<String, String> request = (Map<String, String>) castRequest(Map.class, body);
^
required: Map<String,String>
found: Map
5 warnings.
[INFO] compile in 51.8 s
[INFO]
[INFO] --- compiler:3.11.0:testCompile (default-testCompile) @ comet-spark-spark3.5_2.13 ---
[INFO] Not compiling test sources
[INFO]
[INFO] --- surefire:3.5.4:test (default-test) @ comet-spark-spark3.5_2.13 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- scalatest:2.2.0:test (test) @ comet-spark-spark3.5_2.13 ---
[INFO] ScalaTest report directory: /home/michael.taranov/datafusion-comet/spark/target/surefire-reports
[INFO] Tests are skipped.
[INFO]
[INFO] --- jacoco:0.8.11:report (report) @ comet-spark-spark3.5_2.13 ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO]
[INFO] --- jar:3.2.0:jar (default-jar) @ comet-spark-spark3.5_2.13 ---
[INFO] Building jar: /home/michael.taranov/datafusion-comet/spark/target/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT.jar
[INFO]
[INFO] --- source:3.3.0:jar-no-fork (create-source-jar) @ comet-spark-spark3.5_2.13 ---
[INFO] Building jar: /home/michael.taranov/datafusion-comet/spark/target/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT-sources.jar
[INFO]
[INFO] --- source:3.3.0:test-jar-no-fork (create-source-jar) @ comet-spark-spark3.5_2.13 ---
[INFO] Building jar: /home/michael.taranov/datafusion-comet/spark/target/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT-test-sources.jar
[INFO]
[INFO] --- shade:3.2.4:shade (default) @ comet-spark-spark3.5_2.13 ---
[INFO] Including org.apache.datafusion:comet-common-spark3.5_2.13:jar:1.1.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.arrow:arrow-vector:jar:18.3.0 in the shaded jar.
[INFO] Including org.apache.arrow:arrow-format:jar:18.3.0 in the shaded jar.
[INFO] Including org.apache.arrow:arrow-memory-core:jar:18.3.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.25.5 in the shaded jar.
[INFO] Including com.google.guava:guava:jar:33.2.1-jre in the shaded jar.
[INFO] Including org.apache.arrow:arrow-memory-unsafe:jar:18.3.0 in the shaded jar.
[INFO] Including org.apache.arrow:arrow-c-data:jar:18.3.0 in the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-databind:jar:2.15.2 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-core:jar:2.15.2 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.18.3 from the shaded jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.18.0 from the shaded jar.
[INFO] Excluding com.google.flatbuffers:flatbuffers-java:jar:25.2.10 from the shaded jar.
[INFO] Excluding org.scala-lang.modules:scala-collection-compat_2.13:jar:2.12.0 from the shaded jar.
[INFO] Excluding org.scala-lang:scala-library:jar:2.13.16 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-annotations:jar:2.15.2 from the shaded jar.
[INFO] No artifact matching filter org.apache.parquet:parquet-hadoop:tests
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] arrow-c-data-18.3.0.jar, arrow-format-18.3.0.jar, arrow-memory-core-18.3.0.jar, arrow-memory-unsafe-18.3.0.jar, arrow-vector-18.3.0.jar, comet-common-spark3.5_2.13-1.1.0-SNAPSHOT.jar, comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT.jar, guava-33.2.1-jre.jar, protobuf-java-3.25.5.jar define 1 overlapping resource:
[WARNING] - META-INF/MANIFEST.MF
[WARNING] arrow-c-data-18.3.0.jar, arrow-format-18.3.0.jar, arrow-memory-core-18.3.0.jar, arrow-memory-unsafe-18.3.0.jar, arrow-vector-18.3.0.jar, comet-common-spark3.5_2.13-1.1.0-SNAPSHOT.jar, comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT.jar define 2 overlapping resources:
[WARNING] - META-INF/DEPENDENCIES
[WARNING] - META-INF/NOTICE
[WARNING] arrow-c-data-18.3.0.jar, arrow-format-18.3.0.jar, arrow-memory-core-18.3.0.jar, arrow-memory-unsafe-18.3.0.jar, arrow-vector-18.3.0.jar, comet-common-spark3.5_2.13-1.1.0-SNAPSHOT.jar, comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT.jar, guava-33.2.1-jre.jar define 1 overlapping resource:
[WARNING] - META-INF/LICENSE
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
[WARNING] arrow-c-data-18.3.0-sources.jar, arrow-format-18.3.0-sources.jar, arrow-memory-core-18.3.0-sources.jar, arrow-memory-unsafe-18.3.0-sources.jar, arrow-vector-18.3.0-sources.jar, comet-common-spark3.5_2.13-1.1.0-SNAPSHOT-sources.jar, comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT-sources.jar, guava-33.2.1-jre-sources.jar, protobuf-java-3.25.5-sources.jar define 1 overlapping resource:
[WARNING] - META-INF/MANIFEST.MF
[WARNING] arrow-c-data-18.3.0-sources.jar, arrow-format-18.3.0-sources.jar, arrow-memory-core-18.3.0-sources.jar, arrow-memory-unsafe-18.3.0-sources.jar, arrow-vector-18.3.0-sources.jar, comet-common-spark3.5_2.13-1.1.0-SNAPSHOT-sources.jar, comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT-sources.jar define 2 overlapping resources:
[WARNING] - META-INF/DEPENDENCIES
[WARNING] - META-INF/NOTICE
[WARNING] arrow-c-data-18.3.0-sources.jar, arrow-format-18.3.0-sources.jar, arrow-memory-core-18.3.0-sources.jar, arrow-memory-unsafe-18.3.0-sources.jar, arrow-vector-18.3.0-sources.jar, comet-common-spark3.5_2.13-1.1.0-SNAPSHOT-sources.jar, comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT-sources.jar, guava-33.2.1-jre-sources.jar define 1 overlapping resource:
[WARNING] - META-INF/LICENSE
[WARNING] arrow-c-data-18.3.0-sources.jar, arrow-format-18.3.0-sources.jar, arrow-memory-core-18.3.0-sources.jar, arrow-memory-unsafe-18.3.0-sources.jar, arrow-vector-18.3.0-sources.jar define 1 overlapping resource:
[WARNING] - module-info.java
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing /home/michael.taranov/datafusion-comet/spark/target/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT.jar with /home/michael.taranov/datafusion-comet/spark/target/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT-shaded.jar
[INFO] Replacing original source artifact with shaded source artifact.
[INFO] Replacing /home/michael.taranov/datafusion-comet/spark/target/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT-sources.jar with /home/michael.taranov/datafusion-comet/spark/target/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT-shaded-sources.jar
[WARNING] artifact org.apache.datafusion:comet-spark-spark3.5_2.13:java-source:sources:1.1.0-SNAPSHOT already attached, replace previous instance
[INFO] Dependency-reduced POM written at: /home/michael.taranov/datafusion-comet/spark/dependency-reduced-pom.xml
[INFO]
[INFO] --- site:3.7.1:attach-descriptor (attach-descriptor) @ comet-spark-spark3.5_2.13 ---
[INFO] Skipping because packaging 'jar' is not pom.
[INFO]
[INFO] --- failsafe:3.1.0:integration-test (default) @ comet-spark-spark3.5_2.13 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- failsafe:3.1.0:verify (default) @ comet-spark-spark3.5_2.13 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- apache-rat:0.16.1:check (default) @ comet-spark-spark3.5_2.13 ---
[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 445 licenses.
[INFO]
[INFO] --- install:2.5.2:install (default-install) @ comet-spark-spark3.5_2.13 ---
[INFO] Installing /home/michael.taranov/datafusion-comet/spark/target/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT.jar to /root/.m2/repository/org/apache/datafusion/comet-spark-spark3.5_2.13/1.1.0-SNAPSHOT/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT.jar
[INFO] Installing /home/michael.taranov/datafusion-comet/spark/dependency-reduced-pom.xml to /root/.m2/repository/org/apache/datafusion/comet-spark-spark3.5_2.13/1.1.0-SNAPSHOT/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT.pom
[INFO] Installing /home/michael.taranov/datafusion-comet/spark/target/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT-sources.jar to /root/.m2/repository/org/apache/datafusion/comet-spark-spark3.5_2.13/1.1.0-SNAPSHOT/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT-sources.jar
[INFO] Installing /home/michael.taranov/datafusion-comet/spark/target/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT-test-sources.jar to /root/.m2/repository/org/apache/datafusion/comet-spark-spark3.5_2.13/1.1.0-SNAPSHOT/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT-test-sources.jar
[INFO]
[INFO] ----< org.apache.datafusion:comet-spark-integration-spark3.5_2.13 >-----
[INFO] Building comet-spark-integration 1.1.0-SNAPSHOT [4/4]
[INFO] from spark-integration/pom.xml
[INFO] --------------------------------[ pom ]---------------------------------
[WARNING] Parameter 'requiresOnline' is unknown for plugin 'exec-maven-plugin:3.5.0:exec (check-jar-contents)'
[INFO]
[INFO] --- enforcer:3.3.0:enforce (enforce-maven-version) @ comet-spark-integration-spark3.5_2.13 ---
[INFO]
[INFO] --- enforcer:3.3.0:enforce (no-duplicate-declared-dependencies) @ comet-spark-integration-spark3.5_2.13 ---
[INFO] Rule 0: org.codehaus.mojo.extraenforcer.dependencies.BanCircularDependencies passed
[INFO] Rule 1: org.apache.maven.enforcer.rules.BanDuplicatePomDependencyVersions passed
[INFO] Adding ignore: module-info
[INFO] Adding ignore: META-INF/versions/*/module-info
[INFO] Adding ignore: org.apache.spark.unused.UnusedStubClass
[INFO] Adding ignorable dependency: org.apache.spark:spark-sql_2.13:null
[INFO] Adding ignore: javax.annotation.meta.TypeQualifier
[INFO] Adding ignore: javax.annotation.Nonnull
[INFO] Adding ignore: javax.annotation.meta.When
[INFO] Adding ignore: javax.annotation.Nonnull$Checker
[INFO] Adding ignore: javax.annotation.meta.TypeQualifierValidator
[INFO] Adding ignore: org.apache.parquet.filter2.predicate.SparkFilterApi
[INFO] Adding ignore: org.apache.spark.sql.ExtendedExplainGenerator
[INFO] Adding ignorable dependency: com.google.code.findbugs:jsr305:null
[INFO] Adding ignore: javax.annotation.meta.TypeQualifier
[INFO] Adding ignore: javax.annotation.Nonnull
[INFO] Adding ignore: javax.annotation.meta.When
[INFO] Adding ignore: javax.annotation.Nonnull$Checker
[INFO] Adding ignore: javax.annotation.meta.TypeQualifierValidator
[INFO] Adding ignore: javax.annotation.Nullable
[INFO] Adding ignore: javax.annotation.meta.TypeQualifierNickname
[INFO] Adding ignorable dependency: com.google.guava:guava:null
[INFO] Adding ignore: com.google.thirdparty.publicsuffix.TrieParser
[INFO] Adding ignore: com.google.thirdparty.publicsuffix.PublicSuffixPatterns
[INFO] Adding ignore: com.google.thirdparty.publicsuffix.PublicSuffixType
[INFO] Adding ignorable dependency: org.apache.datasketches:datasketches-memory:null
[INFO] Adding ignore: org.apache.datasketches.memory.internal.ResourceImpl
[INFO] Rule 2: org.codehaus.mojo.extraenforcer.dependencies.BanDuplicateClasses passed
[INFO]
[INFO] --- jacoco:0.8.11:prepare-agent (default-prepare-agent) @ comet-spark-integration-spark3.5_2.13 ---
[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.11/org.jacoco.agent-0.8.11-runtime.jar=destfile=/home/michael.taranov/datafusion-comet/spark-integration/target/jacoco.exec -ea -Xmx4g -Xss4m -XX:+IgnoreUnrecognizedVMOptions
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
-Djdk.reflect.useDirectMethodHandle=false
[INFO]
[INFO] --- remote-resources:1.7.0:process (process-resource-bundles) @ comet-spark-integration-spark3.5_2.13 ---
[INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4
[INFO] Copying 3 resources from 1 bundle.
[INFO]
[INFO] --- flatten:1.3.0:flatten (flatten) @ comet-spark-integration-spark3.5_2.13 ---
[INFO] Generating flattened POM of project org.apache.datafusion:comet-spark-integration-spark3.5_2.13:pom:1.1.0-SNAPSHOT...
[INFO]
[INFO] --- scalastyle:1.0.0:check (default) @ comet-spark-integration-spark3.5_2.13 ---
[WARNING] sourceDirectory is not specified or does not exist value=/home/michael.taranov/datafusion-comet/spark-integration/src/main/scala
Saving to outputFile=/home/michael.taranov/datafusion-comet/spark-integration/target/scalastyle-output.xml
Processed 0 file(s)
Found 0 errors
Found 0 warnings
Found 0 infos
Finished in 0 ms
[INFO]
[INFO] --- spotless:2.43.0:check (default) @ comet-spark-integration-spark3.5_2.13 ---
[INFO] Index file does not exist. Fallback to an empty index
[INFO]
[INFO] --- jacoco:0.8.11:report (report) @ comet-spark-integration-spark3.5_2.13 ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO]
[INFO] --- source:3.3.0:jar-no-fork (create-source-jar) @ comet-spark-integration-spark3.5_2.13 ---
[INFO]
[INFO] --- source:3.3.0:test-jar-no-fork (create-source-jar) @ comet-spark-integration-spark3.5_2.13 ---
[INFO]
[INFO] --- site:3.7.1:attach-descriptor (attach-descriptor) @ comet-spark-integration-spark3.5_2.13 ---
[INFO] No site descriptor found: nothing to attach.
[INFO]
[INFO] --- dependency:3.1.1:build-classpath (put-client-artifacts-in-a-property) @ comet-spark-integration-spark3.5_2.13 ---
[INFO] Dependencies classpath:
/home/michael.taranov/datafusion-comet/spark/target/comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT.jar
[INFO]
[INFO] --- failsafe:3.1.0:integration-test (default) @ comet-spark-integration-spark3.5_2.13 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- exec:3.5.0:exec (check-jar-contents) @ comet-spark-integration-spark3.5_2.13 ---
[INFO] Artifact looks correct: 'comet-spark-spark3.5_2.13-1.1.0-SNAPSHOT.jar'
[INFO]
[INFO] --- failsafe:3.1.0:verify (default) @ comet-spark-integration-spark3.5_2.13 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- apache-rat:0.16.1:check (default) @ comet-spark-integration-spark3.5_2.13 ---
[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 2 licenses.
[INFO]
[INFO] --- install:2.5.2:install (default-install) @ comet-spark-integration-spark3.5_2.13 ---
[INFO] Skipping artifact installation
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Comet Project Parent POM 1.1.0-SNAPSHOT:
[INFO]
[INFO] Comet Project Parent POM ........................... SUCCESS [ 6.730 s]
[INFO] comet-common ....................................... SUCCESS [ 3.609 s]
[INFO] comet-spark ........................................ SUCCESS [02:59 min]
[INFO] comet-spark-integration ............................ SUCCESS [ 1.141 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:11 min
[INFO] Finished at: 2026-08-08T09:28:05Z
[INFO] ------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5547193582
There is now a reproducible microbenchmark for this on `main`, which may be useful for validating #5262 without needing a production dataset.
`CometExplodeBenchmark` (#5381) writes a `nested` view holding `array<struct<platform string, entries array<struct<type, ts, page, source>>>>`, four elements per row, 100K rows. Querying that view directly reproduces this issue in a few seconds. Apple M3 Ultra, JDK 17, `local[1]`, Spark 4.1 / Scala 2.13, release native build, Best ms:
| Query | Spark | Comet on `main` | Comet on `df55` |
| ----- | ----- | --------------- | --------------- |
| `count(k)`, flat column, control | 17 / 21 | 10 (1.6X) | 11 (1.9X) |
| `count(events)`, whole struct, 5 leaves | 60 / 65 | 92 (0.7X) | **59 (1.1X)** |
| `count(events.platform)`, 1 leaf | 22 / 25 | 93 (0.2X) | **13 (1.9X)** |
| `count(events.entries)`, inner array | 61 / 63 | 93 (0.7X) | **57 (1.1X)** |
| `explode(events)` then `count(col.platform)` | 23 / 34 | 95 (0.2X) | **21 (1.6X)** |
The two Spark columns are the two branches' runs of the same query, given here so the Comet columns can be read against a control rather than against each other.
The one-leaf row is the clean statement of the bug. Spark drops to about a third of its whole-struct time when only one leaf is needed; Comet on `main` does not move at all, 92ms to 93ms. Both executed plans print the identical pruned `ReadSchema: struct<events:array<struct<platform:string>>>`, so the pruning reaches the plan and then changes nothing about what gets read. Comet's time is flat at 92ms to 95ms across every subset of the column requested.
On `df55` that row goes to 13ms, which is 1.9X Spark, so this confirms @mixermt's production finding on a dataset anyone can regenerate locally. Worth noting separately: the whole-struct row also improves, 92ms to 59ms, where there is no pruning available at all, so the DataFusion bump helps the nested reader beyond leaf selection.
A caveat for anyone rerunning this: `df55` is currently 114 commits behind `main` and has no `explode.rs`, so it predates `CometExplodeExec` and a generator query there falls back to Spark's `GenerateExec`. Only the scan-shaped queries above compare meaningfully across the two branches.
Cross-referencing #5685: this issue turns out to be the dominant term there. In that issue's `depth 1` case, 92ms of Comet's 113ms is spent before the generator exists, so this is what to fix first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5218014766
> Seems like the `df55` branch is okay to test with if you get a chance, [@mixermt](https://github.com/mixermt). Thanks in advance!
@mbutrovich
Just to verify I just checkout, df55 and build it ? any other config involved or anything else to that I may miss ?
I usually build with `make release-nogit PROFILES="-Pspark-3.5 -Pscala-2.13"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5231641186
@mbutrovich
I've built the comet from df55 without any issues, but unfortunately I have jar collision in our internal env, working on fixing it to test the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5221606809
> > Seems like the `df55` branch is okay to test with if you get a chance, [@mixermt](https://github.com/mixermt). Thanks in advance!
>
> [@mbutrovich](https://github.com/mbutrovich) Just to verify I just checkout, df55 and build it ? any other config involved or anything else to that I may miss ? I usually build with `make release-nogit PROFILES="-Pspark-3.5 -Pscala-2.13"`
That should be all you have to do. Thank you for the help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4950227843
@adriangb @mbutrovich thanks for quick response and relevant PRs.
Please update me if you want me to test anything before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5205021335
@mbutrovich if we can compile comet with custom DF, I would happy to compile comet on our side and perform heavy testing on real data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5205332272
It'd be great to test this in depth to make sure we don't ship a large DataFusion change that benefits no one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5205496628
> @mixermt if we can compile comet with custom DF, I would happy to compile comet on our side and perform heavy testing on real data.
> @adriangb It'd be great to test this in depth to make sure we don't ship a large DataFusion change that benefits no one.
I pushed the Comet-side fix to the `df55` branch, along with the DataFusion bump past apache/datafusion#24090. The DataFusion pin in that branch's `native/Cargo.toml` already points at the fixed commit directly, so no custom patching is needed on your end, a normal `cargo build -p datafusion-comet` from `native/` against `df55` picks it up.
CI is still running on the PR (apache/datafusion-comet#5262). I'd rather wait until it's green before asking anyone to build against it, so I'll comment back here once it passes. At that point pulling `df55` and testing against real data would be a great way to validate the leaf pruning end to end, thanks for offering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4916909423
I've spent a bit exploring this issue with Claude and these are the conclusions we came to. I reviewed them and they make sense to me, I'm not sure what the path forward is though. Keep in mind I am not familiar with Spark or Comet.
This largely confirms @mixermt's analysis and adds some detail on *why* the gap exists and why the obvious workarounds don't apply.
### Why nested pruning works in plain DataFusion but not here
DataFusion *can* prune nested leaves β `SELECT s.foo FROM t` in datafusion-cli only reads the `s.foo` leaf. But that works because the pruning is carried **in the expression**: the projection is `get_field(Column("s"), "foo")`, and `build_projection_read_plan` in `datafusion-datasource-parquet/src/row_filter.rs` translates literal `get_field` chains into a `ProjectionMask::leaves`.
Spark communicates nested pruning completely differently. The `SchemaPruning` optimizer rule *consumes* the `GetStructField` expressions and bakes the result into the scan node as a narrower `requiredSchema` (a `Column("events")` whose *type* is a narrower `array<struct<...>>` than the file's); the remaining downstream field accesses are rebased against the pruned scan output. So at the scan boundary the pruning exists only as a **type**, not as expressions β there is nothing for Comet to translate into `get_field`.
### What DataFusion does with a pruned schema today (DF 54, still true on current main)
When the logical file schema says `events: array<struct<3 fields>>` but the physical file has `array<struct<18 fields>>`:
1. The `DefaultPhysicalExprAdapter` rewrites the projection against the physical file schema, turning the type mismatch into a whole-column `CastExpr(Column("events"), narrow_type)` (`datafusion-physical-expr-adapter/src/schema_rewriter.rs`).
2. `build_projection_read_plan` derives the projection mask from the rewritten expressions. Its only sources of leaf-level masks are literal `get_field` chains (and struct-field filter accesses). A `CastExpr(Column)` falls through to the generic path, which records the root column and expands it to **every physical leaf** under `events`.
3. The full column chunks are fetched and decoded; the cast drops the unwanted subfields in memory. This is exactly why @mixermt measured identical `bytes_scanned` with the pruned schema β the output is fixed, the I/O is not.
The parquet reader itself is not the limitation: parquet-rs `ProjectionMask::leaves` handles arbitrary nesting, and DataFusion already uses it for pushed-down struct filters. The gap is purely that nothing translates a *schema-level* diff (logical file schema narrower than physical) into a leaf mask.
### Why Comet can't just re-synthesize `get_field` expressions
Even setting aside the `array<struct>` problem, translating the pruned schema back into DataFusion expressions is lossy:
- **Placement**: the leaf-mask derivation only looks at expressions inside the `DataSourceExec`'s own projection. Comet builds the scan with an index projection and keeps its projection logic in a separate `ProjectionExec` using its own ordinal-based `GetStructField` expression, which DataFusion's `PushdownChecker` can't recognize (it downcasts specifically to `GetFieldFunc`). Comet would have to push DF-native `get_field` exprs into the scan via `FileSource::try_pushdown_projection`.
- **Ordinal vs. name resolution**: Spark resolves struct fields by ordinal and Parquet structs may contain duplicate or case-colliding field names; `get_field` is name-based. (This appears to be why Comet's `GetStructField` is ordinal-based in the first place.)
- **Struct-level nullability**: the scan must still emit the struct column, so the expression encoding is `named_struct('foo', get_field(s, 'foo'))` β but if `s` is NULL that yields a non-null `{foo: NULL}` instead of NULL, silently changing `s IS NULL` semantics. A clipped-schema read preserves struct nullness via the surviving leaves' definition levels.
- **It stops at structs**: there is no expression form at all for pruning subfields of `array<struct>` elements or map values β which is the shape in this report. (Element access like `events[1].foo` is orthogonal: pruning `list<struct<foo>>` means "only `foo`, for every element"; Parquet shreds nested data into one column chunk per leaf, so leaf selection is the only prunable axis at the schema level.)
### What seems needed upstream in DataFusion
Schema-driven nested pruning in the parquet opener: when a required root column's logical file-schema type is a structural subset of its physical type, compute the leaf mask by walking the two type trees and matching fields by name β the equivalent of Spark's `ParquetReadSupport.clipParquetSchema` β instead of expanding the root to all leaves. After clipping, the decoder emits the narrow type directly and the adapter-inserted cast degrades to a cheap adjustment, while still handling schema evolution (missing subfield β null fill, reordering, promotion). Edge cases to spec: map key/value handling, case sensitivity, and field-id-based matching for Iceberg (possibly making the clipping policy pluggable, like the expr adapter factory already is).
This can be reproduced and tested in pure DataFusion with no Comet involved: write a parquet file with a wide `struct` (or `array<struct>`) column, `CREATE EXTERNAL TABLE` over it declaring a *narrower* nested type, select that column, and observe `bytes_scanned` β the full column chunk is read despite the narrow table schema. That isolates the schema-diff path from the `get_field` path that already works.
An interim alternative would be an API for `ParquetSource` callers to supply the leaf-level projection directly (per-file `ProjectionMask` hook or an authoritative "read schema"), which Comet could drive itself β but the diff-based approach would benefit every embedder that hands DataFusion a pre-pruned schema (delta-rs, Iceberg integrations, etc.).
Happy to help move the DataFusion side forward if there's agreement on the direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4917337325
DataFusion PR is up: https://github.com/apache/datafusion/pull/23391
I also have a draft Comet PR up that uses a DF 54.0-based branch with the fix in it, to run Comet CI against the fix: https://github.com/apache/datafusion-comet/pull/4866
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5181871803
New upstream DataFusion PR https://github.com/apache/datafusion/pull/24090
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5203957738
I just ran the tests from #23391 (that passed with that POC fix) against the latest DataFusion with https://github.com/apache/datafusion/pull/24090 in it and it did not resolve the issue:
```
26/08/06 07:15:36 WARN shuffle/src/schema_align.rs: ShuffleWriter input schema mismatch on col[0] 'events': child produced List(Field { data_type: Struct([Field {
name: "id", data_type: Int64, nullable: true }, Field { name: "payload", data_type: Utf8 }]), nullable: true }), catalyst declared List(Field { data_type:
Struct([Field { name: "id", data_type: Int64 }, Field { name: "payload", data_type: Utf8 }]) }). Inserting a cast; please file the upstream function bug at
https://github.com/apache/datafusion-comet/issues/4515.
802983 was not less than 401491 native scan read the whole nested column despite pruned ReadSchema (bytes_scanned: events.id=802983, events.id+payload=802983)
ScalaTestFailureLocation: org.apache.comet.exec.CometNativeReaderSuite at (CometNativeReaderSuite.scala:1035)
org.scalatest.exceptions.TestFailedException: 802983 was not less than 401491 native scan read the whole nested column despite pruned ReadSchema (bytes_scanned:
events.id=802983, events.id+payload=802983)
at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
26/08/06 07:15:37 WARN shuffle/src/schema_align.rs: ShuffleWriter input schema mismatch on col[0] 'events': child produced List(Field { data_type: Struct([Field {
name: "id", data_type: Int64, nullable: true }, Field { name: "inner", data_type: Struct([Field { name: "a", data_type: Int64, nullable: true }, Field { name:
"blob", data_type: Utf8 }]) }]), nullable: true }), catalyst declared List(Field { data_type: Struct([Field { name: "id", data_type: Int64 }, Field { name:
"inner", data_type: Struct([Field { name: "a", data_type: Int64 }, Field { name: "blob", data_type: Utf8 }]) }]) }). Inserting a cast; please file the upstream
function bug at https://github.com/apache/datafusion-comet/issues/4515.
883241 was not less than 441620 native scan read unrequested nested leaves through array<struct<struct>> (bytes_scanned: events.inner.a=883241,
events.inner.a+blob=883241)
ScalaTestFailureLocation: org.apache.comet.exec.CometNativeReaderSuite at (CometNativeReaderSuite.scala:1075)
org.scalatest.exceptions.TestFailedException: 883241 was not less than 441620 native scan read unrequested nested leaves through array<struct<struct>>
(bytes_scanned: events.inner.a=883241, events.inner.a+blob=883241)
at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
```
Looking into it today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5237171655
Hi @mbutrovich @adriangb
Iβve tested nested reads, and it appears that pruning works effectively.
Additionally, Iβve compared the output data to ensure complete correctness.
Input bytes were even lower than what was in Spark (at least by what was reported in Spark UI and metrics)
Unfortunately execution was still slower with Comet (maybe worth for investigation in other issue)
Report was generated by Codex 5.6 Sol Ultra
### Schema comparison
| Layer | Baseline SQL 13 | Comet SQL 6 | Result |
|---|---:|---:|---|
| Source path | Same 10,500 files, 2.825 TiB | Same | Identical input |
| Source schema | 602 top-level fields, 2,609 primitive leaves | Same | Sampled source footers identical |
| Applied scan schema | 4 top-level fields, 13 leaves | 4 top-level fields, 13 leaves | Identical |
| Pushed filters | Same two filters | Same two filters | Identical |
| Scan output | 371,800,464 rows | 371,800,464 rows | Identical |
| Final writer schema | Seven fields | Seven fields | Identical |
### Overall comparison
Metric | Baseline SQL 13 | Comet SQL 6 | Comet change
-- | -- | -- | --
SQL wall time | 69.724 s | 125.843 s | +80.5%
Summed executor run time | 76.476 task-min | 111.609 task-min | +45.9%
Spark-reported executor CPU | 62.869 CPU-min | 33.940 CPU-min | β46.0%*
Actual pod CPU, Thanos/cAdvisor | β63.49 CPU-min | β107.96 CPU-min | β+70.1%
Average CPU cores during SQL | β54.6 | β51.5 | β5.8%
JVM GC task time | 55.144 s | 61.093 s | +10.8%
GC / executor run | 1.20% | 0.91% | lower share
Input records | 371,800,464 | 371,800,464 | identical
Reported input bytes | 51.636 GiB | 15.612 GiB | β69.8%**
Shuffle bytes | 3.890 GiB | 6.981 GiB | +79.5%
Shuffle records | 199,926,600 | 205,748,931 | +2.91%
Final output | 4.292 MiB / 269,461 rows | 4.591 MiB / 269,461 rows | same rows, +6.97% bytes
Output files | 71 | 134 | +88.7%
Spill / failed tasks / retries | 0 / 0 / 0 | 0 / 0 / 0 | equal
`*` Sparkβs task-thread CPU counter misses much of Cometβs native worker CPU. Historical container counters show that Comet consumed more CPU, not less.
`**` Native Comet and Spark input-byte counters have different semantics. Pod network receive was approximately 127.3 GiB baseline vs 91.8 GiB Comet, only about 28% lower overall.
### Where the time was spent
Metric | Baseline SQL 13 | Comet SQL 6 | Comet change
-- | -- | -- | --
SQL wall time | 69.724 s | 125.843 s | +80.5%
Summed executor run time | 76.476 task-min | 111.609 task-min | +45.9%
Spark-reported executor CPU | 62.869 CPU-min | 33.940 CPU-min | β46.0%*
Actual pod CPU, Thanos/cAdvisor | β63.49 CPU-min | β107.96 CPU-min | β+70.1%
Average CPU cores during SQL | β54.6 | β51.5 | β5.8%
JVM GC task time | 55.144 s | 61.093 s | +10.8%
GC / executor run | 1.20% | 0.91% | lower share
Input records | 371,800,464 | 371,800,464 | identical
Reported input bytes | 51.636 GiB | 15.612 GiB | β69.8%**
Shuffle bytes | 3.890 GiB | 6.981 GiB | +79.5%
Shuffle records | 199,926,600 | 205,748,931 | +2.91%
Final output | 4.292 MiB / 269,461 rows | 4.591 MiB / 269,461 rows | same rows, +6.97% bytes
Output files | 71 | 134 | +88.7%
Spill / failed tasks / retries | 0 / 0 / 0 | 0 / 0 / 0 | equal
The reduce/write stage alone adds about 31.35 seconds, roughly 56% of the total wall-time gap.
The native shuffle writes only 2.9% more records, but 79.5% more bytes:
Baseline: 20.9 shuffle bytes/record
Comet: 36.4 shuffle bytes/record
Difference: +74.4% per record
Comet also reports 14.8 task-minutes of native shuffle decoding/decompression. The final tasks have comparable partition sizes, yet median task duration rises from 2.656 s to 17.505 s. There is no meaningful reducer skew; the slowdown is broad across tasks.
### Plan comparison
```
Baseline:
Parquet scan
β ColumnarToRow
β Spark Filter/Project/Generate Γ2
β HashAggregate
β Spark Exchange
β AQE (71 reducers)
β HashAggregate
β Parquet writer
Comet:
CometNativeScan
β Comet Filter/Project/Explode Γ2
β CometHashAggregate
β CometExchange / CometNativeShuffle
β AQE (134 reducers)
β CometHashAggregate
β CometColumnarToRow
β Spark Parquet writer
```
Driver logs say Comet accelerated 13 of 15 eligible operators (86%). The only substantive fallback is DataWritingCommandExec, because spark.comet.parquet.write.enabled=false
Baseline executors were 6/7 in one DC. Comet executors were spread across multiple DCs. This might contribute to the much longer remote shuffle requests but usually we don't see any
CPU resources were equal: seven executors, 12 cores each, 9-core request/12-core limit, 84 task slots.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4913296236
> Could you, potentially, test it against the latest `main` branch?
Not sure I'll have the capacity to test against the `main` branch, but the fix I prepare with LLMs clearly points out that the issue is present under the `main` branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4913616465
Some observations after LLM iterations
`Generated by LLM, reviewed by human`
I tested the fix direction suggested here (option 3: pass the nested-pruned `required_schema` to DataFusion as the scan's file schema, instead of full `data_schema` + top-level projection vector) and measured native `bytes_scanned` directly. Results with DataFusion 54, a ~2 MB file of 2000 rows with a ~1 KB `payload` leaf next to a small `id` leaf:
| shape | bytes_scanned, pruned schema | bytes_scanned, full schema |
|---|---|---|
| `event: struct<id,payload>` β `struct<id>` | 2,045,604 | 2,045,604 |
| `events: array<struct<id,payload>>` β `array<struct<id>>` | 2,045,618 | 2,045,618 |
| top-level control: `(id, payload)` β `id` | 18,794 | 2,045,597 |
The pruned read schema fixes the scan's **output**: batches now carry exactly Spark's pruned `ReadSchema` (this also removes a latent wrong-results hazard, since downstream expressions bind struct fields by ordinal against the pruned schema, which was only accidentally correct when the pruned fields were a positional prefix of the file struct, as they are in this report). But it does **not** reduce Parquet I/O. Unrequested nested leaves are still fetched and decoded, then dropped in memory by the schema adapter.
Root cause: DataFusion's `ParquetOpener` derives the leaf-level `ProjectionMask` from the projection and filter *expressions* (`build_projection_read_plan` in `datafusion-datasource-parquet/src/row_filter.rs`), and only `get_field(column, ...)` chains produce leaf masks. A pruned-struct-vs-full-struct schema difference is instead handled by the `PhysicalExprAdapter` as a whole-column cast, which the pushdown checker cannot see through, so it falls back to `ProjectionMask::roots` and reads the entire column chunk.
The reader is not the limitation. parquet-rs `ProjectionMask::leaves` handles nested pruning, and DataFusion already uses it for pushed-down struct filters. The gap is purely that nothing translates a clipped logical file schema into a leaf mask. So fully fixing this issue needs one of:
1. Upstream DataFusion: derive `ProjectionMask::leaves` from the logical-vs-physical file schema diff (the equivalent of Spark's `clipParquetSchema`). Benefits all DataFusion users.
2. Comet-side: a custom opener/`FileSource` that computes the leaf mask from `required_schema` against the file's `SchemaDescriptor` by path matching. No upstream dependency, more maintenance.
3. Partial only: DF 54's `FileSource::try_pushdown_projection` accepts expression projections, so plain struct roots could be rewritten as `struct(get_field(col,'x'), ...)`. But there is no expression form for pruning through `array<struct>` or map values, which is exactly the shape in this report, so this cannot fix it alone.
Passing the pruned schema is still worth doing for the schema-correctness fix, just with the expectation set that stage input bytes will not improve until the mask-level work lands.
@andygrove @mbutrovich any guide from you will be appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4917172540
Yep sounds good. Maybe you can put up a PR there so we can start looking at specific implementations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4917100051
Thanks @adriangb, this matches what I found independently, and the detail on why re-synthesizing `get_field` is a dead end (ordinal vs name, struct nullability, no expression form for `array<struct>`/map) lines up with what I saw.
I have a working prototype of the clip you describe. It lives in `build_projection_read_plan`: when a projected column's requested type is a nested type narrower than the file's, it walks the two type trees, matches fields by name, and emits a `ProjectionMask::leaves` plus the pruned schema, instead of expanding the root to all leaves. Validated end to end in Comet (nested reader and fuzz suites pass, and `bytes_scanned` for a single nested leaf drops to a fraction of the full column).
One design fork worth settling before a DataFusion PR. I key the clip off the projection expression's output type (`expr.data_type()`), which is the whole-column cast the adapter already inserts, so it needs no opener signature change. Your framing keys it off the logical-vs-physical file schema diff directly, which is more general (works even without an adapter cast) but needs the logical file schema threaded into the mask derivation. Same algorithm either way. I lean toward whichever you think reads best upstream.
Agree the spec needs map key/value, case sensitivity, and field-id matching for Iceberg, and a pluggable clipping policy (mirroring the expr-adapter factory) is the right shape. Your pure-DataFusion repro (`CREATE EXTERNAL TABLE` with a narrower nested type, watch `bytes_scanned`) is exactly the integration test the PR should carry.
I can share the prototype branch and repro as a starting point. Happy to collaborate on the DataFusion side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4915570951
I seem to remember when we first implemented the (at the time) ParquetExec (now DataSourceExec) scan, we couldnβt push projections into nested columns without scanning the whole parent column. It sounds like we might be able to revisit that. Iβll take a look. Thanks for reporting @mixermt!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5204675142
I think I've got a Comet-side fix that will get this working once DF 55.0 is available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5204530888
It has to do with `CometCastExpr` being used while the upstream DataFusion change only works for `CastExpr`. I'm gonna see if we can apply similar logic as https://github.com/apache/datafusion-comet/pull/4730 to the casts that handle pruning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mixermt (via GitHub)" <gi...@apache.org>.
mixermt commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4912260852
Planing to work on the fix and verify similar for Iceberg table highly-nested input
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4916991727
So in conclusion: I think we *could* specialize struct adaptation to physical files by reading only the fields the logical schema needs instead of reading the whole thing and then dropping them.
cc @kosiew since you've work on struct <-> struct casting and IO optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "sandugood (via GitHub)" <gi...@apache.org>.
sandugood commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4913213491
Could you, potentially, test it against the latest `main` branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #4859:
URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5207508782
Seems like the `df55` branch is okay to test with if you get a chance, @mixermt. Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead closed issue #4859: Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema
URL: https://github.com/apache/datafusion-comet/issues/4859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org