You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/05/13 12:21:19 UTC
[PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
LuciferYang opened a new pull request, #55853:
URL: https://github.com/apache/spark/pull/55853
### What changes were proposed in this pull request?
Extend the bulk read+widen pattern introduced in SPARK-56791 to `DowncastLongUpdater` (parquet INT64 + DECIMAL(p<=9) read into a Spark 32-bit `DecimalType`).
A new `readLongsAsInts` default method on `VectorizedValuesReader` does the per-row fallback. `VectorizedPlainValuesReader` overrides it to fetch source bytes once via `getBuffer(total * 8)` and run a tight in-method conversion loop. `DowncastLongUpdater.readValues` becomes a one-line delegation. The narrowing is Java's primitive long-to-int cast (`(int) buffer.getLong()`), which discards the high 32 bits; this is non-lossy in practice because Parquet's DECIMAL(p<=9) encoding bounds the value range to `[-999_999_999, 999_999_999]`.
### Why are the changes needed?
`DowncastLongUpdater.readValues` allocates a fresh `ByteBuffer` slice inside `getBuffer(8)` for every element on the legacy path, and that allocation dominates the loop. Collapsing N allocations into one is the same win SPARK-56791 delivered for the INT32 -> Long sibling.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
(To be updated after the GHA benchmark and test runs complete.)
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #55853:
URL: https://github.com/apache/spark/pull/55853#discussion_r3247491976
##########
sql/core/benchmarks/ParquetVectorUpdaterBenchmark-jdk21-results.txt:
##########
@@ -7,13 +7,13 @@ AMD EPYC 7763 64-Core Processor
Identity Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
BooleanUpdater 0 0 0 17004.4 0.1 1.0X
-ByteUpdater (INT32 -> Byte) 0 0 0 3758.6 0.3 0.2X
-ShortUpdater (INT32 -> Short) 1 1 0 1678.4 0.6 0.1X
-IntegerUpdater 0 0 0 10276.1 0.1 0.6X
Review Comment:
display precision artifact — not a real zero. The per-row cost for these updaters is below 1 ns, so 1M rows × ~0.1 ns finishes in roughly 0.1 ms per iteration, which rounds down to 0 at the ms column's resolution. The real signal lives in Rate(M/s) and Per Row(ns). It's a pre-existing display quirk of the Benchmark framework ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #55853:
URL: https://github.com/apache/spark/pull/55853#discussion_r3246090645
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java:
##########
@@ -706,7 +706,13 @@ public void decodeSingleDictionaryId(
WritableColumnVector values,
WritableColumnVector dictionaryIds,
Dictionary dictionary) {
- values.putLong(offset, dictionary.decodeToLong(dictionaryIds.getDictId(offset)));
+ // Target is a 32-bit decimal column vector (precision <= 9); narrow the dictionary's
+ // long value to int via the same `(int) longValue` cast used by `readValue` and
+ // `readValues`. The pre-existing `putLong` here was a latent bug that did not surface
+ // until the dictionary-decode path was exercised by the new end-to-end test, because
+ // a 32-bit decimal vector stores values in `intData` and `putLong` would NPE on the
+ // unallocated `longData`.
+ values.putInt(offset, (int) dictionary.decodeToLong(dictionaryIds.getDictId(offset)));
Review Comment:
https://github.com/apache/spark/pull/55890
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #55853:
URL: https://github.com/apache/spark/pull/55853#issuecomment-4467691645
Merged into master and branch-4.3. Thanks @peter-toth @yaooqinn @dongjoon-hyun once again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #55853:
URL: https://github.com/apache/spark/pull/55853#discussion_r3248125118
##########
sql/core/benchmarks/ParquetVectorUpdaterBenchmark-jdk21-results.txt:
##########
@@ -7,13 +7,13 @@ AMD EPYC 7763 64-Core Processor
Identity Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
BooleanUpdater 0 0 0 17004.4 0.1 1.0X
-ByteUpdater (INT32 -> Byte) 0 0 0 3758.6 0.3 0.2X
-ShortUpdater (INT32 -> Short) 1 1 0 1678.4 0.6 0.1X
-IntegerUpdater 0 0 0 10276.1 0.1 0.6X
Review Comment:
Agree — pre-existing %5.0f rounding in Benchmark.scala. Maybe we can uniformly switch it to %5.1f or %5.2f to make non-zero time values visible for lightweight cases in all Spark benchmarks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #55853:
URL: https://github.com/apache/spark/pull/55853#discussion_r3247443666
##########
sql/core/benchmarks/ParquetVectorUpdaterBenchmark-jdk21-results.txt:
##########
@@ -7,13 +7,13 @@ AMD EPYC 7763 64-Core Processor
Identity Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
BooleanUpdater 0 0 0 17004.4 0.1 1.0X
-ByteUpdater (INT32 -> Byte) 0 0 0 3758.6 0.3 0.2X
-ShortUpdater (INT32 -> Short) 1 1 0 1678.4 0.6 0.1X
-IntegerUpdater 0 0 0 10276.1 0.1 0.6X
Review Comment:
Not related to this PR, why the time metrics are all zero?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #55853:
URL: https://github.com/apache/spark/pull/55853#issuecomment-4467658179
I will re-merge this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #55853:
URL: https://github.com/apache/spark/pull/55853#issuecomment-4450388097
cc @yaooqinn @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #55853:
URL: https://github.com/apache/spark/pull/55853#discussion_r3248033788
##########
sql/core/benchmarks/ParquetVectorUpdaterBenchmark-jdk21-results.txt:
##########
@@ -7,13 +7,13 @@ AMD EPYC 7763 64-Core Processor
Identity Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
BooleanUpdater 0 0 0 17004.4 0.1 1.0X
-ByteUpdater (INT32 -> Byte) 0 0 0 3758.6 0.3 0.2X
-ShortUpdater (INT32 -> Short) 1 1 0 1678.4 0.6 0.1X
-IntegerUpdater 0 0 0 10276.1 0.1 0.6X
Review Comment:
In a normal case, these measure a batch time with a scale factor N?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #55853:
URL: https://github.com/apache/spark/pull/55853#discussion_r3246941663
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java:
##########
@@ -706,7 +706,13 @@ public void decodeSingleDictionaryId(
WritableColumnVector values,
WritableColumnVector dictionaryIds,
Dictionary dictionary) {
- values.putLong(offset, dictionary.decodeToLong(dictionaryIds.getDictId(offset)));
+ // Target is a 32-bit decimal column vector (precision <= 9); narrow the dictionary's
+ // long value to int via the same `(int) longValue` cast used by `readValue` and
+ // `readValues`. The pre-existing `putLong` here was a latent bug that did not surface
+ // until the dictionary-decode path was exercised by the new end-to-end test, because
+ // a 32-bit decimal vector stores values in `intData` and `putLong` would NPE on the
+ // unallocated `longData`.
+ values.putInt(offset, (int) dictionary.decodeToLong(dictionaryIds.getDictId(offset)));
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #55853:
URL: https://github.com/apache/spark/pull/55853#issuecomment-4454197360
> @peter-toth I don't have my computer on hand right now. Could you please revert it first if it's convenient? thanks
Let me revert them, @LuciferYang .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #55853:
URL: https://github.com/apache/spark/pull/55853#discussion_r3246794177
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java:
##########
@@ -706,7 +706,13 @@ public void decodeSingleDictionaryId(
WritableColumnVector values,
WritableColumnVector dictionaryIds,
Dictionary dictionary) {
- values.putLong(offset, dictionary.decodeToLong(dictionaryIds.getDictId(offset)));
+ // Target is a 32-bit decimal column vector (precision <= 9); narrow the dictionary's
+ // long value to int via the same `(int) longValue` cast used by `readValue` and
+ // `readValues`. The pre-existing `putLong` here was a latent bug that did not surface
+ // until the dictionary-decode path was exercised by the new end-to-end test, because
+ // a 32-bit decimal vector stores values in `intData` and `putLong` would NPE on the
+ // unallocated `longData`.
+ values.putInt(offset, (int) dictionary.decodeToLong(dictionaryIds.getDictId(offset)));
Review Comment:
I've merged https://github.com/apache/spark/pull/55890, feel free to rebase this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #55853:
URL: https://github.com/apache/spark/pull/55853#issuecomment-4456310274
Thank you @dongjoon-hyun and @peter-toth , I re-open this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #55853:
URL: https://github.com/apache/spark/pull/55853#issuecomment-4454122985
@peter-toth I don't have my computer on hand right now. Could you please revert it first if it's convenient? thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #55853: [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater
URL: https://github.com/apache/spark/pull/55853
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #55853:
URL: https://github.com/apache/spark/pull/55853#issuecomment-4454053159
@LuciferYang , can you please check `master` and `branch-4.x` as this PR seems to cause a failure:
```
2026-05-14T16:42:50.4488049Z [0m[[0m[0minfo[0m] [0m[0m[31m- INT64 DECIMAL -> 32-bit DecimalType narrowing end-to-end via vectorized read path *** FAILED *** (187 milliseconds)[0m[0m
2026-05-14T16:42:50.4527577Z [0m[[0m[0minfo[0m] [0m[0m[31m org.apache.spark.SparkException: [FAILED_READ_FILE.NO_HINT] Encountered error while reading file file:///home/runner/work/spark/spark/target/tmp/spark-1c4d4e28-9f68-4167-9999-49bec18dcf2e/dec.parquet. SQLSTATE: KD001[0m[0m
2026-05-14T16:42:50.4571400Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFilesError(QueryExecutionErrors.scala:929)[0m[0m
2026-05-14T16:42:50.4577351Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.v2.FileDataSourceV2$.attachFilePath(FileDataSourceV2.scala:142)[0m[0m
2026-05-14T16:42:50.4593577Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:150)[0m[0m
2026-05-14T16:42:50.4599756Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:777)[0m[0m
2026-05-14T16:42:50.4602659Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)[0m[0m
2026-05-14T16:42:50.4606083Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)[0m[0m
2026-05-14T16:42:50.4608421Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:44)[0m[0m
2026-05-14T16:42:50.4611183Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:50)[0m[0m
2026-05-14T16:42:50.4613580Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:593)[0m[0m
2026-05-14T16:42:50.4615059Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:593)[0m[0m
2026-05-14T16:42:50.4617312Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1835)[0m[0m
2026-05-14T16:42:50.4618884Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.rdd.RDD.$anonfun$count$1(RDD.scala:1320)[0m[0m
2026-05-14T16:42:50.4620339Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.rdd.RDD.$anonfun$count$1$adapted(RDD.scala:1320)[0m[0m
2026-05-14T16:42:50.4621867Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2536)[0m[0m
2026-05-14T16:42:50.4623566Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)[0m[0m
2026-05-14T16:42:50.4625180Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:206)[0m[0m
2026-05-14T16:42:50.4626516Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.Task.run(Task.scala:147)[0m[0m
2026-05-14T16:42:50.4627969Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:894)[0m[0m
2026-05-14T16:42:50.4629763Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:86)[0m[0m
2026-05-14T16:42:50.4631599Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:83)[0m[0m
2026-05-14T16:42:50.4633420Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:97)[0m[0m
2026-05-14T16:42:50.4634897Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:897)[0m[0m
2026-05-14T16:42:50.4636554Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)[0m[0m
2026-05-14T16:42:50.4638371Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)[0m[0m
2026-05-14T16:42:50.4639812Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.base/java.lang.Thread.run(Thread.java:840)[0m[0m
2026-05-14T16:42:50.4641240Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1063)[0m[0m
2026-05-14T16:42:50.4642768Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkContext.runJob(SparkContext.scala:2496)[0m[0m
2026-05-14T16:42:50.4644157Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkContext.runJob(SparkContext.scala:2517)[0m[0m
2026-05-14T16:42:50.4645497Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkContext.runJob(SparkContext.scala:2536)[0m[0m
2026-05-14T16:42:50.4646921Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkContext.runJob(SparkContext.scala:2561)[0m[0m
2026-05-14T16:42:50.4648136Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.rdd.RDD.count(RDD.scala:1320)[0m[0m
2026-05-14T16:42:50.4649345Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTest$.$anonfun$getErrorMessageInCheckAnswer$1(QueryTest.scala:858)[0m[0m
2026-05-14T16:42:50.4650974Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)[0m[0m
2026-05-14T16:42:50.4656126Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:375)[0m[0m
2026-05-14T16:42:50.4657971Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTest$.getErrorMessageInCheckAnswer(QueryTest.scala:858)[0m[0m
2026-05-14T16:42:50.4659474Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTest$.checkAnswer(QueryTest.scala:835)[0m[0m
2026-05-14T16:42:50.4660948Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTestBase.checkAnswer(QueryTest.scala:177)[0m[0m
2026-05-14T16:42:50.4662690Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTestBase.checkAnswer$(QueryTest.scala:159)[0m[0m
2026-05-14T16:42:50.4664735Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite.checkAnswer(ParquetIOSuite.scala:59)[0m[0m
2026-05-14T16:42:50.4666932Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite.$anonfun$new$406(ParquetIOSuite.scala:2070)[0m[0m
2026-05-14T16:42:50.4668776Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)[0m[0m
2026-05-14T16:42:50.4670234Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Assertions.withClue(Assertions.scala:1065)[0m[0m
2026-05-14T16:42:50.4671593Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Assertions.withClue$(Assertions.scala:1052)[0m[0m
2026-05-14T16:42:50.4673132Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuite.withClue(AnyFunSuite.scala:1564)[0m[0m
2026-05-14T16:42:50.4675087Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetTest.$anonfun$withAllParquetReaders$2(ParquetTest.scala:182)[0m[0m
2026-05-14T16:42:50.4676952Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)[0m[0m
2026-05-14T16:42:50.4684410Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.catalyst.SQLConfHelper.withSQLConf(SQLConfHelper.scala:54)[0m[0m
2026-05-14T16:42:50.4686410Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.catalyst.SQLConfHelper.withSQLConf$(SQLConfHelper.scala:38)[0m[0m
2026-05-14T16:42:50.4689977Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite.org$apache$spark$sql$QueryTestBase$$super$withSQLConf(ParquetIOSuite.scala:59)[0m[0m
2026-05-14T16:42:50.4692388Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTestBase.withSQLConf(QueryTest.scala:300)[0m[0m
2026-05-14T16:42:50.4693995Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTestBase.withSQLConf$(QueryTest.scala:298)[0m[0m
2026-05-14T16:42:50.4695962Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite.withSQLConf(ParquetIOSuite.scala:59)[0m[0m
2026-05-14T16:42:50.4699236Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetTest.withAllParquetReaders(ParquetTest.scala:181)[0m[0m
2026-05-14T16:42:50.4701546Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetTest.withAllParquetReaders$(ParquetTest.scala:172)[0m[0m
2026-05-14T16:42:50.4704492Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite.withAllParquetReaders(ParquetIOSuite.scala:59)[0m[0m
2026-05-14T16:42:50.4707830Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite.$anonfun$new$404(ParquetIOSuite.scala:2057)[0m[0m
2026-05-14T16:42:50.4711212Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite.$anonfun$new$404$adapted(ParquetIOSuite.scala:2038)[0m[0m
2026-05-14T16:42:50.4713626Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTest.$anonfun$withTempDir$1(QueryTest.scala:725)[0m[0m
2026-05-14T16:42:50.4715404Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTest.$anonfun$withTempDir$1$adapted(QueryTest.scala:724)[0m[0m
2026-05-14T16:42:50.4717201Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkTestSuite.withTempDir(SparkTestSuite.scala:193)[0m[0m
2026-05-14T16:42:50.4718731Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkTestSuite.withTempDir$(SparkTestSuite.scala:191)[0m[0m
2026-05-14T16:42:50.4720854Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite.org$apache$spark$sql$QueryTest$$super$withTempDir(ParquetIOSuite.scala:59)[0m[0m
2026-05-14T16:42:50.4723335Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTest.withTempDir(QueryTest.scala:724)[0m[0m
2026-05-14T16:42:50.4724939Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.QueryTest.withTempDir$(QueryTest.scala:723)[0m[0m
2026-05-14T16:42:50.4726780Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite.withTempDir(ParquetIOSuite.scala:59)[0m[0m
2026-05-14T16:42:50.4728935Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite.$anonfun$new$403(ParquetIOSuite.scala:2038)[0m[0m
2026-05-14T16:42:50.4731191Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite.$anonfun$new$403$adapted(ParquetIOSuite.scala:2037)[0m[0m
2026-05-14T16:42:50.4733068Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.immutable.List.foreach(List.scala:323)[0m[0m
2026-05-14T16:42:50.4734877Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite.$anonfun$new$402(ParquetIOSuite.scala:2037)[0m[0m
2026-05-14T16:42:50.4736635Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)[0m[0m
2026-05-14T16:42:50.4738074Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)[0m[0m
2026-05-14T16:42:50.4739536Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)[0m[0m
2026-05-14T16:42:50.4741111Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)[0m[0m
2026-05-14T16:42:50.4742989Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)[0m[0m
2026-05-14T16:42:50.4744486Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:30)[0m[0m
2026-05-14T16:42:50.4746026Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:41)[0m[0m
2026-05-14T16:42:50.4747649Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)[0m[0m
2026-05-14T16:42:50.4748923Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)[0m[0m
2026-05-14T16:42:50.4750234Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)[0m[0m
2026-05-14T16:42:50.4751570Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Transformer.apply(Transformer.scala:22)[0m[0m
2026-05-14T16:42:50.4752998Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Transformer.apply(Transformer.scala:20)[0m[0m
2026-05-14T16:42:50.4754502Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)[0m[0m
2026-05-14T16:42:50.4756163Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkTestSuite.withFixture(SparkTestSuite.scala:175)[0m[0m
2026-05-14T16:42:50.4757842Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkTestSuite.withFixture$(SparkTestSuite.scala:169)[0m[0m
2026-05-14T16:42:50.4759360Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:30)[0m[0m
2026-05-14T16:42:50.4761206Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)[0m[0m
2026-05-14T16:42:50.4763168Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)[0m[0m
2026-05-14T16:42:50.4764719Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)[0m[0m
2026-05-14T16:42:50.4766242Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)[0m[0m
2026-05-14T16:42:50.4767939Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)[0m[0m
2026-05-14T16:42:50.4769824Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:30)[0m[0m
2026-05-14T16:42:50.4771659Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)[0m[0m
2026-05-14T16:42:50.4773896Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)[0m[0m
2026-05-14T16:42:50.4775422Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:30)[0m[0m
2026-05-14T16:42:50.4777056Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)[0m[0m
2026-05-14T16:42:50.4778710Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)[0m[0m
2026-05-14T16:42:50.4780225Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.immutable.List.foreach(List.scala:323)[0m[0m
2026-05-14T16:42:50.4781646Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)[0m[0m
2026-05-14T16:42:50.4783253Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)[0m[0m
2026-05-14T16:42:50.4784694Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)[0m[0m
2026-05-14T16:42:50.4786167Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)[0m[0m
2026-05-14T16:42:50.4787886Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)[0m[0m
2026-05-14T16:42:50.4789562Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)[0m[0m
2026-05-14T16:42:50.4814698Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Suite.run(Suite.scala:1114)[0m[0m
2026-05-14T16:42:50.4816677Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Suite.run$(Suite.scala:1096)[0m[0m
2026-05-14T16:42:50.4823424Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)[0m[0m
2026-05-14T16:42:50.4826698Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)[0m[0m
2026-05-14T16:42:50.4828633Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.runImpl(Engine.scala:535)[0m[0m
2026-05-14T16:42:50.4830297Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)[0m[0m
2026-05-14T16:42:50.4833457Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)[0m[0m
2026-05-14T16:42:50.4837012Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:30)[0m[0m
2026-05-14T16:42:50.4841475Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)[0m[0m
2026-05-14T16:42:50.4844238Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)[0m[0m
2026-05-14T16:42:50.4846902Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)[0m[0m
2026-05-14T16:42:50.4849071Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:30)[0m[0m
2026-05-14T16:42:50.4851364Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)[0m[0m
2026-05-14T16:42:50.4853246Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)[0m[0m
2026-05-14T16:42:50.4854647Z [0m[[0m[0minfo[0m] [0m[0m[31m at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)[0m[0m
2026-05-14T16:42:50.4856145Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)[0m[0m
2026-05-14T16:42:50.4857669Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)[0m[0m
2026-05-14T16:42:50.4859351Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)[0m[0m
2026-05-14T16:42:50.4861170Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.base/java.lang.Thread.run(Thread.java:840)[0m[0m
2026-05-14T16:42:50.4863371Z [0m[[0m[0minfo[0m] [0m[0m[31m Cause: java.lang.NullPointerException:[0m[0m
2026-05-14T16:42:50.4866658Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.vectorized.OnHeapColumnVector.putLong(OnHeapColumnVector.java:393)[0m[0m
2026-05-14T16:42:50.4870036Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory$DowncastLongUpdater.decodeSingleDictionaryId(ParquetVectorUpdaterFactory.java:709)[0m[0m
2026-05-14T16:42:50.4873844Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdater.decodeDictionaryIds(ParquetVectorUpdater.java:75)[0m[0m
2026-05-14T16:42:50.4876411Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:288)[0m[0m
2026-05-14T16:42:50.4878920Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:406)[0m[0m
2026-05-14T16:42:50.4882283Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:238)[0m[0m
2026-05-14T16:42:50.4885152Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)[0m[0m
2026-05-14T16:42:50.4887413Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext0(FileScanRDD.scala:139)[0m[0m
2026-05-14T16:42:50.4889405Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:301)[0m[0m
2026-05-14T16:42:50.4893860Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext0(FileScanRDD.scala:139)[0m[0m
2026-05-14T16:42:50.4895777Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:148)[0m[0m
2026-05-14T16:42:50.4897409Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:777)[0m[0m
2026-05-14T16:42:50.4899799Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)[0m[0m
2026-05-14T16:42:50.4902432Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)[0m[0m
2026-05-14T16:42:50.4904454Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:44)[0m[0m
2026-05-14T16:42:50.4906902Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:50)[0m[0m
2026-05-14T16:42:50.4909037Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:593)[0m[0m
2026-05-14T16:42:50.4910585Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:593)[0m[0m
2026-05-14T16:42:50.4912111Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1835)[0m[0m
2026-05-14T16:42:50.4913593Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.rdd.RDD.$anonfun$count$1(RDD.scala:1320)[0m[0m
2026-05-14T16:42:50.4915189Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.rdd.RDD.$anonfun$count$1$adapted(RDD.scala:1320)[0m[0m
2026-05-14T16:42:50.4917027Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2536)[0m[0m
2026-05-14T16:42:50.4918691Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)[0m[0m
2026-05-14T16:42:50.4925586Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:206)[0m[0m
2026-05-14T16:42:50.4932392Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.Task.run(Task.scala:147)[0m[0m
2026-05-14T16:42:50.4937063Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:894)[0m[0m
2026-05-14T16:42:50.4938936Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:86)[0m[0m
2026-05-14T16:42:50.4940994Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:83)[0m[0m
2026-05-14T16:42:50.4942960Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:97)[0m[0m
2026-05-14T16:42:50.4945907Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:897)[0m[0m
2026-05-14T16:42:50.4947824Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)[0m[0m
2026-05-14T16:42:50.4950242Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)[0m[0m
2026-05-14T16:42:50.4952201Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.base/java.lang.Thread.run(Thread.java:840)[0m[0m
2026-05-14T16:42:50.5112360Z 16:42:50.510 WARN org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #55853:
URL: https://github.com/apache/spark/pull/55853#issuecomment-4454211628
This is reverted.
- master: https://github.com/apache/spark/commit/004e626a054ca8c9e9fd86743a6ecc2b473e1604
- branch-4.x: https://github.com/apache/spark/commit/e42c5a00e27faaed0ec70189583ee501a87e0b52
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #55853:
URL: https://github.com/apache/spark/pull/55853#issuecomment-4452110245
Merged into master/branch-4.x thanks @dongjoon-hyun @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #55853:
URL: https://github.com/apache/spark/pull/55853#discussion_r3246089265
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java:
##########
@@ -706,7 +706,13 @@ public void decodeSingleDictionaryId(
WritableColumnVector values,
WritableColumnVector dictionaryIds,
Dictionary dictionary) {
- values.putLong(offset, dictionary.decodeToLong(dictionaryIds.getDictId(offset)));
+ // Target is a 32-bit decimal column vector (precision <= 9); narrow the dictionary's
+ // long value to int via the same `(int) longValue` cast used by `readValue` and
+ // `readValues`. The pre-existing `putLong` here was a latent bug that did not surface
+ // until the dictionary-decode path was exercised by the new end-to-end test, because
+ // a 32-bit decimal vector stores values in `intData` and `putLong` would NPE on the
+ // unallocated `longData`.
+ values.putInt(offset, (int) dictionary.decodeToLong(dictionaryIds.getDictId(offset)));
Review Comment:
This is a long-standing legacy issue. I will fix it in a standalone PR first, then rebase the current PR afterwards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #55853: [SPARK-56803][SQL] Add bulk read+narrow path for INT64 DECIMAL to 32-bit Decimal Parquet vector updater
URL: https://github.com/apache/spark/pull/55853
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org