You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu> on 2026/09/18 21:39:51 UTC
[XL] Change in asterixdb[master]: [ASTERIXDB-3868][EXT] Split large data files across scan tasks
From Hussain Towaileb <hu...@gmail.com>:
Attention is currently required from: Ian Maxon.
Hello Anon. E. Moose #1000171, Ian Maxon, Jenkins,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21819?usp=email
to look at the new patch set (#3).
The following approvals got outdated and were removed:
Contrib-2 by Anon. E. Moose #1000171, Integration-Tests+1 by Jenkins
Change subject: [ASTERIXDB-3868][EXT] Split large data files across scan tasks
......................................................................
[ASTERIXDB-3868][EXT] Split large data files across scan tasks
A data file was always read whole by one partition, so a scan's wall
clock was floored by its largest file however many partitions were
available: a single large file occupied one reader while the rest of the
cluster idled.
Plan each file into tasks of the table's target split size, and weigh
the distribution across partitions by sizeBytes. Iceberg's offsets-aware
splitter ignores the requested target and emits one task per row group,
so adjacent pieces are coalesced back up to the target instead of being
left as hundreds of tiny tasks.
Splitting is gated by a splitScanTasks option, default on, so it can be
turned off without a patch release; the off path reads one task per file
and is the oracle the tests compare against.
Ext-ref: MB-73313
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
Change-Id: I038078da736f7e8dd3d7be5293fed9856f3c75aa
---
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/iceberg/IcebergFileRecordReader.java
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/iceberg/IcebergParquetRecordReaderFactory.java
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
A asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/aws/iceberg/IcebergLargeFileSplittingTest.java
A asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/aws/iceberg/IcebergScanTaskSplittingTest.java
5 files changed, 1,274 insertions(+), 12 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/19/21819/3
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21819?usp=email
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newpatchset
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I038078da736f7e8dd3d7be5293fed9856f3c75aa
Gerrit-Change-Number: 21819
Gerrit-PatchSet: 3
Gerrit-Owner: Hussain Towaileb <hu...@gmail.com>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Hussain Towaileb <hu...@gmail.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Attention: Ian Maxon <im...@apache.org>